Lessons learned bringing a large dataset in-house and where we are going from here.

1. Oh, you’re from Jersey? What exit?
Lessons learned bringing the roads
dataset in house and where we are
headed from here

2. A little background…

3. A little bit more background…

4. Last little bit…

5. Road Centerlines – in the past
&

6. Road Centerlines - now

7. But getting it out to folks…

8. A bit of a change
Feature Class Alternate Labeling LRS Values
Names Table Table Table
SEG_ID SEG_ID SEG_ID SEG_ID
FULLNAME PRE_DIR L1_NAME SRI
ADDR_L_FR PRE_TYPE L2_NAME SLD_MP_ST
ADDR_L_TO PRE_MOD H1_NAME SLD_MP_END
ADDR_R_FR NAME H1_SHLD PAR_MP_ST
ADDR_R_TO SUF_TYPE H1_SUBSHLD* PAR_MP_END
ZIP_L SUF_DIR H1_NUM FLP_MP_ST
ZIP_R SUF_MOD H2_NAME FLP_MP_END
MUNI_L H2_SHLD MMS_MP_ST
MUNI_R H2_SUBSHLD* MMS_MP_END
PRE_DIR H2_NUM TPK_MP_ST
PRE_TYPE H3_NAME TPK_MP_END
PRE_MOD H3_SHLD
NAME H3_SUBSHLD*
SUF_TYPE H3_NUM
SUF_DIR
SUF_MOD
ROUTE_TYPE
STATUS
SURFACE PLUS:
ZLEV_FR Teleatlas/TomTom Flat Format
ZLEV_TO
ACCESS Teleatlas/TomTom Coincident Geometry
JURIS Tiburon Format
Address Locator

9. The goal….
Repeatable
Extract
Repeatable Repeatable
Load Transform

10. The solution….

11. The uber-language

12. Maybe not
• 3 seconds per transaction
• 1.1 million records
• 3.3 million seconds = ~ 38 days

13. I’m not dead yet

14. Parallel Python

15. ppservers = ()
if len(sys.argv) > 1:
ncpus = int(sys.argv[1])
job_server = pp.Server(ncpus, ppservers=ppservers)
else:
job_server = pp.Server(ppservers=ppservers)

16. jobs = [(segment,job_server.submit(segment_handling,(
segment.SEG_ID,segment.ADDR_L_FR,segment.ADDR_L_T
O,segment.ADDR_R_FR,segment.ADDR_R_TO,segment.ZIP_
L,segment.ZIP_R,segment.MUNIC_ID_L,segment.MUNIC_ID
_L),(log, Road_Segment),("arcpy",)))for segment in
segments]
for segment, job in jobs:
result = job()
log('Segment ID: ' + str(result.segment_id) + 'n')

17. We did our best
• 1.1 seconds/transaction
• 1.1 million records
• 1.2 seconds = ~ 14 days

18. What I learned…
• Python is cool as all get out and powerful
• Not a good transactional tool for that many
records
• I have a ton to learn
• But we have code that does what we need it
to in order to stand up the geocoding services
for future use

19. Going back to basics
• SDE spatial views
sdetable -o create_view -T SEG_NAME_ETL -t SEG_NAME,L_STREET_ABBR
-c SEG_NAME.SEG_ID, SEG_NAME.NAME_TYPE_ID,
SEGNAME.NAME_PRE_DIR, SEG_NAME.NAME_PRE_TYPE,
SEG_NAME.NAME_PRE_MOD, SEG_NAME.NAME,
SEG_NAME.NAME_SUF_TYPE, SEG_NAME.NAME_SUF_DIR,
SEG_NAME.NAME_SUF_MOD -a SEG_ID, NAME_TYPE_ID, PRE_DIR,
PRE_TYPE, PRE_MOD, NAME, SUF_TYPE, SUF_DIR, SUF_MOD -w {where
clause} -i 5157 -s ****.state.nj.us -u **** -p ****
• Oracle table views
SELECT (a whole bunch of columns)
FROM ROAD.SEG_NAME_V, ROAD.SEGMENT_NJ
WHERE SEG_NAME_V.SEG_ID = SEGMENT_NJ.SEG_ID

20. Or so we thought…
FULL OUTER JOINs not creating the m:1
relationship in SDE views

21. Query Tables and Virtual ID’s
• ArcPy, Python and Geoprocessing models
• Virtual ID’s

22. So what’s next?
Get it out there!
Keep it up to date!

23. Sean McGinnis
e: sean.mcginnis@oit.state.nj.us
e: seankmcginnis@gmail.com
t: @seankmcginnis
w: www.georamblings.com
w: www.seankmcginnis.com