Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Shrug2017 arcpy data_and_you

57 Aufrufe

Veröffentlicht am

Python, Data and You!

  • Als Erste(r) kommentieren

  • Gehören Sie zu den Ersten, denen das gefällt!

Shrug2017 arcpy data_and_you

  1. 1. 1
  2. 2. AGENDA• Python And the ERSI Platform • Key Items in ESRI Platform o Connection Files o ArcPy Cursors • Branching out with Python Libraries o Pyodbc o Cx_Oracle • Sqlite – A hidden gem • Code Focus – Example 1 • Code Focus – Example 2 2
  3. 3. It’s All Connected… 3 AGOL Open Source Shapefiles Flat Files ArcSDE ArcServer / Portal
  4. 4. Connection Files: Gateways Between Platforms Oracle MS SQL Server 4 ArcSDE ArcServer/ Portal For ArcGIS AGOL External Data
  5. 5. Cursor Function Code arcpy.da.SearchCursor Read-only access arcpy.da.InsertCursor Inserts rows arcpy.da.UpdateCursor Updates or deletes rows 5
  6. 6. 6 import arcpy fc = ‘C:/some.gdb/buildings’ fields = [ ‘OID@’, ‘SHAPE@XY’, ‘STRUCTURE’ ] # Other Options: SHAPE@WKT, SHAPE@WKB, SHAPE@JSON with arcpy.da.SearchCursor(fc, fields) as cursor: for row in cursor: # Perform some task
  7. 7. 7 import arcpy row_values = [( ‘Barn’, (-84.005, 33.002)), ( ‘House’, (-84.899, 33.556))] # Open an InsertCursor cursor = arcpy.da.InsertCursor(‘C:/some.gdb/building', [ ‘STRUCTURE’, ‘SHAPE@XY’ ]) # Insert new rows that include point nane and a x,y coordinate # pair that represents the structure's center for row in row_values: cursor.insertRow(row) # Delete cursor object
  8. 8. 8 import arcpy fc = ‘c:/some.gdb/buildings’ fields = [‘OID@’, ‘SHAPE@XY’, ‘STRUCTURE’] # Create update cursor for feature class with arcpy.da.UpdateCursor(fc, fields) as cursor: for row in cursor: row[2] = row[2].upper() # Update the cursor with the updated list cursor.updateRow(row)
  9. 9. Python Library - PyODBC 9 Pyodbc is an open source Python module specifically designed for connecting to ODBC databases. Based on the DB API 2.0 specification, this library is also packed with even more Pythonic convenience. Pyodbc can connect to these Microsoft databases:
  10. 10. How to use PyODBC 10 # Import python library import pyodbc # Use the specific connection below for each Microsoft database type. # MS Access 2000 connection DBfile = ‘C:/SomeMdb.mdb’ cnmdb = pyodbc.connect(‘‘‘DRIVER={Microsoft Access Driver (*.mdb)};DBQ=’’’ +Dbfile)
  11. 11. 11 # Sql Server Express Connection. Use FQDN for remote server connection cnxpress = pyodbc.connect(‘‘‘Driver= {SQL Server Native Client 10.0}; Server=Instance Name; Database=DB Name; Uid=User; Pwd=Password;’’’) PyODBC continued…
  12. 12. PyODBC continued… 12 # MS SQL Server Connection. Use FQDN for remote server connection cnmsql = pyodbc.connect(‘‘‘DRIVER={SQL Server}; SERVER=ServerName; DATABASE=DB Name; UID=User; PWD=Password’’’)
  13. 13. PyODBC continued… 13 # MS SQL Server Connection cnazure = pyodbc.connect(‘‘‘Driver= {SQL Server Native Client 10.0}; Server= tcp:ServerName.database.windows.net; Database=DB NAME; Uid=[LoginForDb]@[serverName]; Pwd=myPassword;Encrypt=yes;’’’) cur = ConnChoice.cursor()
  14. 14. # Import python library import cx_Oracle # Create connection to Oracle DB. Versions 10g, 11g, and 12c are applicable. Use FQDN for remote server connection. con = cx_Oracle.connect(‘‘‘scott/tiger@server:port/instance’’’) # Create database cursor cur = con.cursor() Python Library – cx_Oracle 14
  15. 15. What is SQLite? 15 SQLite is an open source transactional database that supports ANSI SQL 92. Why use SQLite with GIS? SQLIte can be use as a traditional database on disk or invoked solely in memory. Operations like joins are handled in a single transaction versus multiple steps.
  16. 16. How to use SQLite 16 # Loads python library import arcpy # Execute CreateSQLiteDatabase for disk based database arcpy.CreateSQLiteDatabase_management(‘C:/data/example. sqlite’, ‘SPATIALITE’)
  17. 17. # Loads python library import sqlite3 as lite # Creating a SQLite DB connection in RAM con = lite.connect(‘:memory:’) # Generate cursor cur = con.cursor() How to use SQLite 17
  18. 18. “If you want to succeed you should strike out on new paths, rather than travel the worn paths of accepted success.” 18 John D. Rockefeller
  19. 19. 19 Take the provided CSV file and create a point feature class using the coordinates provide. Be sure to validate all text attributes for spacing issues before creating feature class. The final schema should look like the following: Attribute Name Attribute Type Attribute Size State_Terr Text 75 Capital Text 75 Latitude Double - Longitude Double - Conus Text 3 Type Text 10 Bridging Platforms
  20. 20. 20 Case Example for Bridging Platforms
  21. 21. 21 import arcpy import datetime import sqlite3 as lite # Scripting environment variables arcpy.env.workspace = r‘C:Shrug2017’ arcpy.env.overwriteOutput = True csv = r‘C:Shrug2017Demo1states_terr.csv’ fields = [‘State_Terr’,‘Capital’,‘Latitude’,‘Longitude’,‘Conus’, ‘Type’] # Optional: fields = [‘*’]
  22. 22. 22 # Creating a SQLite DB in RAM con = lite.connect(‘:memory:’) # Generate cursor cur = con.cursor() print(‘Destination database created –’+ str(datetime.datetime.now())) # Creates a table structure for incoming data cur.execute(‘‘‘CREATE TABLE States(StateNme TEXT, Capital TEXT, Lat REAL, Long REAL, Conus TEXT, Typ TEXT);’’’)
  23. 23. 23 print(‘Working table created –’+ str(datetime.datetime.now())) with arcpy.da.SearchCursor(csv, fields) as cursor: for row in cursor: # Inserting records to SQLite DB to_db = [row[0],row[1],row[2],row[3],row[4],row[5] ] cur.execute(‘‘‘INSERT INTO States(StateNme,Capital,Lat,Long, Conus,Typ) VALUES(?,?,?,?,?,?);’’’, to_db) con.commit()
  24. 24. 24 print(‘CSV inserted into SQLite database table –’ + str(datetime.datetime.now())) #Removing white spacing from columns with text data type cur.execute(‘‘‘UPDATE States SET StateNme = TRIM(StateNme), Capital = TRIM(Capital), Conus = TRIM(Conus), Typ = TRIM(Typ);’’’) con.commit() print(‘Validation process completed – ’+ str(datetime.datetime.now()))
  25. 25. 25 cur.execute(‘‘‘ALTER TABLE States ADD ShapeXY TEXT’’’) cur.execute(‘‘‘UPDATE States SET ShapeXY = ‘POINT’||‘(’ ||Long|| ‘ ’||Lat|| ‘)’ ’’’) con.commit() print(‘Spatial column created – ’+ str(datetime.datetime.now())) output_path = r‘C:Shrug2017Scratch.gdb’ out_name = ‘us_states_terr’ geometry_type = ‘Point’
  26. 26. 26 wkt = ‘‘‘GEOGCS['GCS_WGS_1984',DATUM['D_WGS_1984', SPHEROID['WGS_1984',6378137.0,298.257223563]], PRIMEM['Greenwich',0.0], UNIT['Degree',0.0174532925199433]], VERTCS['WGS_1984',DATUM['D_WGS_1984', SPHEROID['WGS_1984',6378137.0,298.257223563]], PARAMETER['Vertical_Shift',0.0], PARAMETER['Direction',1.0],UNIT['Meter',1.0]]; -400 -400 1000000000;-100000 10000;-100000 10000;8.98315284119522E-09; 0.001;0.001;IsHighPrecision’’’ sr = arcpy.SpatialReference()
  27. 27. 27 sr.loadFromString(wkt) result = arcpy.CreateFeatureclass_management(output_path, out_name,geometry_type, ‘’, ‘DISABLED’,‘DISABLED’,sr) fcOut = result.getOutput(0) arcpy.AddField_management(fcOut,‘State_Terr’,‘TEXT’,‘’,‘’,75) arcpy.AddField_management(fcOut,‘Capital’,‘TEXT’,‘’,‘’,75) arcpy.AddField_management(fcOut,‘Latitude’,‘DOUBLE’, 8,‘’, ‘’, ‘’, ‘NULLABLE’)
  28. 28. 28 arcpy.AddField_management(fcOut,‘Longitude’,‘DOUBLE’, 8,‘’,‘’,‘’, ‘NULLABLE’) arcpy.AddField_management(fcOut,‘Conus’,‘TEXT’,‘’,‘’, 3) arcpy.AddField_management(fcOut,'Type','TEXT',‘’,‘’, 10) print( ‘Shell featureclass created – ’+ str(datetime.datetime.now()) ) schema = [ ‘SHAPE@WKT’,‘State_Terr’,‘Capital’, ‘Latitude’,‘Longitude’,‘Conus’,‘Type’ ] curESRI = arcpy.da.InsertCursor(fcOut,schema)
  29. 29. 29 # Query to extract data from SQLITE DB table rows = cur.execute(‘‘‘SELECT ShapeXY,StateNme,Capital, Lat,Long,Conus,Typ FROM States’’’) for row in rows: curESRI.insertRow(row) # close insert cursor del curESRI # close SQLite database connection con.close() print(‘SQLite table insertion completed –’+ str(datetime.datetime.now()))
  30. 30. 30 The point data that is displayed in ArcGIS Pro was created in 0.004 of second. The table below depicts the processing times for three different configurations Arc Map Arc Pro SQLite 1.47 s 0.39 s 0.004 s
  31. 31. 31 Putting It All Together… Prior Steps Before Starting Demo2 Create a MS SQL Server Express database using the SQL scripts provided. Modified the previous script to add data to a MS SQL Server Express database table. Upload ‘states_update1.csv’ to the temp table in MSSQL Express
  32. 32. 32 import pyodbc import os import datetime print(‘Starting Update Process- ’ + str(datetime.datetime.now())) # Sql Server connection cnmsql = pyodbc.connect(‘DRIVER={SQL Server}; Server=Server; Database=ShrugDemo; UID=ScrptKng;PWD=python!’) # Create cursor cur = cnmsql.cursor()
  33. 33. 33 # Issue update commands for states counts = cur.execute(‘‘‘MERGE INTO dbo.States_Territories USING dbo.States_Delta ON dbo.States_Territories.State_Terr = dbo.States_Delta.State_Terr WHEN MATCHED THEN UPDATE SET Capital = dbo.States_Delta.Capital, Latitude = dbo.States_Delta.Latitude, Longitude = dbo.States_Delta.Longitude, Conus = dbo.States_Delta.Conus Type = dbo.States_Delta.Type
  34. 34. 34 WHEN NOT MATCHED THEN INSERT(State_Terr,Capital,Latitude, Longitude,Conus,Type) VALUES(dbo.States_Delta.State_Terr, dbo.States_Delta.Capital, dbo.States_Delta.Latitude, dbo.States_Delta.Longitude, dbo.States_Delta.Conus, dbo.States_Delta.Type);’’’).rowcount print(‘Number of rows added or updated: ’+ str(counts)+ ‘-’ + str(datetime.datetime.now()))
  35. 35. 35 # Commits transactions to database cur.commit() # Closing cursor and database connection cur.close() cnmsql.close()
  36. 36. 36
  37. 37. 37 • Daniel Thomas • ThomasD@leoncountyfl.gov • TLCGIS.org Goto http://shrug-gis.org/workshop/Workshop/Presentation-Resources for this presentation, scripts, code snippets, and more!

×