Python, Data and You!

1. 1

2. AGENDA• Python And the ERSI Platform
• Key Items in ESRI Platform
o Connection Files
o ArcPy Cursors
• Branching out with Python
Libraries
o Pyodbc
o Cx_Oracle
• Sqlite – A hidden gem
• Code Focus – Example 1
• Code Focus – Example 2
2

3. It’s All Connected…
3
AGOL
Open
Source
Shapefiles
Flat Files
ArcSDE
ArcServer /
Portal

4. Connection Files: Gateways Between Platforms
Oracle
MS SQL
Server
4
ArcSDE
ArcServer/
Portal For
ArcGIS
AGOL
External
Data

5. Cursor Function Code
arcpy.da.SearchCursor
Read-only
access
arcpy.da.InsertCursor Inserts rows
arcpy.da.UpdateCursor
Updates or
deletes rows
5

6. 6
import arcpy
fc = ‘C:/some.gdb/buildings’
fields = [ ‘OID@’, ‘SHAPE@XY’, ‘STRUCTURE’ ]
# Other Options: SHAPE@WKT, SHAPE@WKB, SHAPE@JSON
with arcpy.da.SearchCursor(fc, fields) as cursor:
for row in cursor:
# Perform some task

7. 7
import arcpy
row_values = [( ‘Barn’, (-84.005, 33.002)),
( ‘House’, (-84.899, 33.556))]
# Open an InsertCursor
cursor = arcpy.da.InsertCursor(‘C:/some.gdb/building',
[ ‘STRUCTURE’,
‘SHAPE@XY’ ])
# Insert new rows that include point nane and a x,y coordinate
# pair that represents the structure's center
for row in row_values:
cursor.insertRow(row)
# Delete cursor object

8. 8
import arcpy
fc = ‘c:/some.gdb/buildings’
fields = [‘OID@’, ‘SHAPE@XY’, ‘STRUCTURE’]
# Create update cursor for feature class
with arcpy.da.UpdateCursor(fc, fields) as cursor:
for row in cursor:
row[2] = row[2].upper()
# Update the cursor with the updated list
cursor.updateRow(row)

9. Python Library - PyODBC
9
Pyodbc is an open source Python module
specifically designed for connecting to ODBC
databases. Based on the DB API 2.0 specification,
this library is also packed with even more Pythonic
convenience.
Pyodbc can connect to these Microsoft databases:

10. How to use PyODBC
10
# Import python library
import pyodbc
# Use the specific connection below for each Microsoft
database type.
# MS Access 2000 connection
DBfile = ‘C:/SomeMdb.mdb’
cnmdb = pyodbc.connect(‘‘‘DRIVER={Microsoft Access Driver
(*.mdb)};DBQ=’’’ +Dbfile)

11. 11
# Sql Server Express Connection. Use FQDN for remote server
connection
cnxpress = pyodbc.connect(‘‘‘Driver=
{SQL Server Native Client 10.0};
Server=Instance Name;
Database=DB Name;
Uid=User; Pwd=Password;’’’)
PyODBC continued…

12. PyODBC continued…
12
# MS SQL Server Connection. Use FQDN for remote server
connection
cnmsql = pyodbc.connect(‘‘‘DRIVER={SQL Server};
SERVER=ServerName;
DATABASE=DB Name;
UID=User;
PWD=Password’’’)

13. PyODBC continued…
13
# MS SQL Server Connection
cnazure = pyodbc.connect(‘‘‘Driver=
{SQL Server Native Client 10.0};
Server=
tcp:ServerName.database.windows.net;
Database=DB NAME;
Uid=[LoginForDb]@[serverName];
Pwd=myPassword;Encrypt=yes;’’’)
cur = ConnChoice.cursor()

14. # Import python library
import cx_Oracle
# Create connection to Oracle DB. Versions 10g, 11g, and 12c
are applicable. Use FQDN for remote server connection.
con = cx_Oracle.connect(‘‘‘scott/tiger@server:port/instance’’’)
# Create database cursor
cur = con.cursor()
Python Library – cx_Oracle
14

15. What is SQLite?
15
SQLite is an open source transactional database that supports
ANSI SQL 92.
Why use SQLite with GIS?
SQLIte can be use as a traditional database on disk or invoked
solely in memory. Operations like joins are handled in a single
transaction versus multiple steps.

16. How to use SQLite
16
# Loads python library
import arcpy
# Execute CreateSQLiteDatabase for disk based database
arcpy.CreateSQLiteDatabase_management(‘C:/data/example.
sqlite’, ‘SPATIALITE’)

17. # Loads python library
import sqlite3 as lite
# Creating a SQLite DB connection in RAM
con = lite.connect(‘:memory:’)
# Generate cursor
cur = con.cursor()
How to use SQLite
17

18. “If you want to
succeed you
should strike out
on new paths,
rather than travel
the worn paths of
accepted
success.”
18
John D. Rockefeller

19. 19
Take the provided CSV file and create a point
feature class using the coordinates provide.
Be sure to validate all text attributes for spacing
issues before creating feature class.
The final schema should look like the following:
Attribute Name Attribute Type Attribute Size
State_Terr Text 75
Capital Text 75
Latitude Double -
Longitude Double -
Conus Text 3
Type Text 10
Bridging Platforms

20. 20
Case Example for Bridging Platforms

21. 21
import arcpy
import datetime
import sqlite3 as lite
# Scripting environment variables
arcpy.env.workspace = r‘C:Shrug2017’
arcpy.env.overwriteOutput = True
csv = r‘C:Shrug2017Demo1states_terr.csv’
fields = [‘State_Terr’,‘Capital’,‘Latitude’,‘Longitude’,‘Conus’,
‘Type’]
# Optional: fields = [‘*’]

22. 22
# Creating a SQLite DB in RAM
con = lite.connect(‘:memory:’)
# Generate cursor
cur = con.cursor()
print(‘Destination database created –’+
str(datetime.datetime.now()))
# Creates a table structure for incoming data
cur.execute(‘‘‘CREATE TABLE States(StateNme TEXT, Capital
TEXT, Lat REAL, Long REAL, Conus TEXT, Typ TEXT);’’’)

23. 23
print(‘Working table created –’+ str(datetime.datetime.now()))
with arcpy.da.SearchCursor(csv, fields) as cursor:
for row in cursor:
# Inserting records to SQLite DB
to_db = [row[0],row[1],row[2],row[3],row[4],row[5] ]
cur.execute(‘‘‘INSERT INTO
States(StateNme,Capital,Lat,Long,
Conus,Typ)
VALUES(?,?,?,?,?,?);’’’, to_db)
con.commit()

24. 24
print(‘CSV inserted into SQLite database table –’ +
str(datetime.datetime.now()))
#Removing white spacing from columns with text data type
cur.execute(‘‘‘UPDATE States
SET StateNme = TRIM(StateNme),
Capital = TRIM(Capital),
Conus = TRIM(Conus),
Typ = TRIM(Typ);’’’)
con.commit()
print(‘Validation process completed – ’+
str(datetime.datetime.now()))

25. 25
cur.execute(‘‘‘ALTER TABLE States ADD ShapeXY TEXT’’’)
cur.execute(‘‘‘UPDATE States SET ShapeXY = ‘POINT’||‘(’
||Long|| ‘ ’||Lat|| ‘)’ ’’’)
con.commit()
print(‘Spatial column created – ’+ str(datetime.datetime.now()))
output_path = r‘C:Shrug2017Scratch.gdb’
out_name = ‘us_states_terr’
geometry_type = ‘Point’

26. 26
wkt = ‘‘‘GEOGCS['GCS_WGS_1984',DATUM['D_WGS_1984',
SPHEROID['WGS_1984',6378137.0,298.257223563]],
PRIMEM['Greenwich',0.0],
UNIT['Degree',0.0174532925199433]],
VERTCS['WGS_1984',DATUM['D_WGS_1984',
SPHEROID['WGS_1984',6378137.0,298.257223563]],
PARAMETER['Vertical_Shift',0.0],
PARAMETER['Direction',1.0],UNIT['Meter',1.0]];
-400 -400 1000000000;-100000 10000;-100000
10000;8.98315284119522E-09;
0.001;0.001;IsHighPrecision’’’
sr = arcpy.SpatialReference()

27. 27
sr.loadFromString(wkt)
result = arcpy.CreateFeatureclass_management(output_path,
out_name,geometry_type,
‘’,
‘DISABLED’,‘DISABLED’,sr)
fcOut = result.getOutput(0)
arcpy.AddField_management(fcOut,‘State_Terr’,‘TEXT’,‘’,‘’,75)
arcpy.AddField_management(fcOut,‘Capital’,‘TEXT’,‘’,‘’,75)
arcpy.AddField_management(fcOut,‘Latitude’,‘DOUBLE’, 8,‘’, ‘’,
‘’, ‘NULLABLE’)

28. 28
arcpy.AddField_management(fcOut,‘Longitude’,‘DOUBLE’,
8,‘’,‘’,‘’, ‘NULLABLE’)
arcpy.AddField_management(fcOut,‘Conus’,‘TEXT’,‘’,‘’, 3)
arcpy.AddField_management(fcOut,'Type','TEXT',‘’,‘’, 10)
print( ‘Shell featureclass created – ’+
str(datetime.datetime.now()) )
schema = [ ‘SHAPE@WKT’,‘State_Terr’,‘Capital’,
‘Latitude’,‘Longitude’,‘Conus’,‘Type’ ]
curESRI = arcpy.da.InsertCursor(fcOut,schema)

29. 29
# Query to extract data from SQLITE DB table
rows = cur.execute(‘‘‘SELECT ShapeXY,StateNme,Capital,
Lat,Long,Conus,Typ FROM States’’’)
for row in rows:
curESRI.insertRow(row)
# close insert cursor
del curESRI
# close SQLite database connection
con.close()
print(‘SQLite table insertion completed –’+
str(datetime.datetime.now()))

30. 30
The point data
that is displayed
in ArcGIS Pro was
created in 0.004
of second.
The table below
depicts the
processing times
for three different
configurations
Arc
Map
Arc
Pro
SQLite
1.47 s 0.39 s 0.004 s

31. 31
Putting It All Together…
Prior Steps Before Starting Demo2
Create a MS SQL Server Express
database using the SQL scripts
provided.
Modified the previous script to add data
to a MS SQL Server Express database
table.
Upload ‘states_update1.csv’ to the
temp table in MSSQL Express

32. 32
import pyodbc
import os
import datetime
print(‘Starting Update Process- ’ + str(datetime.datetime.now()))
# Sql Server connection
cnmsql = pyodbc.connect(‘DRIVER={SQL Server};
Server=Server;
Database=ShrugDemo;
UID=ScrptKng;PWD=python!’)
# Create cursor
cur = cnmsql.cursor()

33. 33
# Issue update commands for states
counts = cur.execute(‘‘‘MERGE INTO dbo.States_Territories
USING dbo.States_Delta
ON dbo.States_Territories.State_Terr =
dbo.States_Delta.State_Terr
WHEN MATCHED THEN
UPDATE
SET Capital = dbo.States_Delta.Capital,
Latitude = dbo.States_Delta.Latitude,
Longitude =
dbo.States_Delta.Longitude,
Conus = dbo.States_Delta.Conus
Type = dbo.States_Delta.Type

34. 34
WHEN NOT MATCHED THEN
INSERT(State_Terr,Capital,Latitude,
Longitude,Conus,Type)
VALUES(dbo.States_Delta.State_Terr,
dbo.States_Delta.Capital,
dbo.States_Delta.Latitude,
dbo.States_Delta.Longitude,
dbo.States_Delta.Conus,
dbo.States_Delta.Type);’’’).rowcount
print(‘Number of rows added or updated: ’+ str(counts)+ ‘-’ +
str(datetime.datetime.now()))

35. 35
# Commits transactions to database
cur.commit()
# Closing cursor and database connection
cur.close()
cnmsql.close()

36. 36

37. 37
• Daniel Thomas
• ThomasD@leoncountyfl.gov
• TLCGIS.org
Goto http://shrug-gis.org/workshop/Workshop/Presentation-Resources
for this presentation, scripts, code snippets, and more!