Discover how police data can be safely shared, using automated techniques, for the Police Data Initiative or via any open data portal. In this webinar, you’ll hear lessons learned handling sensitive data from the RCMP in BC and the City of Auburn. Learn tips for integrating disparate data, maintaining victim privacy, performing data validation, and much more.
2024 DevNexus Patterns for Resiliency: Shuffle shards
How to Manage Open Police Data - Tips for Data QA/QC and Automation
1. How to Manage Open Police
Data – Tips for Data QA/QC
and Automation
Alice Pence, City of Auburn
Robert Schultz, RCMP in BC
Cat Gracey, Safe Software
Dale Lutz, Safe Software
2. Who are we?
Connect Your Applications | Transform Your Data | Automate Your Workflows
3. Police Data Initiative
53 U.S. jurisdictions. 41 million people. 90 datasets. And growing!
www.publicsafetydataportal.org
4. Why Should Police Data be Shared?
● Increase transparency and build trust
● Help provide improved public services
● Fuel more intelligent policing decisions
The Ethics:
● Need to protect victim identity
● Only share when it’s lawful to do so
● Must be accurate, free of errors
10. Quick Bio
• BS in Geography from University of Idaho w/Certificate in GIS
• In GIS field for 3+ years – been in Auburn, WA for over 1 year
Alice Pence
11. Our Crimes Database
• Stored in a Proprietary Linux system
• Raw data with only addresses
• Can’t directly dump into GIS- needs to have data transformed for public/internal viewing, QA/QC, a
spatial component to view on maps and applications, attributes renamed…
13. Our Mission
How to view all this
data spatially for
analysis and reporting?
Our first idea…
14. A Python Script
The Good
• It worked. Can add new offenses, change and update as needed.
• Ran in windows scheduler to automate.
The Bad
• Long, cumbersome script.
• Needed multiple databases (.gbd → SDE) to clean and move data.
The Ugly
• Can’t easily change connection to Crimes Database, no direct connection to source (connected
through ArcCatalog -then built through Model Builder- then exported out as a Python Script).
• No custom alerts or in depth logging capabilities.
• Not user friendly for non-python users.
18. Attribute Mapper
Transformer
• As an example: RKBK
Crime Type= Felony Crimes Against Person
Crime Name= Robbery
1 2
3
*double bonus:
I was able to
automatically import
codes by a CSV file,
AND…FME
automatically read in
all the source values
for me
25. Comparison
Python
• Cumbersome/long script
• Not very easy to
customize alerts or logs
• Hard to edit- other users
might not know the
python language
• No friendly user interface
FME
• Easy to update codes and
field changes
• Other users can easily edit-
even w/minimal knowledge of
FME
• Share directly to FME Server
for logs and alerts
• Friendly user interface for
FME Server and easy to set up
30. Thank you!
Alice Pence
GIS/Database Specialist
City of Auburn, WA
apence@auburnwa.gov
Links to sites discussed in slides:
Socrata Open Data Portal: https://data.auburnwa.gov/
Auburn Crimes Map:
https://maps.auburnwa.gov/html5viewer/Index.html?
viewer=crimes
37. Open Data in San Francisco
The new ETL job platform.
38. Background.
● City’s official open data portal is SF OpenData.
The Vision.
● Empower use of City data.
“Our vision is that the City’s data is understood, documented, and high quality.”
39. The Project: Increase number and timeliness of datasets
● Few workspace authors
● Safe place to try out workspaces
The Results.
● Improved ETL job platform, design and architecture
● 100-500 ETL jobs
● Simple ETL Jobs
● Use best-practices
View the original presentation
by Samuel Valdez and
Janine Heiser
(Presented at the FME World Tour 2016)
41. Background.
● The Citywide Street Centerline Database (CSCL) is the authoritative source
for location data used by New York City's public safety agencies (NYPD,
FDNY) for 911 call-taking and dispatch.
The Vision.
Make CSCL data easily accessible for the day-to-day GIS
user.
View the original presentation
by Chris Rado
(Presented at the FME World Tour 2016)
42. The Results.
New version of CSCL created that is better suited for general GIS tasks
and made it available to all city agencies, to the public via
NYC Open Data and to Batman.
44. Thank you!
Live chat us at www.safe.com
Free Resources:
FME Desktop Training (June 7-8)
FME Server Training (June 14-15)
Blog post: Guide to Open Data:
Using it, Sharing it, and Creating a
Portal
Open Data Webinar
Open Data eBook