Slides from the first New York Data Brewery meetup. Overview of the meetup data warehousing topics.

1. New York
№ 1: Introduction
data brewery
Štefan Urbánek, @Stiivi
stefan.urbanek@gmail.com
January 2015

2. Who are you?
Why are you here?
What would you like to ask or say?

3. Few Questions…

4. How do you know that
the data you are looking at
is the data you are looking at?

5. How fast can a domain expert
(very likely a technical layperson)
get understandable answers to his/her questions
if there are so many ways and places where your
data is stored?
?

6. How do you embrace unknown
and change?
Requirements (questions) are changing, labelling (naming things) is changing, source
systems (structures) are changing, …
A
B
C
A
B
C
2014 2015
??
∑✦ ?

7. Data Warehouse
traditional, conceptual perspective

8. Governance
Analytics and
Visualisation
Data
Sources
Technologies
Cleaning, Transforming, Conforming

9. Governance
Analytics and
Visualisation
Cleaning, Transforming, Conforming
Data
Sources
Technologies!!!
trust? provenance? reproducibility? maintainability?

10. No GPU powered distributed cluster of real-time
generator of heat-maps and bubble charts is
going to help you!

11. + +people concepts technology

12. Data Processing Data Use
Governance
Technologies
Quality
Management
ETL
Management
Extraction
Discovery and
Acquisition
Cleansing and
Conforming
Analytical
Modeling
Presentation,
Analysis
Language
Master Data
Management
sources
Storage Infrastructure
…
…

13. Example Topics

14. Two views, two needs …
numerical categorical
T1[s] T2[s] T3[s] T4[s]
P1 634.90 439.14 950.16 80.42
P2 179.32 331.75 6.12 227.86
P3 885.67 960.45 650.60 780.52
P4 262.40 390.02 442.23 325.17
P5 410.44 639.54 913.06 220.39
vs

15. Transformations
■ how to make data understandable to the users?
■ mappings – structures and mechanisms
■ how to handle manual corrections?
■ metadata based transformations

16. Multidimensional Modeling
■ why? how it can help?
■ conceptual hierarchies, drill-downs
■ ★ star and ❄ snowflake schemas
logical approach, query generators
■ relational database implementation (SQL)
fact and dimension table modelling, stars in a cluster
(as a concept)

17. Slowly Changing Dimensions
■ why? what is it?
■ how to load dimensions
UPSERT on steroids
■ what about not-so-slowly changing dimensions?

18. Data Quality Indicators
■ what is data quality?
■ when I need it?
■ how to measure data quality?

19. Master Data Management
■ why?
■ how and where to integrate it in the pipeline

20. Image by: LPLT / Wikimedia Commons

21. ehm…
“Data warehouse is obsolete! We
have new technologies!”

23. Python and Data
Warehousing
need for language and ecosystem

24. Theme
Data
Harvesting
Data
Cleansing
Analyzing
Data
Crawling
Web
Scraping
API
Visualisa-
tion
Plotly
Matplotlib
NumPy
SciPy Pandas
NLTK
IPython
Machine
Learning
Scikit
Learn
Scikit
Image
Data
Reporting,
Publish
CKAN
Cubes
ETL
mETL
Bubbles
Luigi
Numba
Bokeh
Python
SymPy
mply
PyBrain
Vispy
NetworkX
from “Python for Data Science” presentation (2014) by Bence Faludi – @bfaludi

25. mETL
MAIN COMPONENTS
All processes start with a source file
from which the data are retrieved.
There are unique types, which all
have their own settings.
After the data is read from the
source, and the transformations
are completed, the finalized record
gets to the Target which will write
and create the file with the final
data.
github.com/ceumicrodata/mETL
: :

26. Cubes
∑❄
: :
github.com/DataBrewery/cubes

27. Open Data Utilities
mostly light-weight, category oriented, one-purpose
github.com/pudo
OKFN Labs
github.com/okfn

28. Don’t forget …
*related to data warehousing in a similar way NumPy is to data science
*

29. … and just very few lonely others
Not enough.

30. Who is going to fix your COBOL Java tool
if you have only Python guys around?
PyData NYC 2012

31. data brewery.org

33. umbrella for Python data
warehouse toolkits
databrewery

34. The New Data Brewery
■ projects helping to solve data warehouse problems
■ curated and incubated catalogue
■ community
knowledge sharing, tool development, professional help
■ complementary to the NumPy/SciPy ecosystem
not competitive – different purpose
b

35. The Meetup

36. Topics Summary
■ categorical data
■ multi-dimensional modeling
star and snowflake schemas, metadata, dimension modelling, slowly changing dimensions
■ data quality management
approaches, data quality indicators
■ master data management
concepts and their implementation

37. Meetup
■ meet experts in the domain
■ share stories, ask for stories
■ discuss a problem, get feedback
■ get questions answered
?

38. … in the Data Brewery
tech ❤️ non-tech

39. Let’s brew some data
http://www.meetup.com/New-York-Data-Brewery

40. Thank You
@Stiivi

41. Questions, suggestions?

42. https://www.flickr.com/photos/britishlibrary
Scanned images are from the British Library album at