A traditional data team has roles including data engineer, data scientist, and data analyst. However, many organizations are finding success by integrating a new role – the analytics engineer. The analytics engineer develops a code-based data infrastructure that can serve both analytics and data science teams. He or she develops re-usable data models using the software engineering practices of version control and unit testing, and provides the critical domain expertise that ensures that data products are relevant and insightful. In this talk we’ll talk about the role and skill set of the analytics engineer, and discuss how dbt, an open source programming environment, empowers anyone with a SQL skillset to fulfill this new role on the data team. We’ll demonstrate how to use dbt to build version-controlled data models on top of Delta Lake, test both the code and our assumptions about the underlying data, and orchestrate complete data pipelines on Apache Spark™.

1. The modern data team for the
modern data stack:
dbt & the role of the analytics engineer

2. Welcome
Jeremy Cohen
Associate Product Manager
he/him
jeremy@fishtownanalytics.com
@jerco (community.getdbt.com)

3. The modern data stack

4. The modern data team
▪ Custom ingestion
▪ Orchestration
▪ ML endpoints
▪ Platform, architecture,
tooling: inform build vs.
buy
▪ Provide lean,
transformed data
ready for analysis
▪ SWE practices to
analytics code
▪ Maintain data
documentation
Analytics EngineerData Engineer Data Analyst
▪ Deep insights &
forecasting
▪ Close partnership
with business users
▪ Build & guarantee
critical reporting

5. What is dbt?
A. A python program
B. The heart of the modern data
stack
C. An analytics engineer’s best
friend
D. A community of top-class data
professionals
E. All of the above

6. What is dbt, actually?
▪ Define, test, document, and reuse complex data transformation
logic—just by writing SQL (and a little bit of YAML).
▪ dbt infers a DAG of transformations and runs models in order.
▪ Auto-generated documentation site, built from the same code as
your transformations.
The power of a framework, not the limitations of a GUI.

8. Extending SQL with Jinja
▪ Loops
▪ Macros
▪ Packages
A pythonic templating engine to write DRYer code and leverage open source innovations.

9. The dbt community, by the numbers
▪ 2800+ companies running dbt in production across 12+ databases
▪ 48 open source packages of reusable macros and models
▪ 23k views: our opinionated best practices for dbt project design
▪ 7k data professionals at the top of their game in dbt Slack

10. +

11. dbt +
▪ Open source plugin
▪ pip install dbt-spark
▪ Write business logic in
SparkSQL
▪ Dynamically template repetitive
SQL with Jinja
▪ Connect to any Spark cluster +
dbt run

12. Analytics engineering meets Delta Lake
▪ Access all core dbt features when you materialize models as Delta
tables
▪ Use merge to build incremental models + snapshot slowly changing
dimensions
▪ optimize zorder with hooks, operations, macros...
The power of a data lake, the flexibility of a modern data warehouse, the intuition of a common
modeling framework.

13. Announcing: dbt Cloud + Databricks
▪ Hosted IDE
▪ Compile + run SQL in real time
▪ Straightforward git flow
▪ No installation hassle
▪ Configurable job scheduler
▪ Continuous integration
▪ Host data documentation
▪ Persist dbt artifacts
DeployDevelop
Now in closed beta

14. Demo

15. How to deploy dbt?
▪ SaaS: up & running in minutes
▪ Enterprise: Fishtown-managed VPC, client-managed VPC, airgapped
on-prem, …
▪ You! dbt, the Spark plugin, the documentation site: it’s all open
source and can be deployed using standard infrastructure.
Build, buy, or balance

16. Feedback
Your feedback is important to us.
Don’t forget to rate
and review the sessions.