Quby is a leading company offering data driven home services technology across European markets, known for creating the in-home display and smart thermostat Toon. In this talk, Erni will take you on a tour of how Quby leverages the full Databricks stack to quickly prototype, validate, scale and launch data science products. We will explore the technical workflow of a Data Science project from end to end. Starting from developing a notebook prototype and tracking the Machine Learning Model performance with ML Flow, we move towards production-grade Databricks jobs with a CI/CD pipeline, debugging production code with Databricks Connect, and finally setting up a monitoring system for the jobs.

1. Making Homes Efficient and
Comfortable using AI, IoT data
and the full Databricks stack
Erni Durdevic
Quby’s (customer) story
Making Homes Efficient and
Comfortable using AI, IoT data
and the full Databricks stack

2. We believe that the future can be better.
Easier, more comfortable, and more sustainable
without compromising on the important things in life.

3. 109,136,000,000,000 kWh
Worldwide annual energy consumption
Source: IEA
But how much is just wasted?

4. Source: Eurostat
44%
Households use 44% of all natural gas
and 27% of all electricity in the EU
27%

5. 400,000 connected homes across Europe

6. Boiler
adapters
Central heating
system
Z-Wave
Meter adaptors
Gas sensor
Electricity sensor
Gas
Water
Water sensor
Electricity
Z-Wave
Smart plugs & smoke detectors
Smart lighting 3rd party appsSmart speakers
Solar panels and power
storage

7. Outline
• Introducing Quby
• Data Driven Services
• Databricks Stack
• The Team

8. Our data-driven services
IoT Data Models Outcomes

9. Central heating
system
Water sensorElectricity sensorGas sensor
Our data-driven services
IoT Data Models Outcomes
Terabytes of IoT data daily
60+
ETLs

10. Our data-driven services
IoT Data Models Outcomes
Signal processing
Piecewise models
6 2
Deep Learning KNN GMMRandom ForestLightGBM
1 1 000 000+1 1
Offline training Daily re-training

11. Personalised advice on
how to avoid wasting
energy
Our data-driven services
IoT Data Models Outcomes

12. Outline
• Introducing Quby
• Data Driven Services
• Databricks Stack
• The Team

13. AWS S3
Our unified data analytics setup
IoT stream
Click data
Batch processing
Exploratory Data
Science Notebooks
Real time services
Live dashboards
Batch
Streaming
R & D
SQL
Services API
SQL / NoSQL

14. Our unified data analytics setup
IoT stream
Batch processing
AWS S3
Batch
Services API
SQL / NoSQL

15. One example: Thermostat schedule advice
Cooling down and warming up rate Humidity sensor

16. Batch processing
Unit tests
Preprocess
Electricity
Water
Presence
detection
Schedule
advice
…

17. Unit test tips

18. Unit test tips

19. Batch processing
Integration tests
Preprocess
Electricity
Gas
Presence
detection
Schedule
advice
Mock Mock

20. CI/CD
DB connect

21. Example

22. Data pipeline – lessons learned
• Enforce idempotent constraints
• Enforce reproducibility
• Let data transformations be chainable
• Leverage partitioning and data locality

23. Our unified data analytics setup
IoT stream
Click data
Batch processing
Exploratory Data
Science Notebooks
Batch
R & D
AWS S3
SQL
Services API
SQL / NoSQL

24. AWS S3
Our unified data analytics setup
IoT stream
Click data
Batch processing
Exploratory Data
Science Notebooks
Real time services
Live dashboards
Batch
Streaming
R & D
SQL
Services API
SQL / NoSQL

25. Apache Hive vs Delta Lake
Apache Hive Delta Lake
Transaction log

26. Outline
• Introducing Quby
• Data Driven Services
• Databricks Stack
• The Team

27. United Teams of Data
Use-case proposition
Software & Data
ML & Algorithms
Analytics Translators Data Scientists Machine Learning Engineers Data Engineers

28. Team skillset
0
0.5
1
1.5
2
2.5
0
10
20
30
40
50
60
70
80
90
100
DevOpsSoftware
Engineering
Data
Engineering
Machine
Learning
Engineering
Data AnalyticsML modelingData Science
proposition
Teamskillpoints(FTE)
Individualskillsetdistribution(%)
E E J K R B G S A Team skillpoints (FTE)

29. Why it is important?
We want to work in a
healthy and balanced team.
So we can build, deliver and
maintain great products and
services for our customers

30. Why does it work?
“developing and deploying ML systems is relatively fast and cheap, but
maintaining them over time is difficult and expensive.”

31. Why does it work?

32. Make a happy team!

33. Erni Durdevic
Senior Machine Learning Engineer
Erni.durdevic@quby.com
https://www.linkedin.com/in/ernidurdevic/