Wrangling data like a boss
Effective data wrangling starts with data collection. Make sure to think about Front-end validation, limiting free text, GDPR, security, accessibility, inclusion, and designing to minimise the amount of data collected. The next part is effective data storage. You can store data in relational or non-relational data stores. Relational data solutions in Azure include Azure SQL Database, Azure Database for MariaDB, and Azure Database for PostgreSQL. Non-relational data solutions include Azure Storage and Azure Cosmos DB. You might wrangle data during a data movement process between systems. A batch process moves multiple records, typically on a schedule. A streaming process operates on each individual event/message as it happens. In ETL (Extract Transform Load) you run a process to retrieve data from a system, process it, and then insert it into a new data store. In ELT (Extract Load Transform) you run a process to retrieve data from a system, insert it into a new data store, and the process it. The modern data warehouse stack (used to combine data from multiple sources) includes Azure Data Factory, Azure Databricks, Azure Synapse Analytics and Azure Analysis Services. Power Query is a useful language for processing data and can be used in Excel, Power BI, and Azure Data Factory. Power BI is a self-service data modelling and visualisation tool that can connect to data stored in lots of different places. It has a data modeling area where Power Query can be used to build a dataset. The reports allow you make multi-page analysis of a dataset. A dashboard combines visuals from one or more reports to provide a broader and simpler view. SQL is a language that works with many relational and non-relational systems. You can use it interact with data, data structures, and the access model for the data objects. Finally you can write code to wrangle data with Python, R, and dotnet being common data wrangling languages.
Recomendados
Más contenido relacionado
Was ist angesagt?
Similar a Wrangling data like a boss
Más de Stephanie Locke
![Help There’s Too Many [Something]Ops!](https://cdn.slidesharecdn.com/ss_thumbnails/developervelocitywebinar3-210622151800-thumbnail.jpg?width=1600&height=908&fit=bounds)
Último
![[PositConf 2023] How Data Scientists Broke A/B Testing (and How We Can Fix It)](https://cdn.slidesharecdn.com/ss_thumbnails/posit-conf-ab-testing-no-notes2-230920192507-43c9986c-thumbnail.jpg?width=384&height=256&fit=bounds)
![[PositConf 2023] How Data Scientists Broke A/B Testing (and How We Can Fix It)](https://cdn.slidesharecdn.com/ss_thumbnails/posit-conf-ab-testing-no-notes2-230920192507-43c9986c-thumbnail.jpg?width=1600&height=908&fit=bounds)
Wrangling data like a boss
- 1. Wrangling data like a boss @theStephLocke
- 2. #DRF21 @theStephLocke @NightingaleHQai CEO @ Nightingale HQ Data & AI specialist Microsoft MVP, 5+ years Tech founder T: @theStephLocke Li: /stephanielocke
- 3. The plan
- 5. Garbage In, Garbage Out Front-end validation Avoid free text GDPR Security Accessibility Inclusion Attention spans
- 6. Examples
- 7. Data storage
- 9. Relational databases Database Schema Table Key Columns … Index Tables … Views Stored procedures Schema …
- 11. Non-relational databases { "id": 1, "film": {"name" : "Interstellar", "year" : 2014, "runtime" : 169, "oscars": 1}, "director":{ "id" : 1, "first_name": "Christopher", "last_name": "Nolan" } , "actors":[ { "id" : 1, "first_name": "Matthew", "last_name": "McConaughey" } , { "id" : 2, "first_name": "Anne", "last_name": "Hathaway" } , … ] } { "id": 2, "film": {"name" : "Batman Begins", "year" : 2005, "runtime" : 140, "oscars": 0}, "director":{ "id" : 1, "first_name": "Christopher", "last_name": "Nolan" } , "actors":[ { "id" : 5, "first_name": "Christian", "last_name": "Bale" } , { "id" : 3, "first_name": "Michael", "last_name": "Caine" } , … ] }
- 12. Azure non-relational data stores Data stores Azure Table Azure Blob Azure File Azure Cosmos DB
- 13. Cosmos DB • Multi-model NoSQL database Account Container 1 Partition 1 Object 1 … Object n … Partition n … Container n
- 14. Data movement
- 15. Batch processing
- 17. Extract Transform Load (ETL)
- 18. Extract Load Transform (ELT)
- 19. The modern data warehouse Azure Databricks Azure Data Factory Azure Synapse Analytics Azure Analysis Services
- 20. Power Query
- 21. Power Query Behind the scenes data wrangling for Power BI, Excel, and more!
- 22. Power BI • Self-service data modelling and visualisation tool • Connect to many different data sources • Host online
- 23. Power BI components Datasets • Power Query • DAX Reports • Visuals Dashboards • Tiles
- 26. Power BI report development
- 28. SQL
- 30. Types of SQL statements DML • Data Manipulation Language • Interact with data • CRUD: • SELECT • INSERT • UPDATE • DELETE DDL • Data Definition Language • Interact with database objects • CREATE, ALTER, DROP DCL • Data Control Language • Manage access to database • GRANT, REVOKE, DENY
- 31. Querying a Database • SELECT is the statement that retrieves data from tables • FROM dictates what table to retrieve data from • We can also add additional elements: • JOINs • WHERE • GROUP BY • ORDER BY • HAVING
- 32. Query tools Azure portal SQL Management Studio SQL Server Data Tools Azure Data Studio SQLCMD Azure CLI/Cloud Shell
- 33. Code
- 34. Code • Python, R, and dotnet are common data wrangling languages • Perform ETL • Perform analysis / data science • Stream & batch processes • Event-driven activities
- 35. Code example "somedirectory" |> list.files(pattern = "*.csv") |> map_df(read_csv) -> consolidated_data
- 36. Wrap up
- 37. Next steps Learn • Azure Data Fundamentals (MSFT) • Power BI Fundamentals (MSFT) • R Fundamentals (me) • Programmer Fundamentals (Li) Resources • Sqlfiddle.com • Microsoft Learn • Azure for Students • Meetup.com • R-Ladies