Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

An Introduction to Amazon Lake Formation - AWS Summit Sydney

103 Aufrufe

Veröffentlicht am

No code required! If you don't include SQL. Come and learn more about Amazon Lake Formation and get an understanding of how this service helps you to create a best practice modern data platform, assisting you in building capability for enhanced BI, data science, and machine workloads.

  • Als Erste(r) kommentieren

An Introduction to Amazon Lake Formation - AWS Summit Sydney

  1. 1. S U M M I T SYDNEY
  2. 2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Introduction to AWS Lake Formation Karsten Ploesser Solution Architect Amazon Web Services
  3. 3. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Data lakes play a pivotal role in your analytics strategy
  4. 4. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T But the actual engineering is still hard Create Amazon Simple Storage Service (Amazon S3) locations Configure access policies Map tables to Amazon S3 locationsETL jobs to load and clean data Create metadata access policies Configure access from analytics services Rinse and repeat for other: data sets, users, and end-services And more: manage and monitor ETL jobs update metadata catalog as data changes update policies across services as users and permissions change manually maintain cleansing scripts create audit processes for compliance … Manual | Error-prone | Time consuming Find sources
  5. 5. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Enforce security policies across multiple services Gain and manage new insights Identify, ingest, clean, and transform data Build a secure data lake in days
  6. 6. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T How does it work? Setup storage1 Move data2 Cleanse, prep, and catalog data 3 Configure and enforce security and compliance policies 4 Make data available for analytics 5
  7. 7. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T How does it work? Setup storage1 Move data2 Cleanse, prep, and catalog data 3 Configure and enforce security and compliance policies 4 Make data available for analytics 5
  8. 8. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Register existing data or import new data Amazon S3 forms the storage layer for AWS Lake Formation Data is stored in your account. You have direct access to it. No lock-in. Simply register existing Amazon S3 buckets that contain your data Ask AWS Lake Formation to create the required Amazon S3 buckets and import data into them Data Lake Storage Data Catalog Access Control Data import Crawlers ML-based data prep AWS Lake Formation Amazon Simple Storage Service (S3)
  9. 9. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Easily load data to your data lake logs DBs Blueprints one-shot incremental Data Lake Storage Data Catalog Access Control Data import Crawlers ML-based data prep AWS Lake Formation Amazon Simple Storage Service (S3) Amazon Aurora Amazon RDS Amazon Kinesis Data Firehose AWS CloudTrail Amazon CloudFrontElastic Load Balancing (ELB)
  10. 10. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T With blueprints You 1. Point us to the source 2. Tell us the location to load to in your data lake 3. Specify how often you want to load the data Blueprints 1. Discover the source table(s) schema 2. Automatically convert to the target data format 3. Automatically partition the data based on the partitioning schema 4. Keep track of data that was already processed 5. You can customise any of the above
  11. 11. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Blueprints build on AWS Glue
  12. 12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Machine Learning Transforms and AWS Lake Formation
  13. 13. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T What kinds of problems does this help address? Data Integration Finding the relationships between multiple datasets, even when those datasets do not share an identifier (or when their identifier is unreliable) Deduplication Transforming a dataset that has multiple rows referring to the same actual thing into a dataset where no two rows refer to the same actual thing
  14. 14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T How does it work? Setup storage1 Move data2 Cleanse, prep, and catalog data 3 Configure and enforce security and compliance policies 4 Make data available for analytics 5
  15. 15. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Secure once, access in multiple ways Data Lake Storage Data Catalog Access Control AWS Lake Formation 2. User tries to access data via one of the services 3. Service sends user credentials to Lake Formation 4. Lake Formation returns temporary credentials allowing data access 1. Set up user access in Lake Formation Admin Amazon Simple Storage Service (S3) Amazon Athena AWS GlueAmazon Redshift Amazon QuickSight Amazon EMR Amazon SageMaker
  16. 16. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Security permissions in AWS Lake Formation Control data access with simple grant and revoke permissions Specify permissions on tables and columns rather than on buckets and objects Easily view policies granted to a particular user Audit all data access at one place
  17. 17. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Security permissions in AWS Lake Formation Search and view permissions granted to a user, role, or group in one place Verify permissions granted to a user Easily revoke policies for a user
  18. 18. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Grant table and column-level permissions User 1 User 2
  19. 19. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Security deep dive User Principals can be AWS IAM users, roles Active Directory users via federation End-services retrieve underlying data directly from S3 AWS Lake Formation query T 1 request access T2 3short-term creds. for T 4 request objs comprising T return objs of T 5 Amazon Athena AWS Glue Amazon EMR Amazon Redshift Amazon Simple Storage Service (S3)
  20. 20. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T How does it work? Setup storage1 Move data2 Cleanse, prep, and catalog data 3 Configure and enforce security and compliance policies 4 Make data available for analytics 5
  21. 21. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Search and collaborate across multiple users Text-based, faceted search across all metadata Add attributes like data owners, stewards, and other as table properties Add data sensitivity level, column definitions, and others as column properties Text-based search and filtering Query data in Amazon Athena
  22. 22. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Audit and monitor in real time See detailed alerts in the AWS Lake Formation console Download audit logs for further analysis Data ingest and data catalog notifications are also published to Amazon CloudWatch events
  23. 23. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  24. 24. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Step 1: Blueprints to ingest data
  25. 25. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Monitor the import 1
  26. 26. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Imported data as table in the data lake
  27. 27. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Step 2: Grant permissions to securely share data
  28. 28. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Step 3: Run query in Amazon Athena
  29. 29. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Enforce security policies across multiple services Gain and manage new insights Identify, ingest, clean, and transform data Build a secure data lake in days
  30. 30. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Sign up for AWS Lake Formation preview today https://aws.amazon.com/lake-formation/
  31. 31. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Pricing of AWS Lake Formation No additional charges – Only pay for the underlying services used.
  32. 32. Thank you! S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Karsten Ploesser karstep@amazon.com AWS Lake Formation PM team lakeformation-pm@amazon.com

×