AWS Government, Education, Nonprofit Symposium 2015
•Als PPTX, PDF herunterladen•
2 gefällt mir•3,672 views
The document summarizes a presentation given at the AWS Government, Education, and Nonprofit Symposium on June 25-26, 2015 in Washington DC. The presentation discusses AWS as a data platform, highlighting the growing size and complexity of data as well as the various AWS services that can be used to store, process, analyze and gain insights from data at different scales. These services include S3, Glacier, DynamoDB, Redshift, EMR, Kinesis and Machine Learning among others. The presentation emphasizes that AWS provides a flexible suite of tools that can be used together to effectively manage the full data lifecycle and derive value from data.
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Andere mochten auch
Andere mochten auch (20)
Ähnlich wie AWS Government, Education, Nonprofit Symposium 2015
Ähnlich wie AWS Government, Education, Nonprofit Symposium 2015 (20)
Mehr von Amazon Web Services
Mehr von Amazon Web Services (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
AWS Government, Education, Nonprofit Symposium 2015
- 1. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 AWS as a Data Platform Joe Healy Sr. Consultant – AWS WWPS Professional Services ©2015, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 2. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 GB TB PB ZB EB Unconstrained growth: Big data is moving fast • 2.8 trillion GB in 2012 • 40 trillion GB in 2020 • 95% unstructured • 70% user-generated Source: IDC
- 3. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 It’s not just about size data velocity variety volume structured, unstructured, text, binary gigabytes, terabytes, petabytes millisecond, second, minute, hour, day
- 4. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 ease of uselower costs Why AWS?
- 5. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 no capital investment pay as you go no subscriptions pay only for what you use ease of uselower costs
- 6. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 programmable zero admin easy to configure integrates with existing tools ease of uselower costs
- 7. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 one tool to rule them all
- 8. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 II Use the right tools
- 9. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Movement and coordination Data PipelineDirect Connect Storage GatewayImport/Export
- 10. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Storage and analysis services EC2EBS Instance Storage RedshiftRDS SQL Stores EMR hadoop DynamoDB NOSQL Amazon Kinesis stream S3 Storage Services CloudFrontAmazon Glacier EFS machine Amazon Machine Learning
- 11. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Movement and coordination
- 12. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Movement and coordination: Plumbing ship us your disks Direct Connect Storage Gateway Import/ Export dedicated network pipes storage backup and archiving
- 13. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 AWS Data Pipeline resource management scheduling, execution, and retry dependency tracking failure notification AWS and on-premises Movement and coordination: Orchestration
- 14. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Data storage and analysis
- 15. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Storage services: Object store Amazon S3 Store objects (like “files”) Objects are stored in buckets Buckets keep data in a single AWS region, replicated across multiple facilities - cross-region replication highly durable, highly available, highly scalable - 99.999999999% durability
- 16. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Storage services: Archive storage low-cost, durable archive “cold storage” tape replacement infrequently accessed data integrated S3 lifecycle policies 99.999999999% durability immutable Amazon Glacier
- 17. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Storage services: Content delivery simple to use with global footprint (50+ edge locations) streaming support large file distribution private content S3, EC2, and ELB integration geo restrictions static and dynamic content Amazon CloudFront
- 18. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Instance storage: Options Ephemeral storage (“local”) you manage backup/restore free! high storage instances available i2.8xlarge – 6.4 TB SSD (350K IOPS) d2.8xl, hs1.8xl – 48 TB disk storage Amazon EC2 Elastic Block Store (EBS) “network attached storage” snapshot, encryption provisioned throughput (IOPS) magnetic or SSD up to 16 TB and 20,000 IOPS per volume
- 19. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Instance storage: Build your own Amazon EC2 NFS MongoDB Cassandra GraphLab Titan Kafka Luster Gluster Flume Scribe Presto …and more
- 20. PUBLIC MATERIAL | TOAN DO
- 21. Migrating to AWS: A proven method Refactor and develop applications for C2S cloud securely with the following requirements: portable across infrastructure, stable platform, and agile. stable platform enhanced security elastic stable platform enhanced security agile PUBLIC MATERIAL | TOAN DO
- 23. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Instance storage: Elastic File System Amazon EFS fully managed file system for EC2 instances works with standard operating system APIs sharable across thousands of instances elastically grows to petabyte scale delivers performance for a variety of workloads highly available and durable NFS v4–based
- 24. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 We focused on changing the game EFS is simple EFS is elastic EFS is scalable 1 2 3
- 25. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 EFS is simple • fully managed – no hardware, network, file layer – create a scalable file system in seconds • seamless integration with existing tools and apps – NFS v4—widespread, open – standard file system semantics – works with standard OS file system APIs • Simple pricing = simple forecasting 1
- 26. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 EFS is elastic • file systems grow and shrink automatically as you add and remove files • no need to provision storage capacity or performance • you pay only for the storage space you use, with no minimum fee 2
- 27. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 • file systems can grow to petabyte scale • throughput and IOPS scale automatically as file systems grow • consistent low latencies regardless of file system size • support for thousands of concurrent NFS connections EFS is scalable 3
- 28. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 MySQL, Aurora, Oracle, SQL Server, PostgreSQL backup/restore, high availability, encryption push-button scalability up to 3 TB and 30,000 IOPS Amazon RDS SQL stores - Managed relational DB
- 29. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 If you host your databases on-premises power, HVAC, net rack and stack server maintenance OS patches DB software patches database backups scaling high availability DB software installs OS installation app optimization
- 30. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 If you choose a managed DB service power, HVAC, net rack and stack server maintenance OS patches DB software patches database backups app optimization high availability DB software installs OS installation scaling
- 31. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 relational data warehouse massively parallel petabyte scale fully managed $1,000/TB/year Amazon Redshift SQL stores - Petabyte data warehouse
- 32. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 NoSQL database seamless scalability zero admin single-digit millisecond latency Amazon DynamoDB NoSQL - Dial up capacity
- 33. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 flexible tool and framework support - Hive, Impala, Pig, MapReduce, Presto, Spark easy to use; fully managed on-demand and Spot pricing persistent and transient clusters deep integration with S3 and other AWS services Amazon Elastic Map Reduce Hadoop - Managed
- 34. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 real-time data collection seamlessly scale to gigabytes/s low-cost managed service EMR integration Streaming at scale Amazon Kinesis
- 35. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Streaming - Amazon Kinesis architecture Amazon Web Services AZ AZ AZ durable, highly consistent storage replicates data across three data centers (Availability Zones) millions of sources producing 100s of terabytes per hour front end authentication authorization ordered stream of events supports multiple readers inexpensive: $0.028 per million puts aggregate analysis in Hadoop or data warehouse machine learning algorithms or sliding window analytics real-time dashboards and alarms aggregate and archive to S3
- 36. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 managed machine learning service built for developers robust technology based on Amazon’s internal systems create models using your existing data in AWS deploy models to production in seconds Machine learning made simple Amazon Machine Learning
- 37. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Smart applications by example based on what you know about the user: Will the user use your product? based on what you know about an order: Is this order fraudulent? based on what you know about a news article: What other articles are interesting?
- 38. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Three supported types of predictions • binary classification – predict the answer to a yes/no question • multiclass classification – predict the correct category from a list • regression – predict the value of a numeric variable
- 39. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 The right tool. At the right time. At the right scale.
- 40. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Open data on AWS …One more related topic
- 41. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 What is open data? Open data is data that can be used by anyone for any purpose for free. Many of our customers, such as Esri, the Weather Company, and the Climate Corporation, rely on quality open data as much as they rely on our computing, storage, and other web services.
- 42. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Data Enrichment Sensemaking Data at Rest (Object storage) Basic APIs Complex APIs Consumer applications Algorithmic policy Data-driven journalism Data Catalogs Focused data dashboards Predictive modeling Visualizations Lower cost of knowledge (Efficiency) Open data as a platform
- 43. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Data Creation Data Enrichment Sensemaking Data at Rest (Object storage) Basic APIs Complex APIs Consumer applications Algorithmic policy Data-driven journalism Data Catalogs Focused data dashboards Predictive modeling Visualizations Efficiency Open data as a platform
- 44. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Data Enrichment Sensemaking Amazon Kinesis Amazon EC2 Amazon EC2 AWS Data Pipeline Amazon S3 Amazon RDS Amazon EMR Amazon Redshift Amazon DynamoDB AWS Lambda Open data as a platform
- 45. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Public datasets on AWS To enable more innovation, AWS hosts a selection of datasets that anyone can access for free. Data in our public datasets is available for rapid access to our flexible and low-cost computing resources. earth science NASA Earth Exchange (NEX) life sciences 1000 Genomes project Internet science Common Crawl corpus
- 46. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Thank You. This presentation will be loaded to SlideShare the week following the Symposium. http://www.slideshare.net/AmazonWebServices AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015
Hinweis der Redaktion
- Notes: Due to the convergence of many technologies of cloud, mobile, social, and advancements in many field such as genomics, life sciences, space, the size of the digital universe is growing at an ever increasing rate. Customers have also found tremendous value in being able to mine this data to make better medicine, tailored purchasing recommendations, detect fraudulent financial transactions in real time, provide on-demand digital content such as movies and songs, predict weather forecasts, the list goes on and on. (IDC) for 2.8/40 t gb According to IDC, 95% of the 1.2 zettabytes of data in the digital universe is unstructured; and 70% of of this is user-generated content. Unstructured data is also projected for explosive growth, with estimates of compound annual growth (CAGR) at 62% from 2008 - 2012. Various analysts have attempted to quantify the gap between data generated by applications and data that makes it’s way into an analytical environment. The general trend is that the gap is large and growing – people simply cannot capture and process the volume of data they are generating. Big data services like EMR and Redshift are specifically designed to eliminate two of the major barriers to broad adoptions of analytics… effort and cost. Operational data often does not find it’s way to the data warehouse (or analytical environment) because the cost of a data warehouse prevents smaller, less-proven data sets from gaining entry. On top of this, the effort required to bring data into the warehouse is substantial relative to it perceived value. AWS data services like Redshift and EMR has been built and priced to change this – and allow customers to close the gap and start analyzing all their data. 1.2 zetabytes…1000 exabytes….1 billion terrabytes…one sextilllion
- OBAMA for America -> In the system, Ruby on Rails (RoR), Python/Django, PHP, and a host of other front- and mid-tier technologies intermingled, creating a robust heterogeneous design. Below that, the use of 10 different structured storage systems reflected a focus on bringing tools suited to the data itself. Intermingling technologies included relational database management services like Amazon Relational Database Service (Amazon RDS) for MySQL, PostgreSQL, and Microsoft SQL Server; NoSQL software like MongoDB, Apache Hadoop, Vertica, and LevelDB; and Amazon S3, Amazon DynamoDB, and Amazon SimpleDB.
- OBAMA for America -> In the system, Ruby on Rails (RoR), Python/Django, PHP, and a host of other front- and mid-tier technologies intermingled, creating a robust heterogeneous design. Below that, the use of 10 different structured storage systems reflected a focus on bringing tools suited to the data itself. Intermingling technologies included relational database management services like Amazon Relational Database Service (Amazon RDS) for MySQL, PostgreSQL, and Microsoft SQL Server; NoSQL software like MongoDB, Apache Hadoop, Vertica, and LevelDB; and Amazon S3, Amazon DynamoDB, and Amazon SimpleDB.
- Talk about how volume, velocity, and variety relate
- Your jobs' scheduling, execution, and retry logic Tracking the dependencies between your business logic, datasources, and previous processing steps to ensure that your logic does not run until all of its dependencies are met Sending any necessary failure notifications Creating and managing any temporary compute resources your jobs may require
- Support for many HTTP verbs GET, HEAD, POST, PUT, DELETE, PATCH and OPTIONS
- Talk about volume/velocity/variety
- After reviewing some possible self managed storage options built on top of EC2 and EBS, hand off to Red Hat for a showcase of their capabilities
- Bring copy of proposal -
- Who out there is responsible for managing a database on premises at the moment? Or maybe part of an app dev team with joint responsibility for a DB If you had your hand up you’re responsible for a lot… Everything from fundamental infrastructure responsibilities, all the way up to the bit that people actually notice Schema optimization = increases app performance = credit! On the other hand, further down this stack = routine, should be invisible, only get noticed when something goes wrong! Ton of work here. Wouldn't’t it be great if we could offload a bunch of it
- With managed database services you can. Managed database services like RDS, Redshift and DynamoDB take away all of this additional work like: Installing the database keeping it patched making sure its backed up regularly Leaves you to focus on the things that really add value to your customers experience and less so on the ‘routine stuff’ Doesn't mean that EC2 is no longer a viable platform to run databases. We have lots of of customers using EC2 today So what's right for you?
- More context – Mongo DB, Cassandra Name-value pairs 64k record size Hash and range key Very high velocity, with very stable performance. JPL -> Alongside the traditional cloud value props, we chose Dynamo for performance, ease of indexing our static data model, integration with cloud search, and flexibility to accept new (e.g. Crowd sources) data as we grew. This will soon be NASA's new public mars image viewer.
- More context – Mongo DB, Cassandra Variety – can process many different types, custom serdes, etc. Velocity – certain pacakages that run on Hadoop help with real time data injestion, like flume, storm, kafka, spark streaming Volume – designed to work on massive data sets.
- Volume – pretty high Velocity – very high Variety – good if it fits into 40k, otherwise need to do some lifting.
- Binary Classification: predict the answer to a yes/no question Is this order fraudulent? Will this customer convert? Which article should I show next? Multi-class classification: predict the correct category from list: What is the genre of this movie? What is the root cause of this customer contact? Regression: predict the value of a numeric variable How many units of this item will sell next week? How long will this user session last?
- We use a very basic definition of open data. The beauty of open data – if it’s good – is that it gives people something to do with our computing and storage resources. We have very significant customers who rely on open data for their work. Their products wouldn’t exist or wouldn’t be very useful without the availability of high quality, license-free, government information. Open data matters to our customers, so it matters to us.
- This graph is heavily inspired by Socrata, who created a similar graph to describe an “open data odyssey.” On the Y-axis, we plot advances in what I call “sensemaking,” which are innovations in user experience, visualization, and simple story telling that make it easier for people to understand findings from data. On the X-axis we plot data enrichment, which can start with something as simple as normalization or data clean up, but then advances into data mashups, and sophisticated analysis of many data sources. As we progress along these two axes, new products are developed, from basic APIs and visualizations, and all the way out to mass consumer applications and services. The Weather Channel could be considered an application of open data. What does this mean? I call it a “lower cost of knowledge,” which leads to more efficient decision making. It reduces the cost of learning the answers to questions like how much should I irrigate? Is this land a good investment? Is this a good road for a trucking route? Will the people who live in this neighborhood be able to afford my services? Should I apply for this patent? Is this drug safe? It allows businesses to create better products and deliver them more efficiently. At a very basic level, it means you can know whether or not you need to bring a jacket before you step out the door or when planning a trip – it’s very difficult to put a price on that, but imagine not being able to know that.
- So, that’s how we see it, but there’s another part of this chart that’s also very interesting to us, which is data creation. Data has to come from somewhere, and we have a lot to offer there too.
- In reality, this graph would be much messier because these products can be used in many different ways all over the graph (and I’m leaving out many of our services that apply here). What I want to communicate here is that we have products that help at every step of the way. From data production, storage, sharing, databases, and analysis.