This document discusses Coursera's use of AWS services like Amazon Redshift, EMR, and Data Pipeline to consolidate their data from various sources, make the data easier for analysts and users to access, and increase the reliability of their data infrastructure. It describes how Coursera programmatically defined ETL pipelines using these services to extract, transform, and load data between sources like MySQL, Cassandra, S3, and Redshift. It also discusses how they built reporting and visualization tools to provide self-service access to the data and ensure high data quality and availability.

1. November 13, 2014 | Las Vegas | NV
Roy Ben-Alta (Big Data Analytics Practice Lead, AWS) Thomas Barthelemy (Software Engineer, Coursera)

2. Amazon
Redshift
Amazon
EMR
Amazon
EC2
Process & Analyze
Store
AWS Direct
Connect
Amazon
S3
Amazon
Kinesis
Amazon
Glacier
AWS
Import/Export
DynamoDB
Collect

3. Billing
DB
CSV files
CRMDB
Extract
Transform
Load
DWH
DB
DB
DB
ETL

5. Amazon
Redshift
Amazon
RDS
Amazon S3 as
Data Lake
Amazon
DynamoDB
Amazon S3 as Amazon EMR
Landing Zone
AWS Data Pipeline
Data
Feeds
Amazon EC2

9. http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp- program-pipeline.html

10. Amazon S3 as Data
Lake/Landing Zone
Amazon EMR
as ETL Grid
Amazon Redshift –
Production DWH
Visualization
Logs

12. Define
aws datapipeline create-pipeline –name myETL--unique-id token
--output: df-09222142C63VXJU0HC0A
Import
aws datapipeline put-pipeline-definition --pipeline-id df- 09222142C63VXJU0HC0A --pipeline-definition /home/repo/etl_reinvent.json
Activate
aws datapipeline activate-pipeline --pipeline-id df-09222142C63VXJU0HC0A

19. copy weblogs
from 's3://staging/'
credentials 'aws_access_key_id=<my access key>;aws_secret_access_key=<my secret key>'
delimiter ','

28. Coursera Big Data Analytics Powered by AWS
Thomas Barthelemy
SWE, Data Infrastructure
thomas@coursera.org

29. Overview
●About Coursera
●Phase 1: Consolidate data
●Phase 2: Get users hooked
●Phase 3: Increase reliability
●Looking forward

30. Coursera at a Glance

31. About Coursera
●Platform for Massive Open Online Courses
●Universities create the content
●Content free to the public

32. Coursera Stats
●~10 million learners
●+110 university partners
●>200 courses open now
●~170 employees

33. The value of data at Coursera
●Making strategic and tactical decisions
●Studying pedagogy

34. Becoming More Data-Driven
●Since the early days, Coursera has understood the value of data
oFounders came from machine learning
oMany of the early employees researched with the founders
●Cost of data access was high
oeach analysis requires extraction, pre-processing
odata only available to data scientists + engineers

35. Phase 1:
Consolidate data

36. Sources
●MySQL
oSite data
oCourse data sharded across multiple database
●Cassandra increasingly used for course data
●Logged event data
●External APIs
Classes
Classes
SQL
Sharded SQL
NoSQL
Logs
External APIs

37. (Obligatory meme)

38. What platforms to use?
●Amazon Redshifthad glowing recommendations
●AWS Data Pipelinehas native support for various Amazon services

39. ETL development was slow :(
●Slow to do the following in the console:
ocreate one pipeline
ocreate similar pipelines
oupdate existing pipelines

40. Solution: Programmatically create pipelines
●Break ETL into reusable steps
oExtract from variety of sources
oTransform data with Amazon EMR, Amazon EC2, or within Amazon Redshift
oLoad data principally into Amazon Redshift
●Use Amazon S3 as intermediate state for Amazon EMR-or Amazon EC2-based transformations
●Map stepsto set of AWS Data Pipeline objects

41. Example step: extract-rds
●Parameters: hostname, database name, SQL
●Creates pipeline objects:
S3 Node can be used by many other step types

42. Example load definition
steps:
-type:extract-from-rds
sql:| SELECT instructor_id, course_id, rank
FROM courses_instructorincourse;
hostname:maestro-read-replica
database:maestro
-type:load-into-staging-table
table: staging.maestro_instructors_sessions
-type:reload-prod-table
source:staging.maestro_instructors_sessions
destination:prod.instructors_sessions
Extract data from Amazon RDS
Load intermediate table in Amazon Redshift
Reload target table with new data

43. ETL –Amazon RDS
SQL
AmazonS3
AmazonRedshift
Extract
Load

44. ETL –Sharded RDS
AmazonS3
AmazonS3
AmazonRedshift
SQL
AmazonS3
AmazonEMR
AmazonS3
AmazonEMR
AmazonEMR
Transform
Extract
Load

45. ETL –Logs
Logs
Amazon EC2
Amazon S3
Amazon S3
AmazonEMR
AmazonRedshift
Amazon S3
Transform
Load
Extract

46. Reporting Model, Dec 2013

47. Reporting Model, Sep 2014

48. AWS Data Pipeline
●Easily handles starting/stopping of resources
●Handles permissions, roles
●Integrates with other AWS services
●Handles “flow” of data, data dependencies

49. Dealing with large pipelines
●Monolithic pipelines hard to maintain
●Moved to making pipelines smaller
oHooray modularity!
●If pipeline B depended on pipeline A, just schedule it later
oAdd a time buffer just to be safe

50. Setting cross-pipeline dependencies
●Dependencies accomplished using a script that would wait until dependencies finished
oShellCommandActivityto the rescue

51. The beauty of ShellCommandActivity
●You can use it anywhere
oAccomplish tasks that have no corresponding activity type
oOverride native AWS Data Pipeline support if it does not meet your needs

52. ETL library
●Install on machine as first step of each pipeline
oWith ShellCommandActivity
●Allows for even more modularity

53. Phase 2:
Get users hooked

54. We have data. Now what?
●Simply collecting data will not make a company data-driven
●First step: make data easier for the analysts to use

55. Certifying a version of the truth
●Data Infrastructure team creates a 3NF model of the data
●Model is designed to be as interpretable as possible

56. Example: Forums
●Full-word names for database objects
●Join keys made obvious through universally- unique column names
oe.g. session_id joins to session_id
●Auto-generated ER-diagram, so documentation is up-to-date

57. Some power users need data faster!
●Added new schemas for developers to load data
●Help us to remain data driven during early phases of…
oProduct release
oAnalyzing new data

58. Lightweight SQL interface
●Access via browser
oUsers don’t need database credentials or special software
●Data exportable as CSV
●Auto-complete for DB object names
●Took ~2 days to make

59. Lightweight SQL interface

60. But what about people who don’t know SQL?
●Reporting tools to the rescue!
●In-house dashboard framework to support:
oSupport
oGrowth
oData Infrastructure
oMany other teams...
●Tableau for interactive analysis

61. But what about people who don’t know SQL?

62. Phase 3:
Increase reliability

63. If you build it, they will come
●Built ecosystem of data consumers
oAnalysts
oBusiness users
oProduct team

64. Maintaining the ecosystem
●As we accumulate consumers, need to invest effort in...
oModel stability
oData quality
oData availability

65. Persist Amazon Redshift logs to keep track of usage
●Leverage database credentials to understand usage patterns
●Know whom to contact when
oNeed to change the schema
oNeed to add a new set of tables similar to an existing set (e.g. assessments)

66. Data Quality
●Quality of source systems (GIGO)
oEncourage source system to fix issues
●Quality of transformations
oResponsibility of the data infrastructure team

67. Quality Transformations
●Automated QA checks as part of ETL
oAgain, use ShellCommandActivity
●Key factors
oCounts
oValue comparisons
oModeling rules (primary keys)

68. Ensuring High Reliability
●Step retries catch most minor hiccups
●On-call system to recover failed pipelines
●Persist DB logs to keep track of load times
oDelayed loads can also alert on-call devs
oBonus: users know how fresh the data is
oBonus: can keep track of success rate
●AWS very helpful in debugging issues

69. Looking Forward

70. Current bottlenecks
●Complex ETL
oHow do we easily ETL complex, nested structures?
●Developer bottleneck
oHow do we allow users to access raw data more quickly?

71. Amazon S3 as principal data store
●Amazon S3 to store raw data as soon as it’s produced (data lake)
●Amazon EMR to process data
●Amazon Redshift to remain as analytic platform

72. Thank you!

73. Want to learn more?
●github.com/coursera/dataduct
●http://tech.coursera.org
●thomas@coursera.org

75. http://blogs.aws.amazon.com/bigdata/ http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-concepts-schedules.html

76. http://bit.ly/awsevals