Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
© 2015 GridPoint, Inc. Proprietary and Confidential 1
Managing (Schema) Migrations in Cassandra
Mitch Gitman
senior softwa...
© 2015 GridPoint, Inc. Proprietary and Confidential 2
10/23/2015
© 2015 GridPoint, Inc. Proprietary and Confidential 3
10/23/2015
© 2015 GridPoint, Inc. Proprietary and Confidential 4
10/23/2015
migration
A word with many meanings.
© 2015 GridPoint, Inc. Proprietary and Confidential 5
10/23/2015
disclaimer…
image © Ana Camamiel
© 2015 GridPoint, Inc. Proprietary and Confidential 6
What I mean by migrations
• Live-data migrations
10/23/2015
One-off ...
© 2015 GridPoint, Inc. Proprietary and Confidential 7
What I mean by migrations
• Source-driven migrations
− Schema migrat...
© 2015 GridPoint, Inc. Proprietary and Confidential 8
Database refactoring
10/23/2015
© 2015 GridPoint, Inc. Proprietary and Confidential 9
• Integration test & functional test automation (bootstrap-ability)
...
© 2015 GridPoint, Inc. Proprietary and Confidential 10
We need tools!
• Built into web application frameworks
• Standalone...
© 2015 GridPoint, Inc. Proprietary and Confidential 11
What do (perhaps) all these tools have in common?
10/23/2015
They’r...
© 2015 GridPoint, Inc. Proprietary and Confidential 12
NoSQL Distilled
10/23/2015
Chapter 12. Schema Migrations
"We have s...
© 2015 GridPoint, Inc. Proprietary and Confidential 13
10/23/2015
CREATE TABLE entities (
doc_id int,
attribute_name Strin...
© 2015 GridPoint, Inc. Proprietary and Confidential 14
The Cassandra migration tools landscape
10/23/2015
• Flyway: First-...
© 2015 GridPoint, Inc. Proprietary and Confidential 15
What’s the secret behind DB migration tools?
10/23/2015
The migrati...
© 2015 GridPoint, Inc. Proprietary and Confidential 16
Migration tool philosophies
10/23/2015
© Martha Stewart Living Omni...
© 2015 GridPoint, Inc. Proprietary and Confidential 17
Flyway for Cassandra
10/23/2015
• First-class Flyway• Faked-out Fly...
© 2015 GridPoint, Inc. Proprietary and Confidential 18
The tradeoff
10/23/2015
• Store the migrations tracking table in an...
© 2015 GridPoint, Inc. Proprietary and Confidential 19
Programmatically invoke Flyway
10/23/2015
© 2015 GridPoint, Inc. Proprietary and Confidential 20
10/23/2015
© 2015 GridPoint, Inc. Proprietary and Confidential 21
CassandraFlywayCallback
10/23/2015
implements FlywayCallback
© 2015 GridPoint, Inc. Proprietary and Confidential 22
Two-step process
10/23/2015
source control
artifact repository
Migr...
© 2015 GridPoint, Inc. Proprietary and Confidential 23
The migrations source
10/23/2015
The input to
MigrationsBuilder
© 2015 GridPoint, Inc. Proprietary and Confidential 24
10/23/2015
Run MigrationsBuilder for CQL:
Run MigrationsBuilder for...
© 2015 GridPoint, Inc. Proprietary and Confidential 25
The generated
migrations
10/23/2015
The output from
MigrationsBuild...
© 2015 GridPoint, Inc. Proprietary and Confidential 26
The generated SQL script
10/23/2015
Faking out Flyway
© 2015 GridPoint, Inc. Proprietary and Confidential 27
10/23/2015
Run FlywayMigrator for CQL:
Run FlywayMigrator for SQL:
...
© 2015 GridPoint, Inc. Proprietary and Confidential 28
10/23/2015
flyway-migrator-postgresql.jarflyway-migrator-cassandra....
© 2015 GridPoint, Inc. Proprietary and Confidential 29
The migrations version tracking table
10/23/2015
The Cassandra inca...
© 2015 GridPoint, Inc. Proprietary and Confidential 30
Best practices
10/23/2015
• Variations on versions
− Version contro...
© 2015 GridPoint, Inc. Proprietary and Confidential 31
Schema agreement
10/23/2015
https://datastax.github.io/java-driver/...
© 2015 GridPoint, Inc. Proprietary and Confidential 32
Cassandra… migrations… limitations
10/23/2015
• Limitations of our ...
© 2015 GridPoint, Inc. Proprietary and Confidential 33
Static vs. dynamic tables
10/23/2015
© 2015 GridPoint, Inc. Proprietary and Confidential 34
Deploy time vs. runtime
10/23/2015
Dedicated migration application ...
© 2015 GridPoint, Inc. Proprietary and Confidential 35
Source-driven, but…
10/23/2015
• The orchestration is in source con...
© 2015 GridPoint, Inc. Proprietary and Confidential 36
Embracing polyglot persistence
10/23/2015
A unified migrations solu...
© 2015 GridPoint, Inc. Proprietary and Confidential 37
Takeaways
10/23/2015
•challenging
•exciting
•routine
•boring
© 2015 GridPoint, Inc. Proprietary and Confidential 38
10/23/2015
Thank you!
Mitch Gitman
 mgitman@gridpoint.com
 mgitma...
Nächste SlideShare
Wird geladen in …5
×

Managing (Schema) Migrations in Cassandra

12.113 Aufrufe

Veröffentlicht am

I'm going to cover something which could be seen as essential for Cassandra but which hasn't gotten much attention in the Cassandra community and literature. It's schema migrations--how you go about pushing out and versioning changes to your keyspace and table definitions across environments. This is an area that has established solutions in the relational database world, with tools like Liquibase(http://www.liquibase.org/) and Flyway (http://flywaydb.org/) and in web frameworks like Rails and Grails.

I'll explain the different types of migrations but then focus, for most of the talk, on schema migrations. I'll explain how schema migrations have been done in the Cassandra community and the roadblocks teams have faced trying to use Liquibase and Flyway to manage Cassandra migrations.

Then I'll share an elegant, lightweight schema migrations system that we at GridPoint built on top of Flyway. I'll use our system as a context for discussing schema migration best practices for Cassandra and the various choices teams have for their migrations and table definitions, including when NOT to use a tool like Flyway. I'll also touch on the other types of migrations besides keyspace and table definitions that can be versioned and driven off source control.

Veröffentlicht in: Technologie
  • Dating direct: ❤❤❤ http://bit.ly/39pMlLF ❤❤❤
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier
  • Follow the link, new dating source: ❶❶❶ http://bit.ly/39pMlLF ❶❶❶
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier

Managing (Schema) Migrations in Cassandra

  1. 1. © 2015 GridPoint, Inc. Proprietary and Confidential 1 Managing (Schema) Migrations in Cassandra Mitch Gitman senior software engineer GridPoint, Inc.
  2. 2. © 2015 GridPoint, Inc. Proprietary and Confidential 2 10/23/2015
  3. 3. © 2015 GridPoint, Inc. Proprietary and Confidential 3 10/23/2015
  4. 4. © 2015 GridPoint, Inc. Proprietary and Confidential 4 10/23/2015 migration A word with many meanings.
  5. 5. © 2015 GridPoint, Inc. Proprietary and Confidential 5 10/23/2015 disclaimer… image © Ana Camamiel
  6. 6. © 2015 GridPoint, Inc. Proprietary and Confidential 6 What I mean by migrations • Live-data migrations 10/23/2015 One-off as opposed to ETL
  7. 7. © 2015 GridPoint, Inc. Proprietary and Confidential 7 What I mean by migrations • Source-driven migrations − Schema migrations − Reference data migrations − Test/sample data migrations • CQL commands as opposed to real data (sstables), generally 10/23/2015 source control versioning artifact versioning publish
  8. 8. © 2015 GridPoint, Inc. Proprietary and Confidential 8 Database refactoring 10/23/2015
  9. 9. © 2015 GridPoint, Inc. Proprietary and Confidential 9 • Integration test & functional test automation (bootstrap-ability) • CI server pipelines • Containerization?? • Consistency & repeatability across environments − Local developer box − Dev environments − Integration & QA environments − Staging − Production Source-driven DB refactoring—the benefits 10/23/2015
  10. 10. © 2015 GridPoint, Inc. Proprietary and Confidential 10 We need tools! • Built into web application frameworks • Standalone 10/23/2015
  11. 11. © 2015 GridPoint, Inc. Proprietary and Confidential 11 What do (perhaps) all these tools have in common? 10/23/2015 They’re relational. They’re for SQL.
  12. 12. © 2015 GridPoint, Inc. Proprietary and Confidential 12 NoSQL Distilled 10/23/2015 Chapter 12. Schema Migrations "We have seen that developing and maintaining an application in the brave new world of schemaless databases requires careful attention to be given to schema migration." either/or: • RDBMS = strong schema • NoSQL = no schema
  13. 13. © 2015 GridPoint, Inc. Proprietary and Confidential 13 10/23/2015 CREATE TABLE entities ( doc_id int, attribute_name String, attribute_value String, ... PRIMARY KEY(doc_id, attribute_name) ); • partition keys & clustering keys • table-per-query denormalization • shift from Thrift to CQL • Thrift: super columns & super column families • CQL: collection types “metadata-driven documents in columnar storage:” Does Cassandra like weak schemas? So how have teams been managing their keyspace & table definitions?
  14. 14. © 2015 GridPoint, Inc. Proprietary and Confidential 14 The Cassandra migration tools landscape 10/23/2015 • Flyway: First-class Cassandra support. − Requires JDBC. − https://github.com/flyway/flyway/issues/823 • Pillar: Scala tool. • mutagen-cassandra: Java tool, Astyanax driver. • Trireme: Python tool. • cql-migrate: Python tool. • mschematool: Python tool.
  15. 15. © 2015 GridPoint, Inc. Proprietary and Confidential 15 What’s the secret behind DB migration tools? 10/23/2015 The migrations version tracking table
  16. 16. © 2015 GridPoint, Inc. Proprietary and Confidential 16 Migration tool philosophies 10/23/2015 © Martha Stewart Living Omnimedia Inc. © Harpo Print, LLC
  17. 17. © 2015 GridPoint, Inc. Proprietary and Confidential 17 Flyway for Cassandra 10/23/2015 • First-class Flyway• Faked-out Flyway migrations (in SQL) CQL
  18. 18. © 2015 GridPoint, Inc. Proprietary and Confidential 18 The tradeoff 10/23/2015 • Store the migrations tracking table in an RDBMS
  19. 19. © 2015 GridPoint, Inc. Proprietary and Confidential 19 Programmatically invoke Flyway 10/23/2015
  20. 20. © 2015 GridPoint, Inc. Proprietary and Confidential 20 10/23/2015
  21. 21. © 2015 GridPoint, Inc. Proprietary and Confidential 21 CassandraFlywayCallback 10/23/2015 implements FlywayCallback
  22. 22. © 2015 GridPoint, Inc. Proprietary and Confidential 22 Two-step process 10/23/2015 source control artifact repository MigrationsBuilder FlywayMigrator
  23. 23. © 2015 GridPoint, Inc. Proprietary and Confidential 23 The migrations source 10/23/2015 The input to MigrationsBuilder
  24. 24. © 2015 GridPoint, Inc. Proprietary and Confidential 24 10/23/2015 Run MigrationsBuilder for CQL: Run MigrationsBuilder for SQL:
  25. 25. © 2015 GridPoint, Inc. Proprietary and Confidential 25 The generated migrations 10/23/2015 The output from MigrationsBuilder
  26. 26. © 2015 GridPoint, Inc. Proprietary and Confidential 26 The generated SQL script 10/23/2015 Faking out Flyway
  27. 27. © 2015 GridPoint, Inc. Proprietary and Confidential 27 10/23/2015 Run FlywayMigrator for CQL: Run FlywayMigrator for SQL: java -classpath /…/flyway-migrator-postgresql.jar com.gridpoint.tools.migrator.flyway.FlywayMigrator postgresql java -classpath /…/flyway-migrator-cassandra.jar com.gridpoint.tools.migrator.flyway.FlywayMigrator cassandra
  28. 28. © 2015 GridPoint, Inc. Proprietary and Confidential 28 10/23/2015 flyway-migrator-postgresql.jarflyway-migrator-cassandra.jar
  29. 29. © 2015 GridPoint, Inc. Proprietary and Confidential 29 The migrations version tracking table 10/23/2015 The Cassandra incarnation
  30. 30. © 2015 GridPoint, Inc. Proprietary and Confidential 30 Best practices 10/23/2015 • Variations on versions − Version control: f94c7d7f8b130df360a4e9e4f586eafc618ddc50 − Artifact repository: 3.5.1 − Migration tool: 201505270800 or 10 or whatever you want − Effective contract versions—multiple versions can coexist at runtime • Consistent deployment across environments • Failure handling • Baselining • Rollbacks? • Check schema agreement
  31. 31. © 2015 GridPoint, Inc. Proprietary and Confidential 31 Schema agreement 10/23/2015 https://datastax.github.io/java-driver/2.1.8/features/metadata/
  32. 32. © 2015 GridPoint, Inc. Proprietary and Confidential 32 Cassandra… migrations… limitations 10/23/2015 • Limitations of our Flyway-based solution − You need a relational database − Not open-sourced • Limitations of source-driven migrations, in general
  33. 33. © 2015 GridPoint, Inc. Proprietary and Confidential 33 Static vs. dynamic tables 10/23/2015
  34. 34. © 2015 GridPoint, Inc. Proprietary and Confidential 34 Deploy time vs. runtime 10/23/2015 Dedicated migration application vs. part of main application
  35. 35. © 2015 GridPoint, Inc. Proprietary and Confidential 35 Source-driven, but… 10/23/2015 • The orchestration is in source control • Actual data rather than CQL commands − Not necessarily live data − Maybe doesn’t need to be in source control
  36. 36. © 2015 GridPoint, Inc. Proprietary and Confidential 36 Embracing polyglot persistence 10/23/2015 A unified migrations solution
  37. 37. © 2015 GridPoint, Inc. Proprietary and Confidential 37 Takeaways 10/23/2015 •challenging •exciting •routine •boring
  38. 38. © 2015 GridPoint, Inc. Proprietary and Confidential 38 10/23/2015 Thank you! Mitch Gitman  mgitman@gridpoint.com  mgitman@nilistics.net  mgitman@gmail.com  skeletal presence @ LinkedIn

×