Divine into Complex Systems with Vayacondios
•
0 gefällt mir•1,104 views
This presentation given by Flip Kromer and Huston Hoburg on March 24, 2014 at the MongoDB Meetup in Austin. Vayacondios is a system we're building at Infochimps to gather metrics on highly complex systems and help humans make sense of their operation. You can think of it as a "data goes in, the right thing happens" machine: send in facts from anywhere about anything, and Vayacondios will promptly process and syndicate them to all consumers. Producers don't have to (or get to) worry about the needs of those who will use the data, or the details of transport, storage, filtering or anything else: the data will go where it needs to go. Each consumer, meanwhile, finds that everything they need to know is available to them, on the fly or on demand, without crufty adapters or extraneous dependencies. They don't have to (or get to) worry about the distribution of their sources, the tempo of update, or how the data came to be. Vayacondios was built for our technical ops team to monitor all the databases and systems they superintend, but it suggests a better way to build database driven applications of any kind. The quiet tyranny of developing against a traditional database has left us with many bad habits: not duplicating data, using models that serve the query engine not the user, assembling application objects from raw parts on every page refresh. Combining streaming data processing systems with distributed datastores like MongoDB let you do your query on the way _in_ to the database -- any number of queries, decoupled, of any complexity or tempo. The resulting approach is simpler, fault-tolerant, and scales in terms of machines and developers. Most importantly, your data models are purely faithful to the needs of your application, uncontaminated by differing opinions of other consumers or by incidentals of the robots that gather and process and store the data.
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Ähnlich wie Divine into Complex Systems with Vayacondios
Ähnlich wie Divine into Complex Systems with Vayacondios (20)
Mehr von Infochimps, a CSC Big Data Business
Mehr von Infochimps, a CSC Big Data Business (17)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Divine into Complex Systems with Vayacondios
- 1. Vayacondios: Divine into Complex Systems Huston Hoburg &Flip Kromer Infochimps, a CSC Company MongoDB Austin 2014 March 24th
- 2. Infochimps • Big Data Platform for Large Companies • Cloud::Queries (ElasticSearch,MongoDB,HBase) • Cloud::Hadoop (Dynamic Hadoop) • Cloud::Streams (Storm+Trident) • Managed Service, Enterprise Features • Recently sold to CSC, and it’s quite awesome • We’re Hiring (natch)
- 3. Vayacondios • Built for ourVisibility Stack… • … but we think it has wider use ! • “Data Goes In, the Right Thing Happens” • Prompt, Comprehensive and Faithful
- 7. OK, Glass “OK Glass, Show me a skeuomorphism”
- 10. Target INR = 2-3 Low Platelets = H.I.T. (bad) Heparin (Blood Thinner)
- 11. Low Platelets • Folic Acid,Vitamin B12 • Medication (Valproic Acid, Singulair, Heparin) • Sepsis • HIV • (about three dozen others)
- 12. Systems • Anatomical Systems: Circulatory, Immune, etc • Interventions: Drugs, Surgeries, … • Course of Treatment: topline progress indicators • Diagnosis • Practitioner • Medical Devices
- 13. ICU • Model the patient, not the data source • Highlight Interactions among systems • Highlight Interactions among numbers • Broaden your view of “systems”
- 14. Monitoring Sucks
- 15. Operations
- 16. System != Machine • Whole-System MongoDB: • Machines it runs on,Volumes it uses • Systems writing to it • Applications and Collections • Data Files, Logs, Repl Sets, Oplog, Arbiters • Codebase repo, Cookbooks, Configuration • Issue Tracker Tickets, Change Events
- 17. Operations • Cognitive model for Humans, not from Robots • Go beyond the Time-series Graph • Highlight Interactions • Link to Systems that write to this DB • Link to Github for Repos & Cookbooks • Drill into System • Issues in Issue Tracker • Broaden your view of “systems”
- 18. • 15 clients, 15 architectures • < 1 operator per client, 2 continents • 1500 machines in 150 clusters • 30+ technologies (HBase, MongoDB, Storm, …) • 4 Providers (AWS, Metal,VCE, OpenStack) • 3Virtualizations (AWS,VMWare, OpenStack) • Max 21 minutes downtime / month (99.95% SLA) Our Challenge
- 19. Systems to Instrument • WholeSystems: ZookeeperSystem, ElasticsearchSystem, HbaseSystem, HadoopMapredSystem, HadoopHdfsSystem, KafkaSystem, MysqlSystem, MysqlClientSystem, ListenerSetSystem, StormTridentSystem, MongodbSystem, NfsSystem, VayacondiosSystem,TachyonSystem, SplunkSystem, S3System, RdsSystem, PigSystem, HiveSystem, HueSystem • Machines: ZookeeperMachine, ElasticsearchDatanodeMachine, HBaseRegionserverMachine, HBaseMasterMachine, HadoopDnttMachine, HadoopTtonlyMachine, HadoopNamenodeMachine, HadoopJobtrackerMachine, HadoopSecondaryNamenodeMachine, HadoopFailoverMonitorMachine, MysqlServerMachine, KafkaBrokerMachine, PlatformListenerMachine, StormBolterMachine, StormMasterMachine, MongodbMachine, NfsServerMachine, VayacondiosServerMachine, PlatformApiMachine,TachyonServerMachine, HueMachine • Daemons: n, ElasticsearchDaemon, HbaseRegionserverDaemon, HbaseMasterDaemon, HadoopDatanodeDaemon, HadoopTasktrackerDaemon, HadoopNamenodeDaemon, HadoopJobtrackerDaemon, HadoopSecondaryNamenodeDaemon, HadoopFailoverDaemon, KafkaBrokerDaemon, MysqlDaemon, PlatformListenerDaemon, StormNimbusDaemon, StormUiDaemon, StormSupervisorDaemon, MongodbDatanodeDaemon, NfsServerDaemon, NtpDaemon, NfsClientDaemon, VayacondiosServerDaemon,TachyonServerDaemon, PlatformApiServerDaemon, HueBeeswaxDaemon • Providers:AwsProvider, CloudTrailProvider, OpenstackProvider, VceProvider, ChefServerProvider, Route53Provider, ElbProvider • Manifests: most of the above have a planned version and the realized version • Events: MachineLifecycle, CronJobLifecycle, ChefClientLifecycle • Build Artifacts:: FitDeployArtifact, DebArtifact, RpmArtifact, GemArtifact,AmiArtifact, OpenstackImageArtifact, VceTemplateArtifact, NpmArtifact,TarballArtifact • PlatformApps: HadoopJobLifecycle (Hive, Pig,Wukong),TridentJobLifecycle, MountweaselLifecycle • OpsProcesses: IncidentLifecycle, ChangeRequestLifecycle, FiredrillLifecycle, GitCommitLifecycle, ProblemLifecycle (JIRA), LunchladyLifecycle
- 20. Vayacondios • Visibility Stack for our operations team • Open-sourcing this summer • Internals in Ruby • Access anywhere (HTTP or log file) • MongoDB (but now please forget that fact)
- 22. Cognitive Model • MongoDB: • is_a Data store • has_many Network Services • has_many Daemons • has_many Machines • has_manyVolumes • has_many Collections • …etc
- 25. Faithful • Whiteboard rule: how do folks talk about system? • If you need it,it’s in the system Prompt • As fast as joint laws of Economics & Physics allow Comprehensive
- 30. Faithful to Source • crap data => well-formed data • uniform JSON-ready hash • syntax cleaned up • semantically unchanged • encouraged to model it, but let Wookiee win
- 32. Write Contract • Vaya Con Dios,“Go with God”. As the kingdom of heaven is unknowable, so is further fate of data: • How used • By Whom • How Processed • Where Stored
- 34. Reporters/Reports • Assemble Biographies into Reports • Faithful to application • Don’t know when will be run, why, etc
- 35. Presentation
- 36. Dashboarding
- 37. text metric text metric text metric text metric text metric text metric text metric text metric Model-Driven Templates
- 39. Model/Presenter/View • Report == Model • Reporter == Presenter • Dashboard .xml ==View
- 40. Model/Presenter/View • More targets that just dashboard! • Splunk+PagerDuty Alerts • Cucumber tests • Auditing reports (Security, Good Manners)
- 41. System Checks • Correctness, Consistency • Attached Directly to the Model • No worthwhile distinction between QA (integration tests) and live Alerts • Drive Splunk+Pager Duty for Alerts • Author Cucumber specs(!) for QA tests
- 42. Safe Systems
- 43. System Drift • Cognitive Model • Discoverable Interface • Testable Contract
- 44. Inevitability • If configured and reported, consistency checks • If reported, dashboard exists • If is_a generic system (eg filesystem), gets correctness tests (eg “capacity < 75%”) • If system A discovers system B: • dashboard has link from A to B • connectivity & security checks from A to B
- 45. Interaction • Monitoring systems do a terrible job here • Hard sources of failure: • Drift conceived != realized • Interaction unexpected consequences • Change oops
- 47. Application Design • Visibility into complex systems: • Biography of raw parts (raw Model) => Reporter (Presenter) => Summary of Systems (View-ready Model) • Database-driven Application • Model => Presenter => View
- 48. Simple Blog
- 49. Blog: Views Author Page Post Page Index Page
- 50. Blog: Views Author Page Post Page Index Page PostSynopsisReport PostReport UserReport CommentReport
- 51. “Query on the way In” ! • New/Updated Post: Update Post triggers… • Update PostReport • Update SynopsisReport • Update UserReport
- 52. “Query on the way In” ! • User fullname changes: Update User triggers… • Update UserReport • Update their SynopsisReports • Update their PostReports • Update their CommentReports
- 54. Faithful • Whiteboard rule: how do folks talk about system? • If you need it,it’s in the system Prompt • As fast as joint laws of Economics & Physics allow Comprehensive
- 55. Faithful • Single concern: subject of the biography • look at what’s offered,look at what reports need Prompt • Run as often as needed (not your concern) Comprehensive
- 56. Faithful • One Reporter per Application (*) & Topic • USCE Method:Utiliz’n,Saturat’n,Connections,Errors Prompt • Run as often as needed (not your concern) Comprehensive
- 57. Benefits • Separation of concerns: • Source complexity (API, parsing, translation) • Timing • Transport • Individual Applications • Reliability
- 58. Benefits • Separation of concerns: Source,Timing, Transport, Individual Applications, Reliability • No external libraries in application • Uniform access times • Reduce risk from multiple-dependencies
- 59. So What? • There’s not much to it: shims and conventions • VCD is not MongoDB • just like MongoDB is not mmap tables • Power through constraint: