SlideShare ist ein Scribd-Unternehmen logo
1 von 38
Building a next-generation database

    david [dot] rosenthal@foundationdb.com
            Twitter: @FoundationDB
Motivation
Ease of building successful applications:
• High performance
• Ease scaling out
• Ease of building abstractions
• Ease of operation
History
Tools
Design
Results
Historical Perspective: 2008
                           Future




 NoSQL doesn’t really exist yet
Databases in 2008
Relational is entrenched; NoSQL emerging
with some interesting advantages:
• Voldemort
• Cassandra
• HBase
 …but the fine print about data guarantees
            doesn’t look so good.
The CAP2008 theorem
• Brewer: Pick 2 out of 3
• Werner Vogels (CTO Amazon.com): “Data
  inconsistency in large-scale reliable
  distributed systems has to be tolerated …
  [for performance and to handle faults]”
• Wrong descriptions all over the web: “The
  availability property means that the system
  is ‘online’ and the client of the system can
  expect to receive a response for its
  request.”
CAP2008 Conclusions?
• Scaling requires distributed design
• Distributed requires high availability
• Availability requires no C

 So, if we want scalability we have to give up C,
            the cornerstone of ACID.

                      Right?
Thinking about CAP2008
• Is a partition worse than a failure?
• Three computers can’t agree?
• Keyword: Availability…

       Availability != high availability
Flash forward to CAP2012
• Brewer: “Why ‘2 of 3’ is misleading”
• Brewer: “CAP prohibits … perfect availability”
• Vogles: “Achieving strict consistency can come at
  a cost in update or read latency, and may result in
  lower throughput…”
• Google (Spanner): “…it is better to have
  application programmers deal with performance
  problems due to overuse of transactions as
  bottlenecks arise, rather than always coding
  around the lack of transactions.“
The FoundationDB concept
• Attack CAP2008 and deliver transactions at
  NoSQL performance and scale
• Reduce core to minimal feature set
• Add features back with higher-level
  abstractions—“Layers”
• Decouple choice of data model and
  choice of storage technology
FoundationDB
Database software:        Application

•Ordered key-value API    Layer

•Scalable
                         Key-value API
•Transactional
•Fault tolerant
History
Tools
Design
Results
Engineering pressures
Engineering Challenge              Strategy
Engineering for extreme reliability Simulation
and fault tolerance of large clusters
under adverse conditions
Many asynchronous                     Erlang?
communicating processes
Fast algorithms; efficient I/O        C++

              We need new tools!
First tool: Flow
• A new programming language
• Adds actor-model concurrency to C++11
• New keywords: ACTOR, future, promise,
  wait, choose, when, streams
• Flow code -> C++11 code -> binary

               Seriously?
Flow allows…
• Testability by enabling simulation.
• Performance by compiling to native.
• Easier ACTOR-model coding.
Flow eases development
Flow output
Flow performance
Joe Armstrong (author of “Programming Erlang”):

“Write a ring benchmark. Create N processes in a ring.
Send a message round the ring M times so that a total
of N * M messages get sent. Time how long this takes
for different values of N and M. Write a similar
program in some other programming language you are
familiar with. Compare the results. Write a blog, and
publish the results on the internet!”
Flow performance
                 (N=1000, M=1000)
•   Ruby (using threads): 1990 seconds
•   Ruby (queues): 360 seconds
•   Objective C (using threads): 26 seconds
•   Java (threads): 12 seconds
•   Stackless Python: 1.68 seconds
•   Erlang: 1.09 seconds
•   Google Go: 0.87 seconds
•   Flow: 0.075 seconds
Second Tool: Lithium
•   Enabled by Flow
•   Simulate physical interfaces
•   Simulate failures modes
•   Deterministic simulation of entire system
Testability: Quicksand
Third tool: Magnesium
History
Tools
Design
Results
Traditional approaches
• Glue together smaller transactional
  systems
  – Two-phase-commit (Open/X XA)
  – Paxos
• Build on a distributed file system
  – BigTable/HBase
The FoundationDB approach
• Deconstruct a traditional transactional
  database and scale the individual parts
• Each part must also be fault tolerant
• The parts:
  – Accept requests
  – Check for transaction conflicts
  – Log transactions
  – Store data
Key insight
Checking for transaction conflicts
• Problem is scalable
• When highly optimized, is a small
  amount of the total % of work.
• Is tricky to make fault tolerant…
Training montage
•   Paxos coordination algorithm
•   Multi-versioned data structures
•   SSD optimizations
•   Application-managed page cache
•   Prioritization deeply integrated
•   Control theory for queue sizes
•   Testing, testing, testing
History
Tools
Design
Results
Did we reach our big goals?
•   High performance
•   Ease scaling out
•   Ease of building abstractions
•   Ease of operation
High performance
FoundationDB
delivers performance
exceeding other
NoSQL databases, but
with transactions!
Ease of scaling out
• Add and remove nodes on-the-fly
• Single key-space with global transactions
• Validated to 96-cores, 48-SSDs
Ease of building abstractions
• Transactions enable abstraction
• Abstractions very hard to build on non-
  transactional systems
• Ordered data model for performance

     Abstractions built on a scalable, fault
tolerant, transactional foundation inherit those
                   properties.
Examples of “ease”
• SQL database in one day
• Indexed table layer (3 days * 1 intern)
• Fractal spatial index in 200 lines:
Ease of operation
• Automatic data partitioning/replication
• Highly fault-tolerant
• Minimal management



          Try to break it yourself!
Conclusion
• Our mission is to solve the problem of state
  management so that developers can focus on
  building their applications
• 3+ years in the making, now ready for your
  applications
• Bindings for C, Python, JVM, Node.js, Ruby
Free at foundationdb.com
Join our Alpha community
Building a next-generation database

    david [dot] rosenthal@foundationdb.com
            Twitter: @FoundationDB

Weitere ähnliche Inhalte

Was ist angesagt?

Apache Kafka vs RabbitMQ: Fit For Purpose / Decision Tree
Apache Kafka vs RabbitMQ: Fit For Purpose / Decision TreeApache Kafka vs RabbitMQ: Fit For Purpose / Decision Tree
Apache Kafka vs RabbitMQ: Fit For Purpose / Decision TreeSlim Baltagi
 
Monitoring patterns for mitigating technical risk
Monitoring patterns for  mitigating technical riskMonitoring patterns for  mitigating technical risk
Monitoring patterns for mitigating technical riskItai Frenkel
 
Confluent Workshop Series: ksqlDB로 스트리밍 앱 빌드
Confluent Workshop Series: ksqlDB로 스트리밍 앱 빌드Confluent Workshop Series: ksqlDB로 스트리밍 앱 빌드
Confluent Workshop Series: ksqlDB로 스트리밍 앱 빌드confluent
 
ksqlDB: A Stream-Relational Database System
ksqlDB: A Stream-Relational Database SystemksqlDB: A Stream-Relational Database System
ksqlDB: A Stream-Relational Database Systemconfluent
 
Logstash-Elasticsearch-Kibana
Logstash-Elasticsearch-KibanaLogstash-Elasticsearch-Kibana
Logstash-Elasticsearch-Kibanadknx01
 
Practical Elasticsearch - real world use cases
Practical Elasticsearch - real world use casesPractical Elasticsearch - real world use cases
Practical Elasticsearch - real world use casesItamar
 
Apache Kafka Introduction
Apache Kafka IntroductionApache Kafka Introduction
Apache Kafka IntroductionAmita Mirajkar
 
The RabbitMQ Message Broker
The RabbitMQ Message BrokerThe RabbitMQ Message Broker
The RabbitMQ Message BrokerMartin Toshev
 
Apache Cassandra Multi-Datacenter Essentials (Julien Anguenot, iLand Internet...
Apache Cassandra Multi-Datacenter Essentials (Julien Anguenot, iLand Internet...Apache Cassandra Multi-Datacenter Essentials (Julien Anguenot, iLand Internet...
Apache Cassandra Multi-Datacenter Essentials (Julien Anguenot, iLand Internet...DataStax
 
Data Pipelines with Kafka Connect
Data Pipelines with Kafka ConnectData Pipelines with Kafka Connect
Data Pipelines with Kafka ConnectKaufman Ng
 
Using Redis Streams To Build Event Driven Microservices And User Interface In...
Using Redis Streams To Build Event Driven Microservices And User Interface In...Using Redis Streams To Build Event Driven Microservices And User Interface In...
Using Redis Streams To Build Event Driven Microservices And User Interface In...Redis Labs
 
Decouple and Scale Applications Using Amazon SQS and Amazon SNS - July 2017 A...
Decouple and Scale Applications Using Amazon SQS and Amazon SNS - July 2017 A...Decouple and Scale Applications Using Amazon SQS and Amazon SNS - July 2017 A...
Decouple and Scale Applications Using Amazon SQS and Amazon SNS - July 2017 A...Amazon Web Services
 

Was ist angesagt? (20)

HAProxy
HAProxy HAProxy
HAProxy
 
Apache kafka
Apache kafkaApache kafka
Apache kafka
 
Apache Kafka vs RabbitMQ: Fit For Purpose / Decision Tree
Apache Kafka vs RabbitMQ: Fit For Purpose / Decision TreeApache Kafka vs RabbitMQ: Fit For Purpose / Decision Tree
Apache Kafka vs RabbitMQ: Fit For Purpose / Decision Tree
 
Monitoring patterns for mitigating technical risk
Monitoring patterns for  mitigating technical riskMonitoring patterns for  mitigating technical risk
Monitoring patterns for mitigating technical risk
 
Confluent Workshop Series: ksqlDB로 스트리밍 앱 빌드
Confluent Workshop Series: ksqlDB로 스트리밍 앱 빌드Confluent Workshop Series: ksqlDB로 스트리밍 앱 빌드
Confluent Workshop Series: ksqlDB로 스트리밍 앱 빌드
 
Como reducir costos en AWS
Como reducir costos en AWSComo reducir costos en AWS
Como reducir costos en AWS
 
ksqlDB: A Stream-Relational Database System
ksqlDB: A Stream-Relational Database SystemksqlDB: A Stream-Relational Database System
ksqlDB: A Stream-Relational Database System
 
Apache kafka
Apache kafkaApache kafka
Apache kafka
 
Logstash-Elasticsearch-Kibana
Logstash-Elasticsearch-KibanaLogstash-Elasticsearch-Kibana
Logstash-Elasticsearch-Kibana
 
Deep Dive: AWS CloudFormation
Deep Dive: AWS CloudFormationDeep Dive: AWS CloudFormation
Deep Dive: AWS CloudFormation
 
Practical Elasticsearch - real world use cases
Practical Elasticsearch - real world use casesPractical Elasticsearch - real world use cases
Practical Elasticsearch - real world use cases
 
Apache Kafka Introduction
Apache Kafka IntroductionApache Kafka Introduction
Apache Kafka Introduction
 
The RabbitMQ Message Broker
The RabbitMQ Message BrokerThe RabbitMQ Message Broker
The RabbitMQ Message Broker
 
Apache Kafka - Overview
Apache Kafka - OverviewApache Kafka - Overview
Apache Kafka - Overview
 
Apache Cassandra Multi-Datacenter Essentials (Julien Anguenot, iLand Internet...
Apache Cassandra Multi-Datacenter Essentials (Julien Anguenot, iLand Internet...Apache Cassandra Multi-Datacenter Essentials (Julien Anguenot, iLand Internet...
Apache Cassandra Multi-Datacenter Essentials (Julien Anguenot, iLand Internet...
 
REST API
REST APIREST API
REST API
 
Data Pipelines with Kafka Connect
Data Pipelines with Kafka ConnectData Pipelines with Kafka Connect
Data Pipelines with Kafka Connect
 
Using Redis Streams To Build Event Driven Microservices And User Interface In...
Using Redis Streams To Build Event Driven Microservices And User Interface In...Using Redis Streams To Build Event Driven Microservices And User Interface In...
Using Redis Streams To Build Event Driven Microservices And User Interface In...
 
Decouple and Scale Applications Using Amazon SQS and Amazon SNS - July 2017 A...
Decouple and Scale Applications Using Amazon SQS and Amazon SNS - July 2017 A...Decouple and Scale Applications Using Amazon SQS and Amazon SNS - July 2017 A...
Decouple and Scale Applications Using Amazon SQS and Amazon SNS - July 2017 A...
 
Postman.ppt
Postman.pptPostman.ppt
Postman.ppt
 

Andere mochten auch (20)

Deterministic simulation testing
Deterministic simulation testingDeterministic simulation testing
Deterministic simulation testing
 
NoSQL and ACID
NoSQL and ACIDNoSQL and ACID
NoSQL and ACID
 
Непобедимая Москва
Непобедимая МоскваНепобедимая Москва
Непобедимая Москва
 
Presentation for donation 31 aug 2011 website
Presentation for donation 31 aug 2011 websitePresentation for donation 31 aug 2011 website
Presentation for donation 31 aug 2011 website
 
Vazta Music y Quiero Club
Vazta Music y Quiero ClubVazta Music y Quiero Club
Vazta Music y Quiero Club
 
Itm.leasing
Itm.leasing Itm.leasing
Itm.leasing
 
Resistor colour code
Resistor colour codeResistor colour code
Resistor colour code
 
Непобедимая Москва
Непобедимая МоскваНепобедимая Москва
Непобедимая Москва
 
Treball de plàstica
Treball de plàsticaTreball de plàstica
Treball de plàstica
 
Ubileinye daty
Ubileinye datyUbileinye daty
Ubileinye daty
 
Sistemes operatius
Sistemes operatiusSistemes operatius
Sistemes operatius
 
Музей
МузейМузей
Музей
 
Donation Presentation
Donation Presentation Donation Presentation
Donation Presentation
 
Google
GoogleGoogle
Google
 
RAJAN_RESUME -update
RAJAN_RESUME -updateRAJAN_RESUME -update
RAJAN_RESUME -update
 
Rha profile
Rha profileRha profile
Rha profile
 
Colagens
ColagensColagens
Colagens
 
Resumen vlan configuracion basica
Resumen vlan configuracion basicaResumen vlan configuracion basica
Resumen vlan configuracion basica
 
AP Government Practice Exam [5]
AP Government Practice Exam [5]AP Government Practice Exam [5]
AP Government Practice Exam [5]
 
AP Government 2002 MC Exam
AP Government 2002 MC ExamAP Government 2002 MC Exam
AP Government 2002 MC Exam
 

Ähnlich wie Building FoundationDB

Lessons from Building Large-Scale, Multi-Cloud, SaaS Software at Databricks
Lessons from Building Large-Scale, Multi-Cloud, SaaS Software at DatabricksLessons from Building Large-Scale, Multi-Cloud, SaaS Software at Databricks
Lessons from Building Large-Scale, Multi-Cloud, SaaS Software at DatabricksDatabricks
 
Hpc lunch and learn
Hpc lunch and learnHpc lunch and learn
Hpc lunch and learnJohn D Almon
 
Microservices - opportunities, dilemmas and problems
Microservices - opportunities, dilemmas and problemsMicroservices - opportunities, dilemmas and problems
Microservices - opportunities, dilemmas and problemsŁukasz Sowa
 
SpringPeople - Introduction to Cloud Computing
SpringPeople - Introduction to Cloud ComputingSpringPeople - Introduction to Cloud Computing
SpringPeople - Introduction to Cloud ComputingSpringPeople
 
Building a highly scalable and available cloud application
Building a highly scalable and available cloud applicationBuilding a highly scalable and available cloud application
Building a highly scalable and available cloud applicationNoam Sheffer
 
Architectural Decisions: Smoothly and Consistently
Architectural Decisions: Smoothly and ConsistentlyArchitectural Decisions: Smoothly and Consistently
Architectural Decisions: Smoothly and ConsistentlyComsysto Reply GmbH
 
Architectural Decisions: Smoothly and Consistently
Architectural Decisions: Smoothly and ConsistentlyArchitectural Decisions: Smoothly and Consistently
Architectural Decisions: Smoothly and ConsistentlyComsysto Reply GmbH
 
Measure and Increase Developer Productivity with Help of Serverless at JCON 2...
Measure and Increase Developer Productivity with Help of Serverless at JCON 2...Measure and Increase Developer Productivity with Help of Serverless at JCON 2...
Measure and Increase Developer Productivity with Help of Serverless at JCON 2...Vadym Kazulkin
 
Software Architecture and Architectors: useless VS valuable
Software Architecture and Architectors: useless VS valuableSoftware Architecture and Architectors: useless VS valuable
Software Architecture and Architectors: useless VS valuableComsysto Reply GmbH
 
Антон Бойко "Разделяй и властвуй — набор практик для построения масштабируемо...
Антон Бойко "Разделяй и властвуй — набор практик для построения масштабируемо...Антон Бойко "Разделяй и властвуй — набор практик для построения масштабируемо...
Антон Бойко "Разделяй и властвуй — набор практик для построения масштабируемо...Marina Peregud
 
What's New in .Net 4.5
What's New in .Net 4.5What's New in .Net 4.5
What's New in .Net 4.5Malam Team
 
Stay productive while slicing up the monolith
Stay productive while slicing up the monolithStay productive while slicing up the monolith
Stay productive while slicing up the monolithMarkus Eisele
 
Strata SC 2014: Apache Mesos as an SDK for Building Distributed Frameworks
Strata SC 2014: Apache Mesos as an SDK for Building Distributed FrameworksStrata SC 2014: Apache Mesos as an SDK for Building Distributed Frameworks
Strata SC 2014: Apache Mesos as an SDK for Building Distributed FrameworksPaco Nathan
 
Leapfrog into Serverless - a Deloitte-Amtrak Case Study | Serverless Confere...
Leapfrog into Serverless - a Deloitte-Amtrak Case Study | Serverless Confere...Leapfrog into Serverless - a Deloitte-Amtrak Case Study | Serverless Confere...
Leapfrog into Serverless - a Deloitte-Amtrak Case Study | Serverless Confere...Gary Arora
 
Cloud-native Data: Every Microservice Needs a Cache
Cloud-native Data: Every Microservice Needs a CacheCloud-native Data: Every Microservice Needs a Cache
Cloud-native Data: Every Microservice Needs a Cachecornelia davis
 
Data Lake and the rise of the microservices
Data Lake and the rise of the microservicesData Lake and the rise of the microservices
Data Lake and the rise of the microservicesBigstep
 
Azure Cosmos DB - The Swiss Army NoSQL Cloud Database
Azure Cosmos DB - The Swiss Army NoSQL Cloud DatabaseAzure Cosmos DB - The Swiss Army NoSQL Cloud Database
Azure Cosmos DB - The Swiss Army NoSQL Cloud DatabaseBizTalk360
 
Stay productive_while_slicing_up_the_monolith
Stay productive_while_slicing_up_the_monolithStay productive_while_slicing_up_the_monolith
Stay productive_while_slicing_up_the_monolithMarkus Eisele
 
Stay productive while slicing up the monolith
Stay productive while slicing up the monolithStay productive while slicing up the monolith
Stay productive while slicing up the monolithMarkus Eisele
 

Ähnlich wie Building FoundationDB (20)

Lessons from Building Large-Scale, Multi-Cloud, SaaS Software at Databricks
Lessons from Building Large-Scale, Multi-Cloud, SaaS Software at DatabricksLessons from Building Large-Scale, Multi-Cloud, SaaS Software at Databricks
Lessons from Building Large-Scale, Multi-Cloud, SaaS Software at Databricks
 
Hpc lunch and learn
Hpc lunch and learnHpc lunch and learn
Hpc lunch and learn
 
Windows Azure introduction
Windows Azure introductionWindows Azure introduction
Windows Azure introduction
 
Microservices - opportunities, dilemmas and problems
Microservices - opportunities, dilemmas and problemsMicroservices - opportunities, dilemmas and problems
Microservices - opportunities, dilemmas and problems
 
SpringPeople - Introduction to Cloud Computing
SpringPeople - Introduction to Cloud ComputingSpringPeople - Introduction to Cloud Computing
SpringPeople - Introduction to Cloud Computing
 
Building a highly scalable and available cloud application
Building a highly scalable and available cloud applicationBuilding a highly scalable and available cloud application
Building a highly scalable and available cloud application
 
Architectural Decisions: Smoothly and Consistently
Architectural Decisions: Smoothly and ConsistentlyArchitectural Decisions: Smoothly and Consistently
Architectural Decisions: Smoothly and Consistently
 
Architectural Decisions: Smoothly and Consistently
Architectural Decisions: Smoothly and ConsistentlyArchitectural Decisions: Smoothly and Consistently
Architectural Decisions: Smoothly and Consistently
 
Measure and Increase Developer Productivity with Help of Serverless at JCON 2...
Measure and Increase Developer Productivity with Help of Serverless at JCON 2...Measure and Increase Developer Productivity with Help of Serverless at JCON 2...
Measure and Increase Developer Productivity with Help of Serverless at JCON 2...
 
Software Architecture and Architectors: useless VS valuable
Software Architecture and Architectors: useless VS valuableSoftware Architecture and Architectors: useless VS valuable
Software Architecture and Architectors: useless VS valuable
 
Антон Бойко "Разделяй и властвуй — набор практик для построения масштабируемо...
Антон Бойко "Разделяй и властвуй — набор практик для построения масштабируемо...Антон Бойко "Разделяй и властвуй — набор практик для построения масштабируемо...
Антон Бойко "Разделяй и властвуй — набор практик для построения масштабируемо...
 
What's New in .Net 4.5
What's New in .Net 4.5What's New in .Net 4.5
What's New in .Net 4.5
 
Stay productive while slicing up the monolith
Stay productive while slicing up the monolithStay productive while slicing up the monolith
Stay productive while slicing up the monolith
 
Strata SC 2014: Apache Mesos as an SDK for Building Distributed Frameworks
Strata SC 2014: Apache Mesos as an SDK for Building Distributed FrameworksStrata SC 2014: Apache Mesos as an SDK for Building Distributed Frameworks
Strata SC 2014: Apache Mesos as an SDK for Building Distributed Frameworks
 
Leapfrog into Serverless - a Deloitte-Amtrak Case Study | Serverless Confere...
Leapfrog into Serverless - a Deloitte-Amtrak Case Study | Serverless Confere...Leapfrog into Serverless - a Deloitte-Amtrak Case Study | Serverless Confere...
Leapfrog into Serverless - a Deloitte-Amtrak Case Study | Serverless Confere...
 
Cloud-native Data: Every Microservice Needs a Cache
Cloud-native Data: Every Microservice Needs a CacheCloud-native Data: Every Microservice Needs a Cache
Cloud-native Data: Every Microservice Needs a Cache
 
Data Lake and the rise of the microservices
Data Lake and the rise of the microservicesData Lake and the rise of the microservices
Data Lake and the rise of the microservices
 
Azure Cosmos DB - The Swiss Army NoSQL Cloud Database
Azure Cosmos DB - The Swiss Army NoSQL Cloud DatabaseAzure Cosmos DB - The Swiss Army NoSQL Cloud Database
Azure Cosmos DB - The Swiss Army NoSQL Cloud Database
 
Stay productive_while_slicing_up_the_monolith
Stay productive_while_slicing_up_the_monolithStay productive_while_slicing_up_the_monolith
Stay productive_while_slicing_up_the_monolith
 
Stay productive while slicing up the monolith
Stay productive while slicing up the monolithStay productive while slicing up the monolith
Stay productive while slicing up the monolith
 

Kürzlich hochgeladen

The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
99.99% of Your Traces Are (Probably) Trash (SRECon NA 2024).pdf
99.99% of Your Traces  Are (Probably) Trash (SRECon NA 2024).pdf99.99% of Your Traces  Are (Probably) Trash (SRECon NA 2024).pdf
99.99% of Your Traces Are (Probably) Trash (SRECon NA 2024).pdfPaige Cruz
 
Valere | Digital Solutions & AI Transformation Portfolio | 2024
Valere | Digital Solutions & AI Transformation Portfolio | 2024Valere | Digital Solutions & AI Transformation Portfolio | 2024
Valere | Digital Solutions & AI Transformation Portfolio | 2024Alexander Turgeon
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXTarek Kalaji
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Brian Pichman
 
UiPath Studio Web workshop series - Day 5
UiPath Studio Web workshop series - Day 5UiPath Studio Web workshop series - Day 5
UiPath Studio Web workshop series - Day 5DianaGray10
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 

Kürzlich hochgeladen (20)

The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
99.99% of Your Traces Are (Probably) Trash (SRECon NA 2024).pdf
99.99% of Your Traces  Are (Probably) Trash (SRECon NA 2024).pdf99.99% of Your Traces  Are (Probably) Trash (SRECon NA 2024).pdf
99.99% of Your Traces Are (Probably) Trash (SRECon NA 2024).pdf
 
Valere | Digital Solutions & AI Transformation Portfolio | 2024
Valere | Digital Solutions & AI Transformation Portfolio | 2024Valere | Digital Solutions & AI Transformation Portfolio | 2024
Valere | Digital Solutions & AI Transformation Portfolio | 2024
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBX
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
 
UiPath Studio Web workshop series - Day 5
UiPath Studio Web workshop series - Day 5UiPath Studio Web workshop series - Day 5
UiPath Studio Web workshop series - Day 5
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 

Building FoundationDB

  • 1. Building a next-generation database david [dot] rosenthal@foundationdb.com Twitter: @FoundationDB
  • 2. Motivation Ease of building successful applications: • High performance • Ease scaling out • Ease of building abstractions • Ease of operation
  • 4. Historical Perspective: 2008 Future NoSQL doesn’t really exist yet
  • 5. Databases in 2008 Relational is entrenched; NoSQL emerging with some interesting advantages: • Voldemort • Cassandra • HBase …but the fine print about data guarantees doesn’t look so good.
  • 6. The CAP2008 theorem • Brewer: Pick 2 out of 3 • Werner Vogels (CTO Amazon.com): “Data inconsistency in large-scale reliable distributed systems has to be tolerated … [for performance and to handle faults]” • Wrong descriptions all over the web: “The availability property means that the system is ‘online’ and the client of the system can expect to receive a response for its request.”
  • 7. CAP2008 Conclusions? • Scaling requires distributed design • Distributed requires high availability • Availability requires no C So, if we want scalability we have to give up C, the cornerstone of ACID. Right?
  • 8. Thinking about CAP2008 • Is a partition worse than a failure? • Three computers can’t agree? • Keyword: Availability… Availability != high availability
  • 9. Flash forward to CAP2012 • Brewer: “Why ‘2 of 3’ is misleading” • Brewer: “CAP prohibits … perfect availability” • Vogles: “Achieving strict consistency can come at a cost in update or read latency, and may result in lower throughput…” • Google (Spanner): “…it is better to have application programmers deal with performance problems due to overuse of transactions as bottlenecks arise, rather than always coding around the lack of transactions.“
  • 10. The FoundationDB concept • Attack CAP2008 and deliver transactions at NoSQL performance and scale • Reduce core to minimal feature set • Add features back with higher-level abstractions—“Layers” • Decouple choice of data model and choice of storage technology
  • 11. FoundationDB Database software: Application •Ordered key-value API Layer •Scalable Key-value API •Transactional •Fault tolerant
  • 13. Engineering pressures Engineering Challenge Strategy Engineering for extreme reliability Simulation and fault tolerance of large clusters under adverse conditions Many asynchronous Erlang? communicating processes Fast algorithms; efficient I/O C++ We need new tools!
  • 14. First tool: Flow • A new programming language • Adds actor-model concurrency to C++11 • New keywords: ACTOR, future, promise, wait, choose, when, streams • Flow code -> C++11 code -> binary Seriously?
  • 15. Flow allows… • Testability by enabling simulation. • Performance by compiling to native. • Easier ACTOR-model coding.
  • 18. Flow performance Joe Armstrong (author of “Programming Erlang”): “Write a ring benchmark. Create N processes in a ring. Send a message round the ring M times so that a total of N * M messages get sent. Time how long this takes for different values of N and M. Write a similar program in some other programming language you are familiar with. Compare the results. Write a blog, and publish the results on the internet!”
  • 19. Flow performance (N=1000, M=1000) • Ruby (using threads): 1990 seconds • Ruby (queues): 360 seconds • Objective C (using threads): 26 seconds • Java (threads): 12 seconds • Stackless Python: 1.68 seconds • Erlang: 1.09 seconds • Google Go: 0.87 seconds • Flow: 0.075 seconds
  • 20. Second Tool: Lithium • Enabled by Flow • Simulate physical interfaces • Simulate failures modes • Deterministic simulation of entire system
  • 24. Traditional approaches • Glue together smaller transactional systems – Two-phase-commit (Open/X XA) – Paxos • Build on a distributed file system – BigTable/HBase
  • 25. The FoundationDB approach • Deconstruct a traditional transactional database and scale the individual parts • Each part must also be fault tolerant • The parts: – Accept requests – Check for transaction conflicts – Log transactions – Store data
  • 26. Key insight Checking for transaction conflicts • Problem is scalable • When highly optimized, is a small amount of the total % of work. • Is tricky to make fault tolerant…
  • 27. Training montage • Paxos coordination algorithm • Multi-versioned data structures • SSD optimizations • Application-managed page cache • Prioritization deeply integrated • Control theory for queue sizes • Testing, testing, testing
  • 29. Did we reach our big goals? • High performance • Ease scaling out • Ease of building abstractions • Ease of operation
  • 30. High performance FoundationDB delivers performance exceeding other NoSQL databases, but with transactions!
  • 31. Ease of scaling out • Add and remove nodes on-the-fly • Single key-space with global transactions • Validated to 96-cores, 48-SSDs
  • 32. Ease of building abstractions • Transactions enable abstraction • Abstractions very hard to build on non- transactional systems • Ordered data model for performance Abstractions built on a scalable, fault tolerant, transactional foundation inherit those properties.
  • 33. Examples of “ease” • SQL database in one day • Indexed table layer (3 days * 1 intern) • Fractal spatial index in 200 lines:
  • 34. Ease of operation • Automatic data partitioning/replication • Highly fault-tolerant • Minimal management Try to break it yourself!
  • 35. Conclusion • Our mission is to solve the problem of state management so that developers can focus on building their applications • 3+ years in the making, now ready for your applications • Bindings for C, Python, JVM, Node.js, Ruby
  • 37. Join our Alpha community
  • 38. Building a next-generation database david [dot] rosenthal@foundationdb.com Twitter: @FoundationDB