The document discusses how databases work. It begins with introducing the speaker and their background. It then outlines the agenda which includes discussing ACID properties, application interaction, backend operations, concurrent operations, OS/disk interaction, and NoSQL databases. It dedicates sections to explaining each of these concepts at a high level, with ACID properties ensuring atomicity, consistency, isolation, and durability of transactions.

1. How Databases Work
• Gurjeet Singh | 09.18.14
© 2013 EDB All rights reserved. 1

2. Who Am I
Database Architect @ EDB
– Index Advisor
– Optimizer Hints
– Postgres Enterprise Manager
– Postgres Plus Cloud Database
Postgres Contributor
– Postgres Hibernator and others
Learnt From Mistakes
© 2013 EDB All rights reserved. 2

3. Goal
Impart some new knowledge, that helps you do your
job better.
(The “Full Monty” slides is where the meat of the talk
is)
© 2013 EDB All rights reserved. 3

4. Agenda
• ACID
• Application Interaction
• Backend Operations
• Concurrent/Cooperating Operations
• OS/Disk Interaction
• NoSQL
© 2013 EDB All rights reserved. 4

5. ACID
© 2013 EDB All rights reserved. 5

6. ACID
Traits of a transaction in an RDBMS
• Atomicity
– Changes done by a transaction are one unit; all or
nothing.
• Consistency
– Transaction end leaves database in a consistent
state; constraints cannot be violated.
• Isolation
– Transactions behave as if only one transaction is
in progress at a time.
• Durability
– Transactions, once committed, will not be lost.
© 2013 EDB All rights reserved. 6

7. NoSQL
© 2013 EDB All rights reserved. 7

8. NoSQL
• We are not going to discuss NoSQL databases.
• They compromise on at least one of ACID properties
© 2013 EDB All rights reserved. 8

9. Application Interaction
© 2013 EDB All rights reserved. 9

10. Application Interaction
• Applications use a database driver
• The driver uses a custom protocol (language)
• Enables Client-Server/n-Tier architectures
Java
Ruby
Python
JavaScript
C/C++
© 2013 EDB All rights reserved. 10

11. Backend Operations
© 2013 EDB All rights reserved. 11

12. Backend Operations
• Query Processing
– Tokenize/Parse
– Parse Analysis
– Plan/Optimize
– Execute
© 2013 EDB All rights reserved. 12

13. Backend Operations – Tokenize/Parse
SELECT e.id, d.name
FROM emp e JOIN dept d
ON e.dept = d.id
WHERE emp.name = 'Agent Smith';
SELECT e . id , d . name
FROM emp e JOIN dept d
ON e . dept = d . id
WHERE emp . name = 'Agent Smith' ;
© 2013 EDB All rights reserved. 13

14. Backend Operations – Tokenize/Parse
SELECT List FROM List WHERE List
e.id d.name emp dept =
=
© 2013 EDB All rights reserved. 14
e.name 'Agent Smith'
Parse Tree
ON
e.dept d.id

15. Backend Operations – Parse Analysis
• Ensure that
– The objects (tables, columns, functions, etc.) exist
– Resolve/expand views
• Type Coercion
© 2013 EDB All rights reserved. 15

16. Backend Operations – Plan/Optimize
Generate all possible plans to access objects
• Compare cost of
– Accessing tables using various methods
● Full Table Scan, Index Scan, Bitmap Index, etc.
– Joining tables using various methods
● Nested Loop join, Hash Join, Sort-Merge Join
– Other operations like distinct, ordering, grouping.
• Choose the best plan
• Best may mean
– Fastest (e.g. LIMIT clause)
– Cheapest
© 2013 EDB All rights reserved. 16

17. Backend Operations – Plan/Optimize
Plan/Execution Tree
Result
Nested Loop
Index Scan - emp
Condition: name =
'Agent Smith'
Filter: e.dept =
© 2013 EDB All rights reserved. 17
Full Table Scan - dept
d.id
To Client

18. Backend Operations - Execute
• Traverse the Execution Tree recursively
• Pull rows from each execution node, until exhausted
• Insert/Update rows
– Delete in Postgres is just an update of row header
– Update is similar to Delete + Insert
– Multi-Version Concurrency Control (MVCC)
• Send result/rows to the client
© 2013 EDB All rights reserved. 18

19. Concurrent/Cooperating
Operations
© 2013 EDB All rights reserved. 19

20. Concurrent/Cooperating Operations
• Read data from disk
• Share data
• Locks to prevent concurrent update
• Write data to disk
© 2013 EDB All rights reserved. 20

21. Block Structure
© 2013 EDB All rights reserved. 21
Free Space
Block Header
Row Pointers
Row Header

22. MVCC
Isolation; A transaction will:
• Ignore changes done by newer transactions
• Ignore changes done by running transactions
• Ignore changes done by rolled-back transactions
• Only “see” changes by older and committed
transactions
Effectively:
• Readers don't block writers
• Writers don't block readers
© 2013 EDB All rights reserved. 22

23. MVCC
Provides
• Atomicity; with the help of commit log (clog)
• Isolation
Consistency
• Ensured by the individual backends
© 2013 EDB All rights reserved. 23

24. Full Monty
Client 1 Client 2 Client 3 Client n
Network
Backend 1 Backend 2 Backend n Postmaster
© 2013 EDB All rights reserved. 24
Checkpointer
Autovacuum
RAM
Disk Cache
Shared Buffers
WAL Buffers
Controller Cache Controller Cache
BGWriter

25. Full Monty
• WAL – Write Ahead Log
– Record change before the actual change
– Sequential, always growing infinite log
• Checkpoint
– Flush dirty data blocks to database
– Truncate WAL; prevent infinite growth
• Autovacuum
– Clean up dead row versions
• BGWriter
– Write dirty blocks so backends don't have to
© 2013 EDB All rights reserved. 25

26. Be Curious
Thank You
© 2013 EDB All rights reserved. 26