SlideShare ist ein Scribd-Unternehmen logo
1 von 168
Eat My Data: (now with 20% more rant!) How everybody gets file I/O wrong Stewart Smith [email_address] Senior Software Engineer, MySQL Cluster MySQL AB
What I work on ,[object Object],[object Object],[object Object],[object Object]
Overview ,[object Object]
Overview ,[object Object],[object Object],[object Object]
Overview ,[object Object],[object Object],[object Object],[object Object]
Overview ,[object Object],[object Object],[object Object],[object Object],[object Object]
Overview ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
In the beginning ,[object Object]
In the beginning ,[object Object],[object Object]
In the beginning ,[object Object],[object Object],[object Object]
A world without failure
A world without failure ,[object Object]
A world without failure ,[object Object],[object Object]
A world without failure ,[object Object],[object Object],[object Object]
A world without failure ,[object Object],[object Object],[object Object],[object Object]
A world without failure ,[object Object],[object Object],[object Object],[object Object],[object Object]
A world without failure ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
A world without failure ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Data Consistency ,[object Object]
User Expectations ,[object Object]
User Expectations ,[object Object],[object Object]
User Expectations ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Databases
Databases ,[object Object]
Databases ,[object Object],[object Object]
Databases ,[object Object],[object Object],[object Object],[object Object]
Databases ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Databases ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Databases ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Databases ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
What databases are good at ,[object Object]
What databases are good at ,[object Object],[object Object]
What databases are good at ,[object Object],[object Object],[object Object]
What databases are good at ,[object Object],[object Object],[object Object],[object Object],[object Object]
Easy solution to data consistency ,[object Object],[object Object],[object Object]
Revelation #1 ,[object Object]
Revelation #2 ,[object Object]
Revelation #3 ,[object Object]
Revelation #3 ,[object Object],[object Object]
Eat my data ,[object Object]
Where data can be ,[object Object],[object Object]
Where data can be ,[object Object],[object Object],[object Object],[object Object]
Where data can be ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Where data can be ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Data flow ,[object Object],[object Object]
Data flow ,[object Object],[object Object],[object Object],[object Object]
Data flow ,[object Object],[object Object],[object Object],[object Object],[object Object]
Data flow ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Data flow ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Data flow ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Data flow ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Simple Application: Save==on disk ,[object Object],[object Object],[object Object]
Saving a simple document ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Bug #1 ,[object Object],[object Object]
Word Processor Saving -1 Bug ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Bug #2, 3 and 4 ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
File System Integrity ,[object Object]
File System Integrity ,[object Object],[object Object]
File System Integrity ,[object Object],[object Object],[object Object]
File System Integrity ,[object Object],[object Object],[object Object],[object Object]
File System Integrity ,[object Object],[object Object],[object Object],[object Object],[object Object]
Data journaling ,[object Object]
Atomic write(2)
Atomic write(2) ,[object Object]
Atomic write(2) ,[object Object],[object Object]
Atomic write(2) ,[object Object],[object Object],[object Object]
Atomic write(2) ,[object Object],[object Object],[object Object],[object Object]
Atomic write(2) ,[object Object],[object Object],[object Object],[object Object],[object Object]
Eat My Data ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],CRASH
Write to Temp file, rename ,[object Object]
Write to Temp file, rename ,[object Object],[object Object],[object Object]
Write to Temp file, rename ,[object Object],[object Object],[object Object],[object Object]
Write to Temp file, rename ,[object Object],[object Object],[object Object],[object Object],[object Object]
Write to Temp file, rename ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Temp file, rename ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Now all is good with the world...
Now all is good with the world... ,[object Object]
Now all is good with the world... ,[object Object],[object Object]
[object Object]
Now all is good with the world... ,[object Object],[object Object],[object Object]
Now all is good with the world... ,[object Object],[object Object],[object Object],[object Object]
File System Integrity ,[object Object],[object Object],[object Object]
File System Integrity ,[object Object],[object Object],[object Object],[object Object]
File System Integrity ,[object Object],[object Object],[object Object],[object Object],[object Object]
data=ordered ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
other systems ,[object Object],[object Object],[object Object],[object Object],[object Object]
flush and sync ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Flush the buffers! Sync to disk  before  rename
A tale of libxml2 ,[object Object]
A tale of libxml2 ,[object Object],[object Object]
A tale of libxml2 ,[object Object],[object Object],[object Object]
A tale of libxml2 ,[object Object],[object Object],[object Object],[object Object]
A tale of libxml2 ,[object Object],[object Object],[object Object],[object Object],[object Object]
so, replace ,[object Object]
gint common_save_xml(xmlDocPtr doc, gchar *filename) { FILE  *fp; char  *xmlbuf; int  fd, n; fp = g_fopen(filename, &quot;w&quot;); if(NULL == fp) return -1; xmlDocDumpFormatMemory(doc, (xmlChar **)&xmlbuf, &n, TRUE); if(n <= 0) { errno = ENOMEM; return -1; } if(fwrite(xmlbuf, sizeof (xmlChar), n, fp) < n) { xmlFree (xmlbuf); return -1; } xmlFree (xmlbuf); /* flush user-space buffers */ if (fflush (fp) != 0) return -1; if ((fd = fileno (fp)) == -1) return -1; #ifdef HAVE_FSYNC /* sync kernel-space buffers to disk */ if (fsync (fd) == -1) return -1; #endif fclose(fp); return 0; }
Nearing Nirvana ,[object Object],[object Object]
Except if you want to be portable... ,[object Object],[object Object]
Except if you want to be portable... ,[object Object],[object Object],[object Object]
Except if you want to be portable... ,[object Object],[object Object],[object Object]
on fsync, POSIX Says... ,[object Object]
on fsync, POSIX Says... ,[object Object]
POSIX compliant fsync ,[object Object]
POSIX compliant fsync ,[object Object],[object Object]
POSIX compliant fsync ,[object Object],[object Object],[object Object]
POSIX compliant fsync ,[object Object],[object Object],[object Object],[object Object]
POSIX compliant fsync ,[object Object],[object Object],[object Object],[object Object],gcc
POSIX compliant fsync ,[object Object],[object Object],[object Object],[object Object],pushl  %ebp movl  %esp, %ebp movl  $0, %eax popl  %ebp ret gcc
Tale of a really fast database server ,[object Object]
Tale of a really fast database server ,[object Object],[object Object]
Tale of a really fast database server ,[object Object],[object Object],[object Object]
Tale of a really fast database server ,[object Object],[object Object],[object Object],[object Object]
fsync() doesn't have to sync ,[object Object]
fsync() doesn't have to sync ,[object Object],[object Object]
Standards are great ,[object Object]
Standards are great ,[object Object],[object Object]
Standards are great ,[object Object],[object Object],[object Object]
Standards are great ,[object Object],[object Object],[object Object],[object Object]
#ifdef HAVE_DARWIN_THREADS # ifdef F_FULLFSYNC /* This executable has been compiled on Mac OS X 10.3 or later. Assume that F_FULLFSYNC is available at run-time. */ srv_have_fullfsync = TRUE; # else /* F_FULLFSYNC */ /* This executable has been compiled on Mac OS X 10.2 or earlier.  Determine if the executable is running on Mac OS X 10.3 or later. */ struct utsname utsname; if (uname(&utsname)) { fputs(&quot;InnoDB: cannot determine Mac OS X version!&quot;, stderr); } else { srv_have_fullfsync = strcmp(utsname.release, &quot;7.&quot;) >= 0; } if (!srv_have_fullfsync) { fputs(&quot;InnoDB: On Mac OS X, fsync() may be&quot; &quot; broken on internal drives,&quot; &quot;InnoDB: making transactions unsafe!&quot;, stderr); } # endif /* F_FULLFSYNC */ #endif /* HAVE_DARWIN_THREADS */
#if defined(HAVE_DARWIN_THREADS) # ifndef F_FULLFSYNC /* The following definition is from the Mac OS X 10.3 <sys/fcntl.h> */ #  define F_FULLFSYNC 51 /* fsync + ask the drive to flush to the media */ # elif F_FULLFSYNC != 51 #  error &quot;F_FULLFSYNC != 51: ABI incompatibility with Mac OS X 10.3&quot; # endif /* Apple has disabled fsync() for internal disk drives in OS X. That caused corruption for a user when he tested a power outage. Let us in OS X use a nonstandard flush method recommended by an Apple engineer. */ if (!srv_have_fullfsync) { /* If we are not on an operating system that supports this, then fall back to a plain fsync. */ ret = fsync(file); } else { ret = fcntl(file, F_FULLFSYNC, NULL); if (ret) { /* If we are not on a file system that supports this, then fall back to a plain fsync. */ ret = fsync(file); } } #elif HAVE_FDATASYNC ret = fdatasync(file); #else /*  fprintf(stderr, &quot;Flushing to file %p&quot;, file); */ ret = fsync(file); #endif
Yes, some OS Vendors hate you ,[object Object]
Big Files
Big Files ,[object Object]
Big Files ,[object Object],[object Object]
Big Files ,[object Object],[object Object],[object Object]
Big Files ,[object Object],[object Object],[object Object],[object Object]
Big Files ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Big Files ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Large directories ,[object Object]
Large directories ,[object Object],[object Object]
Large directories ,[object Object],[object Object],[object Object]
Large directories ,[object Object],[object Object],[object Object],[object Object],[object Object]
Large directories ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Where data is after being written ,[object Object]
Where data is after being written ,[object Object],[object Object],[object Object]
Where data is after being written ,[object Object],[object Object],[object Object],[object Object]
Where data is after being written ,[object Object],[object Object],[object Object],[object Object],[object Object]
Where data is after being written ,[object Object],[object Object],[object Object],[object Object],[object Object]
Where data is after being written ,[object Object],[object Object],[object Object],[object Object],[object Object]
sqlite ,[object Object]
sqlite ,[object Object],[object Object]
sqlite ,[object Object],[object Object],[object Object]
sqlite ,[object Object],[object Object],[object Object],[object Object]
sqlite ,[object Object],[object Object],[object Object],[object Object],[object Object]
sqlite ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
sqlite ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Performance of Large files ,[object Object],[object Object]
Performance of Large files ,[object Object],[object Object],[object Object]
Performance of Large files ,[object Object],[object Object],[object Object],[object Object]
Performance of Large files ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
inode Infos Direct blocks Indirect Blocks Double indirect blocks
Extent ,[object Object],[object Object],[object Object],[object Object],[object Object]
Parallel writers ,[object Object]
Parallel writers ,[object Object],[object Object],[object Object]
Parallel writers ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Parallel writers ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Preallocation ,[object Object],[object Object],[object Object],[object Object],[object Object]
Tablespace allocation in NDB ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Improvements in mysql-test-run ,[object Object],[object Object],[object Object]
Improvements in mysql-test-run ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Improvements in mysql-test-run ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Library Developers ,[object Object]
Library Developers ,[object Object],[object Object]
Library Developers ,[object Object],[object Object],[object Object]
Library Developers ,[object Object],[object Object],[object Object],[object Object]
Library Developers ,[object Object],[object Object],[object Object],[object Object],[object Object]
Library Developers ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
There is hope ,[object Object],[object Object],[object Object]
Good Luck!
Good Luck! ,[object Object]

Weitere ähnliche Inhalte

Was ist angesagt?

Jetpack datastore入門
Jetpack datastore入門Jetpack datastore入門
Jetpack datastore入門furusin
 
MS Sql Server: Joining Databases
MS Sql Server: Joining DatabasesMS Sql Server: Joining Databases
MS Sql Server: Joining DatabasesDataminingTools Inc
 
SQL Queries
SQL QueriesSQL Queries
SQL QueriesNilt1234
 
MySQL 5.7が魅せる新しい運用の形
MySQL 5.7が魅せる新しい運用の形MySQL 5.7が魅せる新しい運用の形
MySQL 5.7が魅せる新しい運用の形yoku0825
 
z/OS Communications Server: z/OS Resolver
z/OS Communications Server: z/OS Resolverz/OS Communications Server: z/OS Resolver
z/OS Communications Server: z/OS ResolverzOSCommserver
 
Trees In The Database - Advanced data structures
Trees In The Database - Advanced data structuresTrees In The Database - Advanced data structures
Trees In The Database - Advanced data structuresLorenzo Alberton
 
SQLite3
SQLite3SQLite3
SQLite3cltru
 
Jetpack Compose - A Lightning Tour
Jetpack Compose - A Lightning TourJetpack Compose - A Lightning Tour
Jetpack Compose - A Lightning TourMatthew Clarke
 
Materialized Views and Secondary Indexes in Scylla: They Are finally here!
Materialized Views and Secondary Indexes in Scylla: They Are finally here!Materialized Views and Secondary Indexes in Scylla: They Are finally here!
Materialized Views and Secondary Indexes in Scylla: They Are finally here!ScyllaDB
 
Scylla Summit 2022: ScyllaDB Embraces Wasm
Scylla Summit 2022: ScyllaDB Embraces WasmScylla Summit 2022: ScyllaDB Embraces Wasm
Scylla Summit 2022: ScyllaDB Embraces WasmScyllaDB
 
Full-Stack Development
Full-Stack DevelopmentFull-Stack Development
Full-Stack DevelopmentDhilipsiva DS
 

Was ist angesagt? (20)

Jetpack datastore入門
Jetpack datastore入門Jetpack datastore入門
Jetpack datastore入門
 
MS Sql Server: Joining Databases
MS Sql Server: Joining DatabasesMS Sql Server: Joining Databases
MS Sql Server: Joining Databases
 
SQL Queries
SQL QueriesSQL Queries
SQL Queries
 
Sql joins
Sql joinsSql joins
Sql joins
 
Clean code
Clean codeClean code
Clean code
 
EMF - Beyond The Basics
EMF - Beyond The BasicsEMF - Beyond The Basics
EMF - Beyond The Basics
 
MySQL 5.7が魅せる新しい運用の形
MySQL 5.7が魅せる新しい運用の形MySQL 5.7が魅せる新しい運用の形
MySQL 5.7が魅せる新しい運用の形
 
z/OS Communications Server: z/OS Resolver
z/OS Communications Server: z/OS Resolverz/OS Communications Server: z/OS Resolver
z/OS Communications Server: z/OS Resolver
 
Trees In The Database - Advanced data structures
Trees In The Database - Advanced data structuresTrees In The Database - Advanced data structures
Trees In The Database - Advanced data structures
 
SQL JOIN
SQL JOINSQL JOIN
SQL JOIN
 
SQL Commands
SQL Commands SQL Commands
SQL Commands
 
SQLite3
SQLite3SQLite3
SQLite3
 
Jetpack Compose - A Lightning Tour
Jetpack Compose - A Lightning TourJetpack Compose - A Lightning Tour
Jetpack Compose - A Lightning Tour
 
Data Modeling
Data ModelingData Modeling
Data Modeling
 
Learn C Programming Language by Using GDB
Learn C Programming Language by Using GDBLearn C Programming Language by Using GDB
Learn C Programming Language by Using GDB
 
MySQL lecture
MySQL lectureMySQL lecture
MySQL lecture
 
Materialized Views and Secondary Indexes in Scylla: They Are finally here!
Materialized Views and Secondary Indexes in Scylla: They Are finally here!Materialized Views and Secondary Indexes in Scylla: They Are finally here!
Materialized Views and Secondary Indexes in Scylla: They Are finally here!
 
Scylla Summit 2022: ScyllaDB Embraces Wasm
Scylla Summit 2022: ScyllaDB Embraces WasmScylla Summit 2022: ScyllaDB Embraces Wasm
Scylla Summit 2022: ScyllaDB Embraces Wasm
 
InnoDB Locking Explained with Stick Figures
InnoDB Locking Explained with Stick FiguresInnoDB Locking Explained with Stick Figures
InnoDB Locking Explained with Stick Figures
 
Full-Stack Development
Full-Stack DevelopmentFull-Stack Development
Full-Stack Development
 

Ähnlich wie Eat my data

Java File I/O Performance Analysis - Part I - JCConf 2018
Java File I/O Performance Analysis - Part I - JCConf 2018Java File I/O Performance Analysis - Part I - JCConf 2018
Java File I/O Performance Analysis - Part I - JCConf 2018Michael Fong
 
Sequential file programming patterns and performance with .net
Sequential  file programming patterns and performance with .netSequential  file programming patterns and performance with .net
Sequential file programming patterns and performance with .netMichael Pavlovsky
 
Latihan8 comp-forensic-bab5
Latihan8 comp-forensic-bab5Latihan8 comp-forensic-bab5
Latihan8 comp-forensic-bab5sabtolinux
 
High Availability in 37 Easy Steps
High Availability in 37 Easy StepsHigh Availability in 37 Easy Steps
High Availability in 37 Easy StepsTim Serong
 
The care and feeding of a MySQL database
The care and feeding of a MySQL databaseThe care and feeding of a MySQL database
The care and feeding of a MySQL databaseDave Stokes
 
Purdue CS354 Operating Systems 2008
Purdue CS354 Operating Systems 2008Purdue CS354 Operating Systems 2008
Purdue CS354 Operating Systems 2008guestd9065
 
Keeping data-safe-webinar-2010-11-01
Keeping data-safe-webinar-2010-11-01Keeping data-safe-webinar-2010-11-01
Keeping data-safe-webinar-2010-11-01MongoDB
 
Troubleshooting: The Two Laws - IXIASOFT User Conference 2016
Troubleshooting: The Two Laws - IXIASOFT User Conference 2016Troubleshooting: The Two Laws - IXIASOFT User Conference 2016
Troubleshooting: The Two Laws - IXIASOFT User Conference 2016IXIASOFT
 
File Handling In C++(OOPs))
File Handling In C++(OOPs))File Handling In C++(OOPs))
File Handling In C++(OOPs))Papu Kumar
 
Mastering InnoDB Diagnostics
Mastering InnoDB DiagnosticsMastering InnoDB Diagnostics
Mastering InnoDB Diagnosticsguest8212a5
 
Harrison fisk masteringinnodb-diagnostics
Harrison fisk masteringinnodb-diagnosticsHarrison fisk masteringinnodb-diagnostics
Harrison fisk masteringinnodb-diagnosticsguest8212a5
 
Computer basics--basic comp-oper
Computer basics--basic comp-operComputer basics--basic comp-oper
Computer basics--basic comp-operSabbir Alam
 

Ähnlich wie Eat my data (20)

Edubooktraining
EdubooktrainingEdubooktraining
Edubooktraining
 
Java File I/O Performance Analysis - Part I - JCConf 2018
Java File I/O Performance Analysis - Part I - JCConf 2018Java File I/O Performance Analysis - Part I - JCConf 2018
Java File I/O Performance Analysis - Part I - JCConf 2018
 
Sequential file programming patterns and performance with .net
Sequential  file programming patterns and performance with .netSequential  file programming patterns and performance with .net
Sequential file programming patterns and performance with .net
 
Latihan8 comp-forensic-bab5
Latihan8 comp-forensic-bab5Latihan8 comp-forensic-bab5
Latihan8 comp-forensic-bab5
 
High Availability in 37 Easy Steps
High Availability in 37 Easy StepsHigh Availability in 37 Easy Steps
High Availability in 37 Easy Steps
 
The care and feeding of a MySQL database
The care and feeding of a MySQL databaseThe care and feeding of a MySQL database
The care and feeding of a MySQL database
 
Purdue CS354 Operating Systems 2008
Purdue CS354 Operating Systems 2008Purdue CS354 Operating Systems 2008
Purdue CS354 Operating Systems 2008
 
Measuring Firebird Disk I/O
Measuring Firebird Disk I/OMeasuring Firebird Disk I/O
Measuring Firebird Disk I/O
 
Ch23 system administration
Ch23 system administration Ch23 system administration
Ch23 system administration
 
file_c.pdf
file_c.pdffile_c.pdf
file_c.pdf
 
Filehandlinging cp2
Filehandlinging cp2Filehandlinging cp2
Filehandlinging cp2
 
Keeping data-safe-webinar-2010-11-01
Keeping data-safe-webinar-2010-11-01Keeping data-safe-webinar-2010-11-01
Keeping data-safe-webinar-2010-11-01
 
File Handling In C++
File Handling In C++File Handling In C++
File Handling In C++
 
Ext 0523
Ext 0523Ext 0523
Ext 0523
 
Troubleshooting: The Two Laws - IXIASOFT User Conference 2016
Troubleshooting: The Two Laws - IXIASOFT User Conference 2016Troubleshooting: The Two Laws - IXIASOFT User Conference 2016
Troubleshooting: The Two Laws - IXIASOFT User Conference 2016
 
File Handling In C++(OOPs))
File Handling In C++(OOPs))File Handling In C++(OOPs))
File Handling In C++(OOPs))
 
Mastering InnoDB Diagnostics
Mastering InnoDB DiagnosticsMastering InnoDB Diagnostics
Mastering InnoDB Diagnostics
 
Harrison fisk masteringinnodb-diagnostics
Harrison fisk masteringinnodb-diagnosticsHarrison fisk masteringinnodb-diagnostics
Harrison fisk masteringinnodb-diagnostics
 
Computer basics--basic comp-oper
Computer basics--basic comp-operComputer basics--basic comp-oper
Computer basics--basic comp-oper
 
Ungooglable
UngooglableUngooglable
Ungooglable
 

Kürzlich hochgeladen

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024The Digital Insurer
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationSafe Software
 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdfhans926745
 
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationGenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationMichael W. Hawkins
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoffsammart93
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonAnna Loughnan Colquhoun
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slidevu2urc
 
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024The Digital Insurer
 
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century educationjfdjdjcjdnsjd
 
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)wesley chun
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfsudhanshuwaghmare1
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024The Digital Insurer
 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...apidays
 
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUnderstanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUK Journal
 
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptxHampshireHUG
 
Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024The Digital Insurer
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsJoaquim Jorge
 
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...Neo4j
 

Kürzlich hochgeladen (20)

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
 
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationGenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day Presentation
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
 
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024
 
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century education
 
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
 
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUnderstanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
 
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
 
Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
 
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
 

Eat my data

  • 1. Eat My Data: (now with 20% more rant!) How everybody gets file I/O wrong Stewart Smith [email_address] Senior Software Engineer, MySQL Cluster MySQL AB
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.
  • 9.
  • 10.
  • 11. A world without failure
  • 12.
  • 13.
  • 14.
  • 15.
  • 16.
  • 17.
  • 18.
  • 19.
  • 20.
  • 21.
  • 22.
  • 24.
  • 25.
  • 26.
  • 27.
  • 28.
  • 29.
  • 30.
  • 31.
  • 32.
  • 33.
  • 34.
  • 35.
  • 36.
  • 37.
  • 38.
  • 39.
  • 40.
  • 41.
  • 42.
  • 43.
  • 44.
  • 45.
  • 46.
  • 47.
  • 48.
  • 49.
  • 50.
  • 51.
  • 52.
  • 53.
  • 54.
  • 55.
  • 56.
  • 57.
  • 58.
  • 59.
  • 60.
  • 61.
  • 62.
  • 64.
  • 65.
  • 66.
  • 67.
  • 68.
  • 69.
  • 70.
  • 71.
  • 72.
  • 73.
  • 74.
  • 75.
  • 76. Now all is good with the world...
  • 77.
  • 78.
  • 79.
  • 80.
  • 81.
  • 82.
  • 83.
  • 84.
  • 85.
  • 86.
  • 87.
  • 88.
  • 89.
  • 90.
  • 91.
  • 92.
  • 93.
  • 94. gint common_save_xml(xmlDocPtr doc, gchar *filename) { FILE *fp; char *xmlbuf; int fd, n; fp = g_fopen(filename, &quot;w&quot;); if(NULL == fp) return -1; xmlDocDumpFormatMemory(doc, (xmlChar **)&xmlbuf, &n, TRUE); if(n <= 0) { errno = ENOMEM; return -1; } if(fwrite(xmlbuf, sizeof (xmlChar), n, fp) < n) { xmlFree (xmlbuf); return -1; } xmlFree (xmlbuf); /* flush user-space buffers */ if (fflush (fp) != 0) return -1; if ((fd = fileno (fp)) == -1) return -1; #ifdef HAVE_FSYNC /* sync kernel-space buffers to disk */ if (fsync (fd) == -1) return -1; #endif fclose(fp); return 0; }
  • 95.
  • 96.
  • 97.
  • 98.
  • 99.
  • 100.
  • 101.
  • 102.
  • 103.
  • 104.
  • 105.
  • 106.
  • 107.
  • 108.
  • 109.
  • 110.
  • 111.
  • 112.
  • 113.
  • 114.
  • 115.
  • 116.
  • 117. #ifdef HAVE_DARWIN_THREADS # ifdef F_FULLFSYNC /* This executable has been compiled on Mac OS X 10.3 or later. Assume that F_FULLFSYNC is available at run-time. */ srv_have_fullfsync = TRUE; # else /* F_FULLFSYNC */ /* This executable has been compiled on Mac OS X 10.2 or earlier. Determine if the executable is running on Mac OS X 10.3 or later. */ struct utsname utsname; if (uname(&utsname)) { fputs(&quot;InnoDB: cannot determine Mac OS X version!&quot;, stderr); } else { srv_have_fullfsync = strcmp(utsname.release, &quot;7.&quot;) >= 0; } if (!srv_have_fullfsync) { fputs(&quot;InnoDB: On Mac OS X, fsync() may be&quot; &quot; broken on internal drives,&quot; &quot;InnoDB: making transactions unsafe!&quot;, stderr); } # endif /* F_FULLFSYNC */ #endif /* HAVE_DARWIN_THREADS */
  • 118. #if defined(HAVE_DARWIN_THREADS) # ifndef F_FULLFSYNC /* The following definition is from the Mac OS X 10.3 <sys/fcntl.h> */ # define F_FULLFSYNC 51 /* fsync + ask the drive to flush to the media */ # elif F_FULLFSYNC != 51 # error &quot;F_FULLFSYNC != 51: ABI incompatibility with Mac OS X 10.3&quot; # endif /* Apple has disabled fsync() for internal disk drives in OS X. That caused corruption for a user when he tested a power outage. Let us in OS X use a nonstandard flush method recommended by an Apple engineer. */ if (!srv_have_fullfsync) { /* If we are not on an operating system that supports this, then fall back to a plain fsync. */ ret = fsync(file); } else { ret = fcntl(file, F_FULLFSYNC, NULL); if (ret) { /* If we are not on a file system that supports this, then fall back to a plain fsync. */ ret = fsync(file); } } #elif HAVE_FDATASYNC ret = fdatasync(file); #else /* fprintf(stderr, &quot;Flushing to file %p&quot;, file); */ ret = fsync(file); #endif
  • 119.
  • 121.
  • 122.
  • 123.
  • 124.
  • 125.
  • 126.
  • 127.
  • 128.
  • 129.
  • 130.
  • 131.
  • 132.
  • 133.
  • 134.
  • 135.
  • 136.
  • 137.
  • 138.
  • 139.
  • 140.
  • 141.
  • 142.
  • 143.
  • 144.
  • 145.
  • 146.
  • 147.
  • 148.
  • 149. inode Infos Direct blocks Indirect Blocks Double indirect blocks
  • 150.
  • 151.
  • 152.
  • 153.
  • 154.
  • 155.
  • 156.
  • 157.
  • 158.
  • 159.
  • 160.
  • 161.
  • 162.
  • 163.
  • 164.
  • 165.
  • 166.
  • 168.