5. Considerations
Private / single-tenant system
Process
Disk
Kernel IO Interface
Disk Controller Disk Controller Disk Controller
DiskDisk
Cache
Cache
Cache
6. Considerations
Shared multi-
tenant system
(simplified view)
Caches... Caches
Everywhere
Process Kernel
Virtualization
Layer
KernelNetwork Interface
Process
Kernel
IO Interface
Disk Controller
Process
Kernel
IO Interface
Disk Controller
Process
Kernel
IO Interface
Disk Controller
Process Kernel
Process Kernel
Disk Disk Disk
Disk Controller
SSD
Disk
Controller
Disk
Disk
Controller
Disk
7. • Linux Buffers & Caches
• Buffers: Filesystem Metadata + Active in-flight pages
• Caches: File contents
15. What it does?
• Serial Write
• Serial Read
• Random Rewrite
• Random Read
• Mixed Random Read & Write
16. Mitigating buffers
• It is impossible to avoid buffering at all
levels in a non-proprietary system
• But we can use posix_fadvise & Direct IO
to mitigate local kernel buffers
17. posix_fadvise
int posix_fadvise(int fd, off_t offset, off_t len, int
advice);
“Programs can use posix_fadvise() to
announce an intention to access file data in a
specific pattern in the future, thus allowing the
kernel to perform appropriate optimizations.”
19. posix_fadvise
/* *TRY* to minimize buffer cache effect */
/* There's no guarantee that the file will be
removed from buffer cache though */
/* Keep in mind that buffering will happen at
some level */
if (fadvise == true)
{
rc = posix_fadvise(fd, 0, 0,
POSIX_FADV_DONTNEED);
...
20. posix_fadvise
Test DONTNEED NORMAL Difference
Write
Read
Rewrite
Reread
Random
5,82 6,05 0,96
0,163 0,017 9,59
3,037 2,993 1,01
1,244 0,019 65,47
2,403 1,559 1,54
100Mbytes file - 4k BS -XFS - 20 run average
21. posix_fadvise
Test DONTNEED NORMAL Difference
Write
Read
Rewrite
Reread
Random
5,82 6,05 0,96
0,163 0,017 9,59
3,037 2,993 1,01
1,244 0,019 65,47
2,403 1,559 1,54
100Mbytes file - 4k BS -XFS - 20 run average
6,0 GB/s
5,2 GB/s
22. Transfer Rates
• SSD transfer rates typically range from
100MB/s to 600MB/s
• Something is wrong...
23. Synchronous IO
int open(const char *pathname, int flags,
mode_t mode);
O_SYNC
The file is opened for synchronous I/O. Any
write(2)s on the resulting file descriptor will
block the calling process until the data has been
physically written to the underlying hardware.
But see NOTES below.
24. Direct IO
int open(const char *pathname, int flags,
mode_t mode);
O_DIRECT
Try to minimize cache effects of the I/O to and
from this file. In general this will degrade
performance, but it is useful in special
situations, such as when applications do their
own caching. File I/O is done directly to/from
user space buffers.
25. Direct IO
flags = O_RDWR | O_CREAT | O_TRUNC | O_SYNC;
if (directIO == true)
{
myWarn(3,__FUNCTION__, "Will try to enable
Direct IO");
flags = flags| O_DIRECT;
}
26. Notes below
Most Linux file systems don't actually
implement the POSIX O_SYNC semantics,
which require all metadata updates of a write to
be on disk on returning to userspace, but only
the O_DSYNC semantics, which require only
actual file data and meta-data necessary to
retrieve it to be on disk by the time the system
call returns.
27. Results
Test -Direct IO (s) +Direct IO (s) Difference
Write
Read
Rewrite
Reread
Random
5,82 6,640 0,88
0,163 2,197 0,07
3,037 2,905 1,05
1,244 2,845 0,44
2,403 2,941 0,82
100Mbytes file - 4k BS -XFS - 20 run average
28. Results
Test +Direct IO (s) MB/s
Write
Read
Rewrite
Reread
Random
6,640 15,79
2,197 47,72
2,905 36,09
2,845 36,85
2,941 35,64
100Mbytes file - 4k BS -XFS - 20 run average
29. iomelt
IOMELT Version 0.71
Usage:
-b BYTES Block size used for IO functions (must be a power of two)
-d Dump data in a format that can be digested by pattern processing
commands
-D Print time in seconds since epoch
-h Prints usage parameters
-H Omit header row when dumping data
-n Do NOT convert bytes to human readable format
-o Do NOT display results (does not override -d)
-O Reopen worload file before every test
-p PATH Directory where the test file should be created
-r Randomize workload file name
-R Try to enable Direct IO
-s BYTES Workload file size (default: 10Mb)
-v Controls the level of verbosity
-V Displays version number
-b and -s values can be specified in bytes (default), Kilobytes (with 'K'
suffix), Megabytes (with 'M'suffix), or Gigabytes (with 'G' suffix)
Unless specified, block size value is the optimal block transfer size for
the file system as returned by statvfs
30. iomelt
• Available at http://iomelt.com
• Fork it on GitHub:
• https://github.com/camposr/iomelt
• Artistic License 2.0
• http://opensource.org/licenses/artistic-license-2.0
31. Methodology
How to measure the performance of
several instance types on different
regions for long periods of time?
32. Methodology
1. Create a single AMI
1.1.Update Kernel, compiler and libraries
2. Replicate it in several regions and different
instance types:
2.1.m1.small
2.2.m1.medium
2.3.m1.large
42. Conclusions
• Shared multi-tenant environments create
new challenges for performance analysis
• Traditional benchmark methodologies are
not suitable for these environments
• Excessive versatility in most available tools
make it hard to get reproducible
measurements