Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Minio Cloud Storage

2.060 Aufrufe

Veröffentlicht am

Slidedeck presented on GopherCon 2017 by Frank Wessels.

Veröffentlicht in: Technologie
  • I think you need a perfect and 100% unique academic essays papers have a look once this site i hope you will get valuable papers, ⇒ www.HelpWriting.net ⇐
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier
  • You have to choose carefully. ⇒ www.WritePaper.info ⇐ offers a professional writing service. I highly recommend them. The papers are delivered on time and customers are their first priority. This is their website: ⇒ www.WritePaper.info ⇐
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier

Minio Cloud Storage

  1. 1. DEEP DIVE
  2. 2. • Object Storage Server • S3 Compatible • Open Source (github.com/minio/minio) • Founder: Anand Babu Periasamy ◦ Formerly GlusterFS ▪ Distributed File System ▪ Now part of Red Hat MINIO BACKGROUND
  3. 3. MINIO STACK MINIO SERVER MINIO CLIENT MINIO SDK
  4. 4. MINIO FLAVORS 1 BINARY / 3 FLAVORS FS backend $ minio server /dir XL backend with Erasure code & bitrot protection $ minio server /disk1 /disk2 /disk3 /disk4 ... /disk16 Distributed up to 16 servers $ minio server host1:/disk host2:/disk host3:/disk ... host16:/disk
  5. 5. DISTRIBUTED SETUP JBOD JBOD JBOD JBOD MINIO 1 MINIO 2 MINIO 3 MINIO 16 ERASURE CODED BACKEND Erasure Code (2x2, 4x4, 6x6, 8x8) - Maximum 16 Disks / Setup
  6. 6. ERASURE CODING
  7. 7. BLAKE2
  8. 8. PLAN 9/ GOLANG ASSEMBLY • ‘Pseudo’ assembly language ◦ Generalized instructions like MOV, ADD, CMP ◦ Actual instructions are generated platform specifically (obviously) ◦ Mostly this translation is self-explanatory/logical, sometimes it is not ◦ Some architecture aspects shine through • Data flows from left to right ◦ Eg: MOV R1, R2 // R2 = R1 • Some ‘pseudo’ registers ◦ FP, SP, PC, SB • Not all instructions available ◦ However BYTE/WORD/LONG/QUAD constructs for inserting opcodes directly
  9. 9. PLAN 9/ GOLANG ASSEMBLY • Benefits ◦ Fast compilation ◦ Enhanced execution speed ◦ No need for `cgo` dependency ◦ Avoid runtime call stack saving overhead etc. • Allows to take advantage of SIMD (Intel) or NEON (ARM) capabilities ◦ Single Instruction Multiple Data • You are (or will be) a bit on your own… ◦ Learn / Take a look at existing code (grep for *.s in go/src/) ◦ Or see minio repos • Full details, eg for ARM: src/cmd/internal/obj/arm/a.out.go
  10. 10. ASM2PLAN9S: GENERATE BYTE SEQUENCES FOR GO ASSEMBLY • Developed as part of blake2b-simd • Uses YASM behind the scenes • Support for use within defines • Works nicely together with asmfmt • See github.com/minio/asm2plan9s • Planning ARM support // VPADDQ XMM0,XMM1,XMM8 $ asm2plan9s example.s LONG $0xd471c1c4; BYTE $0xc0 // VPADDQ XMM0,XMM1,XMM8
  11. 11. ACCELERATING BLAKE2B USING SIMD • BLAKE2b hash ◦ SHA3 competition contender ◦ Characteristics: speed, simplicity, and security ◦ Optimized for 64-bit platforms (BLAKE2b) • github.com/minio/blake2b-simd ◦ SIMD accelerated pure Go implementation using Go Assembly ◦ Three flavors: AVX2, AVX and SSE ◦ Speed up over pure go (non-assembly): 3.94x • For bitrot detection (minio XL version)
  12. 12. TECH DETAILS BLAKE 2B - SIMD GOLANG v0 += v4 v1 += v5 v2 += v6 v3 += v7 AVX USE 128-BIT WIDE XMM INSTRUCTIONS VPADDQ XMM0,XMM0,XMM2 /* v0 += v4,v1 += v5 */ VPADDQ XMM1,XMM1,XMM3 /* v2 += v6,v3 += v7 */ AVX2 USE 256-BIT WIDE YMM INSTRUCTIONS VPADDQ YMM0,YMM0,YMM1 /* v0 += v4,v1 += v5,v2 += v6,v3 += v7 */ SSE USE 128-BIT WIDE XMM INSTRUCTIONS (NON ‘VEX’-INSTRUCTIONS)
  13. 13. BLAKE 2B ROUNDS 12 Rounds of Processing
  14. 14. BLAKE 2B MACROS Example G1 macro
  15. 15. GATEWAY
  16. 16. MINIO GATEWAY • Recent addition Google Cloud Storage Amazon S3 Azure Blob Storage
  17. 17. MINIO GATEWAY FEATURES • Serve as S3 compatible proxy for different back-ends ◦ Azure / GCS / S3 ◦ Others to follow (plugin model?) • Support full S3 API ◦ Multi-part uploads ◦ Any SDK (Minio, AWS, etc.) ◦ Any language • Transparent for client application • Minio browser for gateway
  18. 18. MULTI-CLOUD STRATEGY • Single API to access any object • Universal access via mc (minio client) • Deployment ◦ Multiple gateways on single server ◦ Multiple servers each with own gateway • Caching ◦ Cache objects on local disk ◦ Faster delivery / create your own CDN ◦ Allow savings on eg bandwidth costs
  19. 19. OTHER RESOURCES
  20. 20. • Blog ◦ Golang Internals Part 1: Autogenerated functions ◦ Golang Internals Part 2: Named return values • dsync ◦ distributed sync package ◦ github.com/minio/dsync • c2goasm ◦ embed C/C++ generated assembly into Go ◦ github.com/minio/c2goasm • asm2plan9 ◦ generate BYTE sequences for Go assembly ◦ github.com/minio/asm2plan9 OTHER RESOURCES
  21. 21. QUESTIONS? slack.minio.io

×