Azure data store with storage and cosmo db

National Head - Consulting & Services Business at COMPAREX India Pvt. Ltd. um COMPAREX India Pvt. Ltd.
4. Mar 2019

Más contenido relacionado


Azure data store with storage and cosmo db

  1. Build,SkillandEnableBuild,SkillandEnable Microsoft Azure Data Storage Bipin Sinhaa Innovate, Collaborate & Accelerate
  2. Build,SkillandEnable Topics • Azure Storage – Introduction • VM(Disk) Storage • Bob Storage • Table Storage • CosmoDB • Queue • Page Storage • Storage Encryption
  3. Build,SkillandEnable Introduction to Store Storage
  4. Build,SkillandEnable Azure VM Storage
  5. Build,SkillandEnable Azure VM Storage
  6. Build,SkillandEnable VM Storage Type
  7. Build,SkillandEnable Azure Managed Disks is an elastic disk storage system that is offered as an alternative to the current storage account-based system. Managed Disks eliminates the complexity of storage accounts and offers a simple scalable and highly available solution for creating and managing disks .It store the .VHD files in Page Blobs. Simple: Abstract storage accounts from customers Better Performance: Storage accounts limits does nit apply Big Scale : Up to 20,000 disk per region per subscription. Encryption: Support VM level disk encryption, secure data at rest Secure- No public access to underlying blob Azure Managed Disk for VM Storage
  8. Build,SkillandEnable Unmanaged disk also store the .VHD files in Page Blobs. Management : Disk created and managed by customers Proper Sizing Storage accounts limited to 20,000 IOPs Sparse Storage (Standard): Only pay for actual data then the size Manage by customer Unmanage Storage Accounts Azure Unmanaged Disk for VM Storage
  9. Build,SkillandEnable Managed Vs Unmanaged Disk
  10. Build,SkillandEnable Storage Accounts
  11. Build,SkillandEnable General-purpose v2 accounts General-purpose v2 storage accounts support the latest Azure Storage features and incorporate all of the functionality of general-purpose v1 and Blob storage accounts. General-purpose v2 storage accounts support these Azure Storage services: •Blobs (all types: Block, Append, Page) •Files •Disks •Queues •Tables General-purpose v1 accounts General-purpose v1 accounts provide access to all Azure Storage services, but may not have the latest features or the lowest per gigabyte pricing. General-purpose v1 storage accounts support these Azure Storage services: •Blobs (all types) •Files •Disks •Queues •Tables Blob storage accounts A Blob storage account is a specialized storage account for storing unstructured object data as block blobs. Blob storage accounts provide the same durability, availability, scalability, and performance features that are available with general-purpose v2 storage accounts. Blob storage accounts support storing block blobs and append blobs, but not page blobs. Types of Storage Accounts •Storage account names must be between 3 and 24 characters in length and may contain numbers and lowercase letters only. •Your storage account name must be unique within Azure. No two storage accounts can have the same name
  12. Build,SkillandEnable Storage account type Supported services Supported performance tiers Supported access tiers Replication options Deployment model1 Encryption2 General- purpose V2 Blob, File, Queue, Table, and Disk Standard, Premium Hot, Cool, Archive 3 LRS, ZRS 4 , GRS, RA-GRS Resource Manager Encrypted General- purpose V1 Blob, File, Queue, Table, and Disk Standard, Premium N/A LRS, GRS, RA- GRS Resource Manager, Classic Encrypted Blob storage Blob (block blobs and append blobs only) Standard Hot, Cool, Archive 3 LRS, GRS, RA- GRS Resource Manager Encrypted • All storage accounts are encrypted using Storage Service Encryption (SSE) • The Archive tier is available at level of an individual blob only, not at the storage account level. Only block blobs and append blobs can be archived • Zone-redundant storage (ZRS) is available only for standard general-purpose v2 storage accounts. Types of Storage Accounts- Comparison
  13. Build,SkillandEnable Performance tiers General-purpose storage accounts may be configured for either of the following performance tiers: •A standard performance tier for storing blobs, files, tables, queues, and Azure virtual machine disks. •A premium performance tier for storing unmanaged virtual machine disks only. Replication/Redundancy LRS- Local Redundant Storage – 3 Copies in single DC ZRS- Zonal Redundant Storage – 3 copies in different DCs in same region or different region GRS- Geo Redundant Storage – 6 Copies . 3 in LRS and 3 in DCs in different geo regions. Geo- redundant storage (GRS) replicates your data to a secondary region that is hundreds of miles away from the primary region. If your storage account has GRS enabled, then your data is durable even in the case of a complete regional outage or a disaster in which the primary region is not recoverable. RA GRS- Read Access Geo Redundant Storage – 6 copies same as GRS but 3 copies in different geo DC also have read access. Read-access geo-redundant storage (RA-GRS) maximizes availability for your storage account, by providing read-only access to the data in the secondary location, in addition to the replication across two regions provided by GRS. Storage Accounts- Performance tiers and Replication
  14. Build,SkillandEnable Storage account endpoints A storage account provides a unique namespace in Azure for your data. Every object that you store in Azure Storage has an address that includes your unique account name. The combination of the account name and the Azure Storage service endpoint forms the endpoints for your storage account. For example, if your general-purpose storage account is named mystorageaccount, then the default endpoints for that account are: Blob storage: Table storage: Queue storage: Azure Files: Storage Account Endpoints The URL for accessing an object in a storage account is constructed by appending the object's location in the storage account to the endpoint. For example, a blob address might have this format:
  15. Build,SkillandEnable Blobs Simple Interface to store and retrieve large amount of unstructured data like files, images, videos, etc Tables No SQLDatastore Used for storing structured, non- relational data. Queues A queue contains many messages. Decouple frontend role from backend. Disks Persistent standard and premium storage for IaaS VM built on PageBlobs . Files SMB 3.0 access to azure storage. Map to file share . Lift and shift legacy apps. Azure Import/Export Ship data in hard disk drive to azure datacentre AZCopy High Performance data uploads , downloads and copy Azure Storage Offerings
  16. Build,SkillandEnable Abstraction of Storage • Blobs – Simple Interface to store and retrieve files in cloud. • Data Sharing : Share documents , picture, video , music etc. • Big Data : Store rwa data/logs • Backups-Data and device Backup • Disks– Network mounted durables disks for VMs in Azure • Data Sharing : Share documents , picture, video , music etc. • Mounted disks and VHDs stored in Azure Blobs • Move on-premised applications to cloud • Tables– Massively scalable and extremely easy to use NoSQL system that auto scales • Store users information, device information and any type of metadata for your service. • Queues– Reliable messaging system • Decouple components/roles • Web role to worker role communication • Allow roles to scale independently • Implement scheduling of asynchronous tasks • Building process/workflows
  17. Build,SkillandEnable Accounts Container Tables Queue Blobs Entities Messages https://<account><container> https://<account><table> https://<account><queue> Abstraction of Storage
  18. Build,SkillandEnable Accessing Windows Storage Via Explorer Cloud storage studio by Cerebrata Azure Storage Explorer by Neudistic CloudXplorer by ClumsyLeaf CloudBerry Explorer by Cloudberry Lab
  19. Build,SkillandEnable How the Azure Storage Used by Microsoft Xbox: Uses Blobs, Tables and Queues for cloud game saves, Xbox Music, Xbox live etc Skype: Uses Blobs, Tables and Queues for Skype video messages and to keep metadata to allow skype clients to connect with each other. Bing: Uses Blobs, Tables and Queues to provide a near real time ingestion engine that consume Twitter and Facebook feeds, indexes them which is the folded inti Bing search SkyDrive: Uses Blobs to store pictures , documents , video, files etc.
  20. Build,SkillandEnable Blob Storage
  21. Build,SkillandEnable Blob Storage Simple Interface to store and retrieve large amount of unstructured data like files, images, videos, etc . A single blob can be hundreds of gigabytes in size, and a single storage account can contain up to 200TB of blobs. Azure Storage supports three types of blobs: Block blobs store text and binary data, up to about 4.7 TB. Block blobs are made up of blocks of data that can be managed individually. Append blobs are made up of blocks like block blobs, but are optimized for append operations. Append blobs are ideal for scenarios such as logging data from virtual machines. Page blobs store random access files up to 8 TB in size. Page blobs store the virtual hard drive (VHD) files serve as disks for Azure virtual machines
  22. Build,SkillandEnable Blob Storage is designed for : • Serving images or documents directly to a browser. • Storing files for distributed access. • Streaming video and audio. • Writing to log files. • Storing data for backup and restore, disaster recovery, and archiving. • Storing data for analysis by an on- premises or Azure-hosted service. Blob Storage
  23. Build,SkillandEnable Blob Storage Resources Blob storage offers three types of resources: •The storage account- A storage account provides a unique namespace in Azure for your data. Every object that you store in Azure Storage has an address that includes your unique account name. An account can have many Blob Containers •A container in the storage account-A container organizes a set of blobs, similar to a directory in a file system. A storage account can include an unlimited number of containers, and a container can store an unlimited number of blobs. Sharing policies are set at the container level, where a container can be set to private or to be publicly accessible •A blob in a container
  24. Build,SkillandEnable  The Hot access tier, which is optimized for frequent access of objects in the storage account. Accessing data in the Hot tier is most cost-effective, while storage costs are somewhat higher.  The Cool access tier, which is optimized for storing large amounts of data that is infrequently accessed and stored for at least 30 days. Storing data in the Cool tier is more cost-effective, but accessing that data may be somewhat more expensive than accessing data in the Hot tier.  The Archive tier, which is available only for individual block blobs. The Archive tier is optimized for data that can tolerate several hours of retrieval latency and will remain in the Archive tier for at least 180 days. The Archive tier is the most cost-effective option for storing data, but accessing that data is more expensive than accessing data in the Hot or Cool tiers. Access tiers for block blob data
  25. Build,SkillandEnable Archive Access tiers for block blob data
  26. Build,SkillandEnable Table Storage
  27. Build,SkillandEnable Table Storage • Azure Table storage is a service that stores structured NoSQL data in the cloud, providing a key/attribute store with a schemaless design. • Because Table storage is schemaless it's easy to adapt your data as the needs of your application evolve. • Access to Table storage data is fast and cost-effective for many types of applications, and is typically lower in cost than traditional SQL for similar volumes of data
  28. Build,SkillandEnable Table Storage Components Storage Account – All access to Windows Azure Storage is done through a storage account. This is the highest level of the namespace for accessing tables. An account can have many Tables Table: A table is a collection of entities. Tables don't enforce a schema on entities, which means a single table can contain entities that have different sets of properties. Entity: An entity is a set of properties, similar to a database row. An entity in Azure Storage can be up to 1MB in size. An entity contains a set of properties. Each table has two properties, “PartitionKey and RowKey”, which form the unique key for the entity Properties: A property is a name-value pair. An entity can hold up to 255 properties. Each entity also has three system properties that specify a partition key, a row key, and a timestamp.
  29. Build,SkillandEnable PartitionKey – The first key property of every table. The system uses this key to automatically distribute and load balance the table’s entities over many servers. RowKey – A second key property for the table. This is the unique ID of the entity within the partition it belongs to. The PartitionKey combined with the RowKey uniquely identifies an entity in a table. Timestamp – Every entity has a version maintained by the system. This is used for optimistic concurrency. Table Storage Properties
  30. Build,SkillandEnable How to Partition
  31. Build,SkillandEnable Schema of Table
  32. Build,SkillandEnable CosmoDB
  33. Build,SkillandEnable CosmoDB- NoSQL No data is is born relational, it’s born dirty and messy, in whatever shape or structure it’s created. NoSQL databases used by companies such as Facebook, Google and Amazon as they working with big data problems and/or real time web applications. This set of technology is coined “Not Only SQL” since they might support a SQL syntax for retrieving data. The Azure Tables are extremely scalable and cheap, but if you start querying on any other attributes then you start running into problems.
  34. Build,SkillandEnable NoSQL - How It Store the data
  35. Build,SkillandEnable CosmoDB- Introduction Azure Cosmos DB is a globally distributed, multi-model database service that supports document, key- value, wide- column, and graph databases.
  36. Build,SkillandEnable CosmoDB -Turnkey global distribution
  37. Build,SkillandEnable The Database is hosted in US region while the users are in different regions . This add latency and delay . In application like stock market it is big problem. One possible solutions is to use the read only copy to different regions and aloo the users to read it, but this also lead to inconsistency because of sync dely. Another solutions is to use the eventual commit where data cannot be read until it commit to all database. CosmoDB Consistency- A situational scenario
  38. Build,SkillandEnable No single machine bottleneck. Server Side partition Management GB to PB Storage selectable throughput Automatic Document expiration Pay by the hour , customized throughput cost Support for request per second or minute CosmoDB Abstract
  39. Build,SkillandEnable Turnkey global distribution • Cosmos DB enables you to build highly responsive and highly available applications worldwide. • Cosmos DB transparently replicates your data wherever your users are, so your users can interact with a replica of the data nearest to them. • Cosmos DB allows you to add or remove any of the Azure regions to your Cosmos account at any time, with a click of a button. Always On Cosmos DB provides 99.999% high availability for both reads and writes. Cosmos DB also provides you with the ability to programmatically (or via Portal) invoke the regional failover of your Cosmos account. This capability helps you ensure that while a Cosmos database may automatically failover if there is a regional disaster. Guaranteed low latency worldwide Cosmos DB guarantees less than 10- ms latencies for both, reads and (indexed) writes at the 99th percentile, all around the world. Cosmos DB is available in all Azure regions worldwide, including 54+ regions in public cloud, Azure China 21Vianet, Azure Germany, Azure Government, and Azure Government for Department of Defense (DoD). CosmoDB Advantage
  40. Build,SkillandEnable CosmoDB Components
  41. Build,SkillandEnable It stores the data in JSON. It’s massively scalable but fully query-able by all parts of JSON tree. CosmoDB - Structure
  42. Build,SkillandEnable The Azure portal has three explorers that can be used to manipulate collection. Document Explorer – It allows you to add, update and delete documents from a collection. Query Explorer - Allows you to select document data from the collection using a SQL like syntax. Script Explorer- Allows you to define JavaScript objects to support advanced functionality CosmoDB –Azure Portal Explorer
  43. Build,SkillandEnable Azure Queue
  44. Build,SkillandEnable It’s all about a collection that has two primary methods: enqueue (to insert new items to the queue at the bottom) and dequeue (to get the top most item from the queue). Storage QUEUE Microsoft Azure Queues are a ready-to-use service that loosely connects components or applications through the cloud. Azure Queues are a part of the Storage Service that unites Blobs, Tables and Queues under one storage account. There are two queue technologies available in Microsoft Azure: Azure Queues and Service Bus Queues. The latter is newer and has a focus on publish/subscribe mechanisms (i.e. the ability to receive messages without having to poll the queue) and integration with the WCF communication stack. On the other hand, Azure Queues are faster and more applicable when storing more than 5 GB of messages in a queue.
  45. Build,SkillandEnable Azure Queues are queues located in the Microsoft cloud which you can use for exchanging messages between components either in the cloud or on premise. A message typically represents a task created by someone (“producer”) that has to be processed by someone else (“consumer”). Each message has a small body and some attributes, such as time-to-live, which you can use for configuring your service. As Azure ensures that a dequeued message is invisible to other listeners, you could imagine many producers and many consumers as well as one-to-one scenarios. The main architectural benefit is loose coupling Storage QUEUE- Architecture
  46. Build,SkillandEnable There are two options available at this stage: 1. you can either connect with the storage account name and storage account key [everyone who gets the name/key credentials has full access to all operations in your storage account. Those people can use that access to add and delete queues, read and write messages, and even access blob and table storages] 2. With a temporary token. [create a temporary token that is valid for a particular queue, and limited by functionality and time. Obviously, this way you can give someone restricted access (e.g. read-only) for a specific time period (e.g. 5 minutes from now), and maintain much tighter control over who has access to your application.] Storage QUEUE- Security
  47. Build,SkillandEnable Storage QUEUE- Example
  48. Build,SkillandEnable Azure FileShare
  49. Build,SkillandEnable Azure FileShare
  50. Build,SkillandEnable Azure FileShare -Advantages
  51. Build,SkillandEnable Azure Storage Encryption
  52. Build,SkillandEnable Storage Service Encryption (SSE)
  53. Build,SkillandEnable Azure VM Disk Encryption
  54. Build,SkillandEnable Index A- Sample Diagram
  55. Build,SkillandEnable Index B- Azure Data Store
  56. Build,SkillandEnable Thank You Innovate, Collaborate & Accelerate