Más contenido relacionado

Presentaciones para ti(20)


A guide through the Azure Messaging services - Update Conference

  1. A guide through the Azure Messaging services Eldert Grootenboer
  2. Eldert Grootenboer @egrootenboer
  3. Messaging?
  4. Messaging in Azure Event Hubs Streaming data Service Bus Enterprise messaging Event Grid Cross cloud reactive eventing Storage Queues Simple task queues Relay Secure sockets without changes to your network
  5. Why so many choices?
  6. Command Report Measurement Job Query Handover Transfer Update Assignment Request Notification Trace
  7. Facts Command Report Measurement Job Query Handover Transfer Update Assignment Request Notification Trace Intents EventingMessaging
  8. Events Independent Report State Change Actionable Time Ordered Context Partitioned Report Condition Analyzable Discrete Series
  9. EventingMessaging
  10. Let’s have a look at the services
  11. Simple Queues
  12. Storage queues are task queues Coordinate work across compute (VMs, Functions, etc.) Low cost Pay as you go Easy to use Messages are generally homogeneous Unlimited number of queues Unlimited number of concurrent connections Maximum queue size 500TB Maximum message size 64KB Few features
  13. Few features No headers/metadata No pub/sub No Тime To Live No scheduling No dead-letter queue Peeking no more than 32 messages only No de-duplication 7 days retention period No transactional guarantees
  14. A simple queue Sender sends message to queue Queue ACKs receipt Receiver connects to queue & retrieves message Receiver ACKs complete (or other action)
  15. Enterprise Messaging
  16. Enterprise Messaging as a Service Queues & Topics Reliable asynchronous communication Rich features for temporal control Message headers Routing and filtering Convoys & Sessions (related messages with state)
  17. Features of Service Bus Scheduled delivery Time To Live ForwardTo Defer Sessions Batching Ordering Auto-delete on idle OnMessage Duplicate detection Actions Transactions Poison message handling
  18. Features of Service Bus Premium Dedicated resources Geo-DR Availability Zones Virtual Network Service Endpoints
  19. Topics & Subscriptions Sender only knows about Topic Receivers only know about Subscriptions Filters and Actions exist on Subscriptions Topic Subscription 1 Subscription 2 type=order type=quote Filter Filter “type” = “order” “type” = “quote”
  20. Where does Service Bus fit in? Decoupling and providing durability with transactional consistency Queue
  21. Push based publish-subscribe event routing
  22. What is an event? Something happened Notification Triggers a reaction Based on facts Can be series based or discrete
  23. IoTHub BlobStorage ResourceGroups AzureSubscriptions EventHubs CustomTopics Storage(GPv2) AzureFunctions LogicApps AzureAutomation WebHooks EventHubs Container Registry StorageQueues DatalakeStore Azure Event Grid
  24. Concepts Events: what happened Event Publishers: where it took place Topics: where publishers send events Event Subscriptions: how you receive events Event Handlers: the app or service reacting to the event Events
  25. Trigger a functionto run CognitiveAPI when a file is added to storage Use a function to run a compliance checkon each newly created SQL database Tag newly provisioned VMs with AzureAutomation and add to metadata store Use custom“drive start” and “driveend” events to log vehicle performancemetrics What is Event Grid For?
  26. "topic": "/subscriptions/{id}/resourceGroups/{rg}/providers/Microsoft.Storage/storageAccounts/{storageacc}", "subject": "/blobServices/default/containers/monitored/blobs/blob.json", "eventType": "Microsoft.Storage.BlobCreated", "eventTime": "2017-09-20T01:12:35.8066557Z", "id":"9ac0de1a-001e-00da-23ad-311d4a06bb2f", "data":{ "api":"PutBlob", "clientRequestId": "9ac0de1a-9da0-11e7-9a3b-51cc40a2ca27", "requestId": "9ac0de1a-001e-00da-23ad-311d4a000000", "eTag":"0x9AC0FC4ADA5997D", "contentType": "application/json", "contentLength": 268, "blobType": "BlockBlob", "url":"https://{storageacc}", "sequencer": "00000000000019200000000000F12B37", "storageDiagnostics": { "batchId": "9ac0de1a-e1a1-44c5-a0a1-a66b2a191fc4“ } What does an Event look like?
  27. Publishing custom events Custom Topics Topic: <Name of Topic> Subject: <whatever you choose> EventType: <whatever you choose> sub1 sub2 mytopic Subject: /products/beverages EventType: skuCreated Subject: /products/snacks EventType: skuUpdated Publisher
  28. The challenge with Custom Topics Having many topics forces you to do routing This is what Grid was made for There is a better way! sub1 sub2 mytopic2 sub1 sub2 mytopic3 sub1 sub2 mytopic4 sub1 sub2 mytopic1 Publisher
  29. Event Domains All events to single endpoint Sender sets topic Subscribers are authenticated via AAD (cross org support) Subscribers identity not required to be Azure Entire service can be white labeled
  30. Advanced Filters Works on the data portion of the event Supports ten levels of nesting Supports one or more values Up to 10 advanced filters can be combined in OR Numbers NumberLessThan NumberLessThanOrEquals NumberGreaterThan NumberGreaterThanOrEquals NumberIn–thevaluefordata.keyequalsavalueintheset[0,2.08,3.14] NumberNotIn–thevaluefordata.keyisnotintheset[1,11,112,1124] Strings StringContains–thevaluefordata.keycontains“the” StringIn–thevaluefordata.keyequalsavalueintheset[“small”,“brown”,“fox”] StringNotIn–thevaluefordata.keyequalsavalueintheset[“jumped”,“over”,“the”] StringBeginsWith–thevaluefordata.keybeginswith“lazy” StringEndsWith–thevaluefordata.keyendswith“dog” Boolean BoolEquals
  31. Engineered for scale and reliability Retry intervals 10 seconds 30 seconds 1 minute 5 minutes 10 minutes 30 minutes 1 hour Once an hour up to 24 hours Defaults: 30 delivery attempts / 24 hours Dead-lettering Requires Storage account + container Dead-lettered events stored as blobs
  32. Storage Queues Flexible Interoperability Multi-Cloud Solutions IoT Hub Blob Storage Resource Groups Azure Subscriptions Event Hubs Custom Topics Storage (GPv2) Container Registry Service Bus Azure Event Grid + CNCF CloudEvents Azure Functions Logic Apps Azure Automation WebHooks Event Hubs
  33. Distributed log streaming
  34. Records a stream Recording moves forward only You can play the tape over and over again Can play from anywhere on the tape FastFwd or Rewind then Play Multiple channels with different data Event streaming
  35. Event Hubs conceptual architecture Kafka HTTP AMQP
  36. Where does data streaming fit in? Cloud gateways (web APIs) Field gateways Applications LegacyIOT (customprotocols) Devices IP-capabledevices (Windows/Linux) Low-powerdevices (RTOS) Search and query Cortana Analytics PowerBI Dashboards Devices to take action
  37. Event Hubs Capture: slow data stream Policy based push to own storage Avro format Does not impact throughput Better batching and archival scenarios Offloads batch processing from your real-time stream (no pressure drop)
  38. How does Capture work?
  39. How are Event Hubs and Kafka similar? Both use partitioned consumer Partitions are independent Both use a client side cursor concept Both can scale to very high workloads Conceptually they are nearly the same
  40. Azure Event Hubs for Apache Kafka™ Kafka 1.0 compatible endpoint backed by Event Hubs Use your existing Kafka application & tools with Event Hubs Only need to change your connection string All defaults supported
  41. Cloud relayed sockets
  42. Cloud relayed communication Hybrid Connections & Relay Securely connect to on premises data and services from anywhere Load balance multiple sites behind a single endpoint Protect internal services by projecting a cloud endpoint Does not require any network changes
  43. Where does Relay fit in? When you have scenarios that are too complex or costly for VPN When VNET integration is not possible When you have to keep your data where it is
  44. How does it work? Service registers cloud endpoint Client connects to cloud endpoint Network magic happens
  45. Relay as a gateway Can front multiple service instances Round robin controlled in Relay service Sticky sessions ServiceInstances
  46. Almost done
  47. Serverless Streaming Data Enterprise Key Takeaways Segment Simple Queuing Eventing PubSub Streaming Enterprise Messaging Cloud Socket Product Storage Queues Event Grid Event Hubs Service Bus Relay & HC What do you care about • Communication within an app • Individual message • Queue (polling) semantics • Easy to use • Pay as you go • Communication between apps / orgs • Individual message • Push semantics • Filtering and routing • Pay as you go • Fan out • Many messages in a Stream (think in MBs) • Ease of use and operation • Low cost • Fan in • Strict ordering • Works with other tools • Instantaneous consistency • Strict ordering • JMS • Non-repudiation & Security • Geo-Replication & Availability • Rich features (de-dupe, scheduling, etc.) • Raw TCP socket that can traverse firewalls • Load balanced live traffic What you care less about • Ordering of messaging • Instantaneous consistency • Ordering of messaging • Instantaneous consistency • Individual message semantics • Server-side cursor • Only Once • Cost • Simplicity • Buffered messaging • Queue semantics Focus point • Single partition lightweight queue • Reliable fan out push at massive scale • High scale distributed log • Highly indexed full featured message broker • High density TCP tunneling Hybrid
  48. @egrootenboer Thank You

Hinweis der Redaktion

  1. Exchange of messages between systems NOT SMS, WhatsApp etc.
  2. Commands are an instruction, telling a system to "do something". Approve the invoice. Reject the policy. Turn the light off. Hop on one foot. Commands can be validated, approved/rejected, processed, replied. Events on the other hand reflect an immutable fact. Invoice approved. Policy rejected. Light turned off. Ankle broken. We can't approve/reject events in the past, even if we would love to turn back the clock on certain recent elections.
  3. What the service is What is the key scenario How we address that challenge
  4. Rename from Azure Storage Queue to Azure Simple Queue or Elastic Queue
  5. - A reference to MSMQ (when on-premises solutions roamed the world) - Mention it's HTTP-based implementation - Tasks queues coordinate work across compute (VMs, Functions, etc.) - The messages in a queue are generally homogeneous - Low cost - Pay as you Go billing model - Easy to use - Have as many queues as you like (or connections) - Maximum queue size is insane while message size is small - mention claim check that's required to be implemented - Very few features – next slide
  6. Event Grid adds a small randomization to all retry intervals. After one hour, event delivery is retried once an hour. By default, Event Grid expires all events that aren't delivered within 24 hours. You can customize the retry policy when creating an event subscription. You provide the maximum number of delivery attempts (default is 30) and the event time-to-live (default is 1440 minutes). [30=7 in the first hour + 23 with one-per-hour]
  7. CloudEvents simplifies interoperability by providing a common event schema for publishing, and consuming cloud based events. This schema allows for uniform tooling, standard ways of routing & handling events, and universal ways of deserializing the outer event schema. With a common schema, you can more easily integrate work across platforms.
  8. Kill heart. Data sovereignty and accessing on-prem data securely from remote locations. Law firms with client and case notes in their office that needs to be accessed by attorneys remotely. Same with medical data by doctors and nurses. Mom and pop shop could use this as a cheaper alternative to azure vpn.