In this presentation we review the database development with SQL Azure and Windows Azure. Regards Eduardo Castro Martinez

1. Cloud Application Architecture with SQL Azure and Windows Azure Dr. Eduardo Castro Martínez Microsoft MVP ecastro@mswindowscr.org http://comunidadwindows.org http://ecastrom.blogspot.com

2. Session Objectives Provide a big-picture view of the SQL Server in the Cloud Provide a big-picture view of the Windows Azure platform Describe typical scenarios for using the Windows Azure platform

3. Agenda Cloud Computing Platform Understanding Cloud Computing Cloud Computing Security Cloud Computing and Standards Creating Cloud Aware Applications Windows Azure Platform

5. Collaboration

6. High level of interconexion between users

7. Web platform growing

8. Pressure over companies data centre.4

9. Issues Facing Developers 70% of Web Development Effort is “Muck”: Data Centers Bandwidth / Power / Cooling Operations Staffing Scaling is Difficult and Expensive: Large Up-Front Investment Invest Ahead of Demand Load is Unpredictable

10. Solution: Web-Scale Computing Scale capacity on demand Turn fixed costs into variable costs Always available Rock-solid reliability Simple APIs and conceptual models Cost-effective Reduced time to market Focus on product & core competencies

11. The Cloud General concepts

12. Typical Application Grow? Users Unused capacity Buy more of everything Servers Rearchitect Database to scale out Buy bigger SQL Box Add some Web Servers

13. Workload Patterns Optimal For Cloud “On and Off “ “Growing Fast“ Inactivity Period Compute Compute Average Usage Usage Average Time Time On & off workloads (e.g. batch job) Over provisioned capacity is wasted Time to market can be cumbersome Successful services needs to grow/scale Keeping up w/ growth is big IT challenge Complex lead time for deployment “Unpredictable Bursting“ “Predictable Bursting“ Compute Compute Average Usage Average Usage Time Time Unexpected/unplanned peak in demand Sudden spike impacts performance Can’t over provision for extreme cases Services with micro seasonality trends Peaks due to periodic increased demand IT complexity and wasted capacity

14. What is the ‘Cloud’? The Cloud – metaphor for the internet ‘Cloud Computing’ Exploiting resources ‘out there’ Scalable computational power Data management and storage Enabling services Applications

15. What is the ‘Cloud’? The Cloud “Packaged” Application An application that I buy “off the shelf” and run myself Hosted “Packaged” An application that I buy “off the shelf” and run at a host site “Packaged” Using cloud services An application that I buy “off the shelf” and host using cloud platform “Software as a Service” A hosted application that I buy as a service from a vendor Buy Build vs. Buy “Custom Built” Application An application that I develop and run myself Hosted “Custom Built” An application that I develop myself and run at a host site “Custom Built” using cloud services An application that I develop myself and host using cloud platform “Platform as a Service” A vendor hosted development and runtime environment Build

16. 3 Cloud Service Models Cloud Software as a Service (SaaS) Use provider’s applications over a network Cloud Platform as a Service (PaaS) Deploy customer-created applications to a cloud Cloud Infrastructure as a Service (IaaS) Rent processing, storage, network capacity, and other fundamental computing resources To be considered “cloud” they must be deployed on top of cloud infrastructure that has the key characteristics 12

17. Service Model Architectures 13

18. Cloud App Architecture

19. High Scale Application Archetype Intelligent Network Load Balancer Network Activation M Tiers of N stateless machines w/ Forward Caching Stateless ‘Worker’ Machines Queue Activation Shared Filesystem Partitioned Relational DB Async Dispatch Queue ‘Monolithic’ State Tier

21. Sample Cloud Application Architecture Source: eleniamapi@yahoo.com

22. Sample Cloud Architecture Source: eleniamapi@yahoo.com

23. The ‘Why’ and ‘How’ of Cloud Migration There are many benefits that explain why to migrate to clouds Cost savings, power savings, green savings, increased agility in software deployment Cloud security issues may drive and define how we adopt and deploy cloud computing solutions 19

24. Balancing Threat Exposure and Cost Effectiveness 20 Private clouds may have less threat exposure than community clouds which have less threat exposure than public clouds. Massive public clouds may be more cost effective than large community clouds which may be more cost effective than small private clouds. Doesn’t strong security controls mean that I can adopt the most cost effective approach?

25. 21 Cloud Migration and Cloud Security Architectures Clouds typically have a single security architecture but have many customers with different demands Clouds should attempt to provide configurable security mechanisms Organizations have more control over the security architecture of private clouds followed by community and then public This doesn’t say anything about actual security Higher sensitivity data is likely to be processed on clouds where organizations have control over the security model

26. 22 Putting it Together Most clouds will require very strong security controls All models of cloud may be used for differing tradeoffs between threat exposure and efficiency There is no one “cloud”. There are many models and architectures. How does one choose?

27. Migration Paths for Cloud Adoption Use public clouds Develop private clouds Build a private cloud Procure an outsourced private cloud Migrate data centers to be private clouds (fully virtualized) Build or procure community clouds Organization wide SaaS PaaS and IaaS Disaster recovery for private clouds Use hybrid-cloud technology Workload portability between clouds 23

28. Possible Effects ofCloud Computing Small enterprises use public SaaS and public clouds and minimize growth of data centers Large enterprise data centers may evolve to act as private clouds Large enterprises may use hybrid cloud infrastructure software to leverage both internal and public clouds Public clouds may adopt standards in order to run workloads from competing hybrid cloud infrastructures 24

29. An Overview of the Windows Azure Platform

30. The Windows Azure Platform Windows Azure SQL Azure Windows Azure platform AppFabric Developer tools

31. Windows Azure Basics Windows Azure can potentially provide various kinds of Windows-based environments The current release supports both .NET and unmanaged applications Windows Azure provides Operating System like concepts Process Management Computation Storage

32. Azure™ Services Platform

33. A set of connected servers On which developers can: Install and run services Store and retrieve data What Is The Cloud in Windows Azure?

34. It is an operating system for the cloud It is designed for utility computing It has four primary features: Service management Compute Storage Developer experience What Is Windows Azure?

35. The Windows Azure Platform SQL Azure Cloud Applications Windows Azure AppFabric Windows Azure On-Premises Applications Others Windows

36. Windows AzureAn illustration Application Compute Storage Config Fabric … SQL Azure Applications Windows Azure AppFabric Windows Azure Applications Others Windows

37. Windows Azure Datacenter Your Service Windows Azure Building Blocks Web Site (ASPX, ASMX, WCF) Web Site (ASPX, ASMX, WCF) Web Role (ASPX, WCF) LB Tables Worker Service Worker Role Internet Storage Queue LB Blobs

38. Windows Azure Applications, Storage, and Roles n m Web Role Worker Role LB Cloud Storage (blob, table, queue)

39. Windows Azure, In One Picture Business Portal Developer Portal Service Management Service REST User Code … … Runtime API Storage Cluster VS Tools Compute Cluster … WA SDK VM Setup REST MSFT Datacenters Desktop Cloud VM

40. Azure Partial Trust “Dev Fabric” Dev Environment

41. Windows Azure Compute Service A closer look VMs VMs HTTP/ HTTPS Worker Role Instance Web Role Instance Load Balancer IIS Application Storage Compute Fabric …

42. Storage Web Role Instance Worker Role Instance Fabric Agent Fabric Agent Fabric Controller Windows Azure Fabric

43. Cloud Simple Application Worker Role Web Role Queue Blob

44. Windows Azure for Application Developers Hosted Service Portal SDK Storage Develop Run Deploy

45. Roles Role Code Configuration

46. Load Balanced Instances Role Instance 1 http://yourapp.cloudapp.net Instance 2

47. Web vs Worker Role Worker Role Web Role System Host IIS Host Your Code Your Code

48. Compute

49. Windows Azure Compute Service The VMs are provided by a cloud-optimized hypervisor Not your average VM They run 64-bit Windows Server 2008 Each VM has a one-to-one relationship with a processor core For developers: It’s mostly standard Windows and .NET A few things require accessing the Windows Azure Agent, e.g., logging A desktop facsimile of Windows Azure in the cloud is provided for development via the SDK

50. Windows Azure Compute Service A closer look VMs VMs Worker Role Instance Web Role Instance HTTP IIS Load Balancer RoleEntryPoint.Start() Agent Agent Windows Azure Fabric Application Storage Compute Fabric …

51. Storage

52. Windows Azure Storage Storage types: Blobs: a simple hierarchy of binary data Tables: entity storage (not relational tables) Queues: high performance quasi-transactional queue structures Access: Data is exposed via a RESTful interface Data can be accessed by: Windows Azure applications Any other internet connected applicationon-premises or other cloud applications

53. Windows Azure Storage ServiceA closer look HTTP/ HTTPS RESTServices Blobs Tables Queues Application Storage Compute Fabric …

54. Business Analytics Reporting Data Sync Database

55. SQL AzureA relational DB in the cloud SQL Azure Database Data Hub Others (Future) Relational database as a service Highly available, automatically maintained Extension of the SQL Server Data Platform .NET Services SQL Services Applications Live Services Windows Azure Applications Windows Mobile Windows Vista/XP Windows Server Others

56. Database Relational database, provided as a service Highly symmetrical development and tooling experience (use TDS protocol and T-SQL) Highly scaled out, on commodity hardware Built on the SQL Server technology foundation Beyond “hosted database” High availability, DB provisioning, and DB management are provided by the service Pay for what you use

58. Highly symmetrical

59. Highly scaled

61. Symmetry w/ on-premises database platform

62. Automatic high-availability and fault-tolerance

63. Automated DB maintenance (infrastructure)

64. Simple, flexible pricing – “pay as you grow”T-SQL (TDS) T-SQL (TDS) SQL Azure Database - AD Federation (LiveId /AppFabric AC)

65. Application Primary FocusDesigning for the Cloud Fundamentals Automated Service Management High Availability Scale out Services Multi-tenancy Resource Optimization

66. Transitioning Cloud Driven Process

67. Transitioning to the cloud Move Entire Applications to the Cloud Extend Application Components to the Cloud Store Data in the Cloud Create hybrid applications by Combining Cloud Services Start new applications based on Cloud Services

68. Migration Paths for Cloud Adoption Use public clouds Develop private clouds Build a private cloud Procure an outsourced private cloud Migrate data centers to be private clouds (fully virtualized) Build or procure community clouds Organization wide SaaS PaaS and IaaS Disaster recovery for private clouds Use hybrid-cloud technology Workload portability between clouds 57

69. Infrastructure Transition

70. Cloud Reference Architecture Cloud Computing for Financial Institutions. VivekKejriwal

71. Transitioning Cloud Driven Architectures

72. Transitioning Cloud Driven Architectures

73. Transitioning Cloud Driven Architectures

74. Transitioning Cloud Driven Architectures

75. Application Architecture for the Cloud

76. Application Basic Building Blocks n m Presentation Role Service Role LB Cloud Storage (blob, table, queue) 65

77. N-Tier Application Transition Presentation Business Move to the cloud On Premises Data Access Data Data

78. N-Tier Application Transition Presentation Business Move to the cloud On Premises Data Access Data Data

79. N-Tier Application Transition Presentation Move to the cloud Business Move to the cloud Data Data Access Data On Premises Data Sync

80. Simple Cloud Application Services Presentation Queue Blob

82. Service Bus 2) Discover endpoints Registry 4) Upgrade to direct connection 1) Register endpoints 3) Access application Endpoints Application Application Organization X Organization Y Access Control Service Bus Cloud Communication Pattern

83. Transitioning to the CloudDesign Requirements

84. Basic cloud development principles

85. Basic cloud development principles

86. Cloud Transitioning ArchitectureIdentify Patterns Consider: Basic components Messaging Interoperable Message Format and Protocols Communication or Integration Bus Resource Deployment Storage Infrastructure (Processing, Creation, …)

87. Scaling Out using a Queue Based Pattern Process Message Process Role Submit Message Queue Msg 1 Msg 2 Msg 2 Msg 1 Application Worker Role Process Role Msg 3 Msg 4 Msg 2

88. Table Incident Table Queue A Alarm Processor Pos Rpt Table NewMsg Parse & Route Event Table Timer Table Implement Queues for Reliability and Continuity Processor A Table Queue B Processor B Table

89. Building Blocks Key takeaways Use the cloud provider service configuration to scale Design how to communicate your application components Built your application to scale up/down roles in production

90. Windows Azure Storage Service

91. Windows Azure Storage ServiceA closer look HTTP/ HTTPS Blobs Tables Queues Application Storage Compute Fabric …

92. Fundamental Storage Abstractions Tables– Provide structured storage. A Table is a set of entities, which contain a set of properties Queues– Provide reliable storage and delivery of messages for an application Blobs – Provide a simple interface for storing named files along with metadata for the file Drives – Provides durable NTFS volumes for Windows Azure applications to use (new) 81

93. Windows Azure StorageTables: Strengths Massive scalability By effectively allowing scale-out data Perspective: Applied to the right problem, Windows Azure Tables are a beautiful thing But they’re not the optimal solution for a majority of data storage scenarios Amazon, Google, and others provide similar cloud storage mechanisms All of these can be seen as part of the NOSQL movement

94. Windows Azure StoragePoints of interest Storage types: Blobs: store binary data Tables: provide scale-out, entity-based storage Not relational tables Queues: allow message-based communication Access: Data is grouped into storage accounts Data can be accessed by: Windows Azure apps Other on-premises or cloud apps

95. Windows Azure StorageA closer look at blobs Options for accessing blobs: From Windows Azure applications or other applications: via RESTful requests From Windows Azure applications: via Windows Azure Drives Drives make a blob look like an NTFS file system to the application Blobs can also be used with the Windows Azure Content Delivery Network (CDN) For faster access from distributed locations to frequently accessed content

96. Windows Azure StorageA closer look at tables . . . Table Table Table Entity . . . Entity Entity Storage Accounts Property Property Property Name Type Value

97. Windows Azure Tables Provides Structured Storage Massively Scalable Tables Billions of entities (rows) and TBs of data Can use thousands of servers as traffic grows Highly Available & Durable Data is replicated several times Familiar and Easy to use API ADO.NET Data Services – .NET 3.5 SP1 .NET classes and LINQ REST – with any platform or language 86

98. Windows Azure StorageTables: Challenges No SQL A new approach for developers to learn No aggregates Access via REST You can’t use ordinary ADO.NET An unfamiliar structure for data You can’t easily move relational data in or out Supporting services are scarce, e.g., reporting No schema And no views

99. Windows Azure StorageA closer look at queues main() { … } Web Role Worker Role 1) Receive work 4) Do work ASP.NET, WCF, etc. 2) Put message in queue 3) Get message from queue 5) Delete message from queue Queue

100. Windows Azure Tables

101. Windows Azure Tables Provides Structured Storage Massively Scalable Tables Billions of entities (rows) and TBs of data Can use thousands of servers as traffic grows Highly Available & Durable Data is replicated several times Familiar and Easy to use API ADO.NET Data Services – .NET 3.5 SP1 .NET classes and LINQ REST – with any platform or language 90

102. Table Storage Concepts Entities Tables Accounts Email =… Name = … Users Email =… Name = … moviesonline Genre =… Title = … Movies Genre =… Title = … 91

103. Table Data Model Table A storage account can create many tables Table name is scoped by account Set of entities (i.e. rows) Entity Set of properties (columns) Required properties PartitionKey, RowKey and Timestamp 92

104. Required Entity Properties PartitionKey & RowKey Uniquely identifies an entity Defines the sort order Use them to scale your application Timestamp Read only Optimistic Concurrency 93

105. PartitionKey And Partitions PartitionKey Used to group entities in the table into partitions A table partition All entities with same partition key value Unit of scale Control entity locality Row key provides uniqueness within a partition 94

106. Partitions and Partition Ranges Server A Table = Movies [Action - Comedy) Server A Table = Movies Server B Table = Movies [Comedy- Western) 95

107. Table Operations Table Create Query Delete Entities Insert Update Merge – Partial Update Replace – Update entire entity Delete Query Entity Group Transaction (new)

108. Table Schema Define the schema as a .NET class [DataServiceKey("PartitionKey", "RowKey")] publicclassMovie { ///<summary> /// Category is the partition key ///</summary> publicstringPartitionKey { get; set; } ///<summary> /// Title is the row key ///</summary> publicstringRowKey { get; set; } publicDateTime Timestamp { get; set; } publicintReleaseYear { get; set; } publicstring Language { get; set; } publicstring Cast { get; set; } } 97

109. Table SDK Sample Code StorageCredentialsAccountAndKeycredentials = newStorageCredentialsAccountAndKey( “myaccount", “myKey"); stringbaseUri = "http://myaccount.table.core.windows.net"; CloudTableClienttableClient = newCloudTableClient(baseUri, credentials); tableClient.CreateTable(“Movies"); TableServiceContextcontext = tableClient.GetDataServiceContext(); CloudTableQuery<Movie> q = (from movie incontext.CreateQuery<Movie>(“Movies") wheremovie.PartitionKey == “Action" && movie.RowKey == "The Bourne Ultimatum" select movie).AsTableServiceQuery<Movie>(); MoviemovieToUpdate = q.FirstOrDefault(); // Update movie context.UpdateObject(movieToUpdate); context.SaveChangesWithRetries(); // Add movie context.AddObject(new Movie(“Action" , movieToAdd)); context.SaveChangesWithRetries(); 98

110. Key Selection: Things to Consider Scalability Distribute load as much as possible Hot partitions can be load balanced PartitionKeyis critical for scalability Query Efficiency & Speed Avoid frequent large scans Parallelize queries Entity group transactions (new) Transactions across a single partition Transaction semantics & Reduce round trips 99

111. Key Selection: Case Study 1 Table for listing all movies Home page lists movies based on chosen category 100

112. Movie Listing – Solution 1 Why do I need multiple PartitionKeys? Account name as Partition Key Movie title as RowKey since movie names need to be sorted Category as a separate property Does this scale? 101

113. Movie Listing – Solution 1 Single partition - Entire table served by one server All requests served by that single server Does not scale Client Client Request Request Request Request Server A 102

114. Movie Listing – Solution 2 All movies partitioned by category Allows system to load balance hot partitions Load distributed Better than single partition Server A Client Client Request Request Request Request Request Request Request Request Server B 103

115. Key Selection: Case Study 2 Log every transaction into a table for diagnostics Scale Write Intensive Scenario Logs can be retrieved for a given time range 104

116. Logging - Solution 1 Timestamp as Partition Key Looks like an obvious choice It is not a single partition as time moves forward Append only Requests to single partition range Load balancingdoesnot help Server may throttle Server A Applications Client Server B Request Request Request Request 105

117. Logging Solution 2 - Distribute "Append Only” Prefix timestamp such that load is distributed Id of the node logging Hash into N buckets Write load is now distributed Better throughput To query logs in time range Parallelize it across prefix values Server A Applications Client Server B Request Request Request Request 106

118. Key Selection: Query Efficiency & Speed Select keys that allow fast retrieval Reduce scan range Reduce scan frequency 107

119. Single Entity Query Where PartitionKey=‘SciFi’ and RowKey = ‘Star Trek’ Efficient processing No continuation tokens Server A Client Request Server B Result 108

120. Table Scan Query Select * from Movies where Rating > 4 Returns Continuation token 1000 movies in result set Partition range boundary Serial Processing: Wait for continuation token before proceeding Returns 1000 movies Partition range boundary hit Server A Cont. Cont. Return continuation Client Request Request Cont. Request Cont. Server B Cont. 109

121. Make Scans Faster Split “Select * from Movies where Rating > 4” into Where PartitionKey >= “A” and PartitionKey < “D” and Rating > 4 Where PartitionKey >= “D” and PartitionKey < “I” and Rating > 4 Etc. Execute in parallel Each query handles continuation Server A Cont. Cont. Request Client Request Request Server B Cont. 110

122. Query Speed Fast Single PartitionKey and RowKey with equality Medium Single partition but a small range for RowKey Entire partition or table that is small Slow Large single scan Large table scan “OR” predicates on keys => no query optimization => results in scan Expect continuation token for all except in 1 111

123. Make Queries Faster Large Scans Split the range and parallelize queries Create and maintain own views that help queries “Or” Predicates Execute individual query in parallel instead of using “OR” User Interactive Cache the result to reduce scan frequency 112

124. Expect Continuation Tokens – Seriously! Maximum of 1000 rows in a response At the end of partition range boundary Maximum of 5 seconds to execute the query 113

125. Entity Group Transactions (EGT) (new) Atomically perform multiple insert/update/deleteover entities in same partition in a single transaction Maximum of 100 commands in a single transaction and payload < 4 MB ADO.Net Data Service Use SaveChangesOptions.Batch 114

126. Key Selection: Entity Group Transaction Case Study Maintain user account information Account ID, User Name, Address, Number of rentals Maintain information of checked out rentals Account ID, Movie Title, Check out date, Due date Solution 1 – Maintain two tables – Users & Rentals Handle Cross table consistency Insert into Rentals table succeeds Update to Users table fails Queue to maintain consistency 115

127. Solution 2 Store Account Information and Rental details in same table Maintain same PartitionKey to enforce transactions Account ID as PartitionKey Update total count and Insert new rentals using Entity Group Transaction Prefix RowKey with “Kind” code: A = Account, R = Rental Row key for account info: [Kind Code]_[AccountId] Row Key for rental info: [Kind Code]_[Title] Rental Properties not set for Account row and vice versa 116

128. Best Practices Select PartitionKey and RowKey that help scale Efficient for frequently used queries Supports batch transactions Distributes load Distribute “Append only” patterns using prefix to PartitionKey Always Handle continuation tokens Client can maintain their own cache/views instead of frequent scans Future Feature - Secondary Index Execute parallel queries instead of “OR” predicates Implement back-off strategy for retries 117

129. Windows Azure Queues Queue are performance efficient, highly available and provide reliable message delivery Simple, asynchronous work dispatch Programming semantics ensure that a message can be processed at least once Access is provided via REST 118

130. Queue Storage Concepts Messages Queues Accounts 128 x 128 http://... thumbnailjobs 256 x 256 http://... sally http://... traverselinks http://... 119

131. Account, Queues and Messages An account can create many queues Queue Name is scoped by the account A Queue contains messages No limit on number of messages stored in a queue Set a limit for message expiration Messages Message size <= 8 KB To store larger data, store data in blob/entity storage, and the blob/entity name in the message Message now has dequeue count 120

132. Queue Operations Queue Create Queue Delete Queue List Queues Get/Set Queue Metadata Messages Add Message (i.e. Enqueue Message) Get Message(s) (i.e. Dequeue Message) Peek Message(s) Delete Message 121

133. Queue Programming Api CloudQueueClientqueueClient = newCloudQueueClient(baseUri, credentials); CloudQueue queue = queueClient.GetQueueReference("test1"); queue.CreateIfNotExist(); //MessageCount is populated via FetchAttributes queue.FetchAttributes(); CloudQueueMessage message = newCloudQueueMessage("Some content"); queue.AddMessage(message); message = queue.GetMessage(TimeSpan.FromMinutes(10) /*visibility timeout*/); //Process the message here … queue.DeleteMessage(message); 122

134. Removing Poison Messages Producers Consumers C1 P2 1. GetMessage(Q, 30 s)  msg 1 2 1 1 1 1 1 2 1 3 4 0 3 0 1 1 2 1 1 0 2 0 C2 P1 2. GetMessage(Q, 30 s)  msg 2 123

135. Removing Poison Messages Producers Consumers 1 1 C1 P2 1. GetMessage(Q, 30 s)  msg 1 5. C1 crashed 4 0 1 1 2 1 3 0 1 2 1 1 1 2 3 6. msg1 visible 30 s after Dequeue 2 1 C2 P1 2. GetMessage(Q, 30 s)  msg 2 3. C2 consumed msg 2 4. DeleteMessage(Q, msg 2) 7. GetMessage(Q, 30 s)  msg 1 124

136. Removing Poison Messages Producers Consumers 1. Dequeue(Q, 30 sec)  msg 1 5. C1 crashed 10. C1 restarted 11. Dequeue(Q, 30 sec)  msg 1 12. DequeueCount > 2 13. Delete (Q, msg1) C1 P2 4 0 1 2 3 0 1 3 1 2 1 3 3 1 2 C2 P1 6. msg1 visible 30s after Dequeue 9. msg1 visible 30s after Dequeue 2. Dequeue(Q, 30 sec)  msg 2 3. C2 consumed msg 2 4. Delete(Q, msg 2) 7. Dequeue(Q, 30 sec)  msg 1 8. C2 crashed 125

137. Best Practices & Summary Make message processing idempotent No need to deal with failures Do not rely on order Invisible messages result in out of order Use Dequeue count to remove poison messages Enforce threshold on message’s dequeue count Use message count to dynamically increase/reduce workers Use blob to store message data with reference in message Messages > 8KB Batch messages Garbage collect orphaned blobs 126

138. Storage Summary Table Scalable & Reliable Structured Storage System Partitioning is critical to scalability Entity Group Transactions (new) Queue Scalable & Reliable Messaging System Dequeue count returned with message (new) Use back-off strategy on retries Official Storage Client Library (new) 127

139. What is “SQL Azure”?

140. Review – Conceptual model Subscription Used to map service usage to the billing instrument Users may have many subscriptions Logical Server Akin to SQL Server Instance Unit of Geo-Location & Billing 1:1 Subscription & server User Database Restricted T-SQL surface area Additional catalog views provided e.g. sys.billing, sys.firewall_rules, etc

141. SQL AzureA relational DB in the cloud SQL Azure Database Data Hub Others (Future) Relational database as a service Highly available, automatically maintained Extension of the SQL Server Data Platform .NET Services SQL Services Applications Live Services Windows Azure Applications Windows Mobile Windows Vista/XP Windows Server Others

142. SQL Azure Database Single Logical Database Multiple PhysicalReplicas Replica 1 Single Primary Replica 2 DB - Based on SQL Server 2008 R2 engine Use same tools and data access frameworks Six global datacenters High Availability & Redundancy Reads are completed at the primary Writes are replicated to a quorum of secondaries Replica 3 Multiple Secondaries

144. Future Offerings

145. Additional data platform capabilities: Reporting, BI

147. The Evolution of SDS Evolves BrowserApplication Application Application BrowserApplication Application ODBC, OLEDB, ADO.Net PHP, Ruby, … REST Client SQL Client* REST Client Cloud Cloud Windows Azure REST (Astoria) Web App ADO.Net + EF REST Client HTTP+REST HTTP+REST HTTP TDS HTTP Windows Azure Web App SQL Client* Data Center Data Center TDS + TSQL Model REST/SOAP + ACE Model SDS Next SDS Current * Client access enabled using TDS for ODBC, ADO.Net, OLEDB, PHP-SQL, Ruby, …

148. SQL Azure SQL Azure Database Others (Future) SQL Azure Applications Windows Azure AppFabric Windows Azure Applications Others Windows

149. SQL Azure DatabaseAn Illustration SQL Azure Database Database TDS Application Database Database SQL Azure Database Others (Future)

150. SQL Azure DatabaseUsing one or multiple databases SQL Azure Database Database Application Max database size is 50 GB Database Database Application Database

151. SQL Azure Network Topology Applications use standard SQL client libraries: ODBC, ADO.Net, PHP, … Application Internet Azure Cloud TDS (tcp) Security Boundary Load balancer forwards ‘sticky’ sessions to TDS protocol tier LB TDS (tcp) Gateway Gateway Gateway Gateway Gateway Gateway Gateway: TDS protocol gateway, enforces AUTHN/AUTHZ policy; proxy to CloudDB TDS (tcp) SQL SQL SQL SQL SQL SQL Scalability and Availability: Fabric, Failover, Replication, and Load balancing

152. TDS Gateway TDS Listener Capability negotiation TDS Packet inspection Security Logical->Physical mapping via metadata catalog Enabler for multi-tenet capabilities Isolation layer

153. TDS Gateway Layering Gateway Process TDS Endpoint AdminSvc Endpoint Provisioning Endpoint Protocol Parser Business Logic Services Connection Mgmt SQL SQL SQL SQL SQL SQL Scalability and Availability: Fabric, Failover, Replication, and Load balancing

154. Provisioning Subscription Coordinated across all Azure services Executed in parallel w/retries Server May occur between data centers Point where Geo-location is established Database Always occurs within a single data center Cross node operations executed during this process e.g. add new db to sys.databases on the master

155. Server Provisioning Driven by administrator Portal Provision request is sent to Gateway Metadata catalog entry created DNS record (CNAME) created within LiveDNS service Master DB created On completion metadata catalog updated

156. SQL Azure Server Provisioning Live DNS Cluster Customer Browser Live DNS Svc Datacenter (Sub-Region) 1 5 Portal LB Gateway LB 2 4 3 6 Front-end Node Front-end Node Front-end Node Front-end Node Gateway Gateway Admin Portal Admin Portal 7 Backend Node Backend Node Backend Node SQL Server SQL Server SQL Server Mgmt. Services Mgmt. Services Mgmt. Services Fabric Fabric Fabric

157. Database Provisioning Gateway performs stateful TDS packet inspection Picks out subset of messages Parses out args for create database Makes entry into Gateway metadata catalog Unused replica set located and reserved Replica set (UserDB) is prepped for use Metadata catalog is updated

158. SQL Azure Database provisioning TDS Gateway 1 Front-end Node Protocol Parser TDS Session 2 3 Gateway Logic Master Node Master Cluster Master Node Components 4 7 5 6 8 Backend Node 1 Backend Node 2 Backend Node 3 SQL Instance SQL Instance SQL Instance SQL DB SQL DB SQL DB Scalability and Availability: Fabric, Failover, Replication, and Load balancing Scalability and Availability: Fabric, Failover, Replication, and Load balancing

159. SQL Azure Login Process Login request arrives at the Gateway Gateway locates MasterDb & UserDb replica sets Credentials are validated against MasterDb TDS session is opened to UserDB and requests are forwarded

160. SQL Azure Login Process TDS Gateway 7 1 Front-end Node Protocol Parser TDS Session 2 6 Gateway Logic Master Node Global Partition Map Master Node Components 3 8 4 5 Backend Node 1 Backend Node 2 Backend Node 3 SQL Instance SQL Instance SQL Instance SQL DB SQL DB SQL DB Scalability and Availability: Fabric, Failover, Replication, and Load balancing Scalability and Availability: Fabric, Failover, Replication, and Load balancing

161. Service Resilience Provisioning State machines used to coordinate activities across node (and datacenter) boundaries Failed provisioning attempts cleaned automatically after 10 minutes Login Failovers during the login will be transparent (<30 seconds) Metadata catalog refresh occurs automatically Active Session Surface as connection drops (due to state)

162. Monitoring Service Health Metrics Cluster wide performance counters gather key metrics on the service Used to alert Operations to issues before they become a problem Early warning system Code issues Capacity warnings Health Exercises the service routinely looking for problems When issues are encountered runs deep diagnostics Network connectivity at the node level Validate all dependent services (Live DNS, Live ID, etc) Monitoring from other MSFT DC’s Validates accessibility from multiple geographic locations Alerts fired automatically when test jobs fail

163. Security/Attack Considerations Service Secure channel required (SSL) Denial Of Service trend tracking Packet Inspection Server IP allow list (Firewall) Idle connection culling Generated server names Database Disallow the most commonly attacked user id’s (SA, Admin, root, guest, etc) Standard SQL Authn/Authz mode

164. What’s New In SQL Azure CTPEnhancements SQL Azure Database Windows Azure Platform – Developer Portal Enhanced user experience and reporting Database manager for SQL Azure SQL Azure Reporting CTP Reporting and BI on SQL Azure databases Based on SQL Server Reporting Services SQL Azure Data Sync CTP2 Synchronize SQL Azure databases Synchronize SQL Azure with on-premises SQL Server

165. What’s New In SQL Azure CTPEnhancements SQL Azure Database Windows Azure Platform – Developer Portal Enhanced user experience and reporting Database manager for SQL Azure SQL Azure Reporting CTP Reporting and BI on SQL Azure databases Based on SQL Server Reporting Services SQL Azure Data Sync CTP2 Synchronize SQL Azure databases Synchronize SQL Azure with on-premises SQL Server

166. What’s New In SQL Azure CTPEnhancements SQL Azure Database Windows Azure Platform – Developer Portal Enhanced user experience and reporting Database management SQL Azure Reporting CTP Reporting and BI on SQL Azure databases Based on SQL Server Reporting Services SQL Azure Data Sync CTP2 Synchronize SQL Azure databases Synchronize SQL Azure with on-premises SQL Server

167. SQL Azure Reporting CTP Features: Based on SQL Server Reporting Services Interactive and tabular reporting Data visualizations: charts, graphs, mapping, gauges SQL Azure Database data sources Utilizes BI Developer Studio (free) to author reports Export to Excel, PDF, CSV formats Core scenarios: Operational reporting on SQL Azure data Embed reports into Windows Azure or on-premises applications Web Role Reporting “Introduction to SQL Azure Reporting”

168. What’s New In SQL AzureEnhancements SQL Azure Database Windows Azure Platform – Developer Portal Enhanced user experience and reporting Database management SQL Azure Reporting CTP Reporting and BI on SQL Azure databases Based on SQL Server Reporting Services SQL Azure Data Sync CTP2 Synchronize SQL Azure databases Synchronize SQL Azure with on-premises SQL Server

169. SQL Azure Data Sync CTP2 Elastic Scale Service scales as resources requirements grow No-Code Sync Configuration Easily define data to be synchronized Schedule Sync Choose how often data is synchronized Conflict Handling Handle issues where same data is changed in multiple locations Logging and Monitoring Administration capabilities for tracking data and monitoring potential issues CTP 1 CTP 2 “Introduction to SQL Azure Data Sync”

170. SQL Azure Data Sync – A Closer LookCTP 2 On Premises Windows Azure Data Sync Service Local Agent Sync Orchestrator Sync Orchestrator HTTPS SQL Server Sync Provider SQL Server Proxy Provider SQL Server Provider TDS SQL Server SQL Azure

171. SQL Azure Data Sync – Example Use Cases Move workloads in stages preserving existing infrastructure Move part of the application and sync its data Meet compliance and regulations Control data synchronized off-premises Enable scale-out read or read/write Multiple synchronized databases for scalability Preserve data – geo replication of data Enable new scenarios Spanning enterprise, cloud and remote offices/retail stores

172. SQL Azure Data Sync – Roadmap SQL Azure Database Now Sync Sync Data Sync Service For SQL Azure Remote Offices Retail Stores Sync Sync Sync Sync Sync CTP2 – End of CY10 On-Premises (Headquarters)

173. New SQL Azure Usage Scenarios Fully featured Windows Azure Platform Application Data Sync Web Role Reporting

174. New SQL Azure Usage Scenarios Fully featured Windows Azure Platform Application Globally available, cloud & on-premises data synchronization Data Sync Data Sync Geo-replication of data across Windows Azure Platform datacenters Web Role Reporting Corporate Office Branch Office or Retail Datacenter Synchronize multiple on-premises SQL Server databases in different locations

175. Roadmap Now Visit www.microsoft.com/sqlazure to register for upcoming CTP access 2010 New Windows Azure Developer Portal; Database Manager Limited CTPs of Data Sync CTP2, Reporting CTP Watch for PDC Announcements; visit the SQL Azure blog and website 2011 Backup & Restore SQL Azure Reporting SQL Azure Data Sync

176. Windows Azure AppFabricInfrastructure in the cloud Service Bus Access Control SQL Azure Applications ? Windows Azure AppFabric Windows Azure Applications Others Windows

177. Service Bus 3) Discover Service Bus endpoint 1) Register service endpoint Registry 2) Expose Service Bus endpoint 5) Invoke operation on service endpoint 4) Invoke operation on Service Bus endpoint Client (On-premises or cloud) WCF Service (On-premises) Service Bus Access Control Illustrating Service Bus

178. Windows Azure Platform PricingA summary (in US dollars) Compute: $0.12 to $0.96/hour for each role instance depending on instance size Storage: Windows Azure blobs and tables: Data: $0.15/GB per month Access: $0.01/10,000 operations SQL Azure relational: $9.99/month per GB Bandwidth: Americas/Europe: $0.10/GB in, $0.15/GB out Asia/Pacific: $0.30/GB in, $0.45/GB out

179. Service Level AgreementsKey points: Windows Azure compute When you deploy two or more role instances in different fault and upgrade domains, your Internet-facing roles will have external connectivity at least 99.95% of the time Microsoft will detect within two minutes when a role instance’s process is not running and initiate corrective action

180. Service Level AgreementsKey points: Windows Azure storage At least 99.9% of the time, Microsoft will successfully process correctly formatted requests to add, update, read and delete data At least 99.9% of the time, your storage accounts will have connectivity to Microsoft’s Internet gateway

181. Service Level AgreementsKey points: SQL Azure database SQL Azure will maintain a Monthly Availability of 99.9% during a calendar month. Monthly Availability percentage for a specific customer database is the ratio of the time the database was available to customer to the total time in a month. An interval is marked as unavailable if the customer’s attempts to connect to a database are rejected by the SQL Azure gateway.

182. Using the Windows Azure Platform: Scenarios

183. Applying Azure (1)Some characteristics of Azure-appropriate applications Apps that need massive scale Example: A Web 2.0 application Apps that need high reliability Example: A Software as a Service (SaaS) application Apps with variable load Example: An on-line ticketing application Apps with a short or unpredictable lifetime Example: An app created for a marketing campaign

184. Applying Azure (2)Some characteristics of Azure-appropriate applications Apps that do parallel processing Example: A financial modeling application Apps that must fail fast or scale fast Example: Start-ups Apps that don’t fit well in an organization’s data center Example: A business unit that wishes to avoid its IT department Apps that can benefit from external storage Example: An application that archives data

185. Building on AzureA non-Azure app using Windows Azure storage Blobs SQL Azure On-Premises or Hosted Application

186. Building on Azure A massively scalable web app exposed on the public Internet Tables Web Role Instance Users

187. Building on Azure A massively scalable web app with background processing Tables Blobs Queues Web Role Instance Worker Role Instance Users

188. Building on Azure A web app with relational storage SQL Azure Web Role Instance Users

189. Building on Azure A web app using cloud and on-premises data SQL Azure Web Role Instance Service Bus On-premises Database Users

190. Building on Azure A parallel processing application Blobs Queues Worker Role Instance Web Role Instance User

191. The Windows Azure Platform Developer Experience Use existing skills and tools. platform Relational data Management AppFabric Compute Storage Management CDN Connectivity Access control

192. Conclusions Cloud platforms are here Microsoft is placing a big bet with the Windows Azure platform A new world is unfolding Prepare to be part of it

193. Related Content Comunidad Windows http://comunidadwindows.org http://www.slideshare.net/ecastrom http://ecastrom.blogspot.com Introducing the Windows Azure Platform http://go.microsoft.com/fwlink/?LinkId=158011 Introducing Windows Azure http://go.microsoft.com/?linkid=9682907

194. Presentation SourcesResources SQL Azure website: http://www.microsoft.com/sqlazure Blog: http://blogs.msdn.com/sqlazure Dev Center: http://msdn.microsoft.com/windowsazure/sqlazure Frameworks: http://msdn.microsoft.com/data

195. An Introduction to the Windows Azure Platform Dr. Eduardo Castro Martínez Microsoft MVP ecastro@mswindowscr.org http://comunidadwindows.org http://ecastrom.blogspot.com

196. © 2010 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.