Building SSRS 2008 Large Scale Solutions as per SQLPASS 2008 Summit w/ Lukasz Pawlowski and Denny Lee

1. Denny Lee, Lukasz Pawlowski
SQL Customer Advisory Team
SQL Server Reporting Services
Building SSRS 2008 Large
Scale Solutions PASS Community Summit 2008
November 18 – 21, 2008 Seattle WA

2. SQL Server Customer Advisory Team
(SQLCAT)
 Works on the largest, most complex SQL Server projects worldwide
– US: NASDAQ, Progressive, Premier Bankcard, Hilton Hotels
– Europe: Barclays Capital, Danske Bank, McLaren, Bwin
– Asia/Pacific: Korea Telecom, GMarket, Japan Railways East, China
Mobile
– LATAM: Banco Itau, Oi
– Strategic ISVs: SAP, Siebel, JDE, PeopleSoft, GE Healthcare, SunGard,
Siemens, Dynamics and more
 Drives product requirements back into SQL Server from our customers
and ISVs
 Shares deep technical content with SQL Server community
– SQLCAT.com
– http://blogs.msdn.com/sqlcat
– http://blogs.msdn.com/mssqlisv
– http://technet.microsoft.com/en-us/sqlserver/bb331794.aspx
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 2

3. SQL Server Design Win Program
 Target the Most Challenging and Innovative
Applications on SQL Server
 Investing in Large Scale, Referenceable SQL Server
Projects Across the World
– Provide SQLCAT technical & project experience
– Conduct architecture and design reviews covering performance,
operation, scalability and availability aspects
– Offer use of HW lab in Redmond with direct access to SQL
Server development team
 Work with Marketing Team Developing Case Studies
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 3

4. Session Objectives and Takeaways
 Session Objective(s):
– Provide guidance on how to scale out your Reporting Services
environment
– Provide RS best practices on RS catalogs, scale out deployment,
and performance optimizations
 Agenda:
– Reporting Services Scale Out Architecture
– Report Catalog Best Practices
– Scale Out Deployment Best Practices
– Performance Optimization Configurations
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 4

5. Reporting Services Scale Out
Architecture
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 5

6. Scale Out Architecture:
Overall Architecture
Report Server
RS Scale Out Deployment
Clients
RS Server
Report Catalog
Reporting Data
Flat Files,
OLE DB,
ODBC
NLB
Clients
RS Server RSDB
SQL, AS,
DB2, Oracle,
RS Server Teradata, etc.
Clients
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 6

7. Scale Out Architecture
Read the manuals!
 A lot of documentation on SSRS available online
 Many mistakes in implementation could have been avoided
 Read these:
– Planning for Scalability and Performance with Reporting Services
– Upgrading Reporting Services (SQL Books Online)
– Configuring a Report Server Scale-Out Deployment
 On sqlcat.com
– Building and Deploying Large Scale SQL Server Reporting Services
Environments Series
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 7

8. Scale Out Architecture:
Enterprise Rent-A-Car Customer Scenario
Report Server
RS Server
RS Scale Out Deployment
AS Server
Report Catalog
Reporting Data
RS Server
Teradata
RSDB
RS Server
AS Server
RS Server
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 8

9. Scale Out Architecture:
Enterprise Rent-A-Car Customer Scenario
 Build test system that accurately represented production
– Goal: 1800 concurrent users
 using VS test
 10s think time
 Mean 33-36s txn time
– Testing allowed them to identify blocking issues
 drop down parameter lists of thousands of rows for areas and branches
 Developed accurate workload representation (e.g. Proclarity and SSRS
clients)
 Currently in production
 This presentation incorporates lessons learned from this
and other customers
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 9

10. Scale Out Architecture
Importance of Performance Testing
 Need to understand your scenarios and reports
– Scenarios are defined by user personas & usage patterns
– Reports are either test reports or actual reports
– Tests should isolate Report Server from other systems
 Need tools to automate the testing
– See white paper: Using Visual Studio 2005 to Perform Load Testing
on a SQL Server 2005 Reporting Services Report Server
– Make single incremental changes between tests
– Do not use SQL trace inside VSTE
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 10

11. Scale Out Architecture
Customer Performance Testing
RS User Mean Tx Mean
Servers s Time CPU%
Max # of Concurrent Users
1 Server 608 36.9 99
2500
2 servers 1218 36.8 96
2300
2000 4 servers 2300 30.5 80
1500
 8GB RAM, 2 dual core RS
1218
1000
servers, Windows 2003
500  Graph is max # of users
608
reached for sustained time
0 period (>=15 min)
1 server 2 servers 4 servers
 2x RAM and CPU core, only
Users
1/3 increase in load
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 11

12. Scale Out Architecture
RS 2008 vs. RS 2005: Lessons Learned
 RS 2008 Front-End Server Scales Up Much Better than RS
2005
– Able to respond to 3–4 times the total number of users and their
requests without errors on the same hardware for all renderers
– RS 2008 consistently outperformed RS 2005 with the PDF and XLS
renderers on the four-processor, quad-core hardware platform
 See: Scaling Up RS 2008 vs. RS 2005: Lessoned Learned
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 12

13. Scale Out Architecture
RS 2008 vs. RS 2005: Lessoned Learned
Avg. Response Time (lower is better)
250
4x4 2008 Mix
4x4 2005 Mix
200
4x2 2008 Mix
4x2 2005 Mix
2x2 2008 Mix
150
Avg. Response
Time (ms) 2x2 2005 Mix
100
50
0
User Load
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 13

14. Scale Out Architecture
Scaling Up and Scaling Out with RS 2008 (cont)
 RS 2008
– Scale up front-end server to four-processor, quad-core servers for
performance
– Scale out to a two-node deployment for high availability
– Optimize disk I/O subsystem on all RS 2008 boxes for maximum
performance
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 14

15. Reporting Catalog Best
Practices
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 15

16. Report Catalog Best Practices
Report Server
RS Scale Out Deployment
Clients
RS Server
Report Catalog
Reporting Data
Flat Files,
OLE DB,
ODBC
NLB
Clients
RS Server RSDB
SQL, AS,
DB2, Oracle,
RS Server Teradata, etc.
Clients
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 16

17. Report Catalog Best Practices
Report Server Catalog Breakdown
Report Server Catalog (RSDB)
Stores all report metadata including report
Report Catalog
definitions, report / history snapshots,
scheduling, etc.
RSDB RS Temp DB
Stores temporary snapshots while running
reports
 These databases can be a bottleneck
 Optimize by applying standard SQL DB techniques
 Catalog has a lot of I/O and transactions
– RS2005: Many inserts to ChunkData, SnapshotData, and SessionData tables
– RS2008: Many inserts Segment; takes majority of transactions of RSTempDB
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 17

18. Report Catalog Best Practices
Use a dedicated server
Common scenarios
 Same server as SSRS Server
– Great for small environments
RSDB – In enterprise environments, too much resource contention
 Same server as data source database
– SQL resource contention (TempDB, plan cache, memory
buffer pool) between data source and RS catalogs
– As load increases need to monitor CPU, I/O, network
resources, and buffer pool
 Reduce resource contention by having a dedicated RS
catalog server you can tune.
 Apply standard high availability and disaster recovery
(e.g. clustering, mirroring, log shipping) to protect the
RSDB
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 18

19. Report Catalog Best Practices
High Performance Disk
 Check out Predeployment I/O Best Practices
 Have more smaller size disks with faster rotation speeds
(>=15k RPM) vs. fewer larger disks with slower rotations
RSDB  Maximize/balance I/O across ALL available spindles
 Separate disks between RSDB and RSTempDB
– RSDB a lot of small transactions (report metadata)
– RSTempDB has more (not as many) larger transactions
 Pre-grow your databases
 Stripe dB files to number of cores (0.25 – 1.0)
– Minimize allocation contention
– Easier to rebalance database when new LUNs are available
 Use RAID 10, not RAID 5
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 19

20. Report Catalog Best Practices
Operations Best Practices
 Data in RSTempDB is highly volatile
– Report lifetime policy of data = SessionTimeout value (10min)
– CleanupCycleMinutes guides background cleanup thread
RSDB – Once session timeout reached, cleanup temporary snapshot from tempDB
– This is done every CleanupCycleMinutes
 Data is RSDB is long lived; should be backed up
– Backing Up and Restore Databases in SQL Server
– Optimizing Backup and Restore Performance in SQL Server
– Backing Up and Restore Encryption Keys
 Maintain your RS catalogs
– Remember, these are SQL databses
– E.g. Re-indexing catalog tables or updating stats may improve query
performance
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 20

21. Report Catalog Best Practices
Report Catalog Sizing
 RSDB database size
– Varies by number of reports published and number of history snapshots
– General rule of thumb:
 Moderate size report definition takes 100-200KB of disk space
RSDB  This is larger than the actual RDL
 SSRS persists both RDL and compiled binary
 Assume 5:1 compression ratio (e.g. 10MB of data, snapshot is 2MB in size)
 RSTempDB database size
– Varies by number of users whom are concurrently using the Report Servers
– Each live report execution generates report snapshot persisted in the
RSTempDB
– General rule of thumb:
 10-20% concurrency of user base
 E.g. 1000 users, then max 200 concurrent users.
 If most users are accessing 10MB reports, then you will need 400MB of storage
– 200 users x 10MB reports / 5:1 compression ratio= 400MB
 Want to calculate for the maximum number of concurrent users
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 21

22. Disaster Recovery
Primary Data Center Content Switch
Automatic Failover
SSRS SSRS
Manual Failover
Failover Cluster
RSDB Async RSDB
RSDB Mirroring

23. Scale Out Deployment Best
Practices
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 23

24. Scale Out Deployment Best Practices
Report Server
RS Scale Out Deployment
Clients
RS Server
Report Catalog
Reporting Data
Flat Files,
OLE DB,
ODBC
NLB
Clients
RS Server RSDB
SQL, AS,
DB2, Oracle,
RS Server Teradata, etc.
Clients
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 24

25. Scale Out Deployment Best Practices
RS 2005: File System Snapshots
 RS TempDB has a lot of transactions to keep
report consistency (i.e. cached reports)
 Reduce RS Catalog I/O with File System
RS Server Snapshots
– It will store data on file system
– Unlike RS/IIS setup, will require more disk space
RS Server  To enable, update RSReportServer.config file:
– <Add Key="WebServiceUseFileShareStorage"
Value="true" />
RS Server – <WindowsServiceUseFileShareStorage>True</Wi
ndowsServiceUseFileShareStorage>
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 25

26. Scale Out Deployment Best Practices
RS 2005: File System Snapshots
 RS TempDB has a lot of transactions to keep
report consistency (i.e. cached reports)
 Reduce RS Catalog I/O with File System
RS Server Snapshots
– It will store data on file system
– Unlike RS/IIS setup, will require more disk space
RS Server  To enable, update RSReportServer.config file:
– <Add Key="WebServiceUseFileShareStorage"
Value="true" />
RS Server – <WindowsServiceUseFileShareStorage>True</Wi
ndowsServiceUseFileShareStorage>
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 26

27. Scale Out Deployment Best Practices
RS 2008: Why not File System Snapshots?
 SSRS 2005
– Advantage for SSRS 2005 because enabling feature
allowed less hits to RSTempDB
RS Server
– Entire report was calculated when requesting first page
 SSRS 2008 caches a lot of this data into memory
– Data continually persisted in report catalogs
RS Server
– Local file system acts as a write-through cache
– Does not pre-calculate everything on initial request
– On-demand engine retrieves all of the data and places into
RSTempDB for consistency
RS Server
– But many calculations are done on-demand as needed vs.
pre-calculated and stored.
 Still want to test in your environment
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 27

28. Scale Out Deployment Best Practices
Cache Execution
 Recurring theme on effective user of memory and
minimal I/O
 To help reduce I/O further, enable cache execution
RS Server on your reports.
 By default, reports are live execution
 Turn on cache execution for each report so the
RS Server report is stored in memory (thus reduced disk I/O)
 E.g. Even if you cache report every 5 minutes,
potentially a 80% reduction in I/O
RS Server
– If report is hit every minute, now only I/O hit every 5
minutes, i.e. 20% of the time
 No global setting for cache execution
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 28

29. Scale Out Deployment Best Practices
Load Balance your Network
 Load balancing important for many
client connections to RS servers
RS Scale Out Deployment
Clients  Recommend: Use cookie
RS Server persistence to preserve SSRS-to-
client connection
– IP affinity can work but may be
NLB overload for browser-based
Clients connections
RS Server
– Makes use of SSRS file cache
– Keep round-robin for initial
connections
RS Server  Recommend: dual NIC for RS
Clients – Split browser and AS/DB traffic
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 29

30. Scale Out Deployment Best Practices
Isolate your workloads
Report Server
Interaction
RS Scale Out
Deployment
NLB
Clients
RS Server
Report Catalog
Reporting Data
Scheduling
Benefits:
RSDB
Predictable Workloads
RS Scale Out
Deployment
Helps with Security Model
Isolate Performance Issues
RS Server
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 30

31. Scale Out Deployment Best Practices
Report Data Performance Considerations
 Scale out works for RS but may not work for
underlying Report Data (data source)
 Reporting loads Report Data, limit impact of
large numbers of users
Reporting Data
Flat Files,
OLE DB, – Limit data set size using report filters
ODBC
– SSIS limited data from Operational data sources
– Do not let all users access all of the reports
– E.g. Report Builder against Analysis Services results in
many queries being executed.
SQL, AS,
DB2, Oracle,
Teradata, etc.
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 31

32. Scale Out Deployment Best Practices
Report Data Performance Considerations
Additional resources to scale out SQL and SSAS
 Deploying a Scalable Shared Database
 SQL Server Replication: Providing High
Availability using Database Mirroring
Reporting Data
Flat Files,
OLE DB,
ODBC  Database Mirroring and Log Shipping
 SQL Server Replication Features
 Scale-Out Querying with Analysis Services
SQL, AS,  Scale-Out Querying with Analysis Services Using
DB2, Oracle,
Teradata, etc. SAN Snapshots
 Scaling out an Analysis Services Solution
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 32

33. Performance Optimization
Configurations
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 33

34. Performance Optimization
Handling Large Workloads
 Control Size of reports
– Do you need them?
– Is this really a data feed?
RS Server
– Aggregate reports and remove unused columns
 Recommendations
– Cache Execution
– Report Execution Timeouts
– Scheduled snapshots for large reports with data processing
bottlenecks
– Delivered Rendered reports for non-browser formats
– Pre-populate report cache using data driven subscriptions
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 34

35. Performance Optimization
Large Workload Tuning
 Analyze your reports
– Use ExecutionLog2 View
 Back to Report Catalogs
RS Server
– Increase size of your report catalogs to store more snapshot
data
 Tune the web service
– SSRS 2005: Tune IIS
– SSRS 2008: Tune HTTP.sys
 Windows 2003
 Windows 2008
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 35

36. Performance Optimization
ExecutionLog2 Analysis Checklist
 Sort by ElapsedSec or RowCount  Sort by Instance
for long running reports – Determine if NLB is handling request in
– TimeDataRetrieval: If high, need to balanced fashion
optimize data source  Sort by Report Path & Timestart
– High RowCount: A lot of data to determine report pattern
aggregated by SSRS, have SQL do
this – E.g. Expensive report (takes 5 minutes
to run) running every 10 minutes
 Sort by Request Type
 Sort by Status
– A lot of subscriptions, can determine
bottlenecks and stagger reports – Failures occur before (e.g. incorrect
RDL) or after (e.g. subscription delivery
 Sort by source error) report is processed
– To determine if live data or snapshot – Outdated information or settings (e.g.
– If report can be snapshot (e.g. last expired passwords, missing
week’s report), create snapshot to subreports, etc.)
avoid query execution, report  Data Driven Subscriptions
processing, and rendering
– Errors > 5%
– Continual Scale Mode
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 36

37. Monitoring Reporting Services by the
ExecutionLog2 View

38. Performance Optimization
Should we use 64-bit?
Yes!
 Report Server Catalog
– Standard Database techniques for optimization
RS Server
– Since SQL 2005, database written natively for 64-bit
 Report Server Service
– Most reports memory intensive
– Note, some workloads (e.g. many small reports) 32-bit can
execute faster
– Handle more concurrent report users or more large reports
– Able to more effectively use memory in SSRS 2008
– Will spill to file system if hits memory pressure
– Exceptions:
 Certain data provides not available for 64-bit
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 38

39. Performance Optimization
SSRS 2008 Memory Configurations
 Uses memory more efficiently; under intensive workload pressure, it
uses the file system cache. E.g., small requests will continue to stay
in memory while long running request will go to disk
 Therefore, before looking at the file system, check these memory
RS Server
configurations:
– WorkingSetMinimum / WorkingSetMaximum:
 Minimum / Maximum amount of physical memory that RS will make available to
perform its task;
 KB value within RSReportServer.config
 Increase value to process more requests in memory
 After WorkingSetMaximum is reached and exceeded for a period of time, recycle
app domains to reduce memory utlization
– MemorySafetyMargin:
 Defines boundary between low/medium pressure scenarios
 Default 80% value in RSReportServer.config
– MemoryThreshold:
 Defines boundary between medium/high pressure scenarios
 Default 90% value in RSReportServer.config
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 39

40. Performance Optimization
SSRS 2005 Memory Configurations
Recall, SSRS 2005 does not scale up as well as SSRS 2008
 MemoryLimit Configuration
• Default 60% of physical memory
RS Server
• Increase help process more requests
• Once threshold hit, no new requests are accepted
 MaximumMemoryLimit Configuration
• Default 80% of physical memory
• If this threshold is met, processing is aborted
 Changing values may solve RS only to bring up other contentions
 Recommendation: If constantly hitting memory thresholds, consider
scaling up and then scaling out
PASS Community Summit 2008 BI-401-A Building SSRS 2008 Large Scale Solutions 40

41. Thank you
for attending this session and the
PASS Community Summit 2008
PASS Community Summit 2008
November 18 – 21, 2008 Seattle WA