Run into that annoying SharePoint 5,000 Item Limit List View Threshold? Wondering why Microsoft says you can put 30,000,000 files in a library but also limits your list views to 5,000 items and locks your List/Library in the event it goes over? This session will discuss this issue, why it exists, how to plan around it so your organization can prevent it, and how to fix Libraries that are impacted by this threshold. Best practices, recommendations, and custom tools are included for demonstration.

1. Big data in SharePoint
and the 5,000Item ListView Threshold
Benjamin J. Steinhauser
SPS NYC 2016
1

2. About Me
• Benjamin J. Steinhauser
• SharePoint Solution Architect
• B&R Business Solutions
• BS/MS in CS; MCP, MCTS
• 15+ years as ASP.NET/Web Application Developer
• 10+ years as SharePoint Developer
• Creator of Open Source Projects:
• SPCAMLQueryHelper, SPFileZilla, SPMMDNavigator, and more…
• Partner with AMREIN Engineering Web Part Experts:
• Metro Grid Web Part, Inspired Tiles, Goal Thermometer, Swipe Gallery
• Regular attendee & presenter at SharePoint User Groups, SharePoint
Saturday Events
2

3. The Myth
• “I read online that I can’t save more than 5000 items(files)
in a list(library).” –Linda
• “After my library exceeded the list view threshold my list
was not accessible, I lost all my data!” –Bob
• FACT: Document Libraries and Lists can save LOTS of data.
• Microsoft says we can save 30,000,000 files in a library,
and 30,000,000 items in a list
• Max file size is 2 GB (or 10 GB for SharePoint 2016)
• FACT: 5,000 items or files is NOT a limitation for On
Premise and SharePoint Online.
• Don't write off SharePoint if you encounter or read about
the 5,000 Item List view threshold!!!
3

4. The Problem
• “This view cannot be displayed because it exceeds the list view
threshold (5000 items) enforced by the administrator.”
4

5. The Answer
1. Understand the threshold issue.
2. Plan to avoid the threshold issue.
This doesn’t mean to only save 5000 or less items!
3. Resolve threshold issues if/when they occur.
5

6. 1. Understanding
• What is the List View Threshold?
• “Specifies the maximum number of list or library items that a
database operation can process at the same time.”
• This is typically a List View (default or custom, list page or web
part) that is retrieving list items or files from the database and
displaying them on the page.
• All List Views that are sorting, filtering, grouping, or aggregating
(sum, count) are impacted.
• Can also impact compiled or client side code (ex. C# using
SharePoint Object Model, JavaScript using JavaScript Object
Model or REST APIs, SP Client Object Model C# or JS).
• All APIs are impacted.
• *You can use paged queries (REST, SPOM, JSOM, SPCOM) to get
around this limitation.
6

7. Cont’d
• Why is the number 5,000 important?
• To minimize database contention SQL Server uses row-level locking
to ensure accurate reads/updates without adversely impacting other
users who are accessing other rows.
• However, if a read or write database operation causes more than
5,000 rows to be locked at once, then it's more efficient for SQL
Server to escalate the lock to the entire table.
• When this lock escalation occurs it prevents other users from
accessing the table.
• If this happens too often, then users will experience a degradation of
system performance.
• Therefore, thresholds and limits are essential to help minimize the
impact and balance the needs of all users.
• NOTE: This is only relevant for storing items/files in SQL Server, and that’s why SharePoint has
this problem. Dropbox, OneDrive (not OD for Business, Google Drive, Box, don’t have this
problem. 7

8. Cont’d
• Can the number be changed?
• On Premise Farms: YES!
• Each Web Application can have a different threshold set for regular
users (default 5,000), and for Administrators (default 20,000).
• BEWARE: Increasing this number will impact performance in your
farm and is not recommended as a permanent solution
• However, a temporary change to resolve the threshold issue is possible,
more on that later.
• SharePoint Online: NO!
• Stuck with 5,000.
8

9. Cont’d
• Why is the Threshold important?
• Thresholds can help throttle operations and balance resources
for many simultaneous users.
• Its all about performance!
• Once any user adds an item to a list or file to a library that
exceeds the List View Threshold, you can get stuck!
• Sometimes this puts you in a bit of a Catch-22 situation, because
you are blocked from performing the very operations that can
help you get unstuck.
9

10. Cont’d
• What happens when the Threshold is exceeded?
• Error messages! List or View is unavailable! APIs crash!
• Block Operations:
• Create or remove an index. “This is yuge!” –Trump
• Filter and sort based on non-indexed columns.
• Set List/Library/Folder/Item permissions.
• Use the Open with Explorer command (to bulk move, delete).
• Delete a List or Folder (conflicting advice here, limit may be
100,000 for Office 365).
• Copy or rename a folder.
• Change column details (except a name change).
• Show totals, counts, aggregates in a list view.
• Save a list as a template with data. 10

11. 2. Planning
• Distribute the content:
• Move the content from a Single Library to Multiple Libraries
(same or different Sites).
• Try to organize content into Business Area or Function.
• Create Library Folders allowing you to break up content into
smaller chunks
• Each folder can only contain up to 5000 items, not counting sub
folders.
• Even if you initially break your content into separate Sites, Lists,
and/or Folders, you cannot guarantee that users will follow this
guideline over time.
11

12. Cont’d
• Manage column indexes:
• Add indexes on columns that are filtered, sorted, grouped, etc.
• Know your list, know your fields.
• Try to identify up front the fields that will be critical for filtering,
sorting, etc.
• Automatic Indexes:
• NEW for SharePoint 2016 and SharePoint Online.
• Open List/Library Settings, Advanced Settings, "Automatic Index
Management", by default is turned On.
• “Allows SharePoint to maintain column indexes to provide the best
query performance within views.”
• Limited success while testing, when my test libraries went over 5,000
files the views failed, not sure if timing issue or other.
• This doesn't let us off the hook for proper planning!
12

13. Cont’d
• List View Management:
• Create or update list views that filter, sort, group, or aggregate
using the indexed columns.
• Don’t forget about the Default List View!
• Enable Metadata Navigation:
• When you enabled and configure metadata navigation for the
library SharePoint automatically creates indexes for your
columns.
• Therefore, with metadata navigation you can have queries that
exceed the 5,000 List View Threshold (using those MMD fields).
• Available in Site Settings -> Manage Site Features.
13

14. Cont’d
• Use the Content Organizer:
• The Content Organizer can route files to specific document
libraries, folders, or even other sites.
• Rules can be set to automatically create folders based on
metadata properties and balance content into different folders to
maintain a maximum size for each folder.
• When a specified size limit is reached, a new subfolder is created
to contain the new files.
• The documentation suggests this is for folder sizes, not number of
files in a folder! Needs testing, confirmation.
14

15. 3. Resolving
• On Premise Only:
• Changes in Central Administration -> Web Application Settings ->
Resource Throttling.
• Daily Time Window for Large Queries: daily time window set by the
administrator during which queries are unrestricted.
• Changing the List View Threshold in Central Administration for All
users (5,000).
• Changing the List View Threshold in Central Administration for
Admin users (20,000).
15

16. Cont’d
• On Premise and Cloud:
• Deleting rows (in Lists) and files (in Libraries) until the number of
items is 5,000 or less and regain access to the list or library.
• This strategy can utilize the Recycle Bin, but you can only delete up
to 100,000 items.
• Use Access (items and files) or Excel (items only) to migrate list
data.
• Use PowerShell to move/delete files. (Or C# console
applications, JS scripting, REST, SOAP, ...)
16

17. Cont’d
• Open Source Tools:
• My free custom tool SP5000ItemLimitThresholdHelper.
• Uses the SharePoint File.MoveTo command to bulk move files.
• C# and SharePoint Client Object Model DLLs
• Compatible with SharePoint 2010, 2013, 2016, SharePoint Online.
• Only supports Document Libraries (not regular Lists), must be in
same site, folders are supported, File.CopyTo and Bulk Delete
available too.
• https://sp5000itemlimitthresholdhelper.codeplex.com
• Any migration tool will work too, Metalogix, ShareGate, etc.
17

18. Demo
• List View Threshold in action.
• SP “Super Long Name” in action.
18

19. Research
• Thanks to the following web sites and blogs for valuable
information and tips.
• http://sharepointmaven.com/how-to-overcome-sharepoint-
5000-item-limit-threshold/
• http://www.slideshare.net/gzelfond/sharepoint-5000-item-list-
view-threshold-checklist-and-best-practices
• https://technet.microsoft.com/en-
us/library/cc262813(v=office.14).aspx
• https://support.office.com/en-us/article/Manage-large-lists-and-
libraries-in-Office-365-b4038448-ec0e-49b7-b853-
679d3d8fb784?ui=en-US&rs=en-US&ad=US
19

20. And Thank You!
• Email: bsteinhauser@bandrsolutions.com
• Twitter: @njitben
• Presentation: http://tinyurl.com/jalcg5x
20
Take a picture of this slide!