Many organizations that have deployed SharePoint 2010 or 2013 also have some type of legacy ECM system in place.
However, SharePoint also has rich ECM capabilities with many innovative features not available in other ECM systems. As a result, we have found that many companies that use SharePoint wish to consolidate their IT infrastructure, by migrating content from older ECM systems into SharePoint.
While we think this is a great idea, such a migration requires a good deal of planning and design as well as specialized tools to automate the migration.
This slide deck provide detailed best practice guidance on how to plan for a migration of content from nearly any ECM system into MS SharePoint 2010 or 2013.
14 tips for planning a ecm content migration to share point
1. 14 Tips for Planning an ECM
content migration to SharePoint
2. Agenda
• Overview of SharePoint ECM
• Why migrate content from legacy ECM systems into
SharePoint?
• 14 Planning tips for ECM content migration to SharePoint
3. About Tom Castiglia
• Principal / Director of SharePoint Consulting Services
• Started with HersheyTech in 1998
• Active in the SharePoint community
• Speaker at various SharePoint Saturday conferences
• President of the San Diego SharePoint User Group
• Contact Me
• Twitter: @TomCastiglia
• tom@hersheytech.com
7. Why Migrate Content to SharePoint?
• Many legacy ECM products have been deprecated
• ECM ApplicationXtender is being phase out. EMC is pushing clients to upgrade to
Documentum D6/D7 which is much more expensive
• File Magic from Westbrook Technologies was discontinued in 2006 to favor a new
product called Fortis.
• Many other vendors simply have stopped innovating their products
• Consolidate IT infrastructure
• Reduce maintenance costs
• If IT staff is already maintaining a mature SharePoint farm, do they really need to also
maintain a separate document management system?
8. Why Migrate Content to SharePoint?
• Take advantage of improved taxonomy features in SharePoint
• Content Types
• Site Columns
• Managed Metadata
• BCS
• Lookup columns
• Take advantage of Records Management features in SharePoint
• Take advantage of SharePoint’s flexibility in general
• SharePoint is more extensible and customizable than any other ECM product
• SharePoint has largest market of 3rd party solutions to provide additional features
10. Tip # 1 –
Consider splitting one legacy repository into
multiple SharePoint libraries
Eliminate the
need for item
level
permissions
Reduce number
of documents
per library
(improves
performance)
SharePoint and
XenDocs allows
unified search
across multiple
libraries
11. Tip # 1 –
Take advantage of SharePoint Content types
Vendor PO # Invoice # Division
Alpha, LLC 3456617 74584 ACSC
Bravo, Inc 3456633 88363 ACMO
Charlie Co. 3456641 56546454 ACSC
Alpha, LLC 3456648 74584 ACSC
Delta Signs 3456652 675676 ACTX
Echo Ink 3456661 INV-324454 ACTX
Bravo, Inc. 3456670 456546464 ACMO
Vendor PO # Invoice #
Alpha, LLC 3456617 74584
Charlie Co. 3456641 56546454
Alpha, LLC 3456648 74589
ACSC
Invoices
Vendor PO # Invoice #
Bravo, Inc 3456633 88363
Bravo, Inc. 3456670 456546464
Vendor PO # Invoice #
Delta Signs 3456652 675676
Echo Ink 3456661 INV-324454
ACMO
Invoices
ACTX
Invoices
Legacy Repository
SharePoint Libraries
13. Tip # 2 –
Consider converting legacy “document types”
into SharePoint Content Types.
Document Contract
NDA
Retention: 5
Years
Workflow:
none
MSA
Retention: 7
Years
Workflow:
Approval
Fields
PartyName
ContactNumber
EffectiveDate
ExpirationDate
Fields
Name
Title
14. Tip # 3 –
Consider converting “choice” fields into Managed
Metadata, Lookup or External data columns
Managed
Metadata
Lookup External Data
(BCS)
Hierarchical YES NO NO
Reference
other columns
NO YES YES
Scope Farm Site Collection Enterprise
15. Tip # 4 –
Improve column name consistency
Legacy ECM System
Repository Field NameColumn
Auto Policies PolicyNumber
Medical Policies PolicyNo
PolicyNumber
Policy No
Policy#
SharePoint Site
Policy Number
Home Policies Policy#
16. Tip # 5 – Fix Improper Column Re-Use
Repository Field Name DocType Values
Policies DocType Application
SharePoint Site
Column
Renewal
Cancellation
Legacy ECM System
“Invoice DocType”
Invoices DocType PO Invoice
“Policy DocType”
Non-PO Invoice
Expense Report
Check Request
Claims DocType Accident Report
Estimate
Doc Type
“Claim DocType”
17. Tip # 6 –
Understand file types in legacy ECM system
Scanned
Documents
PDF
• Searchable
• Image-Only
TIF
• Multi-Page (1
file per
document)
• Single-Page
(multiple files
per document)
MS Office
Word
Excel
PowerPoint
Engineering
Auto-Cad
• dwg
Email
msg
eml
Multimedia
Audio
• mp3
• wmv
• wav
Video
• avi
• mpeg
• mp4
18. Tip # 7 –
Understand metrics in legacy ECM system
Total doc count in each repository
Average page count per document (or total page count per repository)
Average and peak file size per repository
Growth rates per document type or repository (# new docs/month)
Frequency of document retrievals per day per repository
19. Tip # 8 –
Understand document capture use cases
Do users validate new documents after they have been exported to the
document management system?
How do we sync terms between the ECM system and the
document capture system?
Does the current document capture system support exporting
documents and metadata to SharePoint?
How is content received? (dedicated scanners? fax? file import?
MFPs? email?)
Are there any specialized indexing scenarios (e.g. One to Many data)
20. Tip # 9 –
Understand ECM security model
By Repository
Document
Level Security
21. Tip # 10 –
Understand how documents are found
How many docs are
returned in typical query?
Save queries for re-use?
Search metadata vs. full-text
(or both)
What business condition
cause users to search for
a document (e.g.
customer service inquiry,
vendor payment, etc.)
How many times per day
is repository queried?
How many users
frequently search for
documents?
22. Tip # 10 –
What do users do with documents?
Read-Only
• View
• Print
• Email
Edit
Metadata
Annotate
• Sticky notes
• redactions
• shapes
• lines
• Stamps
Edit pages
• Rotate
• Re-Order
• Clean-up
• Insert
• Delete
23. Tip # 11 –
Understand any workflow processes related to the
documents
Ad-Hoc
routing
Formalized
workflow
24. Tip # 12 –
Understand any application integration with
current ECM system
Are there any Line of Business apps that have links to
documents in current ECM system?
Are there any custom reports created on data stored
in current ECM system?
Is data from current ECM system ever exported to
other systems?
25. Tip # 13 –
Consider some 3rd party ECM add-ons
Metadata-based
search is less
intuitive
SharePoint formats
Search results like a
“search engine”,
not a DMS.
SharePoint
metadata filtering
does not scale for
large libraries
26. Tip #13 –
XenDocs ECM Search With Vizit™
Intuitively build
precise document
queries
Sortable search
results
Image previews
for scanned
images
27. Tip # 14 –
Understand migration tool options
• Existing 3rd party tools vs Custom tools
• Field mapping (different field names between source system and SharePoint)
• Filters to allow migrations to be performed in chunks
• E.g. Only migrate documents where DocType=‘Expense Report’ and DocDate>’02/13/2009’
• Start/pause/stop/resume –
• not needed for smaller repositories, but critical for larger ones
• Ability to audit status of every record
• File Conversions
• Single Page TIF to Multi-Page TIF
• TIF to PDF
• Image only PDF to Searchable PDF
• PDF to PDF/A
28. Tip # 14 –
Migration tool pricing
• Pricing Models for SharePoint Content Migration Tools
• Per PC or server
• Subscription based (monthly or annual)
• Size based (Per GB of content)
30. In closing
• If you need assistance in planning an ECM content
migration , note that we can offer 2 hours of
complementary consulting services to assist with this or to
provide a demo of various migration tools.
• Upcoming webinars
• Tomorrow at 9:30am – Using Kapow for web content migration
• Sept. 24th on AP Invoice Automation
Hinweis der Redaktion
At Hershey Technologies, we’ve been providing IT solutions for more than 20 years. We have deep, end-to-end SharePoint consulting and development expertise, and we specialize in document capture, document management and workflow solutions!
We’re also resellers for a number of major SharePoint ISVs such as Nintex, AvePoint, and Vizit. We’re also a Diamond partner with Kofax.
I’d like to note that today’s webinar is not going to be a “product pitch”. The content today will focus on how to plan a migration of content from another ECM system to SharePoint. Although I will reference a few software tools that can automate your migration, in my experience the success of most migrations to SharePoint has more to do with how SharePoint is architected and how the migration process was planned than it has to do with the tools used to perform the migration.
I’d like to make today’s presentation more interactive than a typical webinar. So please chime in with any questions using the “chat” window. The content today assume that you have some familiarity with SharePoint. However, if you would like me to explain further about specific SharePoint features and how they might be relevant to your migration plans, please let us know through the chat window.
Before we get started, I want to ensure our terms are clear.
In the SharePoint community, the term “migration” can have different meanings.
Often people use it to refer to an upgrade of SharePoint version X to version Y. However, for this presentation, I am referring to migration of content (documents and metadata) from some other ECM (non-SharePoint) platform into SharePoint
There has been a great deal of consolidation in the ECM market. Many of the larger vendors have purchased smaller competitors. In many cases, the new owner’s primary goal is not to continue innovating on the new product, but to push customers to move to main product line. A classic example of this is EMC, who owns Documentum and ApplicationXtender. Customers on the AX platform tell us how there are no longer new features added, and EMC wants them to move to Documentum.
But if you are an AX shop and you already have a mature SharePoint installation, its far more cost effective to migrate to SharePoint then Documentum.
Over the last few years we have seen increased interest from our clients in migrating content from what I’ll call “legacy” ECM products to SharePoint.
So if you are in this webinar, I presume that you already are using SharePoint or that your company is already planning to use SharePoint. The purpose of this presentation is focus on “how” to plan your migration, not “why” you should migrate to SharePoint.
That being said, I’ll review some of the key reasons why so many companies are deciding to migrate ECM content to SharePoint…
Because of SharePoint’s enormous success, many ECM vendors have struggled to keep up. Some products that were previously strong players are being formally deprecated (e.g. ECM’s ApplicationXtender).
Some products have huge annual maintenance costs
Once an organization already has a mature SharePoint environment, it’s often a no-brainer IT decision to consolidate content from legacy ECM products into SharePoint.
Besides reducing annual maintenance costs, by consolidating number of systems that IT needs to support, internal IT staffing costs can be reduced as well.
Many legacy ECM products lack taxonomy design features that are core to SharePoint, so the existing taxonomy in the legacy ECM system may have been based not on ‘best practice’ but on what features were available. SharePoint features like Content types, site columns, managed metadata, BCS and Lookup columns allow us to design taxonomies that are far more elegant and functional than most other ECM products.
Many legacy ECM products either don’t have Records Management features or support RM as a separate ‘bolt on’ product, which may or may not be well integrated with the core ECM solution.
In cases when you wish to customize your ECM solution, SharePoint offers far more flexibility than any other ECM product… either through native SharePoint features, custom development or the huge market of Off-the-shelf 3rd party solutions.
Content Types in SharePoint let us group related metadata columns into a re-usable template. This allows us to de-couple the metadata schema from document libraries, and gives us flexibility that many other ECM products don’t have.
A content type can be re-used in multiple document libraries and conversely, a single document library can store documents of different content types.
Sometimes a legacy ECM system may have a small number of very large repositories that ideally should be re-factored into multiple SharePoint libraries. Some document management systems might not allow a unified search across multiple repositories, and so documents get grouped into a single large repository to simplify the end user search experience.
This approach often forces systems to use document level security models which may reduce the scalability of the system. There may be business requirements to separate documents into various document libraries, by department, business unit or some other metadata attribute.
Although SharePoint also supports item-level permissions, using item-level permissions can limit scalability. So if your current system is using item level permissions, consider refactoring the system into multiple repositories to avoid use of item level permissions.
When your ECM solution in SharePoint is used for archive (read-only) purposes only, SharePoint can be support millions of documents in a single document library. But if your documents are also used in collaboration (read-write) scenarios or you have workflow processes on your documents, you need to architect your SharePoint taxonomy to keep the number of documents per library to much smaller numbers, which may require creating more libraries that you have in your legacy ECM system.
Many ECM document repositories have a metadata choice field named “Document Type”, which is typically a drop down list (choice) field of possible document types. This design can be migrated directly to SharePoint. However, sometimes documents should be treated differently based on the “document type”. For example, say we have repository for “contracts”, with a field named “contract type” containing options for: NDA, MSA, Maintenance Agreement, Partnership agreement, etc. Perhaps, the MSA contracts should have a different retention period than other contracts, and maybe Partnership Agreements require a different approval process than the others.
In this case, we can define a new “Contract” content type in SharePoint which contains any other contract metadata fields (except for document type). Then we can create additional content types for NDA, MSA, Maintenance Agreement, Partnership agreement – each of these will inherit from the Contract content type, so they all share the same metadata. But we can define unique retention policies, workflow processes and templates for each one.
Many ECM document repositories have a metadata choice field named “Document Type”, which is typically a drop down list (choice) field of possible document types. This design can be migrated directly to SharePoint. However, sometimes documents should be treated differently based on the “document type”. For example, say we have repository for “contracts”, with a field named “contract type” containing options for: NDA, MSA, Maintenance Agreement, Partnership agreement, etc. Perhaps, the MSA contracts should have a different retention period than other contracts, and maybe the MSAs require a custom approval process than the others.
In this case, we can define a new “Contract” content type in SharePoint which contains any other contract metadata fields (except for document type). Then we can create additional content types for NDA, MSA, Maintenance Agreement, Partnership agreement – each of these will inherit from the Contract content type, so they all share the same metadata. But we can define unique retention policies, workflow processes and templates for each one.
We can now these content types in the same document library or multiple document libraries, and the documents will adhere to the retention rules and workflow processes defined by their content type.
All legacy document management systems support “choice fields”, where metadata values are constrained to a list of values, typically presented to users as a drop down list. Of course, SharePoint supports Choice fields as well. But SharePoint has several other options, which could provide long term benefits for certain use cases:
Managed Metadata: If the options in a choice field need to be re-used in many different use cases and across many SharePoint sites and site collections, or if the options in a choice field are hierarchical in nature, then consider defining columns using Managed Metadata.
Lookup columns: behave similar to a choice field, except that the list of choices are sourced from a SharePoint list that is in the same site as the document library. Lookup columns are great when you want to enable end users to dynamically add, remove or change choices available for that field, but the choices don’t need to be re-used across multiple sites.
BCS (External Data): behave similar to Lookup columns, except that the values are sourced from an external database, rather than a list in SharePoint. Columns based on external data can be re-used in multiple sites and site collections.
Note that all of these options complicate the migration process. So these options should not be considered lightly. Remember that your migration is a one time project, but your taxonomy lasts forever. So weigh whether the long time benefit of these features outweighs the short term costs to use these features.
Sometime users have created multiple repositories that have the same logical fields. Some ECM systems don’t support column re-use, and therefore certain columns may have been “manually copied” or re-created in multiple repositories. If users were not very careful, columns that should have shared the identical name were given slightly different names in each repository.
By creating “Site Columns” instead of “list columns”, we can make sure that columns are properly re-used to create a consistent environment. Mistakes happen! But there’s no reason not to correct those mistakes before starting your migration to SharePoint.
Sometimes however, the opposite problem can occur. Where multiple repositories each contain a field with the same name, event though the field has a different purpose within each repository. The most common example of this is the “Document Type” field. Every repository has a field with this name, but the available ‘doc type’ choices in each are different in each case.
This leaves us with two options:
Leave the naming convention as is, and create new list level choice columns in each document library, and copy the same values to each library. However, this means we sacrifice all of the re-usability benefits of SharePoint site columns.
Rename the columns to be more specific and create them as site level columns, which allows us to re-use the identical field settings in multiple libraries. This approach is a vastly improved design!
This information is important to know for a few reasons –
It may impact your choice of migration tools and the approach used for the migration
Depending on the file types, you’ll need to make sure that you have integrated viewers and editors for various file formats with SharePoint
This information is needed to help architect your SharePoint farm:
Number of site collections and content databases
Create one per repository
Put all repositories in one site collection
Group certain repositories in one site collection and others in a different site collection
Warrant an additional Physical SQL server (or multiple SQL servers)
Storage (SAN, NAS, IOPS)
Don’t assume that the current capture system will (or should) work exactly the same way with SharePoint as it does with the legacy ECM system. In many cases, you will be able to keep the identical capture process, and simply swap out an “export connector” for the legacy system with a new connector for SharePoint… but this isn’t always the case. Depending on your SharePoint taxonomy, your capture system may or may not have the features needed to support all of the relevant SharePoint features (e.g. columns types, such as Managed Metadata, BCS and Lookup columns).
If you are using Choice fields, Managed Metadata, Lookup or BCS columns in SharePoint, think about how you will sync these terms with your document capture system, For example, Kofax Capture allows users to reference Managed Metadata termsets from SharePoint while they are validating documents. So the Kofax admins don’t need to worry about syncing the SharePoint terms with Kofax.
Make sure you inventory all of the channels that can deliver content to your legacy ECM system, and validate that each capture sub-system is compatible with SharePoint.
Be on the lookout for any specialized or custom document indexing processes, such as “One to many indexing” where a single document may contain many related metadata records.
When possible, avoid using item level security in SharePoint, especially for larger libraries or libraries that will have a great deal of unique ACLs, as this can inhibit scalability. SharePoint supports up to 50,000 unique ACLs per library. If your current repository is using item level security, consider refactoring it. Note that SharePoint’s security model supports very granular permission configurations. Be sure to document your legacy ECM system’s security model, so that SharePoint can be configured to in the same manner. Optionally, you may find that the current security model no longer reflects your business’ requirements.
Use the migration as a good time to re-architect your security if needed.
Make sure you understand and document exactly how users interact with your current ECM system. Site down and watch how users really do their job on a daily basis. This is especially important for any users that use your ECM system on a daily basis to get their primary job done.
What do users do with the documents after they find and open them?
If its more than read-only, consideration must be given to the file types (PDF, Office, TIF, etc.). While SharePoint treats office documents as a 1st class citizen, enabling all edit processes easily… when it comes to other file types, any but read-only operations may be a challenge unless you have integrated specialized components to support the editing operations that you require. Hershey recommends use of XenDocs Search and Vizit Pro to provide a user experience in SharePoint that closely resembles other legacy ECM products.
Be sure to inventory any workflow processes in use in your legacy ECM system. If it has any workflow components, these workflow need to be re-created in SharePoint. While SharePoint Designer is a great tool to build simple workflows, we typically recommend Nintex Workflow for its powerful, and easily workflow design features combined with reporting and metrics.
Also, be sure to inventory any integration points with your legacy ECM system. Perhaps your Accounting or ERP application has links to documents in your legacy ECM system. You may have custom reports or dashboards that pull metadata from your current system, or maybe the metadata from your current ECM system is exported to other applications to drive some other process. Be sure to document any integration points so that these can be re-created in SharePoint.
There are two major features that most document management systems have:
-intuitive metadata filtering/query builder
-search results displayed in a tabular grid
SharePoint’s native metadata filtering displays resulting a grid, but it doesn’t support column types like single line of text, and it doesn’t scale well in document libraries with more than 5000 documents.
SharePoint Search provides the advanced search web part, but this can be challenging to configure and formats results like a search engine, rather than in a tabular display.
XenDocs Search provides a user experience that is similar to virtually all other document management systems, including intuitive metadata filtering, tabular search results, document sorting, document previews, etc.
XenDocs allows SharePoint users to have users the “best of both worlds” – all of the web content management and document collaboration features AND a great solution for managing transactional document.
There are many migration tools for SharePoint available. Depending on your source ECM system you may find an off-the-shelf migration product that meets all of your needs for a low cost. In many cases, some level of customization may be needed. Features to look for include:
Field mapping, in case columns in SharePoint are different from the legacy system.
Ability to migrate sub-sets of documents in filtered chunks
Ability to Start/Pause/Stop/Resume a migration
Extremely robust auditing of every document that is successfully migrated or that encounters an error. This is critical to troubleshooting any migration problems. This data should be stored in a database, not exported to a text file.
File Conversions (especially to PDF, very common!)
Support for advanced SharePoint column types (e.g. MMS, Lookup, BCS). However, there are work-arounds in SharePoint for transforming data from text columns to other column types, so if the migration tool doesn’t support this, its not a huge problem. Either way, you need to know what column types are supported.
Most content migration tools fall into one of the following three models:
A Flat fee for the software per machine. Note that most migration tools do not install on your SharePoint server. You may be able to install the software on multiple machines to migrate multiple content sources in parallel
Subscription based – charge a fee per month or annually
Size based – charge a fee per GB of content
Annual Support – many migration tools do not charge an annual support fee as it is assumed most migrations shall be completed in less than 1 year. However, so migration tools can be used for ongoing projects.
Our XenDocs ECM Migration toolset is comprised of two major components:
Export modules, which exports batches of content from your legacy ECM systems, such as ApplicationXtender, FileNet, File Magic, Liberty, Sire, LaserFiche, etc.
File Importer for SharePoint which imports these batches into the appropriate sites, libraries, content types and columns. The File Importer can also perform image processing on your documents during the migration, such as OCR, convert to searchable PDF, auto-rotate upside down pages, etc. This tool can also be used after your migration project to integrate multi-function scanners, fax servers or other line of business applications with SharePoint on an ongoing basis.
If you have a migration project in mind we’d be happy to provide you with a personalized demonstration.