Sharing information and collaborating on work has increased productivity as well as fostered creativity. As SharePoint deployments grow with the business, application managers have more data to maintain. But it also poses a challenge to the storage admin who suffers from performance degradation as well as increased acquisition costs. Both impede the benefits of the organization’s collaboration and the value of the document-management solution.

There are five common challenges that application managers face due to SharePoint running sub-optimally on traditional storage subsystems. I’m writing to tell you the suffering can end for application, storage and database administrators.

1. Size of the database
The size of the content database has always been a challenge for SharePoint application managers. It used to be 200GB, and now with SharePoint 2012, it can go up to 4TB. This 4TB scale-up gives a completely different approach to the required storage perspective.

The recommended number of IOPS for performance is 2 IOPS per GB. So this means that for a 4TB content database the storage system needs to deliver 8,000 IOPS per content database. This would mean that for 200TB in a SharePoint farm, you need to have 400,000 IOPS. What kind of impact does this have on your application performance? Better yet, how quickly can the data be retrieved and delivered to the end user? These are important questions to consider in avoiding an unhappy user experience with the phenomenal growth of the storage consumption and the user connectivity.

2. DBCC consistency checks
The main reason to use bigger databases is the reduction of servers and SQL Server licenses. The size of a content database has a negative impact on the database consistency check. The data integrity of the SQL Server needs to be regularly checked. The bigger the database size, the longer it takes.

This is a maintenance task and cannot be run during production hours. Four terabytes is 20x the previous size of the content database. If the previous consistency checks took one hour, will that be 40 hours for a 4TB environment? How will that impact the other maintenance requirements, like backup? It will only leave eight hours in the weekend to get that in order.

3. Index and statistics defragmentation
When the database is indeed growing up to 4TB per content database, it is also clear that the index and statistics defragmentation will take longer. The index has about 20% of the content database as a sizing rule. This means that the defragmentation needs to happen on 800GB rather than on 40GB. This will have a major impact on the amount of time for performing this task. And it cannot be scheduled like the consistency checks; this will occur randomly at the point that SharePoint finds it necessary. The end result is that it will have an impact on the user experience for all the databases running from the same SQL Server.
4. Security
Access to websites, lists, folders and list items is controlled through a role-based membership system by which users and groups of users are assigned to roles that authorize their access to SharePoint objects. For each object in the content database, it will be checked whether the user has access or not. That means that when a user accesses his or her site where the documents are shared, it takes longer before the documents are displayed.

It is all about the process and how things come together. What happens when we supersize the content database by 20x and at the same time have to perform a re-index? Will the data delivery still live up to the end-user satisfaction? Can the security be quick enough, and will the re-indexing process not hinder it?

5. FAST index search
FAST Search Server 2010 for SharePoint uses a different way of storing the index than the traditional index search in SharePoint. The Fast Search is using flat files to construct the required indexes. The Fast engine can be used to build new indexes but also incremental indexes. The incremental Index has a different load perspective than the rebuild of all the indexes. Storage requirements are 2,000–3,000 I/O operations per second (IOPS), 50–100KB average block size, and less than 10ms average read latency. For example, for a farm setup with seven servers, the SAN must be able to serve 15,000–20,000 IOPS to the FAST Search Server 2010 for a SharePoint farm regardless of any other traffic the same storage system may serve. What will be the impact on the user experience? Will queries end anyway?

A change in mindset for application managers
It is time for a change. These five items are not the only challenges in a SharePoint farm, but they certainly are at the heart of the user experience and the total cost of ownership (TCO). Storage subsystems with a cached controlled engine are out of date for a scale-out SharePoint infrastructure.

Stop talking about IOPS. Stop saying it is not the storage subsystem. We have done some internal performance analysis and it performs fine. The storage subsystem will not be able to fully utilize the CPU power of the required SQL Server and they certainly will not help reducing license costs for SQL Server. This is all due to slow response times measured from the database, not inside the storage subsystem.

Start thinking in terms of fast data delivery. In storage terminology, that is “latency.” The quicker the data gets to the CPU, the faster the CPU can respond to end-user requests.

Are you willing to make this change? IT infrastructure and application management come together when they see the possibilities of a flash memory storage infrastructure. By doing so, they will also see the five issues get resolved, decrease TCO and improve user experience. Just imagine: moving the content databases to a flash storage infrastructure with no other changes required to fix the issues. No risks, no requirements to split the content database into multiple databases. Simply imagine the possibilities of business agility and collaboration.

Rob Bloemendal is Director Consulting Engineering for Violin Memory. In his career he has been an ERP consultant, Oracle and SQL Server DBA and pre-sales consultant for Storage.