e-discovery 2.0

Last month, Gartner updated its Magic Quadrant for Email Active Archiving. I have spoken to the analysts (Carolyn DiCenzo and Ken Chin) many times, and have found them to be keen observers of the archiving market. So, it’s interesting to read what they’ve concluded from all their customer interviews, and track what’s changed in the past 12-24 months. Comparing this year’s report to those from 2007 and 2006, I was struck by 3 observations, leading to 1 simple conclusion.

Observation #1: The Status Quo Is Alive And Well
Despite the thousands of hours that I’m sure vendors have spent trying to influence Gartner, about 80% of vendors in the report have seen no meaningful change in their rankings since last year. Most notably, Symantec is still the only company in the “Leaders” quadrant, while Autonomy/ZANTAZ remains rooted in the “Visionary” quadrant where it has been stuck for the past 2 years. Unlike Forrester, Gartner does not believe that Autonomy’s acquisition of ZANTAZ has made a shred of difference to ZANTAZ’s “ability to execute”, which is the main criterion by which it trails Symantec.

Observation #2: To Move Up, You Have To Differentiate
Only two vendors have significantly improved their positions over the past 2 years. One is Mimosa, which has separated from the pack by moving up from “Niche Player” to “Visionary”, primarily because it is “easy to deploy and manage” and offers “near continuous data protection and recovery features”. The other is CommVault, which has overtaken 8 other vendors. Gartner attributes CommVault’s rise to its new release which “supports the archiving of SharePoint and files, and dramatically improves search, single instance store, and ease of use”. In both cases, vendors are only able to improve their ranking by differentiating their products and offering something new.

Observation #3: E-Discovery Is A New Way To Differentiate
Each year, Gartner emphasizes that e-discovery functionality is becoming increasingly important to archive purchase decisions. In 2006, it advised vendors to “integrate with tools to manage the discovery process”. In 2007, it added that vendors must “offer more than just search and packaging, but also provide tools for review and case management.” This year, Gartner expanded its comments by also saying:

“Discovery tools are fast becoming a requirement to play in the enterprise part of the market. Robust search, review and export features are not only required but are the focus for most scalability concerns… many archiving vendors are partnering with e-discovery vendors to provide multiple options.”

To reinforce the point, Gartner moves EMC down from “Challenger” (top left) to “Niche Player” (bottom left), and cites the fact that EMC has “no e-discovery beyond search” as a major factor in its decision.

Conclusion
As the archiving market matures, it’s becoming increasingly hard for vendors to improve their rankings. Symantec is the runaway leader and the only way for someone to catch them is to find new ways to differentiate. Mimosa and CommVault are making progress because that’s exactly what they have done, but everyone else is either stuck or, in EMC’s case, slipping backwards. As those vendors consider how to move up and to the right, they should take a close look at e-discovery as an area of growing importance where meaningful differentiation is possible.

Working my way through a backlog of articles and reports this week, I came across Forrester’s Archiving Wave Report for Q1 2008, which was published last month. I spoke to Barry Murphy, the Forrester analyst who wrote the report, around the time of its publication and it’s clear he views e-discovery as a major driver of archiving sales. It was, therefore, no surprise that Forrester’s evaluation criteria heavily weighted e-discovery. To quote the report: “We focused on value-add functionality like records and retention management and eDiscovery support…vendors need to offer real solutions for eDiscovery” (p.4).

The report’s most striking conclusion is that ZANTAZ is rated ahead of Symantec. Every previous report from Gartner and Forrester has identified Symantec as the market leader, so this is potentially a significant shift. Forrester’s primary criticism of Symantec is that it has “product gaps”, more specifically “a reliance on the end-of-lifed Alta Vista search engine (a major issue, given the importance of search to eDiscovery).” (p.9) By contrast, Forrester writes that ZANTAZ offers: “an EAS product with strong search, analytics,…and advanced eDiscovery capabilities – the value-add features customers are demanding.” (p.8)

It’s worth taking a moment to understand Forrester’s criticism of Symantec in more detail. Companies buy archives for 2 reasons: mailbox management and e-discovery. For mailbox management, search is becoming less important since most users will rely on Windows search in Vista or Google desktop. Rather than provide their own search functionality, archives will instead just integrate with Microsoft and/or Google.

But the reverse is true for e-discovery, where search has become increasingly important. The growing volume of litigation, regulatory inquiries, and corporate investigations requires companies to continually comb through their archives in a highly iterative process that often involves IT and legal. Forrester’s analysis shows that – for this purpose – Symantec’s functionality is insufficient.

There’s no doubt Enterprise Vault is a great product. It has long been a market leader, and generates a huge amount of revenue for Symantec. But archiving is an intensely competitive market. So, if Symantec does not improve its search/indexing capabilities for e-discovery, it leaves itself vulnerable to others who will. To me, that’s the most important point that Forrester is making in its report.

For those in regulated industries like financial services, where data retention policies are mandated, every keystroke is tracked and every phone call recorded, the question of how long you should keep data is moot: you keep it for as long as regulations demand.

But for the rest of us in manufacturing, media, technology, government, and elsewhere, it remains an open question. The answer to “what should our email and document retention policy be?” is often a political hot potato, pitting legal and IT’s goal of lower costs against the broader population’s desire to hang on to all their email, just in case they need it later. In fact, the only thing harder than agreeing a retention policy is enforcing it afterwards, as corporate users habitually keep more data than allowed, unless physically prevented from doing so.The reason this matters is that many people believe creating a data retention policy is a key part of implementing an e-discovery solution. I too used to think this way, viewing retention-policy-creation as a necessary rite of passage for legal, IT, and information security people who want to lower e-discovery costs. After all, if the #1 cause of higher e-discovery costs is too much data, then a policy reducing the amount of data looks like a low cost, no-brainer solution.

But life just does not work that way. Outside of the command-and-control environment of regulated industries, retention policies simply do not work. You cannot fight human nature and force people to delete information they want to keep – especially when Gmail, Yahoo Mail, Hotmail and others are training them to do precisely the opposite (i.e., never delete, keep everything) in their personal email accounts.

So, I have changed my mind: to anyone engaged in implementing an e-discovery solution in a non-regulated industry, I say: forget data retention policies, it is a red herring. Too much data is a fact of life that will only get worse. You can no more get people to delete email and documents than you can stop someone writing them in the first place. Instead, focus on the battle you can win by putting in an e-discovery solution that enables you to do two things:

1. Collect data efficiently, so that you have a reliable (defensible) way of getting the data you need. Implementing an email archive from HP, Symantec or others is a great way of approaching this, as is leveraging forensics tools from Guidance or Access Data.

2. Analyze the data up front, so that you can cull it down to only those documents relevant to the case before a human being has to review them. Clearwell’s e-discovery solution is one approach which has worked for a large number of enterprises.

If your experiences, or conclusions, differ from mine, then feel free to post a comment. I am particularly interested to hear about successful examples of data retention policies at non-regulated companies, since I have yet to see one.

Today, Symantec announced 3 connectors for Enterprise Vault, for analytics, review and content collection. According to the announcement, these will “provide tight integration with third-party case management, review, analytics, forensics and desktop collection tools.”

The idea that archives should integrate with third party products is one I whole-heartedly support and have written about before. My company, Clearwell, has been working with Nick, Scott, and the gang at Symantec on this for over a year. They tell us that we were the first to integrate with Enterprise Vault and, to our knowledge, we are the only ones who have deployed fully integrated e-discovery solutions with Enterprise Vault at several enterprises.

Having said all that (and climbing down from my soapbox), I think Symantec’s customers will need to read this announcement very carefully to understand what it means. To give them a helping hand, let me translate it from corporate-marketing-speak into plain English:

Symantec is releasing 3 connectors which enable customers to ingest files from EnCase and export files to Summation and Ringtail. It is also exposing a new application programming interface (API) so that third-party vendors can more easily build their own connectors to Enterprise Vault.

At this point, most people’s eyes glaze over and they ask “who cares”? Surely, only techies get excited about something as esoteric as a new API. But as the recent excitement over FaceBook’s API has shown, opening up a platform – even in a limited way, as Symantec is doing – can unlock tremendous value. For those customers with Discovery Accelerator v.7.5, the new API will have a huge impact for 2 reasons:

1. It makes integration with Enterprise Vault much easier, so lots more vendors will do it. In their press release, Symantec mentions a handful of companies who are building connectors to the new API and I’m sure more will follow. This increases customer choice, and makes it more likely that Symantec customers will be able to select related products that closely fit their needs;
2. It enables enterprises to have a smooth workflow across all aspects of e-discovery, from collection/preservation to analysis/review to production/presentation. For example, companies can now collect information in Enterprise Vault, preserve it by placing a litigation hold on key information via Discovery Accelerator, and then seamlessly hand off that information to a third party application (like Clearwell) for review and analysis. This saves a lot of time that would otherwise be wasted on importing/exporting data from different systems, and reduces the risk that something gets lost in the shuffle.

Net net: companies do well by giving customers what they want, and customers want end-to-end e-discovery solutions. Symantec is not the only one to have figured this out; stay tuned for more announcements like this from other archiving vendors.

At some point in his blog last year, David Hornik, a venture capitalist, lamented the fact that VideoEgg, one of his investments, had 38 competitors in the online video market – and those were only the ones that he knew about.

A casual observer could be forgiven for seeing the same thing happening in e-discovery. Barely a day goes by without some company in a completely different market announcing that they too now have an “e-discovery solution”. Debra Logan at Gartner, who is fast emerging as one of the leading lights of the e-discovery world, tells me she is speaking to 30 vendors for her forthcoming research – and could easily have covered twice that number. Brian Babinau, the insightful and witty analyst at Enterprise Strategy Group, jokes that: “nowadays, people either build a social networking product or do e-discovery.”

For example, last week Zimbra, an open source email platform which has nothing to do with e-discovery, announced its new “e-discovery features”, which sound a lot like keyword search. Kazeon, which wins the prize for creating the world’s most complex e-discovery workflow diagram, has added e-discovery as one of its primary “solutions”, while Endeca takes a more measured approach, proposing only that its financial services customers use it for e-discovery. The list goes on and on.

Despite the worsening signal-to-noise ratio, all the activity will ultimately make it easier for customers to figure out which e-discovery solution makes sense for them. There’s more coverage from leading analysts, who can help explain the different products; large vendors such as EMC, Symantec, and HP are gradually educating the market; and the industry is coalescing around the Electronic Discovery Reference Model, which breaks e-discovery down into its key elements and explains how they fit together.

If e-discovery follows the path of online video and other fast-growing categories, lots of companies will continue to throw their hat into the ring. But for every hundred “VideoEggs”, there will only be one YouTube.