e-discovery 2.0

HP announced today that it has signed an agreement with Clearwell to resell the Clearwell E-Discovery Platform. The two companies have been partners for some time and have many joint customers such as Constellation Energy, Del Monte, and Universal Music. But, under this new agreement, thousands of HP sales people will now be compensated for selling Clearwell, giving them a powerful incentive to introduce their customer base to Clearwell’s e-discovery solution.

To my knowledge, this is the first time that a major archiving vendor has agreed to resell a partner’s e-discovery solution, and it raises a couple of interesting questions: why did HP do this deal? And, what does it mean for HP customers?

Ask anyone who tracks the email archiving market, and they will tell you that e-discovery is a major driver of archive purchases. As Gartner’s Carolyn DiCenzo observes: “Legal discovery is being mentioned by almost every client evaluating an e-mail archiving solution.” That’s because whenever a company has litigation, regulatory inquiries or internal investigations, IT is required to provide relevant electronic information to legal or information security. Far better to have it in one repository than spread out on user desktops, email servers, and file shares. So, CIOs are partnering with General Counsels to deploy email archives, much as they did – in years gone by – with the VP of Sales to implement CRM systems.

The problem is that, when you look at archives as e-discovery solutions, they only solve 50% of the pain. In EDRM terms, archives are a very effective solution for collection and preservation, but awful for processing, analysis, and review. They provide a bulletproof way to capture and preserve every message, but do not make it easy to perform early case analyses and cull down data to the very small set of documents relevant to the case at hand.

That’s why enterprise customers find it so compelling to pair up an archive, such as HP’s Integrated Archive Platform, with an e-discovery solution, such as the Clearwell E-Discovery Platform. So to summarize, HP is doing this deal because it’s the best way to provide HP customers with an end-to-end solution for e-discovery. The two products integrate out-of-the-box, have been proven to work together at several large enterprises, and can be purchased from a single supplier (HP). That’s a much easier, lower risk decision for many enterprises than purchasing separate point solutions and cobbling them together.

Very few companies have as much mindshare with corporate CIOs as HP. It can only be good news for the e-discovery market as a whole to have one the largest technology companies in the world out there educating its customers on the value of lowering the costs and risks of e-discovery.

For those in regulated industries like financial services, where data retention policies are mandated, every keystroke is tracked and every phone call recorded, the question of how long you should keep data is moot: you keep it for as long as regulations demand.

But for the rest of us in manufacturing, media, technology, government, and elsewhere, it remains an open question. The answer to “what should our email and document retention policy be?” is often a political hot potato, pitting legal and IT’s goal of lower costs against the broader population’s desire to hang on to all their email, just in case they need it later. In fact, the only thing harder than agreeing a retention policy is enforcing it afterwards, as corporate users habitually keep more data than allowed, unless physically prevented from doing so.The reason this matters is that many people believe creating a data retention policy is a key part of implementing an e-discovery solution. I too used to think this way, viewing retention-policy-creation as a necessary rite of passage for legal, IT, and information security people who want to lower e-discovery costs. After all, if the #1 cause of higher e-discovery costs is too much data, then a policy reducing the amount of data looks like a low cost, no-brainer solution.

But life just does not work that way. Outside of the command-and-control environment of regulated industries, retention policies simply do not work. You cannot fight human nature and force people to delete information they want to keep – especially when Gmail, Yahoo Mail, Hotmail and others are training them to do precisely the opposite (i.e., never delete, keep everything) in their personal email accounts.

So, I have changed my mind: to anyone engaged in implementing an e-discovery solution in a non-regulated industry, I say: forget data retention policies, it is a red herring. Too much data is a fact of life that will only get worse. You can no more get people to delete email and documents than you can stop someone writing them in the first place. Instead, focus on the battle you can win by putting in an e-discovery solution that enables you to do two things:

1. Collect data efficiently, so that you have a reliable (defensible) way of getting the data you need. Implementing an email archive from HP, Symantec or others is a great way of approaching this, as is leveraging forensics tools from Guidance or Access Data.

2. Analyze the data up front, so that you can cull it down to only those documents relevant to the case before a human being has to review them. Clearwell’s e-discovery solution is one approach which has worked for a large number of enterprises.

If your experiences, or conclusions, differ from mine, then feel free to post a comment. I am particularly interested to hear about successful examples of data retention policies at non-regulated companies, since I have yet to see one.

At some point in his blog last year, David Hornik, a venture capitalist, lamented the fact that VideoEgg, one of his investments, had 38 competitors in the online video market – and those were only the ones that he knew about.

A casual observer could be forgiven for seeing the same thing happening in e-discovery. Barely a day goes by without some company in a completely different market announcing that they too now have an “e-discovery solution”. Debra Logan at Gartner, who is fast emerging as one of the leading lights of the e-discovery world, tells me she is speaking to 30 vendors for her forthcoming research – and could easily have covered twice that number. Brian Babinau, the insightful and witty analyst at Enterprise Strategy Group, jokes that: “nowadays, people either build a social networking product or do e-discovery.”

For example, last week Zimbra, an open source email platform which has nothing to do with e-discovery, announced its new “e-discovery features”, which sound a lot like keyword search. Kazeon, which wins the prize for creating the world’s most complex e-discovery workflow diagram, has added e-discovery as one of its primary “solutions”, while Endeca takes a more measured approach, proposing only that its financial services customers use it for e-discovery. The list goes on and on.

Despite the worsening signal-to-noise ratio, all the activity will ultimately make it easier for customers to figure out which e-discovery solution makes sense for them. There’s more coverage from leading analysts, who can help explain the different products; large vendors such as EMC, Symantec, and HP are gradually educating the market; and the industry is coalescing around the Electronic Discovery Reference Model, which breaks e-discovery down into its key elements and explains how they fit together.

If e-discovery follows the path of online video and other fast-growing categories, lots of companies will continue to throw their hat into the ring. But for every hundred “VideoEggs”, there will only be one YouTube.

Most large companies face a dilemma. Should they open up their products and invite others to build features on them, creating a “platform” or ecosystem around themselves? Or would that be inviting the proverbial fox into the hen-house, meaning they should instead prevent others from integrating with their product or leveraging it to create add-on functionality?

In the internet world, there is no doubt about the answer: throw open the doors via easy-to-use APIs (“application programming interfaces”) and let a thousand flowers bloom. That’s what FaceBook did a couple of weeks back with their announcement of the FaceBook Platform, and it has already led to hundreds of new applications for their users. It is what Skype did so effectively, creating a mini-industry around themselves of voicemail, skins, ring-tones, and more. Even eBay, which has jealously guarded its feedback ratings and has habitually crushed smaller companies in its cross-hairs, is embracing the open platform mantra, announcing this week that third-party companies can build features that work with eBay in new ways.

By contrast, telecom companies live in a world of closed standards. Even in the wireless industry, which is arguably the most competitive part of the telecom world, the carriers (Cingular, T-Mobile, Verizon, etc.) exact a heavy toll on any application trying to reach their handsets. As friends in the industry tell me, “There’s a reason why there has never been a billion dollar mobile application company.”

In e-discovery, the large technology vendors like EMC, HP, Symantec, and ZANTAZ face the same choice. Their email archiving products store huge amounts of data. Should they let 3^rd party e-discovery software analyze that data, giving their customers more choice? Or should they slam the door shut, and try to force customers to use their own proprietary e-discovery applications?

The answer, it seems, depends on what they want to be when they grow up. As the market leader, Symantec is confident enough to open its archive (Enterprise Vault) to 3^rd party applications while offering customers its own Discovery Accelerator for litigations holds and some document review. Similarly, perhaps because of its powerful brand, HP focuses on storage optimization with HP RISS and partners with e-discovery software, often with huge savings for its customers. On the other side of the coin, smaller companies like ZANTAZ and Mimosa see themselves as e-discovery companies: they seek to leverage their storage products to get customers to also buy their e-discovery applications.

In the long-run, my feeling is that any archive of any stature will have to adopt open standards. Customers will demand it, and (unlike telecom companies) the archive vendors do not have the market power to resist. Over time, they will also come to appreciate (as HP and Symantec do now) that enabling 3^rd party applications to analyze the data they store is to their advantage, since it creates a powerful, additional incentive to store more information in the archive.

In a previous post, I wrote about the forces transforming e-discovery, a phenomenon that has received increasing attention from the press, most recently in this week’s Economist magazine. While everyone agrees that something big has changed, and (generally speaking) on the reasons why, people struggle to put their finger on exactly what e-discovery has become.

That’s why I think the concept of “E-Discovery 2.0” is so helpful. Analogous to Web 2.0, E-Discovery 2.0 is a set of new processes, technologies, and services that enable companies to manage huge volumes of data, lower costs, and meet tight deadlines.

New Processes

When e-discovery meant handing over a few boxes of paper, companies did not need much of a process. But in today’s world, where it involves terabytes of data, teams of reviewers, and precious little time, it is a very different story. To cope with the growing volume and complexity of e-discovery issues, companies have had no choice but to adopt new processes. These include:

• Collect and Preserve: Most companies have now established procedures so that, when the need arises, they can collect all data relevant to a case and ensure that it cannot be changed or deleted.
• Analyze Up Front: When presented with more work than can be done, a company’s only option is to work smarter, not harder. That means analyzing the collected data up front, to cull it down to only those emails and documents directly relevant to the case at hand.
• Collaborate Efficiently: E-Discovery has become a team sport. And whenever you have a team, you need a playbook, or a process, to ensure work is not repeated and that everyone is marching towards the same goal.

New Technologies

If technology created this problem, by making electronic communication so pervasive and voluminous, then it can also solve it. In recent years, several new technologies have arisen that enable companies to store and sift through their data to fulfill e-discovery obligations. The most significant of these trends include:

• From tape to disk: As the cost of disk storage has continued to decline, more and more companies are abandoning tapes and instead keeping their data online. Email archiving software optimizes for storage efficiency, allowing companies to keep hundreds of terabytes of data readily available for e-discovery.
• From search to analysis: Basic keyword search has evolved into sophisticated analysis technology that mines email meta-data for relevance, links messages together into discussion threads, and groups them by topics. These analysis applications allow users to sift through millions of messages in minutes, to rapidly identify, tag, and export relevant data.
• From closed systems to open standards: Until recently, technology providers made no effort to integrate their applications, leaving customers to fend for themselves. But that has started to change. Symantec Enterprise Vault and HP RISS now have open APIs, creating pressure on others to follow suit. George Socha’s Electronic Discovery Reference Model (EDRM), a standards body, has received widespread support, accelerating progress towards creation of an open e-discovery platform.

To anyone working in litigation support, legal, or information security, all this is quite unremarkable. Of course they use technology to address e-discovery. Obviously, there has to be a process. From the company’s perspective, e-discovery has become no different to HR or finance – it is a core competency, part of doing business.

And that, perhaps, is the most remarkable thing about E-Discovery 2.0 – in only a few short years, it has become so widespread and deeply entrenched within the enterprise, that people barely notice it.