e-discovery 2.0

The other day, I came across the fascinating statistic that over 50% of products returned every year to stores across America have absolutely nothing wrong with them. Apparently, consumers used them for an average of 20 minutes and then gave up, because they were too complicated.

At this point, most customers of traditional enterprise software could be forgiven for thinking: “I wish I could do that.” Enterprise applications are notoriously feature-laden, complicated to use, and difficult to install. They make their users feel stupid, by presenting them with complex pictures that look like amoeba or toolbars with 150 different options. Why does enterprise software seek to punish its customers in this way?

Partly, because customers ask for it. Whether they are buying a dishwasher or an accounting application, people habitually over-estimate their ability to figure out how a complicated product works and, as a result, pay more for features that they never use. Partly, it’s because enterprise software is designed by engineers who think everyone is as technically proficient as they are, and by marketing people who view every additional feature as a new selling point.

By contrast, Web 2.0 applications such as FaceBook, Flickr, StumbleUpon, or Meebo are incredibly easy use. Even an idiot who has never seen these applications before can use them without an instruction manual or a training course. You could say that’s because they are trivially simple applications. But I think it’s primarily because, if they were not so easy to use, people would simply click away and try something else – i.e., they would die.

That to me is the real lesson that Web 2.0 apps can teach enterprise software: make something that is easy to use, easy for someone to install, and easy for them to evaluate. Get people addicted to your application because it’s so good (the average FaceBook user spends 4+ hours a day on the site). No doubt, this is harder to do with enterprise applications because they are inherently more complex. But figure out a way to hide the complexity, packaging all the functionality users need into a design that’s easy to use. This is a key characteristic of e-discovery software applications; it’s the genius of salesforce.com’s CRM application and Apple’s iPod; and, it needs to be a core skill of any company creating enterprise applications today.

In a previous post, I wrote about the forces transforming e-discovery, a phenomenon that has received increasing attention from the press, most recently in this week’s Economist magazine. While everyone agrees that something big has changed, and (generally speaking) on the reasons why, people struggle to put their finger on exactly what e-discovery has become.

That’s why I think the concept of “E-Discovery 2.0” is so helpful. Analogous to Web 2.0, E-Discovery 2.0 is a set of new processes, technologies, and services that enable companies to manage huge volumes of data, lower costs, and meet tight deadlines.

New Processes

When e-discovery meant handing over a few boxes of paper, companies did not need much of a process. But in today’s world, where it involves terabytes of data, teams of reviewers, and precious little time, it is a very different story. To cope with the growing volume and complexity of e-discovery issues, companies have had no choice but to adopt new processes. These include:

• Collect and Preserve: Most companies have now established procedures so that, when the need arises, they can collect all data relevant to a case and ensure that it cannot be changed or deleted.
• Analyze Up Front: When presented with more work than can be done, a company’s only option is to work smarter, not harder. That means analyzing the collected data up front, to cull it down to only those emails and documents directly relevant to the case at hand.
• Collaborate Efficiently: E-Discovery has become a team sport. And whenever you have a team, you need a playbook, or a process, to ensure work is not repeated and that everyone is marching towards the same goal.

New Technologies

If technology created this problem, by making electronic communication so pervasive and voluminous, then it can also solve it. In recent years, several new technologies have arisen that enable companies to store and sift through their data to fulfill e-discovery obligations. The most significant of these trends include:

• From tape to disk: As the cost of disk storage has continued to decline, more and more companies are abandoning tapes and instead keeping their data online. Email archiving software optimizes for storage efficiency, allowing companies to keep hundreds of terabytes of data readily available for e-discovery.
• From search to analysis: Basic keyword search has evolved into sophisticated analysis technology that mines email meta-data for relevance, links messages together into discussion threads, and groups them by topics. These analysis applications allow users to sift through millions of messages in minutes, to rapidly identify, tag, and export relevant data.
• From closed systems to open standards: Until recently, technology providers made no effort to integrate their applications, leaving customers to fend for themselves. But that has started to change. Symantec Enterprise Vault and HP RISS now have open APIs, creating pressure on others to follow suit. George Socha’s Electronic Discovery Reference Model (EDRM), a standards body, has received widespread support, accelerating progress towards creation of an open e-discovery platform.

To anyone working in litigation support, legal, or information security, all this is quite unremarkable. Of course they use technology to address e-discovery. Obviously, there has to be a process. From the company’s perspective, e-discovery has become no different to HR or finance – it is a core competency, part of doing business.

And that, perhaps, is the most remarkable thing about E-Discovery 2.0 – in only a few short years, it has become so widespread and deeply entrenched within the enterprise, that people barely notice it.

If there’s one idea that has captivated Silicon Valley in the past 3 years, it is Web 2.0. People may debate its meaning and definition, but the gist of it is clear: a handful of powerful forces have coalesced to make the internet of today fundamentally different to what it was 5 years ago. Opinions vary on which of these forces is most important: the growth of broadband to the home; open source, ajax and other technologies which lower the cost and increase the functionality of web applications; the power of community in a world where more people are on the web. Whichever you choose, there is no doubt that collectively these forces have had a huge impact, powering the growth of now-household names such as Google, MySpace, and YouTube.

I believe that an analogous set of changes is transforming the way companies do e-discovery. Ten years ago, e-discovery was an after-thought – a necessary, but incidental, part of corporate legal expenses. Today, it is a huge line-item in the legal budget, a headache for corporate IT, and the foundation upon which many cases are built.

E-discovery 1.0 was an ad hoc activity; e-discovery 2.0 is a core business process. E-discovery 1.0 was barely noticed; e-discovery 2.0 is driving the news cycle, affecting everyone from Intel to the US Attorney General. In the legal world, e-discovery 2.0 has had every bit as big an impact on enterprises as Web 2.0 has had on the dating lives of teenagers.

What happened? A series of fundamental changes have made e-discovery far more important, expensive, and complex than it was in the 1990s. Chief among these changes are:

1. Email, Not Voicemail: In the past 10 years, companies have switched from voicemail to email as the primary way they communicate. This has created a written record where none previously existed. Just as oral histories eventually die out, every voicemail eventually gets deleted; but emails and the written word live forever. Whatsmore, the convenience and time-efficiency of email makes it addictive, with the result that every meaningful conversation is captured, time-stamped, and attached to a person’s name. Given that many legal cases turn on intent, and proving who knew what when, this makes email a virtual treasure trove for anyone building a case.

2. Electronic Files, Not Paper: Electronic files are fundamentally different to paper documents: they reproduce like rabbits and are far cheaper to store. For example, one laptop is the equivalent of 2,000 boxes of paper; one server corresponds to 8,000-40,000 boxes of paper. The number of servers and laptops holding vast quantities of email is only increasing as the cost of hard disk storage falls, down from $2.04 per GB in 2004 to $0.77 per GB in 2006. Net net: going electronic has vastly increased the amount of data that must be analyzed as part of the discovery process.

3. Sooner, Not Later: Recent changes to the FRCP guidelines have moved e-discovery up in the process, forcing companies to have an e-discovery plan within 99 days of a suit being filed. Since disputes rarely settle that quickly, that means enterprises must now incur the expense of e-discovery on every case, not just the small number that actually make it to court. The result is a massive increase in e-discovery expenses and workload.

Anecdotal evidence of e-discovery 2.0 is everywhere. A few years back, no one would have guessed that every major analyst firm would have people dedicated to tracking e-discovery. Nor would you have expected to find a litigation support manager at every major enterprise.

So what exactly is e-discovery 2.0? Well, I will talk about that in a future post.