e-discovery 2.0

Many readers write in with questions, which we mostly answer offline. But when there’s one of general interest (and the sender consents), we will post both question and response to the blog and invite others to chime in.

I received one such question from Saurabh on Friday:

Subject: Ediscovery: searching/querying incentives

Hi Aaref,

I am regular reader of your blog. My question is what incentives do the defense litigation consultants/counsels have to employ sophisticated search techniques on e-discoverable documents?

Normally plaintiff and defense counsels agree on the search-terms based on which the defendant will produce documents after checking for privilege documents. In this the defendant is supposed to act in good faith with the searches. But in this the defendant has every incentive to use the most rudimentary search capability and still act in good faith.

But on the contrary we see that in the ediscovery domain the demand for sophisticated search techniques is rising everyday.

Regards,
Saurabh

This is a good question as it addresses two common misperceptions: first, that the process by which opposing counsel agree upon keywords is straight-forward; and second, that defense counsel is often well served by doing rudimentary keyword searches, reviewing for privilege, and handing over the results to the plaintiff. Let’s take each of these in turn.

Keywords are typically negotiated at the “meet-and-confer” conference which, under Rule 16(b) of the FRCP, must occur within the first 99 days of the case. In many cases, This is not a collaborative process as some parties will try to skew the list of keywords in its favor. As a result, defense attorneys spend a considerable amount of time preparing for the keyword negotiation by analyzing their clients’ email and documents to formulate their case strategies. The best way of performing this kind of early case analysis is by employing sophisticated e-discovery software.

Whatever keywords are agreed upon, defense counsel is responsible for much more than just performing privilege review prior to handing over the information to plaintiffs. As one GC at a Fortune 100 company told me, “I want to be responsive, but not overly inclusive”. In other words, he wants to hand over whatever he has to, but not one document more. This process of culling data down to precisely the responsive data set is a complex, iterative process. Given today’s massive data volumes, the only way to do it is to employ e-discovery technology to search, filter, cull, tag, review, and export the relevant information.

So to answer Saurabh’s question, defense counsel has a strong incentive to use e-discovery software to perform early case analyses, and cull data down to the specific set of responsive documents. The more adept they are at doing this, the better they can represent their clients or companies. That’s why, as Saurabh observes, demand for e-discovery software is rising everyday.

Craig Ball’s visibility and prominence in the e-discovery space is akin to Brangelina’s in the world of pop culture. So, chances are that those of you who track the space closely have already read his recent post on XML and its still-unproven potential in e-discovery.

Craig does a fantastic job of summarizing why XML has huge potential to ease the exchange of e-discovery content — as well as why significant challenges still lie ahead before we can all leave our custom load files formats, conversion tools, and scripts behind and head to the beach.

Why are standards efforts in general, and the e-discovery effort in particular, so complex? There are two reasons: one is about people and process, and other is about technology. To borrow a page from Google, and quote the person (Joe Kraus) leading Open Social, a standards effort in the social networking world: “If you want to go fast, go alone; if you want to go far, go together.”

How true. Those involved with the EDRM XML effort are well aware that it took two full years of hard work and many, many iterations just to get to the 1.0 version of the specification, which, as all involved would readily acknowledge, primarily just addresses the biggest pain point that customers and providers face today: load file interchange. Ironically, some in the blogosphere have accused the EDRM effort of not being connected enough with larger international standards bodies. The truth, as usual, is far more complex and nuanced. Not much was happening on the wider standards front, and after repeated attempts to help jump-start those efforts, George Socha, Tom Gelbmann and a host of EDRM participants decided it was time to move the ball forward on their own – and were remarkably successful in that effort, achieving compliance across most of the major players in the industry. Now, with the wind at their backs, the EDRM group is planning the next iteration of the spec and will certainly be reaching out to other interested standards bodies as a part of that effort.

On the technology front, the bottom line is that there’s still a long way to go to figure out how to incorporate actual document content into the EDRM XML schema. There are also a lot of interesting open questions around areas such as interoperability that the XML group plans to take up at the big annual EDRM meeting in St. Paul in May.

In the most general sense, XML’s a done deal: The world is moving toward one in which all content is digitally encoded and marked up with metadata. It’s inevitable. However, in the e-discovery specific sense, our industry is just getting started, and it’s going to take unified, concerted effort across the spectrum of e-discovery providers and end-users to make Craig love XML.

Working my way through a backlog of articles and reports this week, I came across Forrester’s Archiving Wave Report for Q1 2008, which was published last month. I spoke to Barry Murphy, the Forrester analyst who wrote the report, around the time of its publication and it’s clear he views e-discovery as a major driver of archiving sales. It was, therefore, no surprise that Forrester’s evaluation criteria heavily weighted e-discovery. To quote the report: “We focused on value-add functionality like records and retention management and eDiscovery support…vendors need to offer real solutions for eDiscovery” (p.4).

The report’s most striking conclusion is that ZANTAZ is rated ahead of Symantec. Every previous report from Gartner and Forrester has identified Symantec as the market leader, so this is potentially a significant shift. Forrester’s primary criticism of Symantec is that it has “product gaps”, more specifically “a reliance on the end-of-lifed Alta Vista search engine (a major issue, given the importance of search to eDiscovery).” (p.9) By contrast, Forrester writes that ZANTAZ offers: “an EAS product with strong search, analytics,…and advanced eDiscovery capabilities – the value-add features customers are demanding.” (p.8)

It’s worth taking a moment to understand Forrester’s criticism of Symantec in more detail. Companies buy archives for 2 reasons: mailbox management and e-discovery. For mailbox management, search is becoming less important since most users will rely on Windows search in Vista or Google desktop. Rather than provide their own search functionality, archives will instead just integrate with Microsoft and/or Google.

But the reverse is true for e-discovery, where search has become increasingly important. The growing volume of litigation, regulatory inquiries, and corporate investigations requires companies to continually comb through their archives in a highly iterative process that often involves IT and legal. Forrester’s analysis shows that – for this purpose – Symantec’s functionality is insufficient.

There’s no doubt Enterprise Vault is a great product. It has long been a market leader, and generates a huge amount of revenue for Symantec. But archiving is an intensely competitive market. So, if Symantec does not improve its search/indexing capabilities for e-discovery, it leaves itself vulnerable to others who will. To me, that’s the most important point that Forrester is making in its report.

I spend a lot of time researching and developing new search functionality, and working with enterprises and law firms to use this functionality to improve their e-discovery outcomes. To this end, I have followed the excellent research performed as part of the TREC legal track. I also recently attended an informative Sedona Conference webinar on “Search and Information Retrieval”, which contained a section on Information Retrieval (IR) Lessons for E-Discovery presented by Ellen Voorhees of NIST.

As I described some of this research to a colleague of mine, he asked me “So, what’s the so what? What’s the most important step our customers can make to improve the way they search in e-discovery matters based on your work with customers and this research?” My answer was a little surprising even to me. While good cases can be made for looking at concept search and newer, more automated ways of performing content analysis, I believe the most important step that customers can take is simply for them to get their “experts” to start iteratively searching the data in a matter as early as possible in a matter. Let me explain.

When I look at Ellen’s presentation and the findings from the TREC legal track 2006 research overview three findings stand out to me:

1. If you want to get more effective results as measured by “recall” (i.e., how many of the relevant documents did you find?) and “precision” (i.e., how many of the documents you found were relevant versus false positives), then the best way to achieve this is to write a better search query.
2. One of the best ways to get better search queries is to commit human resources to improving them, by putting a “human-in-the-loop” while performing searches.
3. The more expert the human, the better results you are going to get.¹

In other words, what Ellen and the other researches have found that is that you get better results if the same person is running searches, evaluating the results, refining those queries, and trying again. The more expert the person, the better results you are going to get.

Now, you may be thinking that this sounds like common sense and I would completely agree with you. However, while this advice is clearly common sense to you and me, in my experience, it is not always followed in our industry. Instead, it’s all too common that at the beginning of a matter someone comes up with set of keyword queries, someone else runs these queries, some other people perform a detailed review of the results and then finally the “expert” or attorney at the end of this process reviews the most important documents and/or a summary of the documents written by someone else. At this point, some new queries may be developed based on the results of the review and the process starts over again.

What’s the problem with this approach? While in the end this approach can be effective, it can be exceedingly costly and time consuming. Instead, getting your “expert”, whether this is inside counsel, outside counsel, a subject matter expert, a litigation support professional, or a hired investigator, to interact with the data will allow you to find the most important information faster enabling you to make critical legal decisions faster and to dramatically reduce the cost and risk associated with e-discovery.

So why don’t more people follow the common sense advice of getting an expert in front of the data experimenting with queries, interacting with the data and developing better queries? In my view, the single biggest reason is that the technology used to perform searches for e-discovery has simply not been easy enough for legal experts to use. As a result, these experts have got used to developing queries without using technology, and not iteratively interacting with the data over a short period of time.

But that’s changing. In the past few years, several intuitive e-discovery solutions have come to market that enable non-technical lawyers to run their own queries. More and more law firms and enterprises are leveraging these solutions to move to “human-in-the-loop” searching. The results are striking: better early case assessment, much shorter turnaround times, lower costs, and more accurate results.

¹ This is my simplified interpretation of the findings of the TREC legal track. What was found was that an expert manual searcher performed well relative to other non-expert manual run results. Baron, J., Lewis, D., and Oard, D. ”TREC-2006 Legal Track Overview.” The TREC research also contained other findings not covered in this post and I recommend reading the full document so that readers can draw their own conclusions.