e-discovery 2.0

I am thrilled to announce that Clearwell has signed an agreement to be acquired by Symantec for $410 million ($390 million, net of our cash balance of $20 million). By bringing together Clearwell’s market leading e-discovery platform with Symantec’s market-leading archiving solution, we are uniquely positioned to provide customers with the next generation of information management solutions.

The e-discovery software industry has matured rapidly in the 6 years since Clearwell was founded. As electronic information has become a key part of all litigation, regulatory inquiries, and internal investigations, companies have had no choice but to adopt e-discovery software to keep their costs down. Some have done so by bringing e-discovery in-house; others prefer to work with law firms and litigation support companies who provide cloud-based solutions. Either way, e-discovery software has become widely adopted by corporations, government agencies, and law firms around the world.

Clearwell has been a major beneficiary of these trends. Our annual sales have grown rapidly to over $50 million, and the company has been profitable since 2009. Today, we have over 400 customers and 75 partners in 14 different countries.

Many of these customers are using Clearwell together with Symantec Enterprise Vault in a single integrated workflow, and they have often requested that we couple our products more tightly to better serve their information management needs. That’s what led us to partner with Symantec for the past several years and ultimately led to this transaction. Over time, we see corporations and government agencies increasingly seeking information management solutions that encompass both e-discovery and archiving, making the combination of Clearwell with Enterprise Vault incredibly compelling.

In the near term, we expect very little to change for our existing customers. The product will continue to be sold on a standalone basis and supported by the Clearwell team. We remain committed to serving law firms and litigation support partners, who are absolutely critical to our success in more ways than we can describe.

This is an exciting time for the e-discovery industry. Last week, Gartner published its first ever Magic Quadrant For eDiscovery Software. Today, Symantec and Clearwell join forces to deliver a seamless, integrated archiving and e-discovery management workflow, benefitting all our customers. You can find more information about the acquisition at: http://www.symantec.com/clearwell.  There are exciting times ahead.

As the buzz around Watson and its foray into human-like (actually super-human) performance subsides, it may be time to take stock of what all the fuss was about. After all, we’re all used to computers doing better than humans in many things and even take its superior store of knowledge for granted. And, on the surface, we get answers to questions on pretty much anything from a simple Google or Bing search. So, what really is the big deal and is it even relevant in the context of electronic discovery?

For those not clued in on this, Watson is a brainchild of a four-year effort from 20-25 researchers at IBM, to build a computing engine that would successfully compete at champions-level at the popular quiz show, Jeopardy. Although it blundered on a couple of answers, it competed very well, with a wide margin of victory. Several industry experts that learned of it and watched the show have lauded this as an accomplishment at the same scale or even better than the IBM Deep Blue beating Chess Grand Champion, Gary Kasparov, in 1997. So, let’s examine if this is indeed worthy of the accolades it has gotten.

Behind Watson is an impressive piece of hardware – a series of 90 IBM Power 750 nodes, adding to 16TB of memory and 2,880 Power7 processor cores delivering a staggering 80 teraflops of peak performance.  All the hardware is highly inter-connected with ability to work on problems in parallel, but still marching to a final result in three seconds or less – just fast enough to beat the human buzzer. Some highlights of the computing infrastructure from the hardware architect, Dr. James Fan, at IBM indicate that the three-second timeframe meant the entire corpus of 200 million pages was loaded into memory. Also, with several processors simultaneously working on pieces of the problem, they place very high I/O requirements. The hardware supports a multi-processing OS, with virtualization, in a workload optimized system. The software drives the hardware using thousands of dense threads, with each thread of execution processing a large chunk of work with minimal context switch. Also, given the large number of cores, each thread is optimally allocated to a core. Branded as DeepQA, the software executes a series of complex algorithms in order to solve a very specific problem: winning on Jeopardy.

First, the Jeopardy game provides categories of clues. Some categories help in understanding the clue, while others are simply misleading to a computer. Next, the clue is revealed and one needs to determine what the clue is really asking, since many clues do not ask for a factoid with a direct question, but rather is a composition of multiple sub-clues, each related to another in some linguistic, semantic, syntactic, temporal or other forms of connection. The decomposition of clues and figuring the relationships is a challenge even for humans. Finally, after one understands the clue, you then have to hone in on an answer with some level of confidence, within a three-second window, and must activate the answer buzzer ahead of the rest of the competitors. Besides individual clues, one has to also devise an overall game strategy for selecting the next category, selecting a clue within that category, how much to wager on the Double Jeopardy and the Final Jeopardy. Overall, the game is a complex amalgamation of knowledge, language analysis, gaming strategy and speed of recall of answers.

The software architecture of the DeepQA system is documented in a paper published in AI Magazine. The team built several components to address each area of the problem, with many independent algorithms in each component.  There are lots of complicated technical details, but the final outcome is a human-like response.

A question on that anyone who examines its inner workings has is whether the system is really natural language processing, or statistical language analysis, or machine learning or some sort of ad-hoc program, which doesn’t fit any traditional area of analytics. It does appear to be an combination of several techniques, which may mirror exactly how humans go about solving these clues. We seem to have a large collection of knowledge, initially unconnected but the category, the clue, the hypothesis all appear to generate word and concept associations and a fuzzy evaluation of confidence measures which all converge into a confidence with which a competitor answers a question. It is the replication of these processes by algorithms that makes it truly an astounding achievement.

Given the success of DeepQA’s performance, a natural question is whether it has any practical value for helping us solve day-to-day problems. More specifically, can it cope with the information overload and the challenges of e-discovery posed by that mass of information?  Use within e-discovery context has been explored by several authors, most notably, Robert C. Weber of IBM and Nick Brestoff in recent Law.com articles. Their analysis is based on the ability to explore vast volumes of knowledge. But really, what DeepQA tackled is something more significant – the inherent ambiguity in human spoken and written communication. Our natural instincts are to employ subtle nuances, indirect references, implicit assumptions, and incomplete sentences. We tend to leverage prior and surrounding context in most of our communications. It’s just the natural way of communications, since doing so is actually very effective. We assume establishing context is redundant, unproductive and unnecessary as it tends to make communication repetitive. By not employing a rigid structure in how we write, we are able to bring to bear concise exchanges that span a large volume of information.

If the last two decades is an indicator, the nature of communication is getting less formal, with emails, instant messages, tweets, and blog posts replacing well-crafted formal letters and memos. And, forcing individuals to communicate using rigid, unambiguous text in order for it to be processed by computers easily would mean a huge change in behavior in how people communicate. Any action that contemplates such a change in behavior across billions of people is simply not going to occur. What this means is that the burden for automated analysis using computing algorithms is even greater. This is what makes the discovery of relevant content in the context of e-discovery a very hard problem, one that is worthy of the sort of technological prowess employed by DeepQA team.

Given that our appetite for producing information is ever-increasing, while its discoverability is getting harder, taking the work of DeepQA and adapting it to solve e-discovery needs has the potential to make significant improvements in how we tackle the search, review and analytical aspects of e-discovery.  DeepQA took an easily articulated goal of answering at least 60% of the clues with 85% precision in order to reach champion levels. That was sufficient to win the game. Note that there was never an attempt to get 100% of all clues, with 100% confidence. In the realm of e-discovery, we would be looking at taking a very general production request such as the TREC 2009 Topic 201 “All documents or communications that describe, discuss, refer to, report on, or relate to the Company’s engagement in structured commodity transactions known as prepay transactions.” and use just such a simple articulation of the request to produce relevant documents. It is the core algorithms of machine learning, multiple scoring methods, managing relevance and confidence levels along with traditional information retrieval methods that form the ingredients of the new frontier of automated e-discovery. Beyond e-discovery, application of DeepQA algorithms for business analytics also has significant potential, where fact and evidence-based decision making using unstructured data is likely the norm. DeepQA’s very public Jeopardy challenge has shown that the ingredients needed for enabling such problem solving is well within the realm of possibility.

I was at Sedona midyear meeting last week and during Ken Withers’ excellent discussion of recent e-discovery case law, a few thoughts occurred to me. First, there are so many cases coming out now each week it’s hard to stay above the fray and mine for useful nuggets. The task is a bit Sisyphean, so folks like Ken (who keep a rolling index of cases) are particularly helpful. Next, I was struck by how hot Pension Committee still is, even after almost a year and a half. Certainly, this ongoing spotlight wasn’t an accident, and it’s almost certain that Judge Scheindlin is pleased by the ongoing debate.

I frequently get questions from enterprise clients regarding which cases they should know about, and so I put together an EDRM oriented (left to right) list for folks who just can’t get to all the latest cases. While it’s not an annual roundup per se, I do think it’s a bit more functional for busy electronic discovery professionals who need to stay current. So, here’s the buzz index of cases arranged by topic:

Preservation: The Legal Hold Gold Standard

Case: Pension Committee of the Univ. of Montreal Pension Plan, et al., v. Banc of America Securities, LLC, et al. (S.D.N.Y. 2010).

Summary: The dispute focused on claims by a group of investors who brought an action to recover losses of $550 million dollars stemming from the liquidation of two British Virgin Islands based hedge funds. Unlike many typical e-discovery disputes, this instant action focused on the conduct of the plaintiffs as they attempted to deal with the often murky landscape of electronically stored information (ESI) preservation, collection and production. Judge Scheindlin goes out of her way to crystallize duties and identify the type of conduct that can cause an e-discovery breach. “After a discovery duty is well established, the failure to adhere to contemporary standards can be considered gross negligence. Thus, after the final relevant Zubulake opinion in July, 2004, the following failures support a finding of gross negligence, when the duty to preserve has attached:

• to issue a written litigation hold;
• to identify all of the key players and to ensure that their electronic and paper records are preserved;
• to cease the deletion of email or to preserve the records of former employees that are in a party’s possession, custody, or control;
• and to preserve backup tapes when they are the sole source of relevant information or when they relate to key players, if the relevant information maintained by those players is not obtainable from readily accessible sources.”

Why it’s (still) important: First of all, Pension Committee is written by Judge Scheindlin, who is the most famous electronic discovery jurist on the planet. Next, since she’s in the Southern District of New York, it means that folks even in other jurisdiction that aren’t bound by her opinions still must take heed given the fact that New York is home to so many multinational organizations. Finally, her opinion is the clearest (even if disputed) articulation regarding the standard of care for the issuance of legal holds and the duty to preserve ESI. She attempts to categorically define conduct that is grossly negligent and therefore susceptible to extreme sanctions, including spoliation inferences and terminating sanctions. Fortunately, she recognizes the numerous challenges associated with electronic discovery. And, so as to blend in a healthy dose of reality Judge Scheindlin also said: “In an era where vast amounts of electronic information is available for review, discovery in certain cases has become increasingly complex and expensive. Courts cannot and do not expect that any party can meet a standard of perfection.”

In the end, Pension Committee, was the case of the year in 2010 and even in 2011 it’s generating an unprecedented level of retrospectives (here and here). It may be because Judge Scheindlin’s relatively bright line standard has created so much debate, but in the end the Pension Committee discussion will likely continue for the foreseeable future (perhaps only ending when/if the culpability rules are amended to create a unified national standard).

Preservation: Why Preserve in Place is Risky?

Case: Wilson v. Thorn Energy, LLC, (S.D.N.Y. 2010).

Summary: In Wilson, the defendant corporation identified a flash drive that contained relevant ESI, but rather than copying that data safely to a centralized evidence repository, the defendant’s employee chose to hold on to the drive, putting it instead into a desk drawer. When the files were requested for review and production, the files could not be read from the drive. The defendant’s employee attempted to recover the ESI contained on it, but those efforts failed. Granting plaintiffs’ motion for sanctions, the court ordered that defendants would be precluded from offering evidence at trial concerning the data contained on the discarded drive.

Why it’s important: In today’s e-discovery world, many organizations are instituting hold processes via manual solutions and then waiting weeks or months to ultimately collect the ESI. Wilson shows the danger of simply preserving data and makes the argument that you should either “collect to preserve” or collect very shortly after the litigation hold notice goes out. While focusing on a certain media type (flash drive), this analysis can be extended to any digital system containing ESI that inherently has some set failure rates or can be imagined to fail without express, conscious action (due to loss, theft, recycling, etc.).

Identification & Collection: “Manual” Collections Come Under Fire

Case: Green v. Blitz U.S.A. (E.D. Tex. Mar. 1, 2011)

Summary: In this case, Plaintiff sought to re-open her lawsuit despite prior settlement upon learning that defendant had failed to produce relevant documents. Finding that defendant had committed discovery abuses, including failing to disclose relevant evidence and failing to issue a litigation hold, the court ordered defendant to pay plaintiff $250,000, to provide a copy of the court’s order to plaintiffs “in every lawsuit proceeding against it” for the past two years and to file the court’s order in every case that defendant is involved in for the next 5 years. It was revealed that the employee “solely responsible for searching for and collecting documents relevant to litigation” issued no litigation hold, conducted no electronic word searches for emails, and made no effort to speak with defendant’s IT department regarding how to search for electronic documents.

Why it’s important: Green is the latest in a line of cases [See also Ford Motor Co. v. Edgewood Properties Inc., 257 F.R.D. 418 (D.N.J. 2009) and Phillip M. Adams & Assoc., LLC v. Dell, Inc., 621 F. Supp. 2d 1173 (D. Utah 2009) ] that have been highly critical of manual (or self) collection efforts by the individual custodians. Historically, if the custodians were monitored/supervised enough by counsel, this manual collection process was largely deemed defensible, but it looks like this behavior is simply too risky for any conservative enterprise. The better practice is to leverage the custodians to point out where relevant ESI might exist and utilize software tools to conduct broad collections from key players. While it’s not necessary to use IT tools to collect data immediately for all custodians who have received a litigation hold notice, it’s probably unreasonable to not quickly collect ESI (via formal, IT based methods) from at least some subset of key players. The main point is that this isn’t an all or nothing calculation. Costs, risks and benefits should all be carefully evaluated and documented, in case there’s a downstream challenge.

Analysis & Review: Failure to Test Keywords and Sample

Case: Mt. Hawley Ins. Co. v. Felman Prod., Inc., (S.D. W. Va., 2010).

Summary: In this case the court examined the reasonableness of plaintiff’s precautions to prevent disclosure of email, which was inadvertently produced by the plaintiff amidst “a massive disclosure of e-discovery.” The Mt. Hawley court applied the five-factor test established in Victor Stanley, Inc. v. Creative Pipe, Inc. (D. Md. 2008) and found that the producing party had not taken reasonable steps during discovery. In particular, the court was unwilling to find that the inadvertent production of 377 privileged documents was “solely attributable” to a technological glitch and instead found that plaintiff and counsel “failed to perform critical quality control sampling to determine whether their production was appropriate and neither over inclusive nor under-inclusive.” This finding meant that their attorney client privilege was waived as to the subject documents.

Why it’s important: Mt. Hawley demonstrates why sampling and keyword search term formulation is critically important to any defensible discovery effort. In many instances where “blind” keyword strategies are used, the producing party is taking on an undue risk, in essence flirting with the “3^rd rail” of electronic discovery (inadvertent production). Blind keyword searching (followed by brute force review and production) is sadly still a very common practice today. My hope is that cases like Mt. Hawley will force the blissfully ignorant practicioners to take stock of their risky practices and get with contemporary best practices like ECA, sampling, iterative search and the like.

Conclusion

Simply by creating such a list, I’m sure to leave off cases other folks think are more buzz worthy. But, for me, having a few good legal chestnuts is better than trying to boil the ocean and synthesize all the available case law. If you have any comments I’d be eager to hear (good, bad or indifferent).

In past blogs I’ve discussed a number of cases that have expressed skepticism over the self collection of electronically stored information (ESI) in the electronic discovery process.  In many of these cases, the reviewing judge or magistrate has looked at this process with an increasingly jaundiced eye, in some cases using the self collection component as part of its rationale for sanctions.

My conclusion up until now has been that this collection methodology (where employees manually select and potentially harvest their own data) could be defensible if properly executed; meaning with the requisite level of attorney guidance and oversight.  And, while this is still technically accurate, I think the pendulum has swung far enough to proclaim that this approach is simply far too dangerous for most enterprises, except perhaps those that are extremely risk tolerant.

While there was no particular straw that broke the camel’s back, the trend in the case law now seems to be moving inextricably in one direction – i.e., that self (or manual) collection is no longer safe enough for average enterprises.  Just like tight rope walking without a safety net, self collection protocols aren’t inherently doomed to failure, but there isn’t much (probably any) margin for error.

In the recent case of Green v. Blitz U.S.A., (E.D. Tex. Mar. 1, 2011) we see yet another example of self collections gone awry.  In Green the Plaintiff sought to re-open her lawsuit despite a prior settlement, once she suspected that the defendant had failed to produce relevant ESI.  Finding that defendant had committed numerous discovery abuses, including not disclosing relevant evidence and failing to properly issue a litigation hold, the court put the hammer down, issuing a wide range of sanctions:

• Defendant had to pay plaintiff $250,000
• Defendant had to provide a copy of the court’s order to plaintiffs “in every lawsuit proceeding against it” for the past two years
• Defendant had to file the court’s order in every case that it is involved in for the next 5 years.

Self collection was again a material culprit in the culpable behavior.  It was revealed that the employee “solely responsible for searching for and collecting documents relevant to litigation” issued no litigation hold, conducted no electronic word searches for emails, and made no effort to speak with defendant’s IT department regarding how to search for electronic documents.  To exacerbate matters, the main individual in charge of collections was also closely tied to the research and development of the “flame arresters” that were at issue in this exploding gas can case.

Adding fuel to the fire (pun intended) was the fact that the responding company failed to locate or collect emails with the search term “flame arrester.”  The court went on to note that some of the smoking gun emails not only contained this “most obvious term to search for in electronic documents in this case”, but in fact “flame arrester” was used in the title of certain emails.

While folks have called this practice an example of a “fox guarding the henhouse,” in my mind it’s less that custodial bias renders self collection too risky for prime time.  While there are certainly examples like the Green case where bias likely was an issue, the bigger problem is that any significant reliance on custodians to direct a collection process (even a well supervised one) has too many failure points.  The most obvious (and innocent) scenario is where custodians simply can’t remember if they had responsive ESI or where such information might reside.  This problem can be particularly acute given the fact that litigation is almost always conducted in the rear view mirror.  Since I often can’t remember what I had for breakfast I don’t find it surprising that a custodian might not recall if they had data relating to a discrete issue 4-5 years ago.

As such, the contemporary “best practice” for the collection of ESI is quickly evolving past the old manual collection workflow.  Technology is rapidly making it quick and painless to conduct searches for blatantly relevant ESI (like emails with “flame arrester” in the title).  Not only can you conduct basic searches with existing technologies, but recent advancements around concept search and other analytical tools makes the failure to leverage these technologies seem that much more unreasonable.  For example, in the recent case of Northington v. H&M Int., (N.D. Ill. Jan. 12, 2011) the defendant was sanctioned, in part, because they didn’t search for minor misspellings of the plaintiff’s name.

When a court sees manual blunders like that in the Green case it’s not surprising to see such missteps cast as at least negligent and perhaps even more culpable.  This conclusion is made even easier when organizations like the Sedona Conference (in its Best Practices Commentary on the Use of Search and Information Retrieval Methods in E-Discovery) state:  “[i]n many settings involving electronically stored information, reliance solely on a manual search process for the purpose of finding responsive documents may be infeasible or unwarranted. In such cases, the use of automated search methods should be viewed as reasonable, valuable, and even necessary.”

Green is now the latest in a line of cases [See also, Ford Motor Co. v. Edgewood Properties Inc., (D.N.J. 2009) and Phillip M. Adams & Assoc., LLC v. Dell, Inc., (D. Utah 2009)] that have been highly critical of self collection efforts by individual custodians.  The better practice is to utilize technology to conduct collections from key players and perhaps leverage the custodians (and technology) to point out where relevant ESI might exist.  As such, a belt and suspenders approach is undoubtedly the safer way to go.  In this “dual protection” scenario “key custodians still search, identify, and self-collect what they think are relevant emails, but, as a fail safe, IT also collects all of the key custodians’ emails. Then attorneys search and identify relevant documents from this full, uncensored, unfiltered, collection. This double effort guards against the intentional and unintentional mistakes that can sometimes arise in self-collection.”

We know that electronic discovery is never going to be a perfect process, but self collections simply inject too much risk into an already complicated process.  Now is the time to change tactics and stop tight rope walking without a safety net.  After all, no enterprise wants to be the next to endure a highly publicized fall.

On February 21, 2011, U.S. Immigration and Customs Enforcement (ICE) filed a declaration as part of its ongoing case with the National Day Laborer Organizing Network (NDLON). That declaration, written by Ms. Pavlik-Keenan, made references to Clearwell that have since been selectively quoted and commented on by several of our competitors. Essentially, these competitors have tried to exploit the declaration and use it in a way which was not intended by ICE.

We realized on reading the Pavlik-Keenan Declaration (PK Declaration) that it contained many mis-statements. Until now, we have not responded publicly because this is an ongoing legal matter involving one of our customers, and we do not want to weaken ICE’s stated position. We also felt that, rather than speak out ourselves, it was more appropriate to approach ICE and ask for its help in correcting the public record. That process is ongoing, but there are now new documents from ICE in the public domain which correct some of the mis-statements in the PK Declaration. This blog post is based on information from these new documents.

When reading the PK Declaration, there are 4 important considerations to keep in mind:

#1: NDLON vs. ICE is a precedent-setting case

This is bleeding-edge stuff. The issue here is whether metadata is part of the public record and therefore discoverable under the Freedom of Information Act (FOIA). Judge Scheindlin says it is; ICE disagrees and is opposing her order to produce it. This is a grey area because the standards for FOIA are quite different from those for electronic discovery. Unlike with litigation, FOIA requests are governed by 9 exemptions which are designed to protect private information from being released to the public, and not by the FRCP. Should government agencies now be required to provide metadata, then they must redact that metadata to remove information covered by these 9 exemptions, which (according to the government) is difficult and expensive to do. To our knowledge, it’s also something that has hardly ever been done before, because people generally don’t redact metadata in a load file. So, prior to Clearwell’s use in this case, e-discovery products have typically not been used in this way.

#2: A declaration is an advocacy document, not a ruling from a judge.

This means the PK Declaration is designed to argue a point of view based on personal opinion, not be a statement of fact or legal conclusion. The stakes for the government in this matter are very high, since there are potentially thousands of FOIA requests which could be impacted by Judge Scheindlin’s ruling. So ICE has every incentive to argue forcefully that, whatever technology is used, the resources needed make it excessively expensive to comply with the court’s order. It just so happens that the technology used in this particular case is Clearwell.

#3: The PK Declaration is factually incorrect in several important areas.

There are many statements in the PK Declaration which are – quite simply – not true. To give 2 specific examples from the document:

A. Claim in PK Declaration: 11. … OPLA [Office of the Principal Legal Advisor] estimates that it was forced to expend more than $270,000.00 in upgrades, including the acquisition of a new $32,000.00 server, during this period in order to have access to and run the application.

A. Fact: Neither OPLA nor any other part of ICE paid a dime for upgrades or a new server. In reality, its use of the product for this matter is covered under ICE’s existing license, and we provided an extra server and services for free to help them meet a tight deadline. Clearwell’s “re-usable” license is specifically designed to allow customers to deal with unexpected cases at no incremental software cost, which is what happened here.

B. Claim in PK Declaration: 19. … ICE has been advised by the software vendor that ICE’s software, as it currently exists, cannot produce a “load file” that is compatible with Concordance 8X and/or Opticon 3X.

B. Fact: There are many customers using Clearwell today to produce “load files” for any of several industry leading formats in its export/production options, including Concordance, Summation, EDRM XML and Opticon. Clearwell also offers “Configurable Templates” to produce in any form that is requested.

To address these inaccuracies, on April 11, 2011, the US Attorneys’ Office took the highly unusual step of filing a supplementary declaration to (in their words) “clarify” its earlier statements. In the newly filed Document 86, Declaration of Ryan Law, it states:

6. … “the $270,000.00, which includes $32,000.00 for acquisition of a new server, has not yet been spent. … Clearwell loaned a new server to ICE for the duration of the January 17, 2011 production.”

7. … “ICE was not stating that Clearwell does not allow for the production of such a load file, just that ICE cannot do it with its current software.” All that’s required is a configuration file.

In addition to specific statements, such as those listed above which are now being clarified, the PK Declaration also makes broad generalizations about Clearwell which are untrue. ICE has released new information to address the most widely circulated of these:

C. Claim in PK Declaration: 14. … the agency should abandon the Clearwell application and discontinue its use.

C. Fact: ICE is a happy Clearwell customer who regularly takes reference calls on our behalf from other Federal agencies. As evidence of this, on April 8, 2011 (i.e., 6 weeks after the PK Declaration was filed), the contracting officer at ICE issued a letter (referenced in Document 86) which states that Clearwell “meets the government’s needs and performed in a satisfactory manner. As a result…ICE exercised the option (effective September 23, 2010) to extend the term of the contract through September 22, 2011.” You can read more about this in today’s press release.

#4: The PK Declaration misrepresents how Clearwell is being used at ICE.

ICE purchased Clearwell in 2009 for use by OPLA (Office of Principal Legal Advisor) on civil litigation matters. FOIA requests are handled by a different department with whom we had no contact until, without our knowledge, ICE FOIA decided to borrow Clearwell from ICE OPLA to respond to the FOIA request from NDLON in December 2010. In 16 working days, Clearwell was used to process a large volume of information and produce nearly 15,000 pages of Opt-Out Records (Document 79, Filed 3/30/11, Declaration by Sarahi Uribe). To help ICE meet its deadline, two Clearwell consultants worked onsite during this period – at absolutely no cost to ICE.

Much of this is recounted in the first Declaration of Ryan Law (Document 68, Filed 3/23/11), in the section entitled “Description of Events that Led to the Use of Clearwell”. I draw from that document below:

21: I [Ryan Day] consulted with other program offices within ICE and determined that using the e-discovery platform “Clearwell”, which is owned by the ICE Office of the Principal Counsel (“OPLA”), would offer the best chance for the Agency to meet its court-ordered disclosure requirement.

24: Clearwell was not obtained by ICE for FOIA applications. During the three-year application procurement and development process, OPLA did not take FOIA needs into consideration in determining the relevant capabilities the application would require.

26: At the time of the Court’s December 9, 2010 order, the Clearwell application was untested and was not yet approved for use. At that time, OPLA was in the beginning stages of establishing protocols for the use of the software and had originally anticipated a pilot phase of testing to begin in February or March 2011.

27: ICE OPLA was able to deploy Clearwell on December 20, 2011 for use by the FOIA Office on a provisional basis specifically to meet the January 17, 2011 deadline imposed for the production of records responsive to the Plaintiff’s request for “opt-out” records.

***

There is still a lot that we cannot say publicly about the PK Declaration, out of respect for ICE (our customer) who’s engaged in active litigation. But we would be happy to provide further information to concerned parties under NDA.

It’s widely known that Microsoft is a Clearwell customer, and uses our product for e-discovery across a wide range of matters. One such matter is the case of Datel Holdings v. Microsoft Corporation, which is presently in District Court for the Northern District of California. As part of those proceedings, Microsoft mentioned Clearwell in its Opposition to Datel’s Motion to Compel that was ruled upon on March 11, 2011:

Defendant explains that after potentially responsive documents were collected from custodians, they were loaded into a computerized document processing system known as “Clearwell.” Clearwell extracted metadata from each document and converted the documents into a format that allowed for text searching. Once the documents were processed through Clearwell, they were entered into an online platform, where they were reviewed by attorneys. For reasons still unknown to Defendant, Clearwell truncated some “Re-auth” documents during processing.

In itself, this sounds unremarkable. But we’ve noticed that some of our small competitors have been using this statement, and particularly the last line of it, to suggest that there are problems with the Clearwell product.

We realize that, as the market leader, there will always be small competitors seeking to leverage any opening to their advantage. Usually, we ignore this nonsense. But this time, to set the record straight, we asked our customer at Microsoft to respond on our behalf

Here’s what Joe Banks, who manages the e-discovery team at Microsoft, wrote about the issue and gave us permission to publish:

Statement from Microsoft:

In regard to the Declaration of Hojoon Hwang referenced in the 3/11/11 Order granting in part and denying in part plaintiff’s Motion to Compel in Datel Holdings LTD v. Microsoft, No.C-0905535EDL in the Northern District of California, the statement ‘For reasons still unknown to Defendant, Clearwell truncated some ‘Re-auth’ documents during processing’ should be corrected.  Microsoft subsequently learned that the cause of the truncation was the Microsoft software (AD/RMS Bulk Protection Tool) employed to decrypt previously encrypted content, and the truncation issue had nothing to do with Clearwell’s technology whatsoever.  Shortly after Mr. Hwang’s declaration was filed, he clarified – on the record in open court on February 22 – that Microsoft’s decryption process was the true cause of the data truncation:

6 A lot of Microsoft documents, including e-mails, are

7 encrypted when they are sent. And for production purposes, we

8 have to decrypt it. In that process, some of the material got

9 cut off.

Microsoft does not use Clearwell technology to decrypt its data.  In actuality, Clearwell’s Engineering and Support teams were instrumental in helping to identify the root cause of the truncation issue.  Microsoft continues to use Clearwell’s processing and analysis technology on this matter and greatly appreciates the partnership and support Clearwell provides without fail.

At Clearwell, we’re constantly ruminating on innovative ways to help make our customers’ e-discovery process more efficient. Given the astronomical growth of social gaming, we began asking ourselves, “How can we harness the power and passion of millions of social gamers for the greater good?”

Questions like this really get our engineers cooking, and what they came back with is, to steal a word from one of our most popular product launches a year ago, simply “magical”.

Starting today, Clearwell’s eDiscovery World leverages the red-hot consumer social gaming trend to provide dramatic and previously unattainable increases in e-discovery technology training and productivity. In fact, the promise of eDiscovery World is so great that we have added social gaming as a core part of our product architecture across all Clearwell modules, from legal hold through production.

And we’re not stopping there. We believe that strategic social gaming delivers such powerful benefits to a best practices e-discovery process, that we’ve proposed modifying the EDRM diagram to account for this critical new requirement for truly end-to-end discovery.

Prior to today, unstructured obsession with social gaming has actually been an obstacle keeping end-to-end e-discovery from becoming a reality in many organizations. Interviews conducted across law firms, service providers, and every major enterprise vertical indicate that the time spent protecting crops from withering and urban blight from descending upon virtual cities has left insufficient hours with which to implement next-generation electronic discovery technology. As a result, legal costs have continued to rise and the risk of sanctions has grown substantially. One Director of E-Discovery at a Fortune 100 company, when grilled about his organization’s failure to implement a robust legal hold process, pleaded, “Can you spare some Facebook credits so I can buy a chicken?”

Now, Clearwell has turned this challenge into a tremendous opportunity. In eDiscovery World, we provide an alternative to traditional social gaming that allows users to perform end-to-end e-discovery in a virtual environment – first in training mode to gain e-discovery process knowledge and experience, and then working with live documents and high-stakes cases. All stages of the e-discovery process are functional in the eDiscovery World environment, which is backed by a robust cloud computing platform able to support the largest and most complex cases. Best of all, in addition to the substantial productivity gains our beta customers have already achieved, many have even found their employees clamoring to forego significant portions of their salaries in order to earn precious Facebook credits, thus delivering dramatic cost savings for the organization.

eDiscovery World is truly a win-win, and we couldn’t be more excited about it. Enjoy!

In many ways sanctions cases are the lifeblood of the electronic discovery industry.  While the FRCP, Sedona Conference and EDRM are all out there as shining examples of what to do, it seems like more practitioners learn from the scared straight cases like Zubulake, Morgan Stanley, Pension Committee, etc.  Well, if you liked those horror stories, Philips Elecs. N. Am. Corp. v. BC Tech., may certainly keep you up at night.

In this intellectual property (IP) case with copyright infringement, misappropriation of trade secrets, and associated claims, the court entered a number of discovery orders compelling both production and preservation. The court found that, despite these orders, a proper litigation hold was not issued until 19 months after the duty to preserve arose and thousands of files were deliberately deleted from five key players’ computers. Not surprisingly, the plaintiff moved for a finding of contempt and for terminating sanctions.

In this decision, Magistrate Judge Samuel Alba goes the extra mile in his initial opinion (which was upheld on appeal) to detail the defendant’s significant errors.  While it doesn’t quite rise to the maliciousness demonstrated in Victor Stanley 2 (by the “gang that couldn’t spoliate straight”), it still provides a textbook example of “worst” practices.

To begin, Magistrate Alba cites Pension Committee for the general notion that “[c]ourts cannot and do not expect that any party can meet a standard of perfection. Nonetheless, the courts have a right to expect that litigants and counsel will take the necessary steps to ensure that relevant records are preserved when litigation is reasonably anticipated, and that such records are collected, reviewed, and produced to the opposing party.”  Despite the early recognition that the electronic discovery process need not be perfect, he then uses most of the 48 page opinion to detail the parade of horrors committed by the defendant:

“BCT’s behavior, such as failing to timely issue a litigation hold, failing to follow up on that litigation hold, failing to request discovery documents from key employees, and so forth, reveals its intentional failure to meet discovery obligations and its flagrant disregard of the obvious great risk that it was highly probable the destruction of relevant documents would result from its behavior, and BCT’s conscious indifference to the consequences of that risk.”

Magistrate Alba honed in on the culpability analysis because terminating sanctions were being sought and the case law requires the judge to consider lesser sanctions before dismissing a case (“Because dismissal with prejudice ‘defeats altogether a litigant’s right to access to the courts,’ it should be used as ‘a weapon of last, rather than first, resort.’”).  The Defendant not surprisingly proffered a host of unpersuasive arguments about their behavior, attempting to portray many of the key player’s actions as merely inept and rogue.  Examples include:

• Employee “deleted files and folders for over 5 1/2 hours” and then started “loading five movies onto his computer” which comprised approximately twenty gigabytes of data, permanently overwriting the files recently deleted, eliminating the possibility of forensic recovery.
• Employee “deleted and intentionally wiped files from his computer” the day after Philips demanded his laptop be turned over.  He also “twice ran a Microsoft program called “’Cipher.exe’.”
• Employee deleted 97,000 files from his My Documents file, of that 41,000 were lost files covered up by AbsoluteShield (a file wiping program).

But, Magistrate Alba found the requisite scienter anyway, despite lacking express malice.  He held: “[b]ad faith, or culpability, ‘may not mean evil intent, but may simply signify responsibility and control.’” This control component was critical since the defendant attempted to convince the court that it did not have sufficient power over its employees during the legal hold and preservation process.

“BCT argues that it is not responsible for the ESI’s destruction because the executives and employees who destroyed the documents were acting individually and contrary to BCT’s express directives not to delete documents from their laptops.”

Magistrate Alba was not convinced:

“Also, other commonsense actions were not taken to preserve evidence, such as interviewing key employees, or even asking them to produce discoverable information. BCT appears to have been merely going through the motions rather than genuinely trying to preserve evidence since this method of communication was known to be unreliable and ineffective within the company; thus, BCT was not fulfilling its responsibility to diligently and thoroughly ensure that relevant documents were preserved. …  BCT’s sparse and ineffective communication with its employees does not relieve it of its responsibility of its employees’ actions in disobeying direct court orders and destroying massive numbers of electronic documents…. BCT is the party; it has the responsibility; it must follow the court’s orders. The justice system would break down if company employees could claim that they did not know about the court orders and simply disregard them.”

To further complicate the spoliation allegations, the timing of the deletions were extremely suspect (“almost all of the deletions took place a day or two before the BCT laptop computers in question were sent to be imaged”).  And, the number of the deletions were significant as well, with thousands of files that were forensically unrecoverable.

At the end of the day, the Defendants’ underlying incompetence was compounded by attempts to obscure the facts, ultimately dooming any shred of credibility.

“BCT’s dishonesty and efforts (perhaps even strategy) to hide and destroy ESI shred BCT’s credibility and reveal BCT’s overall contumacious and dishonest attitude toward this case, this court, and the system of justice. This inexcusable behavior and attitude greatly contribute to this court’s finding of bad faith. BCT must be excoriated for filing false sworn declarations, giving testimony riddled with lies and deceit, and making false representations to this court.”

These repeat offenses shut the door on any potential “do-over” with the Magistrate noting that the integrity of the judicial process cannot function when litigants so blatantly disregard their obligations.

“Civil litigation and discovery demand a level of integrity from the parties in order to properly function. When parties disregard that responsibility and/or ignore the court’s mandates, there must be strong consequences. Having examined the degree of prejudice to Philips, the interference with the judicial process, BCT’s culpability, whether BCT was warned in advance that its non-compliance may result in dismissal, and the efficacy of lesser sanctions, the court concludes that extreme sanctions are warranted in this case where discovery abuses of a serious magnitude involving bad faith and willful disregard of two direct court orders occurred.”

And, to pour further salt on the Defendant’s wound, the Magistrate recommended perjury sanctions for lying under oath:  “[I]n light of the above findings, the court recommends that this matter be referred to the United States Attorney’s Office for investigation and criminal prosecution.”

Searching for a singular takeaway amongst all the culpable behavior is a bit hard.  But certainly it is easy to point out that the Defendant’s legal hold process (and technology) was woefully behind the times.  If a litigant is going to utilize a manual, custodian based litigation hold process it needs to recognize the risks.  This isn’t to say that this type of approach won’t work, but an unsupervised process (via outside counsel) means that internal employees who may have something to hide can easily put their employer in the cross hairs, as was seen in Phillips.  Once files are destroyed, it’s not surprising to see an ensuing cover-up, and then terminating sanctions aren’t far behind.  We’ll see how many folks are scared straight by this latest horror story.

Surprisingly, a large percentage of the population has been captivated by what many characterize as a public melt down by Two and a Half Men star, Charlie Sheen. Following his well-publicized split with the show’s executive producer, Chuck Lorre, Sheen’s media interviews have been harder to avoid than cowboy hats at a Kenny Rogers concert. Regardless of whether or not you’re a pop-media junkie, fan of Two and a Half Men, or completely disinterested in the entire saga, it’s clear that many of Mr. Sheen’s ramblings have stirred controversy.

What do all Mr. Sheen’s seemingly random musings mean? Has he lost his mind? Is he pulling the wool over the eyes of the media by flawlessly executing the biggest Hollywood hoax in history? Maybe, just maybe, Mr. Sheen is a stealth e-discovery expert, secretly providing the legal community with a guide for handling litigation. Don’t agree? Well, maybe you’ll be a believer after reading my interpretation of how some of Mr. Sheen’s most popular quotes can serve as an e-discovery 101 guidebook.

“It was so gnarly I can’t remember.”

It’s hard to remember that the first Zubulake decision was penned by Judge Scheindlin long ago in 2003, but the gnarly $29.2 million jury verdict against UBS Warburg by a single plaintiff, in a fairly routine employment lawsuit, is one that most legal departments in Corporate America won’t soon forget.[1] Many industry experts feel the jury’s massive verdict could have been avoided if it wasn’t for repeated electronic discovery errors that resulted in the jury receiving an adverse jury instruction about UBS Warburg’s failure to produce emails. Eight years later, the incredible growth of electronic information continues to present e-discovery challenges for organizations, even though clearer guidelines have evolved.

“Sorry man, didn’t make the rules.”

Prior to Zubulake, the Federal Rules of Civil Procedure (FRCP) did not squarely address the unique challenges of electronic evidence. Although she didn’t actually make the rules, Judge Scheindlin served as a member of the committee that helped draft the 2006 amendments to the FRCP. The amendments address many electronic evidence challenges faced by legal departments, and topics such as data sampling, proportionality, and data accessibility that were tackled in Zubulake, ultimately made their way into the notes or text of the amendments.

The amendments seek to minimize discovery disputes and provide clarity by, among other things, requiring parties to “discuss any issues about preserving discoverable information” and by outlining a protocol for dealing with electronically stored information (ESI) characterized as “not reasonably accessible because of undue burden or cost.”[2] Despite these guidelines, the rules are not always bright line instructions so the conduct of the parties is typically evaluated based on “reasonableness” standards when a discovery dispute arises. Some are lobbying for further clarification regarding issues such as when the duty to preserve electronic evidence is triggered and there seems to be a movement afoot that could lead to additional Rule amendments as evidenced by last year’s Civil Litigation Review Conference at Duke University.[3]

“Your perimeter’s been breached.  You got work to do bro.”

No lawyer wants to be responsible for having the organization’s perimeter breached as a result of data spoliation. However, failing to take proper data preservation steps continues to be the number one reason organization’s face e-discovery sanctions.[4] In Zubulake IV, Judge Scheindlin explained that an organization has work to do when it “reasonably anticipates” litigation since the anticipation of litigation is enough to trigger counsel’s duty to issue a litigation hold notice to employees.[5] The duty is easy to understand, but determining the “triggering” event and the best approach for preserving data can be challenging. To minimize the risk of spoliation, many organizations are moving away from using email notifications and spreadsheets to track when, who, how, and why employees are notified of a litigation hold in favor of more automated solutions and repeatable workflows. Automated solutions allow notices, reminders, and surveys to be created with easy-to-use templates and the “reasonableness” of the entire litigation hold process can be illustrated since reports can be automatically generated with the click of a button.

“I’ve got tiger blood and Adonis DNA”

Although the line between “reasonable” and “unreasonable” conduct can be very blurry in some cases, in other situations the offending party simply chooses to flagrantly disregard the rules as if they have tiger blood and Adonis DNA. For example, in Daylight, LLC v. Mobilight Inc., the Utah Appellate court upheld the lower court’s entry of a default judgment after defendants threw a laptop off a building, ran it over with a vehicle and stated: “if this gets us into trouble, I hope we’re prison buddies.”[6] Uh, sorry Charlie….

Typically, most parties are not so cavalier about disregarding their legal obligations and the judge’s decision to issue sanctions when evidence is lost or deleted is not a slam dunk. One challenge is that the 2006 FRCP Amendments allow litigants to request any “Electronically Stored Information” stored in “any medium” that is reasonably calculated to lead to the discovery of admissible evidence.[7] That means the scope of the duty to preserve, collect, and produce information as part of litigation may be very broad and very complicated, even though data growth continues to increase exponentially and corporate information technology systems continue to become increasingly complex.

To meet these burdens, many organizations are demanding technology solutions that do more than manage the legal hold process because they also need to collect, analyze, and review ESI to evaluate the case. The holy grail of e-discovery is being able to leverage a single technology solution to manage all these tasks as well as the litigation hold process. The value is twofold. First, automating e-discovery steps related to preservation and collection that have traditionally been managed manually minimizes the risk of human error and makes it easier to demonstrate a repeatable process that is defensible. Second, using the same technology solution to filter, analyze, and review key documents faster results in significant cost savings and strategic advantages.

“You make a choice to win, and you win”

Despite the fact that organizations continue to make e-discovery mistakes, smart organizations choose to leverage a combination of repeatable workflows and legal technology solutions to help them win. Although the new technological era we live in has created new discovery challenges, legal technology can be used to streamline data preservation, collection, processing, and review. Legal technology can also be used to quickly find important documentary evidence earlier in the case, thereby resulting in strategic advantages so smart organizations can “just keep winning.”

[2] See Fed. R. Civ. P. 26(f)(2) and Fed. R. Civ. P. 26(b)(2)

[3] John G. Koeltl, 2010 Civil Litigation Review Conference Introduction: Progress in the Spirit of Rule 1, 60 Duke L.J. 537 (2010).

[4] See Dan H. Willoughby, Jr., Rose Hunter Jones, and Gregory R. Antine, SANCTIONS FOR E-DISCOVERY VIOLATIONS: BY THE NUMBERS, 60 Duke L.J. 789 (2010), at 803 stating (“FAILURE TO PRESERVE ESI IS THE MOST PREVALENT SANCTIONABLE CONDUCT”

[5] Zubulake v. UBS Warburg LLC, 220 F.R.D. 212, 218 (S.D.N.Y. 2003)

[6] Daylight, LLC v. Mobilight Inc., 2011 UT App. 28 (2011)

[7] Fed. R. Civ. P. 34(a)(1)(A).

It seems like just yesterday that I wrote a post discussing Judge Scheindlin’s latest electronic discovery opinion in Nat. Day Laborer Org. Network v. United States Immigration and Customs Enforcement Agency (“NDLON”).  One of the issues that jumped out at me when I read the original opinion was Judge Scheindlin’s application of FRCP principles, rejecting the government’s claim  that “if the requirements of FOIA and the requirements of the Rules conflict, FOIA must trump the Rules.”  Needless to say, she didn’t buy the “trump” argument:

“[T]here is no need to decide this question because FOIA does not conflict with the Rules. FOIA is silent with respect to form of production, requiring only that the record be provided in ‘any form or format requested by the person if the record is readily reproducible by the agency in that form or format.’… Defendants’ productions to date have failed to comply with Rule 34or with FOIA.”

Now that the government has appealed Judge Scheindlin’s ruling I thought I’d consult with a legal guru (in the form of Ralph Losey) to see how he thinks the chips will fall on both the request for a stay and the appeal.  Here’s what started as an impromptu email exchange, which is now paraphrased for your reading enjoyment:

Dean:  ”Ralph, how likely is the government to get stay in the first place?  After all, if they have to produce the metadata (as ordered) winning a downstream appeal won’t help much in the NDLON case?”

Ralph:  ”An interlocutory appeal to the Circuit Court on a discovery issue is always a long shot. Appeals courts usually do not accept appeals of non-final discovery orders. The appellant has to show irreparable harm. Still, this is an unusual case as production of documents goes to the merits of the case itself. They may well take it. I really don’t know. I might depend on someone’s breakfast.”

Dean:  “So, maybe the government loses this initial battle, but my guess would be that they’re still keenly interested in winning the war, since Judge Scheindlin’s metadata production standards will have far reaching affects across any entity responding to FOIA requests.  As an aside, a given agency may handle a multitude of FOIA requests ranging from dozens on the low end to thousands on the high end – so a change in any production protocol is guaranteed to have a material and long lasting impact.”

Ralph:  “Could be, but then again, maybe some agencies are already producing metadata. If they give actual native copies, then they certainly are. I don’t have statistics on that. Certainly, they should have seen this coming. The states that have looked at this all require reasonable metadata production. I probably don’t appreciate the governments real problems here, since in my world, metadata is produced between private parties all of the time without a fuss. Do the feds really have state secrets stashed away in metadata? I seriously doubt it. In my experience, searching for secrets in metadata is usually a big waste of time. It is a fear based on myth, not fact.”

Dean:  “The Supremes issued a ruling recently in Milner v. Dept. of Navy which effectively reigned in an expansionist construction of FOIA Exemption 2.  Aside from containing my new, favorite quote [“Our construction of the statutory language simply makes clear that Low 2 is all of 2 (and that High 2 is not 2 at all).”] do you think this ruling will have any impact on the NDLON outcome?”

Ralph:  “Honestly I don’t know. I consider myself somewhat educated about metadata and production of various metadata fields in load files, but not meta-foia, and I mean that literally, not “metafoicaly” <ouch!>”

Dean:  “Finally, do you think that the Plaintiff’s argument in NDLON that these very agencies ask producing parties for metadata (sometimes in more onerous fashion) will hamper their cause?  Or, is FOIA a different enough creature to throw out the ’good for the goose’ argument?”

Ralph:  “I am reminded of one of my favorite old sayings, ’hoisted by his own petard.’ Certainly many, if not most federal agencies require metadata production to them in e-discovery. Why should the federal government be any different?”