Opinion
[Copyrighted Material Omitted]
Andrew M. Klein, Klein Law Group, PLLC, Washington, DC, Michael D. McNeely, Law Offices of Michael D. McNeely, Washington, DC, for Plaintiff.
Carol Elder Bruce, Lashon K. Kell, Bracewell & Giuliani LLP, Washington, DC, for Defendant.
MEMORANDUM OPINION
JOHN M. FACCIOLA, United States Magistrate Judge.
This case has been referred to me for the resolution of discovery disputes. The issues now before the Court are (1) whether plaintiff Covad Communications Company (" Covad" ) should be permitted to conduct a forensic search of defendant Revonet, Inc.'s (" Revonet" ) computers and servers; (2) how those searches should be conducted; and (3) who should pay for them.
I. Background.
The history of this case appears in previous opinions. See generally Covad Communications Co. v. Revonet, 254 F.R.D. 147 (D.D.C.2008); Covad Communications Co. v. Revonet, 250 F.R.D. 14 (D.D.C.2008). Briefly, Covad hired Revonet to run a marketing campaign for Covad's voice over Internet protocol (" VoIP" ) business. As part of the campaign, Revonet made outbound phone calls to find persons who might be interested in Covad's services and also fielded calls from persons who were seeking information about Covad. These practices were known as generating outbound and inbound leads, respectively. Covad now alleges that Revonet expropriated information that belonged to Covad in violation of the contract between the parties.
As further discussed in the findings of fact, Revonet used a database called the Federated Database to house all of its sales lead information regardless of the source. Covad has alleged that Revonet took sales lead data-stored in the Federated Database-that belonged to Covad and gave it to other customers. Understandably, there is no dispute that the content of the Federated Database and the evidence within that database itself and elsewhere are the very heart of this lawsuit. Thus, I have expressed my view, and the parties have agreed, that the database itself, and not just the data that it holds, should be considered an exhibit in this case and the parties must find a way to examine it so that they can make the appropriate arguments and try this case. Cf. Smith v. CaféAsia, 246 F.R.D. 19, 21 (D.D.C.2007).
The parties first brought the issue of the database to the Court's attention in December 2008, when their attempts to reach an agreement as to how to search it failed. At that time, I ordered the parties to file proposed protocols for searching the Federated Database. See Minute Order (12/3/09). The parties filed their protocols on December 12, 2009, but unfortunately each proposal was based on a number of untested factual assumptions about Revonet's computer systems. Accordingly, I set the matter for an evidentiary hearing for the purpose of resolving those factual disputes. See Memorandum Order (1/13/09) [# 67]. The hearing took place on three days: March 4, 2009, March 24, 2009, and April 15, 2009.
II. The Hearing.
At the hearing, Revonet presented Fred Purdue, who was qualified as an expert in Information Technology and SQL databases. Purdue is a consultant who, though he did not inspect the servers themselves, spoke to a number of Information Technology (" IT" ) personnel at Revonet. Purdue testified about a number of server failures at Revonet in 2007 after this litigation was filed. He also opined that it would be risky to make a forensic copy of the servers because they are very old. Purdue then gave a cost estimate of $47,000 for copying and analyzing Revonet's drives, a figure that did not include legal expenses associated with the actual forensic examination. He did not discuss any individual computers, whether they exist, whether they should be imaged, or how much that might cost. Next, Brice Anderson, a former salesperson who is now Director of Channel Sales Strategies at Revonet, testified. Anderson testified that Revonet tracked phone numbers that people used to call into Revonet's call centers to determine where callers found out about Covad. When calls came in to the call center, Revonet would input information about the caller into a program called RPM. There was no specific field to indicate that a call was an inbound lead, but Anderson testified that one could tell that from looking at the entry in RPM, though he did not explain how.
At some point around the time that Covad and Revonet stopped doing business, Revonet began the Gold campaign, in which it selected certain leads out of the Federated Database and copied them over into the Gold campaign. Anderson conducted a search and concluded that about 2,000 inbound Covad leads were assigned to the Gold campaign, and of that, about 25 of them were sold to other companies. Anderson also testified that in the summer of 2006, Revonet caused all leads from Covad that were designated as inbound leads to be hidden, so they do not appear to most Revonet employees in RPM.
Finally, Covad called Scott Ellis, a forensic computer analyst, to testify as an expert witness. Ellis contradicted Purdue's contention that it would be extremely time consuming and expensive to make forensic copies of the drives. Ellis also testified that the fact that Revonet used a web-based interface for their salespeople to access the Federated Database was important because computers that access the Internet make temporary Internet cache files, so he would want to look at those PCs. He would also want to look at the PCs of IT Specialists who were moving data around. Ellis ultimately gave a rough estimate of $25,000 for the forensic investigation.
During Ellis' testimony, Covad questioned him about some e-mails that indicate that leads may have been distributed by e-mail in addition to through the Gold Campaign. Covad's counsel indicated that only four external e-mails were produced by Revonet.
III. Findings of Fact.
From that testimony, I make the following findings of fact:
1. Revonet has one database, the Federated Database, that contains all of its campaigns. Transcript of Proceedings (3/24/09) (" 3/24/09 Tr." ) 64:19-24.
2. Within the Federated Database, entries are assigned to different campaigns and identified as such. 3/24/09 Tr. 64:22-24. Entries are not specifically designated as " inbound" or " outbound" in the database, however. 3/24/09 Tr. 83:7-11.
3. The Covad campaign dealt with both " inbound" and " outbound" leads. 3/24/09 Tr. 52:24-53:3. Revonet purchased lists of contacts from outside sources and called those contacts to promote Covad's services. Those sources were designated as " outbound." 3/24/09 Tr. 53:5-54:9. Revonet also fielded inquiries about Covad from contacts who called in to Revonet. Those sources were designated as " inbound." 3/24/09 Tr. 55:1-16.
4. In 2006, Revonet created a new campaign called the " Gold Campaign." 3/24/09 Tr. 58:17-20. To populate the Gold Campaign, Revonet selected leads from other campaigns that it considered to be good leads. 3/24/09 Tr. 59:18-60:21. Criteria for inclusion into the Gold Campaign included past success for another client, such as Covad. 3/24/09 Tr. 60:9-13.
5. Approximately 14,000 leads were copied into the Gold Campaign. 3/24/09 Tr. 76:5-6. To create the campaign, Revonet IT personnel made complete copies of leads that already existed in other campaigns and placed the copies into the Gold Campaign. 3/24/09 Tr. 62:9-10. The only difference between the copies was that the field indicating which campaign the lead belonged to was changed to Gold. 3/24/09 Tr. 83:15-25. The entries were not moved. Instead, they were duplicated, so the original leads would still appear in the appropriate campaigns. Id.
6. For inbound leads, Revonet used phone numbers to determine what advertisement or mailer, or other source the caller was responding to. 3/24/09 Tr. 55:17-22.
7. The Gold Campaign leads were marketed in a program called Leads on Demand. 3/24/09 Tr. 63:23-64:9. Through Leads on Demand, Revonet offered five free leads taken from the items in the Gold Campaign to prospective customers. 3/24/09 Tr. 66:11-16. It also sold leads to customers in bundles. 3/24/09 Tr. 67:4-12.
8. Revonet set up a web portal for its customers to access the five free leads and the leads that were purchased in bundles. 3/24/09 Tr. 74:20-75:6. The web portal was called OneView, and it was a SharePoint database. Id. The SharePoint database was housed on the same server as the Federated Database.
9. Sally Lefferts was responsible for putting leads on the OneView portal. 3/24/09 Tr. 75:7-10. Before she made them available to new customers, she removed prior customer data.
10. When Covad was a client of Revonet's, the Federated Database was housed on a server called OVNC SQL 02, which was located in New Canaan, CT. Transcript of Proceedings 3/4/09 AM (" 3/4/09 AM Tr." ) 33:21-34:1. In July 2007, Revonet moved the SQL 02 server to its office in Sioux Falls, SD. 3/4/09 AM Tr. 34:8-22. During the move, the server crashed, and the data on the server was lost. Id. Revonet was able to restore the server using a back-up tape. 3/4/09 AM Tr. 34:23-35:1.
11. When the SQL 02 server data was restored, it was restored onto a server called OVNC SQL 01. The Federated Database still resides on SQL 01. 3/4/09 AM Tr. 34:23-35:1.
12. In July 2007, the e-mail exchange server that Revonet used in Sioux Falls crashed. 3/4/09 AM Tr. 33:3-6. Revonet sought to restore the server from a back-up tape, but determined that the server had not been backed up appropriately. Id. Accordingly, Revonet searched users' PCs and retrieved e-mail files and loaded those onto an exchange server, Revmail two, that was located in New Canaan. 3/4/09 AM Tr. 33:7-11.
13. In the fall of 2007, Revmail two began experiencing problems. 3/4/09 AM Tr. 35:3-7. Revonet IT professionals then purchased some new drives for the retired SQL 02 server, and repurposed some drives from another retired server to build a new server called Revmail. 3/4/09 AM Tr. 35:8-11. The data from Revmail two was then moved to Revmail. Id.
14. Sales associates at Revonet use a program called RPM to access the Federated Database. RPM is a web-based user interface that is capable of running reports, but generally does not contain any data. 3/4/09 AM Tr. 21:6-15. RPM is located on a server called OVNC WEB 01. 3/4/09 AM Tr. 26:8-9.
15. The OneView web portal is currently not operational. 3/4/09 AM Tr. 23:2-16.
IV. Analysis.
A. Forensic Imaging.
As an initial matter, Covad seeks to make forensic images of Revonet's drives and computers to preserve information as it currently exists. The expense associated with taking a forensic image is small compared with the cost of the forensic search itself and provides both parties and the Court with the best possible present depository of the crucial information in the database. The process is obviously not as time consuming as the forensic search itself and lessens some of the cost of forensic searching since the search can be done off-site rather than on the premises housing the servers or computers. Any interference with business operations will be insignificant because I will require the searching to be done over a weekend. While I appreciate that Revonet's servers are nearing or past their life span, I credit Ellis's testimony that age alone does not render the items so fragile that the process of taking the images will destroy the servers. Creating a forensic image is no more burdensome than using the server for everyday business activities and may ultimately benefit Revonet by creating a forensic record of its data before it uploads the data onto new servers, whether it rents the space (the so-called " cloud configuration" ) or buys new ones. For those reasons, Revonet shall permit Scott Ellis to make forensic images of the following items:
There was testimony from Perdue that Revonet is considering transferring its data to servers that are owned by companies that provide space on their servers for a fee. Access to the data is then procured by use of the Internet; hence, the appellation that the data is " in the clouds."
Characteristics
Server
in 2006
Intervening events
Where it is now
Revmail one
Sioux Falls e-mail exchange server
Crashed in 2007, no back-up was available
unknown, but it has not been repurposed-best guess,Sioux Falls in storage
Revmail two
New Canaan e-mail exchange server
Revmail one e-mails that could be recovered from userPCs were loaded onto Revmail two.
unknown, but it has not been repurposed-best guess,Sioux Falls in storage
Revmail two was moved to Sioux Falls
Revmail two began experiencing problems and wasretired.
Revmail
Did not exist
See OVNC SQL 02 (below)
Sioux Falls, Revmail is the e-mail exchange server forRevonet.
OVNC SQL 02
Home of the Federated Database and SharePointdatabases, located in New Canaan
During transport to Sioux Falls, an external drive onthe server crashed or sustained damage and caused thewhole server to crash.
Now called Revmail, SQL 02 is currently in Sioux Fallsand is the e-mail exchange server for Revonet. It'snot clear what happened to the drives that were in theoriginal SQL 02.
Testimony suggests that the drives from SQL 02 wereremoved, and replaced with two drives from a retiredISA server and a few newly purchased drives. Theinformation from Revmail two was then loaded onto SQL02.
To the extent the original drives still exist, theyshould be copied.
OVNC SQL 01
Don't know
When SQL 02 crashed, Revonet used a back-up to restoreit onto SQL 01.
In Sioux Falls, currently houses the FederatedDatabase.
OVNC WEB 01
Home of RPM
N/A
Home of RPM
If there is any dispute about whether a given computer was used during the relevant time period, Ellis shall make a forensic copy of the computer in any event and preserve it pending further order of this Court. The resulting forensic images will be safeguarded by Ellis who will preserve them inviolate. He shall only use these forensic copies of the devices or electronic depositories at issue to conduct forensic searches.
Personal Computers that interfaced with any of theabove mentioned drives between the date when Covad andRevonet entered into a contract and December 31, 2006.
N/A
N/A
N/A
B. Forensic Searching.
1. Federated Database, SharePoint Database, and RPM (OVNC SQL 01, OVNC SQL 02, and OVNC WEB 01).
Revonet has explained that its databases were housed on a back-end database server. In the Federated Database subsets of data are segregated by " campaign." Thus there is a Covad campaign, an XO campaign, a Verizon campaign, a Gold campaign, etc. within the Federated Database. Revonet claims that Covad inbound leads would only appear in the Covad campaigns and the Gold campaign. Accordingly, Revonet argued in its proposed protocol that it should only have to produce those two campaigns because nothing else would be relevant.
Covad maintains that it needs access to full forensic copies of the drives because it needs access to historical data about the database, not just the database itself. Much of that historical data will not be visible by looking at the database alone.
Rule 34 permits a party to " copy ... electronically stored information" but is limited by Rule 26, which allows only for discovery of " any non-privileged matter that is relevant." Fed.R.Civ.P. 26(b)(1). Revonet does not argue that the information at issue is privileged, because it falls within a recognized common law privilege but only that it is confidential because it may expose information that Revonet is obliged by contract to keep confidential. Confidentiality is not a basis for withholding information in the ordinary course if it can be protected by a protective order (which I will gladly sign) restricting access to confidential information to certain persons and prohibiting its use for any purpose other than the limited purpose of the forensic copying and searching that I am ordering. Furthermore, the testimony of Purdue and Ellis suggests that there will likely be pieces of information located on the server but outside of the Federated Database that would tend to show what Revonet did with the data because of the way that SQL databases usually work and how the electronically stored information was used by Revonet. As I have explained, that information is certainly relevant. Indeed, it is the very heart of this lawsuit.
Note that " relevance" is, in turn, defined to include information " reasonably calculated to lead to the discovery of admissible evidence." Fed.R.Civ.P. 26(b)(1).
The advisory committee note to Rule 34 cautions against making forensic examination the default, however, and encourages courts to " guard against undue intrusiveness." Fed.R.Civ.P. 34, Advisory Comm. Note (2006). Because these examinations raise issues of confidentiality and can produce thousands of documents that have to be reviewed for relevance and privilege, " compelled forensic imaging is not appropriate in all cases, and courts must consider the significant interests implicated by forensic imaging before ordering such procedures." John B. v. Goetz, 531 F.3d 448, 460 (6th Cir.2008); see also White v. Graceland Coll. Ctr. for Prof. Dev. & Lifelong Learning, Inc., No. 07-CV-2319, 2009 WL 722056, at *7-8 (D.Kan. Mar. 19, 2009) (quoting Advisory Committee Notes to 2006 amendments for the proposition that forensic imaging is not appropriate in all cases and noting that courts permit imaging when there is trade secret or electronic evidence at issue, or where discovery responses suggest some inconsistency or impropriety).
The precise scope of the limitations suggested by the 2006 Amendment to Rule 34, as it applies to forensic inspection of a relevant object as opposed to a forensic search for documents, is unclear because the sections of that Rule denominated " Procedure" do not specify what, if any requirements, must be met before a court permits the inspection. The request must, however, be a request " within the scope of Rule 26(b)." Fed.R.Civ.P. 34(a).
Rule 26(b)(2)(B) provides that a party " need not provide discovery of electronically stored information from sources the party identifies as not reasonably accessible because of undue burden or cost." Despite that showing, the court may nevertheless order discovery from such sources if the demanding party shows good cause. Id. In making this determination, the court must " consider[ ] the limitations of Rule 26(b)(2)(C)." Id. Those limitations are, of course, those that pertain to all discovery, whatever form it takes, and require a court to limit the extent of discovery if it determines that (1) the discovery sought is unreasonably cumulative or duplicative or can be obtained from other sources that are more convenient, less burdensome or expensive; (2) the requesting party has had ample opportunity to obtain the information by other discovery; (3) " the burden or expense of the proposed discovery outweighs its likely benefit, considering the needs of the case, the amount in controversy, the parties' resources, the importance of the issues at stake in the action, and the importance of the discovery in resolving the issues." Fed.R.Civ.P. 26(b)(2)(C).
Since the latter rule pertains to all discovery, whether forensic inspection of an object under Rule 34(a)(1)(A) should also be viewed through the prism of the data on it not being reasonably accessible under Rule 26(b)(2)(B) (that requires good cause to search for data that is not reasonably accessible) is of little moment. As a form of discovery, forensic inspection is unquestionably subject to the balancing required by Rule 26(b)(2)(C) whenever any discovery is challenged as an undue and unnecessary burden. I am therefore required to use that calculus to ascertain whether the forensic imaging should be permitted.
As I have explained, Revonet argued that forensically searching the Federated Database server is unduly burdensome because the servers to be searched are old and may crash, its business will be disrupted while the search is conducted, and the search may yield data that is subject to confidentiality agreements with other clients. None of these arguments are compelling. The searches will be done on copies of the forensic images, so the servers themselves will not be affected after the initial copies are made. The search will be conducted on the weekend and concerns about confidentiality can easily be alleviated through a protective order.
On the other hand, the " needs of the case ... the amount in controversy ... the importance of the issues at stake in the action, and the importance of the discovery in resolving the issues," Rule 26(b)(2)(C)(iii), surely establish that the potential benefit of the forensic search outweighs its burden.
First, there is simply no other way in which to seek this information, and it should establish once and for all what Revonet did with the Covad data that was in the Federated Database. Indeed, the investigation may have a determinative impact on whether this case will survive. If the forensic imaging shows little use of the data from Covad by Revonet, Covad's financial motivation to continue this lawsuit may disappear or at least lead to an early settlement. If, on the other hand, the data contradicts Revonet's assertions, made throughout the lawsuit, that Revonet made little use of the Covad data, then the shoe will be on the other foot. In either event, the data simply must be found and analyzed before the parties and the Court can get their hands around what this case is truly worth. It is simply one of those perhaps unusual cases where paradoxically " the amount in controversy," " the likely benefit," and the " needs of the case" can only be ascertained by permitting the discovery sought.
Second, while the issues at stake in this commercial controversy may be not be of interest to any one but these litigants, they still have a right to its speedy and just conclusion, which, in my view, will be advanced by the forensic imaging. I know of no other way to examine the data and how it was used by Revonet.
Third, while Revonet's complaints at the hearing that it is in difficult financial circumstances may have merit, the forensic searching may produce the information it needs to decide how much more of its resources (including money for legal fees) to devote to this case.
Thus, on balance, I conclude that Revonet's objections to the forensic searching, while meritorious, are overwhelmed by the necessity of the forensic searching that I am ordering. I will also meet Revonet's concerns about confidentiality with a protective order. I will therefore order that Ellis shall complete a forensic search of the OVNC SQL 01, OVNC SQL 02, and OVNC WEB 01 to capture data of whatever type or kind that would tend to show (1) Revonet's receipt of information from Covad or (2) Revonet's modification of the information from Covad, or (3) Revonet's use of the information from Covad in the period (a) from the date of the contract between Covad and Revonet until December 31, 2006, or (b) from the date of the contract between Covad and Revonet to the present. Ellis shall then provide the parties with a written report of his findings. He shall also retrieve the information and make it available, in a reasonably usable format, to the parties and their counsel.
2. E-mail servers (Revmail one, Revmail two, and Revmail).
In addition to conducting a forensic examination of the database servers, Covad wants to forensically examine Revonet's e-mail exchange servers. Covad offers three justifications in support of its request: (1) Revonet's e-mail production is facially incomplete; (2) Revonet's email production suggests that it was using e-mail to give leads to clients; and (3) Revonet's servers failed after litigation was initiated but before discovery commenced in earnest.
There is certainly authority for the proposition that if a party's e-mail production suggests that she is intentionally hiding things, or failing to take appropriate steps to respond to discovery, a forensic examination may be appropriate. Preferred Care Partners Holding Corp. v. Humana, Inc., No. 08-CV-20424, 2009 WL 982460, at *14-15 (S.D.Fla. Apr. 9, 2009) (allowing forensic exam at producing party's expense because of discovery failings, inconsistencies and defendant's " print and purge" campaign); Peskoff v. Faber, 244 F.R.D. 54, 59 (D.D.C.2007) (allowing forensic examination because documents were conspicuously absent in certain time periods and circumstances suggested that absence was the result of party's misconduct). However, it does not automatically follow that every time a litigant alleges that the other party's electronic production is deficient in some regard the appropriate next step is to conduct a forensic examination. See Exec. Air Taxi Corp. v. City of Bismark, N.D., 518 F.3d 562, 569 (8th Cir.2008) (finding no deficiency with e-mail production to warrant forensic examination; party's mere suspicion not enough); Sup. Prod. P'ship v. Gordon Auto Body Parts Co., Ltd., No. 06-CV916, 2009 WL 690603, at *2-3 (S.D.Ohio Mar. 12, 2009) (refusing to allow forensic search as a sanction for discovery misconduct where party seeking exam had not provided any evidence that relevant e-mails had been lost or destroyed by inadequate litigation hold).
It is the rare case that a litigant does not allege some deficiency in the production of electronically stored information, particularly e-mail. All too many entities lack records management policies that are aggressively enforced, and records keeping may be a function not of an enterprise wide policy but determined by the idiosyncratic habits of the various users. In such a situation, the possibility that one user saved everything while another saved nothing may lead to curious gaps in the e-mails that are produced and an inability to explain why any are missing. While such productions cast little glory on the companies that produce them, I cannot find any authority in the cases to date that permit a court to conclude that allegations of deficiencies in themselves automatically require a forensic search whenever a party claims that there are, for example, fewer e-mails from a person or about a subject or transmitted in a given time than the party expected to find. This would result in forensic examinations in virtually every case, which would increase the cost of litigation involving electronically stored information markedly not only because of the cost of the examination itself, but also because it would yield information that would have to be sorted for relevance and privilege. Experience shows that the latter, involving expensive reviewing of " e-mail strings" by lawyers, paralegals or by vendors to whom the work has been outsourced, may dwarf the cost of the search. Fear of the consequences of consistently ordering forensic searches of e-mail servers should, at a minimum, require a showing that permits the finding, albeit based on circumstantial evidence, that the producing party has not complied with Rule 26(g) because the production is not " complete and correct as of the time it was made." Fed.R.Civ.P. 26(g)(A).
As evidence of Revonet's alleged shortcomings, Covad identified e-mails that had been produced recently, noted that only four external e-mails had been produced, and identified various alleged deficiencies with search terms. Notably, however, Revonet asked Covad's counsel repeatedly to suggest search terms, and Covad's counsel did not respond to any requests. Given that history it is unfair to allow Covad to fail to participate in the process and then argue that the search terms were inadequate. This is not the kind of collaboration and cooperation that underlies the hope that the courts can, with the sincere assistance of the parties, manage e-discovery efficiently and with the least expense possible. See The Sedona Conference Cooperation Proclamation (2008), available at http:// www. thes edonaconference.org/content/tsc_cooperationproclamation.
While there is some evidence to suggest that Revonet's production may have been incomplete, this is a much closer case than Peskoff or Preferred Care Partners (discussed above). Fortunately, I may be in a better position to assess the completeness of Revonet's production because of the forensic search of the databases that I am permitting. That search may yield important information about the use of e-mail by Revonet that may permit more certain conclusions about whether a search of the e-mail servers can be justified under Rule 26(2)(C). Accordingly, I will reserve decision on whether a forensic examination of the e-mail servers to look for items that should have been produced in response to Covad's requests is appropriate until Ellis has submitted his expert reports and the parties have had the opportunity to confer and, if necessary, brief the issue of whether, in light of what he has found, a forensic search of the email servers is appropriate.
Covad does provide compelling justification for one forensic search of e-mail servers that I will permit.
The original Revmail one server that once resided in Sioux Falls crashed after this case had been filed, and there were no back-ups. Revonet had a responsibility to take reasonable steps to preserve relevant information once it had notice of litigation. While the destruction of e-mails here was clearly inadvertent, Revonet has a responsibility to take appropriate steps to try to recover those e-mails, and it is not clear that it has ever done so. Orrell v. Motorcarparts of Am., Inc., No. 3:06-CV-418, 2007 WL 4287750, at *7 (W.D.N.C. Dec. 5, 2007) (" The fact that plaintiff's home computer allegedly ‘ crashed’ -as opposed to having been ‘ wiped’ as the work laptop was-in no way eliminates the Plaintiff's burden to do all she could under those circumstances to preserve evidence." ). There was no evidence presented regarding what efforts Revonet took to extract data from the two e-mail servers that have since been retired, particularly the first one, which crashed completely. Because we need to determine what, if any, data was lost before evaluating what effect the server crash had on the adequacy of production, Ellis should first examine the forensic images of Revmail one and Revmail two to determine whether it is possible to do a forensic search of either of them.
The only e-mail server that is currently operational at Revonet is Revmail. Thus, Revonet's document request responses would have come from searches run on Revmail. Revmail one and Revmail two were both in operation during the time when Covad and Revonet were working together. Thus, it is very possible that there were e-mails created during the time that Revmail one and Revmail two were operational that would be relevant to this case. What is not clear, however, is the degree to which data may exist on Revmail one or Revmail two that was not transferred to Revmail. Thus, a comparison should be made between Revmail one and Revmail and between Revmail two and Revmail to determine what, if any, data exists on the non-operational servers that does not exist on Revmail. These comparisons should yield the universe of e-mails that have not yet been considered for production (if Revmail one and Revmail two can be searched in their present states).
The ultimate goal of any such search is to ascertain whether there is any recoverable data on Revmail one and Revmail two that would tend to show (1) Revonet's receipt of information from Covad, or (2) Revonet's modification of the information from Covad, or (3) Revonet's use of the information from Covad in the period (a) from the date of the contract between Covad and Revonet until December 31, 2006, or (b) from the date of the contract between Covad and Revonet to the present. Thus, Ellis shall search the universe of data on all three servers to ascertain whether there is data on Revmail one and Revmail two that is not on Revmail and that meets the criteria I have just identified. He shall submit a written report to the parties explaining his conclusion that there is, or is not data that fits the descriptions I have laid out in this section. He shall carefully identify the data he has found that meets the criteria (1) and (2) that is on either Revmail one or Revmail two but is not on Revmail. Once he does that we shall address the next step to be taken. I note here only that I appreciate that if a search for additional e-mail becomes appropriate the necessity of reviewing the data for possibly privileged material may arise that may not be eliminated by the protective order I sign because, as I have already noted, there is a difference between privileged and confidential material. I note here my intention to do everything possible and in particular to use the provisions of Rule 502 of the Federal Rules of Evidence to forge an agreement that will reduce the necessity and cost of such a review to the minimum.
3. Individual PCs.
Like the Federated Database, the individual PCs are relevant because they may contain information that demonstrates how the relevant data was used at Revonet. Accordingly, Ellis shall create a forensic image of each PC and conduct a search of it to ascertain if it contains data that would tend to show (1) Revonet's receipt of information from Covad, or (2) Revonet's modification of the information from Covad, or (3) Revonet's use of the information from Covad in the period (a) from the date of the contract between Covad and Revonet until December 31, 2006, or (b) from the date of the contract between Covad and Revonet to the present. If he finds indications that such data exists, he shall report to the parties what indications he has found and whether he believes additional searches will yield additional data.
4. Additional Searching.
Once Ellis has submitted his reports pursuant to this Opinion, the parties shall meet and confer within one week to discuss whether any additional searching would be worthwhile and justified considering the anticipated costs. The parties are certainly free to agree to any further searches, but that agreement must be in writing and provided to Ellis as instructions for the search. If the parties cannot agree, the party who seeks to conduct additional searches must file a motion for additional searching explaining why the results of the search conducted, the history of the controversy between the parties and of this litigation, the controlling law and any other factor, justify the additional search sought. That party shall indicate in the filing whether it will pay the cost of such additional searching and, if not, why the cost should be shifted to its opponent.
C. Costs.
It has long been the presumption in this country that the producing party bears the cost of production and " [a]ny principled approach to electronic evidence must respect that presumption." Zubulake v. UBS Warburg, LLC, 217 F.R.D. 309, 317 (S.D.N.Y.2003). " Production" of electronic information can quickly become quite expensive and disproportionate to what is truly in stake in the lawsuit. For one, retrieving the information, even when not done through forensic methods, may still require the assistance of a technological expert or consultant to provide guidance as to how the search should be conducted to gather the requisite information at the lowest possible cost. Additionally, the universe of items to be considered for production is ever expanding with the ubiquity of e-mail and other forms of electronic communication, such as instant messaging and the recording of voice messages. Electronic data is difficult to destroy and storage capacity is increasing exponentially, leading to an unfortunate tendency to keep electronically stored information even when any need for it has long since disappeared. This phenomenon-the antithesis of a sound records management policy-leads to ever increasing expenses in finding the data and reviewing it for relevance or privilege.
As this case demonstrates, these new technologies have the capacity to be outcome determinative but often at significant expense. Thus the courts are required to strike a balance between allowing the requesting party to take full advantage of the technologies available to it and protecting the producing party from having to pay to leave no stone unturned. Resting all of the costs of electronic discovery on the producing party may create a perverse incentive on the part of the requesting party to dispense with reason and restraint and unleash every new technology under the sun to try and find information that supports the requesting party's claims.
Fortunately for me, the financial burden of " producing" electronic information in this case is relatively low, and I will require Revonet to bear that cost. When I say " producing," however, I am referring to the costs of satisfying Revonet's obligations under the Federal Rules of Civil Procedure to either (1) make items available for examination, or (2) produce documents. See Fed.R.Civ.P. 34. For the devices or depositories that I have determined that Revonet must make available for examination: OVNC SQL 01, OVNC SQL 02, OVNC WEB 01, and user PCs, Revonet satisfies its obligation to produce those items by paying for a forensic copy of those items to be made. Once Revonet has made those items available to Covad's expert, I see no reason why Revonet should be required to pay for its opponent's expert witness to analyze the evidence. Accordingly, I will require Covad to bear the costs of conducting the analyses I described in §§ III.B.1 and III.B.3 of this Opinion once I have signed any Protective Order that Revonet and Covad prepare for me.
For the items that I am ordering searches of for the purpose of producing documents: Revmail one, Revmail two and Revmail, I will order that Revonet bear the costs of imaging these servers and the costs of the limited searches that I have ordered in § III.B.2 of this Opinion. I am requiring Revonet to bear this cost based on my conclusion that Revonet should have taken all reasonable steps to recover the allegedly lost data at the time it was lost.
I will consider the question of who should pay for any additional searches if and when that issue is before the Court.
V. Conclusion.
For the reasons discussed herein, Revonet shall make the items I have identified available for imaging and searching. An Order accompanies this Memorandum Opinion.