Witnesses Describe FBI’s Mishandling of Computer Servers in Backpage Takedown

computer server
Did the FBI bollix Backpage's computer servers on purpose or is incompetence to blame? (Tim Dorr via Flickr)
A recent evidentiary hearing in U.S. v. Lacey and Larkin revealed how the FBI took a wrecking ball to Backpage.com, damaging the most valuable piece of evidence in the case: the site itself.

An October 25 evidentiary hearing in Phoenix federal court before U.S. District Court Judge Susan Brnovich offered additional proof of the government’s bad faith in its prosecution of Michael Lacey and Jim Larkin, the erstwhile owners of the online classifieds site Backpage.com.

Witnesses at the hearing testified that when the federal government seized Backpage on April 6, 2018, the FBI failed to take proper steps to preserve the website as evidence. And though the FBI turned over to the defense what it claims are mirror images of the hard drives from a handful of Backpage’s 106 servers, much of the data on those hard drives has proved unusable in its current form.

The four-hour hearing on Oct. 25 was the continuation of an ongoing pre-trial donnybrook over evidence that began with an all-day court session earlier in the month. The initial Oct. 3 hearing revealed that government agents had bungled the preservation of valuable server evidence, inexplicably opting not to leave Backpage in an easily-searchable, read-only mode after taking it offline.

Front Page Confidential photoillustration that depicts part of the message the FBI used to obscure the Backpage.com homepage when the agency seized the website on April 6, 2018
The FBI seized Backpage on April 6, 2018, but agents curiously opted not to maintain a “read only” version of the site. (Front Page Confidential photoillustration)

Whether or not the data on Backpage’s servers has been irrevocably corrupted remains in dispute.  Since the beginning of the case, the defense has informed the prosecution that it needs access to the databases on those servers in the same condition as when they were seized. In a motion to compel discovery filed in June, defense attorneys argued that only when they had such access could they “search, view, and analyze information about the website and its actual operations.”

The motion to compel emphasized that the server data was “critical to defendants’ ability to prepare their defense.” Indeed, the server data went to the crux of the allegations against Lacey, Larkin and their four co-defendants, who face 100 counts of conspiracy, money laundering and facilitating prostitution across state lines.

Before the government eradicated Backpage from the internet, users posted many millions of ads on the site for everything from puppy sales to rooms for let. There were also listings for legal, adult services, including ads for dating, escorts, massage, fetishes, phone sex, etc. Ads offering sex for money were banned from the site, but prosecutors claim all of the ads actually were advertisements for illicit sex masquerading as lawful adult ads.

Prosecutors also argue that Lacey and Larkin, who sold Backpage to company CEO Carl Ferrer in 2015, are vicariously responsible for any illegal acts that may have been connected to the ads. The government’s superseding indictment cites 50 specific adult advertisements from Backpage as the basis for the allegations against the defendants.

When the website was functional, it allowed for simple, administrative searches of a treasure trove of historical information, including ad moderation, payment methods and whether an ad was referred to the National Center for Missing and Exploited Children (NCMEC), a semi-governmental agency that farms out such reports to law enforcement.

But now the website no longer exists.

Imaging Garbage?

Tami Loehrs, a digital forensics expert employed by the defense, testified toward the end of the Oct. 25 hearing. She told how the government would only let her eyeball the physical, confiscated servers stored at FBI facilities in Idaho and Phoenix. She wasn’t allowed to boot the servers up or even photograph them.

Rather, the FBI has provided defense attorneys with what it says are “imaged” copies of the hard drives from 10 or 11 Backpage servers. The first batch delivered to the defense were in three boxes containing 56 hard drives that supposedly represented five of the site’s 106 servers.

But when the defense expert inspected these hard drives, she found that “some of them were readable, some of them were not.” She utilized a variety of industry-standard tools in an attempt to tap into the data and largely ended up with bupkis.

The defense’s computer forensics expert said she didn’t have access to the FBI’s software and had no way to test or validate the government’s tools to see if the results they obtained were accurate and reliable.

Loehrs tried using the forensic software that the FBI says it employed to create mirror images of the servers: FTK Imager, a product of the Utah-based company AccessData, but this didn’t work for her either.

Then at the Oct. 3 hearing, the defense expert learned through the testimony of Matthew Frost, a forensic examiner for the FBI, that Backpage had used FreeBSD, an open-source software, as its operating system.

Under direct questioning from defense attorney Whitney Bernstein, who along with attorney Thomas Bienert represents defendant Larkin, Loehrs explained how she researched the software involved and contacted AccessData for support.

AccessData told her that FTK Imager “is not validated to image FreeBSD.” She said she then “installed FreeBSD on another virtual machine and tried to actually use FreeBSD to access” the data on the hard drives. But the process was unsuccessful.

There were other complications, the defense expert explained. For instance, Frost testified that he was unaware there had been encryption on the servers. But in emails recently released by the government through the discovery process, Loehrs learned that the servers were “equipped with self-encrypting drives.”

Loehrs explained that if the server’s hard drives remained in the server’s chassis — i.e., the metal box that stores them — the information would “automatically decrypt.” However, if you remove the drives from the chassis, “you’ve just scrambled all the data.”

What would happen, Bernstein asked, if someone mirror-imaged an encrypted file?

“[Y]ou would have either an empty hard drive because it hasn’t been decrypted, or you would have a bunch of data that is unrecognizable,” the forensics guru replied. “It would still be scrambled … [because] you’ve just imaged garbage.”

The FBI claims to have used a “boot CD” containing proprietary software to help extract data on the servers. From this data, the bureau created a complicated, difficult-to-navigate database, which it then turned over to the defense.

Loehrs said she didn’t have access to the FBI’s software and had no way to test or validate those tools to see if the results they obtained were accurate and reliable.

She also said she later obtained a CD produced by the government, purportedly containing the site’s “payment processing island,” a set of databases that kept information related to whether an ad had been paid for (some were free), how it was paid for, who paid for it and when. But this data was similarly damaged, and she concluded that “there is no way that this is ever going to be usable in any format.”

Bernstein asked how a digital forensic examiner should acquire and preserve data, according to industry standards, when dealing with a server system as complex as Backpage’s.

Loehrs described a methodical, step-by-step process that the FBI apparently did not follow in this case, including documenting everything to do with the servers, recording the servers’ IP addresses and noting how the servers were configured to work with each other.

IRS agent Richard Robinson conceded that the government didn’t need to dismantle and cart away the Tucson servers because it had already obtained a sealed indictment against the defendants about a week before seizing the servers.

The savvy specialist said she would have taken the website off the internet by disconnecting the router. The next step would be to put the website in read-only mode, which would have preserved the website and allowed for the sort of functionality the defense needs to acquire exculpatory information and rebut the government’s allegations.

Why the government did not preserve the website in read-only mode remains unexplained. Backpage maintained two major nests of servers, one in Amsterdam and one in Tucson, Arizona. And for the most part, the Amsterdam and Tucson servers mirrored each other.

The prosecution has avowed that there was redundancy throughout the system, and so it didn’t make available all 106 servers to defense attorneys. But Loehrs said this was no excuse for the government to deny the defense access. In other cases she’s worked, she was sometimes told to ignore certain computers because there supposedly was “nothing of evidentiary value” on them.

But when she eventually obtained the computers and analyzed them, she and her team discovered exculpatory information.

“We have found backup information nobody knew was there,” she told Bernstein. “I need every piece of the evidence, so we can make that decision whether or not they’re relevant and what’s going to work.”

Rich in Irony

Also at the Oct. 25 proceeding, defense attorney Bienert questioned Richard Robinson, an agent with the Internal Revenue Service’s Criminal Investigation Division, who had been investigating Backpage since 2016. Robinson was on-site at a Tucson facility in April 2018 when the FBI took the servers into custody pursuant to a federal warrant.

Prompted by Bienert, Robinson testified he had been trained to collect evidence that either the prosecution or the defense might want to review in the course of a case. Bienert then went through a list of ways an easily-searchable database of the server material would have been of assistance to the defense, such as determining the number of adult ads vs. non-adult ads on the site, or calculating the percentage of escort ads that indicated whether a county or state agency had licensed their service.

Robinson conceded that the government had not needed to disconnect and cart away the Tucson servers as evidence because the DOJ had already obtained a sealed indictment about a week before seizing the servers. Further investigation of the site was unnecessary at that point. Robinson also described how he and other agents had not taken steps that might have helped reconstruct the website, if necessary.

Interestingly, Robinson interviewed Backpage’s ex-CEO Ferrer on April 5, 2018, one day before Backpage’s seizure. Ferrer had flipped for the feds, agreeing to testify against his former colleagues while pleading guilty to one federal charge of conspiracy and a couple of state charges in Texas and California.

Ferrer discussed Backpage’s reports to NCMEC with Robinson, telling the agent that he was proud of the site’s cooperation with center.

Bienert asked Robinson if, on the day before the searches, he knew that the site’s system “would have information relating to Backpage trying to help stop underage sex?”

“That’s one way you could put it, yes,” replied Robinson.

Melting Frost

During the all-day Oct. 3 hearing, Bernstein had turned up the heat on the FBI’s computer forensics expert, Matthew Frost, the feds’ point man on preserving digital evidence from Backpage’s servers.

Frost, appearing as a government witness, practically melted under Bernstein’s merciless interrogation. But court adjourned before Bernstein finished questioning the FBI guy.

At the start of the Oct. 25 hearing, Frost was back on the stand, still under oath. Bernstein continued her cross examination, and Ariel Neuman, an attorney for one of Lacey and Larkin’s codefendants, closed out the cross with another line of questioning for Frost.

Defense attorney Ariel Neuman observed that the government had in other cases “left servers and databases intact, imaged them on site, and gone away and thereby not destroyed the data,” pointing to the feds’ infamous 2012 case against the online storage site, MegaUpload.com.

Frost testified that he had been assigned to the case in late April/early May 2018. The Tucson servers were already in the FBI’s possession in Phoenix, where Frost examined them and was impressed by the system’s size and complexity. He told Bernstein that there were “lots of servers,” and that the Tucson servers were “a large puzzle” to him.

Bernstein asked if Frost had “tried to virtualize those puzzle pieces to bring Backpage into a functioning environment,” but Frost said that he was not able to.

Because the Tucson servers were kaput, Frost was hoping to take the Amsterdam servers offline and leave them in read-only mode. But Frost said the Dutch National Police refused to cooperate on this point. So Frost flew to Amsterdam to inspect the disconnected servers there and help facilitate their eventual transfer to U.S. custody. (Note: There has yet to be a full accounting of all Backpage servers seized by the government.)

Frost admitted that the government had not made available to the defense everything that it has seized in the case. The FBI has copied and turned over just a fraction of the 106 servers it’s confiscated. And as Loehrs observed, the defense is currently incapable of viewing much of that material, despite using industry-standard tools.

The government contends that much of the server material is redundant, but says that it has captured all the important data on the website from one master server and one image server. Frost testified that he accessed the information using the open-source software MySQL.

Problem is, Frost’s method of accessing the database is, to put it mildly, cumbersome and difficult to use. And the data does not look as it did to Backpage’s moderators. Nor can Frost’s voluminous series of charts and spreadsheets recreate an actual ad with any degree of verisimilitude.

During his cross examination of Frost, Neuman recounted the multiple steps that the FBI techie went through at the Oct. 3 hearing, as Frost searched a database for the number of car ads that appeared on Backpage in the Washington, DC market at the time of the seizure.

By contrast, Neuman observed, a read-only version of the website would have been a cinch to use.

Neuman noted that the government had in other high-profile cases “left servers and databases intact, imaged them on site, and gone away and thereby not destroyed the data.”  He pointed to the infamous federal case against the online storage website MegaUpload.com, which the U.S. Department of Justice seized in 2012, indicting the site’s owner, New Zealand resident Kim Dotcom, on allegations of criminal copyright infringement.

(Dotcom, who denies the charges, is out on bail and still fighting his extradition from New Zealand.)

In the MegaUpload case, Neuman said there were over 1,100 servers involved, more than ten times the servers Backpage reportedly had. But the government chose not to seize MegaUpload’s servers. Rather, the government took those servers offline to copy them and then left them on-site.

The reason for the government’s stonewalling should be self-evident: Its seizure of Backpage was deeply flawed, and its collection and preservation of evidence was the very definition of amateur hour.

Why didn’t the government do likewise in Tucson, thereby preserving the data exactly as it was on April 6, 2018? After all, the facility where the servers were stored was secure. A mirror image of the entire site could have been created within a matter of hours.

Frost said that he hadn’t been involved in that decision and did not know why the government took down the Tucson servers and shlepped them off to Phoenix. He denied data was destroyed in the process, but he admitted that “technologically,” it would have been possible for the feds to do the same thing in Tucson as they had done in the MegaUpload case.

Rapping Knuckles

The Oct. 25 hearing drew to a close with Assistant U.S. Attorney Kevin Rapp cross-examining Loehrs about three judicial opinions over the course of her 20 years in digital forensics that criticized her work. But the defense expert remained unflappable throughout.

“Every expert is going to have bad opinions,” she shrugged at one point. “You can’t please everybody.”

Loehrs’ answers certainly didn’t please Rapp, who more than once during his cross ran afoul of that old courtroom saw about not asking questions that you don’t know the answer to.

Rapp, whose jet-black pompadour is reminiscent of the bouffant hairdo of Bob’s Big Boy, seemed unsure of some of the technical issues involved, about which Loehrs was more than willing to set him straight.

It’s worth remembering that the government denied the defense access to Backpage’s servers from jump. Not to mention, the feds fought like alley cats against having the evidentiary hearing, which is scheduled to resume on December 2 at 8:30 a.m.

The reason for the government’s stonewalling should be self-evident: Its seizure of Backpage was deeply flawed, and its collection and preservation of evidence was the very definition of amateur hour.

For example, when asked about a Backpage server that stored data on the site’s cooperation with law enforcement, including responses to subpoenas, Frost could not recall which server the info was on. Ironically, he did remember that he had not imaged it.

Such a cavalier attitude toward exculpatory evidence is contemptible. Keep in mind, prosecutors have sworn to put veteran newspapermen Lacey and Larkin, both in their 70s, in prison for the remainder of their lives.

They and their co-defendants have a right to due process and a fair trial. But the government seems hell-bent on denying them their rights under the U.S. Constitution at every turn, putting on a show trial worthy of the McCarthy era instead.

For more on the Lacey and Larkin case, please read:
Feds Bungled Preservation of Server Evidence in U.S. v. Lacey and Larkin
and
Prosecutors Poisoned Grand Jury Process in Lacey and Larkin Case

About Stephen Lemons

Stephen Lemons is an award-winning investigative journalist with more than 20 years of experience covering everything from government corruption to white-supremacist gangs. In addition to Front Page Confidential, his work has appeared in Phoenix New Times, the Los Angeles Times, Salon.com, and the Southern Poverty Law Center’s Intelligence Report magazine.

Leave a Reply

Your email address will not be published.