Being the chief information security officer at the company that’s suffered the biggest (known) data breaches in history isn’t the kind of fame most CISOs would be looking for. But it’s Yahoo’s Bob Lord’s bag. His LinkedIn profile includes the line: “I lead the Paranoids, the information security team at Yahoo.”
“I think I may have broken a record,” said Lord, during an on stage interview with TechCrunch’s Frederic Lardinois here at TechCrunch Disrupt New York, discussing how many breach notification emails the company had to send to users after it uncovered two massive hacks.
“Hundreds of millions of emails — I do not know the exact number,” he added.
Last fall Yahoo revealed that a state-sponsored hack had affected at least 500 million accounts, with (as it turned out) the information stolen at least as early as January 2014 and utilized until at least December 2016.
The news of that huge hack was topped a few months later when Yahoo also revealed it had suffered an even more massive hack, in August 2013, of more than one billion user accounts. This breach was only disclosed last December. While Lord joined Yahoo in November 2015.
So how did it feel when Lord uncovered the first of these massive breaches? Not great, clearly.
More specifically, he said it felt a bit like this:
“If you’re familiar with that effect that Alfred Hitchcock perfected — where things look like they’re sort of telescoping out. And you can still see everything but you still have this weird parallax going on,” he said. “I remember feeling that when I was putting all of the different pieces together. And that’s not a great feeling.”
This March the US Department of Justice announced the indictment of four defendants for the 2014 Yahoo hack — confirming prior reports of Russian intelligence agency involvement. Yahoo had initially reported that the attack was “state-sponsored” — so how had the company known that so early on in their investigation of the breach?
“We have the benefit of having a group within our organization, that’s called the Paranoids, that really specializes in tracking APT attackers against our users. And so we actually had world class experts who knew what sort of things to look for and how to chase down leads to try to figure out who was behind these attacks,” said Lord.
Were they paranoid enough, quipped Lardinois? “I think if you ask other people in our company they will tell you that the Paranoids earn their reputation every day,” responded Lord. “But hopefully we’re strategic and we’re good partners — and not just paranoid delusionals.”
He wouldn’t go into technical specifics about how the attackers broke in — suggesting people read the DoJ indictment — but said they used “numerous tactics”.
“There’s a specific set of steps that attackers have to go through, that they must go through in order for them to achieve their goals… So whatever machine they break into is not the machine that they want 99.9% of the time. So then they have to go move from machine to machine to find the thing that they’re looking for,” he noted.
Given that the attackers got into Yahoo’s systems in 2014, why did it take the company so long to discover the breach?
“These campaigns can run for an extended period of time,” responded Lord. “These aren’t smash and grab attacks. These are long term plays — and when you really start to figure that out if you haven’t done that sort of work before it’s a little startling.”
Yahoo’s board also wanted answers on that question, he added, noting that it commissioned a study “to try to go back in time and put the pieces together”.
Far fewer details have emerged about the massive 2013 Yahoo hack. Lord said the problem for that investigation is a lack of evidence because of how much time passed between the intrusion and its discovery.
“We know very little. To date, we’ve turned over as many rocks as we can possibly find, to further than investigation but to date we’ve not been able to find the source of that intrusion, to understand how it happened, to understand who it was. It is likely to stem from the 2014 attack — but again, there’s not enough information, not enough evidence for us to really say anything more at this point,” he said.
“Part of it has to with with logs and other information that’s acquired… You really have to find ways to keep logs for a much longer period of time than you would normally do. And in fact if the average time between intrusion and detection is six months, depending on who you listen to, you’re going to need to have to double that in order to account for other factors in your investigations.”
Because of this lack of evidence, Lord said Yahoo may “potentially” never know how the 2013 intrusion happened.
For the 2014 hack, Russian cybercriminals have been accused by the DoJ of working alongside FSB agents. Although one apparently also managed to manipulate Yahoo search results for the phrase “erectile dysfunction medications” to funnel clicks to an online pharmacy that paid commissions to traffic-drivers — in order to make himself some money on the side.
So how exactly was the hacker able to manipulate Yahoo search results?
Lord again wasn’t keen to provide too much detail — reiterating the extended campaign of activity the attackers engaged in in order to work through systems to gain access to different credentials.
“Again, these are long term compromises where they worked hard flying under the radar, they worked hard to get the access that they were specifically tasked with. But it is now clear that in hindsight that these guys could have got actual tech jobs — they were very good,” he said.
“Modifying production systems is hard when you’re trained and under supervision. One would imagine that’s a difficult thing to pull off without detection and to do that for a period of time so you have to say that — I stay away from the word ‘sophisticated’ because I think that word is very loaded… but I think that these were definitely skilled individuals,” he added.
It’s harder for people to say this kind of person is attacking you… Because now we have more evidence that there’s a spectrum in place.
“And moving back and forth between their criminal activities and their state-sponsored activities is now part of that conversation that we should be having. And it muddies the water — because it’s harder for people to say this kind of person is attacking you, this kind of person is attacking you. Because now we have more evidence that there’s a spectrum in place. So I think that makes the conversation much more interesting. But it does muddy the waters a fair amount.”
The reputational damage to Yahoo associated with such massive hacks has knocked some $350M off its sale price (the company is in the process of being acquired by Verizon — the parent company of TechCrunch’s parent company, AOL). “Security professionals are rarely surprised by this kind of thing,” said Lord, when asked what it was like going to Verizon with details about the breaches.
“If you’ve been in this business for more than a few years you’ve had your skirmishes, so I think the question is always really can you get enough of a root cause analysis to remediate? Can you demonstrate that there are any improvements in place and that the attackers are out of the network?”
If all these illegal hacks weren’t enough to damage Yahoo on the user trust front, a report last fall revealed it had developed a custom program for US intelligence agencies to scan all users’ incoming emails for specific queries. CEO Marissa Mayer reportedly did not believe Yahoo would win a legal challenge against the demand to develop the custom program and therefore chose not to fight it.
Asked about the security culture under Mayer, Lord said in his experience at least there was never an issue being given adequate resources. “For me the culture was vibrant,” he said.
“What matters is how the business thinks about security from a strategic standpoint, and how people are engaged in their daily activities,” he added, telescoping out to discuss security generally. “So if you think the security team can go off in a corner and secure everything you’re wrong — it has to be a company wide initiative across all the different layers to be able to be effective.”
So is Lord sure there’s no hackers inside Yahoo’s system now, asked Lardinois? “You’re asking me to prove a negative,” he objected. “It’s hard to prove a negative.”
But, on the balance of a “preponderance of circumstantial evidence”, he suggested similar types of attacks have been mitigated — on account of the programs Yahoo now has in place to reduce the chance of an exploit.
“Certainly the specific techniques are technically not possibly today,” he added.
Comment