Q&A: Joe FitzPatrick

Joe FitzPatrick has spent his career digging into the lowest levels of computer hardware, figuring out how things work and how they fail. He knows the kind of mistakes that manufacturers make thst lead to vulnerabilities, and he understands the ways in which an attacker or someone along the supply chain could insert a backdoor. Those are two different things, but can look quite similar to the untrained eye. FitzPatrick discussed the questions that observers should ask when presented with assertions about a hardware implant during a talk at last week's CanSecWest conference, using figures from Greek mythology as guides. This Q&A has been edited and condensed from a podcast with FitzPatrick.

Dennis Fisher: How did you come up with that concept for using Pandora, Cassandra, Aristotle, all these mythic Greek figures to explain hardware implants?

Joe FitzPatrick: I've always been a fan of Greek mythology and there's a lot of stories that explain things in a very similar way to the things we encounter. The two that I already picked out, Pandora, she's curious, she opens the box and how different it is what she did versus, you know, what anybody does when they inspect something and find vulnerabilities. The recent news kind of inspired me to think about the coverage and the reception of some of this information. Cassandra, she's gifted with the gift of prophecy, but also cursed with the fact that no one will believe anything she says. The two of those kind of summed up a lot of my feelings the past fall, when people were making bold claims about hardware implants. But what I didn't see was the rational approach to, you know, understanding what's real, what's not. My background is hardware. I can look at a report of some hardware risk and, and make an informed guess about it, but the vast majority of people don't have that skill set.

Dennis Fisher: I liked the way that you use Pandora specifically to talk about the curiosity thing and you mention curiosity is essentially the core ethic of hacking. You've got to have this base curiosity to even get started. You have to want to know how things work, how they fail, how they succeed. Is that kind of why you got into hardware hacking in the first place?

Joe FitzPatrick: I was fascinated by computers. We were lucky enough to, to have several old PC clones, which I played with. I didn't really get into programming until later than all the people who started programming when they were five. But, I was always fascinated by what made the next level work. And I kept going down through software, you know, I learned more about assembly and then I kind of learned, you know, conceptually like CPU architecture and that really fascinated me. And rather than pick any one of those levels to stick on, I just keep going deeper, which kind of got me where I ended up, which, you know, I started my career doing tools for silicon debug for our server and desktop CPU, which is kind of pretty low level when you get down to it.

Dennis Fisher: There are these debug features in chips and other pieces of hardware and they're essentially back doors in that that's what they're meant to be. They need to function when everything else isn't working right. And you can't really differentiate between a malicious backdoor and debug feature. Is that a concept that you have to explain to people fairly regularly when they're thinking, oh, this is a backdoor?

Joe FitzPatrick: I'm thinking back to a presentation and I don't recall the presenter or the exact detail, but he showed basically the proof that profit and theft are functionally identical. Right? You know, you input stuff to a system, you take stuff out of the system and you get money in the process and, well, that's profit. When someone is unfamiliar with the way of low level hardware and a low level debug, right, it makes perfect sense that they would see a back door and think it's a back door. But then when they have a little more understanding about like, why this functionality has to exist, why in order to bring up a piece of silicon, you need the ability to debug the silicon in order to debug the silicon. You need observability into the silicon. You can't have one without the other. You can't have a manufacturable, sellable product without having a way to debug it. So that's kind of the detachment. One of the things I pointed out, if you encounter something at hardware and you don't understand it, find a domain expert. Because the domain expert, and I don't mean like top of the field of management, it's just a hardware engineer or someone who has built network adapters before or something, you know, who is familiar with the components you're dealing with. There are lots of people who aren't security people, but know about this. What's interesting is, whether we're talking about implants inside of network adapters or we're talking about Rowhammer, right. People who make memory cells have understood Rowhammer for ages, but they didn't put it together as like a software accessible vulnerability. Until we got to put the things together, glue it all together, open the box and take a look at what's inside.

Dennis Fisher: More broadly, how big of a concern do you think this whole supply chain attack thing is? To me it seems like a very big concern, depending on who the customer is and what their threat model is. But, it's something that people in the security community definitely think about. But I don't know how much the broader government and enterprise security communities think about it

Joe FitzPatrick: For a very long time, when I would talk about hardware security, people would assume that I'm talking about supply chain security. But that supply chain security from the government side, the military side of like, oh, we need to make sure we get these tools from this source and that they're reliable and everything. I get a very different approach to supply chain security. My perception of that was like, we have a bunch of people who've been thinking about supply chain security for hundreds of years. And they're looking to apply their approach to think that that's going to solve hardware security and it won't.. You know, the example I use is we could have a trusted network adapter made by trusted people and our trusted fab and putting on a trusted board and shipped by trusted couriers are installed by a trusted person.

But if it's network adapter that takes firmware updates over the wire, we still don't have secure hardware. How big is the problem? It's huge. Like I think the great part about recent news is it's making people aware of this. I've been trying to point out the reaction is what we need to be careful about. You know, the knee-jerk reaction is not good. The, Hey, we need to talk to our supply chain. We need to find out where our servers come from, where our boards come from, where the components on the boards come from. Up until this point, no one's really asked those questions in volume. Consumers haven't asked those questions. If they start asking those questions, then the suppliers are going to have to come up with answers and processes for documenting this stuff. So we're in a spot where we don't really have visibility into supply chain, but we have that from an accountability in a counterfeit avoidance perspective. We don't necessarily have that from a tamper maliciousness perspective.

"A motherboard manufacturer's not going to backdoor all their clients because they're going to lose every one of them if it comes to light."