Popular TV shows such as Mr. Robot and movies have fueled the perception that the Dark Web is a massive network of criminal sites engaged in all manner of illegal activities. The reality is a bit more prosaic: it isn’t all that large, after all.
The Dark Web is shrouded in mystery, helped by the fact that it requires special tools—software and configurations—to access. Tor is one such tool. However, the phrase Dark Web has been overused to refer to any underground marketplace or criminal forum. The term is often conflated with the marketplaces on the Deep Web, which refers to parts of the Web not indexed by search engines, often because they are behind paywalls or some other type of login mechanism.
Research from threat intelligence company Recorded Future found the number of live, accessible .onion sites amount to less than 0.005 percent of surface web domains. To put in context, there are about 200 million surface Web domains. The research focused on estimating the full size of a reachable Tor network by counting .onion sites.
"The popular iceberg metaphor that describes the relationship of the surface web and dark web is upside down," wrote Recorded Future data scientists Garth Griffin and Juan Sanchez, as unlike an iceberg, the part of the web that we can see is much larger than the web we don't see.
Recorded Future researchers used inbound links to map the .onion sites on the Dark Web. On the surface Web—the Web that is familiar to most people—inbound links to a site help determine the site’s popularity. More inbound links mean more ways for people to reach that site. The same goes with the Dark Web—researchers began with a set of onion sites pulled from public lists and from its internal research. They crawled 260,000 pages and found 55,828 different onion domains, but only 8,416 were observed to be live on the Tor network. Only 15 percent were live sites.
Criminals Go Anywhere
Whenever there is any discussion about online criminal activity, or some kind of illegal activity in a marketplace, the immediate conclusion is that it must be happening on the Dark Web. There are places on Dark Web for illicit activity—they form “a tiny portion of onion sites, a set of invitation-only and generally unpublicized communities buried in the most shadowy corners of the internet,” the researchers said.
The unpleasant truth is that criminals get together wherever they can, and wherever makes sense for them, said Chris Camacho, the chief strategy officer of threat intelligence company Flashpoint. Stolen data can turn up in all kinds of places. Some are Tor sites, some are buried away in log-in only sites on the Deep Way, but many others are traded and sold in the open right on surface web. Some of the activity don’t even happen on the Web, as it may happen on IRC channels, chat apps such as WhatsApp, or sites such as Discord.
The bulk of the criminal activity online happens on chat apps, followed by password-protected forums (not on Dark Web), Camacho said.
Recorded Future noted the same, that “much criminal activity happens on sites not requiring any special protocols to access, such as public social media sites like Twitter or messaging services like WhatsApp and Telegram.”
Understanding that the Dark Web is smaller than expected would help manage expectations from the enterprise defender’s side. Just monitoring the Dark Web will uncover only some bad things—perhaps there will be an early alert exposed payment card numbers—but if the forum selling access to compromised servers is not on the Dark Web, that laser-focus on monitoring the Dark Web is not helpful.
Enterprises “need a partner to help make sure the coverage is comprehensive and that they are seeing everything,” Camacho said.
Criminals that are intent on making money are going to do keep moving in order to stay hidden: they will use anonymizing tools and services, and they will pivot to new methods and sites if they are more effective. If one marketplace shuts down, they go to another. If a chat group gets too noisy, they branch off into smaller conversations.
Defenders have the complex task of trying to find all the places they can hide to find out what they are up to without exposing themselves, Camacho said.
Counting the Dark Web
The most “popular” Tor site—the one with the most inbound links—was a market with 3,585 inbound links. A site offering help with hosting onion servers had 279 inbound links. Not all .onion sites are being used for criminal activity. Legitimate companies operate .onion mirrors, such as the New York Times and Facebook.
The sites are also tightly connected—and most sites are accessible from the Hidden Wiki, a Tor-accessible directory of dark web sites. Users starting from Hidden Wiki are within three clicks (average degree of separation: 2.47) away from 82 percent of the live sites Recorded Future found. The remaining 18 percent were “completely disconnected” from the Hidden Wiki, suggesting they were part of isolated communities separate from the rest of the Tor network.
“The data suggests that if you visit the Hidden Wiki onion page, you’d be about three clicks away from 82% of live onion sites,” the researchers wrote.
Counting the number of Dark Web sites is trickier than counting sites on the surface web, since these sites are unreliable, disorganized, and often short-lived. Sites can vanish without a trace and others reappear as brand-new sites without warning. The most popular sites had uptimes between 60 to 90 percent of the time.
There are also fake “look-alike” domains, as well. One Dark Web typosquatting scheme claims to have defrauded visitors of more than 400 popular onion websites and generated thousands of dollars in Bitcoin.
Recorded Future identified eight sites as “top-tier criminal sites” and found they had an average of 8.7 inbound links. The criminal site with the most number of inbound links from this list had only 15 links. These sites have significant barriers to entry since the owners vet each member to make sure they aren’t actually law enforcement or researchers in disguise, that they are there to buy and sell on the site. It’s these sites that may have helped perpetuate the Dark Web’s image as vast and hard to track.
“The idea of a dark web that is hidden and mysterious is more likely an extrapolation of a tiny portion of these onion sites — a set of invitation-only and unpublicized communities buried in the most shadowy corners of this part of the internet,” the researchers wrote.
Shrinking Web?
Law enforcement officials have been making a lot of moves against Dark Web marketplaces recently. Dream Market shut down because it was being hit by distributed denial-of-service attacks and a ransom demand it could not pay. Law enforcement took over the Wall Street Market, the second largest illegal market on the Dark Web, and arrested three of the alleged operators in Germany. The takedown was made possible after a WSM moderator published server information and backdoor administrator credentials to the marketplace as part of a blackmail scheme against WSM users. Valhalla was also taken down alongside WSM.
Over the past few years, a number of marketplaces have either been seized by law enforcement or disappeared for other reasons: Silk Road, (and Silk Road 2.0), Alpha Bay, RAMP, and Hansa Market.
However, that doesn’t mean that law enforcement has stamped out online illegal activity. There are plenty of places online—Dark Web, Deep Web, and even the surface Web, for illegal activity. And with a lot of activity moving to chats and non-Web methods, there are plenty of opportunities for criminals to trade in stolen goods.
Monitoring criminal activity is “not a numbers game,” Camacho said. It isn’t as if the analyst can just look at a certain number of sites and see all the bad activity. As far as the enterprises are concerned, they want analysts to establish themselves anywhere the criminals are and find everything that affects them, regardless of where that activity is happening.
"If they go to Facebook, that's where we have to be," Camacho said.
On the "clear" web, popular sites have millions of inbound links, making it easy for people to access the sites. The dark web, has fewer inbound links, with the most popular market having 3,585 links. Smaller, more exclusive marketplaces popular among criminals tend to have less than 10 inbound links. (Source: Recorded Future)