The dark side of the web

For Freire, exposing this data and giving researchers the tools to share and analyse it could be a key step for the evolution of science. DeepPeep is far from alone. Next-generation search engines such as Kosmix and info-driven harvesters such as BrightPlanet are working hard to pull data from the deep, while Google now has its own automated deep web search program in place.

There’s nothing necessarily secretive about the majority of this hidden content. When asked if the deep web harbours criminal or illicit activities, Dr Freire explains that “underworld” content is just as likely to be found on the “surface web”, and describes the deep web as “a more benign place” than some imagine. There are, however, areas that are more intentionally secretive, and this is where the deep becomes the dark.

The deep web is a more benign place than some imagine

Liam O Murchu, a security expert at Symantec’s Security Technology and Response team (STAR), believes there are three tiers of criminal operating online. The least serious, and most common, will operate in plain sight, on forums that can be found with a conventional search engine.

Beyond this, there are more serious – and paranoid – cybercriminals who “may only work in environments that they consider secure, for example, invite-only forums or secure private chat channels”. These forums will be “harder to find, often by word of mouth in other forums, or by invitation only or via ‘vetting’ and will not be indexed in search engines”. For a higher level of secrecy, however, there’s the third option: the darknet.

Exploring the anonymous web

Often associated with small file-sharing networks, the term darknet refers to any closed, private network that operates on top of the more conventional internet protocols. To join these hidden internets, all you need to do is install a program, such as Freenet or I2P, and browse away, secure in the knowledge that you’re almost impossible to trace.

Freenet is effectively a shadow of the web, with its own sites, forums and email services. A related service, TOR (The Onion Router), provides tools to set up hidden services, including websites, which will be anonymous within TOR and inaccessible from the outside.

Technically, these applications are ingenious. Freenet operates as a network of decentralised nodes, with each system on the network contributing bandwidth.

Since Freenet sites don’t sit on servers, but on data stores spread throughout the network, they can’t be taken down, and because each communication between one computer and another is routed through other nodes, with each one only “knowing” the address of the next node and that of the last, Freenet’s users can maintain high levels of anonymity.

On Freenet, nobody knows who you are, or what you’re looking at. Each system also contributes hard disk space, which is occupied by a data cache containing chunks of heavily encrypted data that the program can reassemble into Freenet forums and sites.

A trip through Freenet can be unsettling. It isn’t hard to find sites offering hard-core porn or such charming tomes as The Terrorist’s Handbook, Arson Around with Auntie ALF and the Mujahideen Poisons Handbook, along with copyrighted software, video and music to download.

And while we didn’t come across any child pornography during our time on Freenet (for obvious reasons, we didn’t look), it’s widely acknowledged that it can be found.

Freenet was the brainchild of a young Irish computer scientist, Ian Clarke, who came up with the idea during his studies at the University of Edinburgh in the mid-1990s. He wanted to “build a communication tool that would realise the things that a lot of people thought the internet was – a place where you could communicate without being watched, and where people could be anonymous if they wanted to be”.