Search results obtained using a traditional search engine such as Google or Bing are gathered from a surprisingly small portion of the internet. This small portion, estimated to be about four percent of the entire web, is referred to as the "Surface Web." So where is the rest of the information? It's located in non-indexed pages of the web - effectively hidden from view.
The Surface Web
Before we dive into the Deep Web and Dark Web, it's important to understand the Surface Web and how internet search engines operate. Search engines utilize spiders that crawl and read content of websites to determine what information to show for specific search requests. A spider is essentially a special software robot that searches a website page returning to the search engine with information that is contained in it. However, if a site or page is not indexed (not allowed to be crawled), these spiders will not have access to them and therefore those sites/pages won’t show up in surface web search results.
Why Access the Deep Web and/or Dark Web?
Both the Deep Web and Dark Web are potential sources of a wealth of information when used mindfully and with knowledge of each. However, each serve different purposes.
The Deep Web contains information varying from academic journals to databases to blog articles that aren't published yet. The Deep Web can be accessed if you know the URL and have the authority to access it or know where and how to search. Some reasons search engines might not be able to access these sites include:
- Password access
- Robots blocked - a specific file can be placed into the main directory of a website to block spiders from crawling the site
- Hidden pages - no hyperlinks to take you to the page
- Form controlled entry - the site requires human based action to turn up results, i.e. dropdown menus
The Dark Web is another story. This can only be accessed through Tor (The Onion Router) or I2P (Invisible Internet Protocol), which utilize masked IP addresses in order to keep users and site owners anonymous. Tor is downloadable software and works by building encrypted connections on servers around the world, creating multiple layers of encryption creating an "onion effect," hence its name. Only at the very end does the traffic come through unencrypted.
Like the Deep Web, the Dark Web can only be searched if you know where you are going. Some have claimed to have created search engines for the Dark Web, but none have been verified yet. As you probably know, the internet consists of domains like .com, .org, .net. Tor allows for access to the Deep Web with page domain .onion, where you might see URLs that look like http://ke2y7fm4mj2qew23.onion.
While many associate the Dark Web with the more nefarious deeds, such as human trafficking and drug sales, it can also be used for some very legitimate purposes. Dissidents who fear prosecution from their government or a particular group can use the Dark Web to anonymously search and post without fear of repercussion. Journalists also find it a safe haven when their sources want to remain private. ProPublica just announced its launch of the first Dark Web version of a news site, allowing users anonymity while accessing full editions of news.
The Dark Web - Legality and Anonymity
Many wonder if merely entering the Dark Web could be considered a criminal offense. The answer is a resounding no, it is legal to surf the Dark Web. However, it's important to use caution when visiting sites or clicking links. The Dark Web is rife with sites offering hit men, firearms and forged papers. While searching online is not illegal in and of itself, the actions you take while on the sites could be perceived as illegal based on the content you are viewing. If you are looking up child adoption, a link could take you to a site involving child pornography - a situation where the act of viewing is an illegal offense.
When it comes to the Dark Web, it is unwise to assume you are completely anonymous without taking additional precautions to prevent being traced. The following are some of the few ways to supplement Tor use in order to maintain anonymity:
- turning off cookies and JavaScript
- not downloading or torrent file sharing
- placing tape over your webcam
- enabling your firewall
There have been cases in the past where security researchers have been able to utilize the very last point where data comes out of the Tor network, called an exit node, to intercept hundreds of emails from various accounts. The emails themselves were not encrypted, creating an exposure risk for users. Leaked IP addresses and man in the middle attacks (where a third party intercepts and sometimes alters communications between two parties who think they're directly communicating with each other) can also put users at risk for exposure.
Use caution and be aware of the risks associated when using Tor:
1) Exposing your computer to malware: people operating one of the nodes can use the device to add malware. So, users who download through Tor expose their network to malware infections.
2) Information theft: Traffic can be sniffed at the exit node, or the point where information leaves the encrypted network and becomes readable again. People operating the nodes can monitor the traffic and capture sensitive information. Note that a higher percentage of Tor transactions are fraudulent when compared to ordinary internet transactions. If you do business on Tor and you run into a problem, or if you're scammed, there may not be an easy recourse.
3) Attention of law enforcement: Using Tor may draw the attention of the NSA, FBI or other law enforcement agencies that specifically target Tor users.
Practical Applications
While it can often be difficult to locate information on the Dark Web, there are several practical applications. For example, for attorneys tasked with protecting trademarks, patents or data, the Dark Web is a potentially excellent source for determining from where counterfeit goods are originating. The Dark Web may also be useful for finding the website from which illegally obtained personal data is being sold. Attorneys should have a general understanding of the Dark Web in the event they are defending a client charged with using it to conduct illegal activities.
Summary
The Deep Web and Dark Web's beneficial information should not be overlooked. However, the Dark Web, should be approached with a level of caution due to potentially serious security and legal implications.
Amanda Sexton is the Director of Corporate Development at On The Lookout Investigation, LLC and DGR - The Source for Legal Support, winners for the past three years of the New Jersey Law Journal's annual 'Best Of' survey. She is currently president of the New Jersey Professional Process Servers Association, a board member of the Legal Vendors Network and attends local and national conferences and training sessions to stay on top of the latest techniques and regulations for both process service and private investigations, including online investigations and social media surveillance.
Roy Zur is a cyber and intelligence expert, the CEO of Cybint Ltd, an Israeli cyber company and Cybint solutions, a BARBRI company (US). Roy has over 13 years of experience in cyber and intelligence operations from the Israeli security forces, and since 2014 he has developed cyber training programs and technology for financial institutes, law firms and government agencies. Prior to his current positions, Roy received his LLM and MBA from Tel-Aviv University and served as a legal adviser in the Israeli Supreme Court, and founded the Israeli Legislation Research Center (OMEK Institute), which includes 150 researchers, who work with the Israeli parliament.