Searching The Surface Web I: Google, Yahoo! And More

What the heck is the surface web? It refers to the thin top layer of the Net that the major search engines are able to spider. So, when you’re working on the surface web, that is, searching only Google, or Yahoo, or MSN, you are only “scratching the surface” so to speak.

<A HREF=”http://searchthehiddenweb.com” TARGET=”_blank”>There’s a lot more available to you on the hidden web</A>… but first, are you sure you’re getting everything you can out of the surface web?

Here are some basic ways you can better search the Surface Web. If you’re doing market research for example,using something like Google TRends, it’s a good idea to start here anyway, since the vast number of online users are using the “big three”.

Once you learn some of these search techniques, you can apply them to the Hidden Web too. First, let’s examine some of the major search engines out there right now.

First is of course is Google. They are growing exponentially, every day indexing more pages, scanning in more documents, and generally sucking up everything they can on the Web. Google’s even trying to add ways to dig through the “deep web” too. (Stay tuned, we’ll keep you up to date on these new features.)

Next up are Yahoo! and MSN’s Live Search as the next biggest competitors. Following those three, there are many others, such as Ask!, Alta Vista, and others, however, Google, Yahoo and MSN Live have acquired and now are the power behind many of the search engines that started out early on the Web.

These three together account for the vast majority of searches conducted on the Web, and webmasters tailor their pages to show up in these results. But sometimes you will get different results for your search depending on which engine you use, and what search terms you use.

Next post, we’ll talk about some basic search terms.

What is the Hidden Web?

<A HREF=”http://searchthehiddenweb.com” TARGET=”_self”>The “hidden web” contains the fastest growing storehouse of information on the Internet</A>. Think you’re getting everything you need when you search Google? We hear how massive it is but it is only about 1/10th of the full amount of data available, for free, on the Internet.

Google and Yahoo, so-called “surface web” engines, are big, yes, but their indexing software (called crawlers, spiders, bots, etc.) is unable to access the vast majority of websites and databases.

Some examples of pages these spiders can’t reach are: databases, dynamically created pages (using Java or other software), newspaper archives, any site that requires a login, certain file formats that are not index-friendly, and so on.

You may have heard of the “deep web” or the “hidden web” – these are the names given to the 99% of information that is available on the Internet but not easily found using the major search engines. It has by far the greatest amount of best quality information available.

And guess what? Luckily for the smart researcher, marketer, or product developer, nearly 95% of this content is accessible by the public, for free!

There is an amazing wealth of ideas here, for new hot products, for high-quality market research and analyses by academic experts, massive stores of government information, both national and international, and tools for finding detailed information on competitors – both their businesses and their personal information, scary as that may sound. And that’s just the beginning; there are materials available to the public for the taking, in the public domain, which you can use to create new and valuable content for your customers.

If you have an ounce of creativity, just try scanning these sources for one of your market niches, and you will begin to see the possibilities immediately. It’s an instant brainstorm!