With over 400 million active websites online, and more websites going up by the minute, search engines, like Google and Bing have a lot of work on their hands. They need to crawl information spread across hundreds of billions of…