June 9, 2010

Google is serving up fresh, Caffeinated search results.

The web giant says it will generate 50% fresher search results, thanks to its new web indexing system called Caffeine.

When you Google something, you're not searching the Internet – you're searching Google's index of the Internet, which until Tuesday night, was organized in layers.

Under the old index, when you did a search, Google would scan the various layers of its index, prioritized by importance. It would search one group of high priority sites, then another less prioritized group of sites, and so on. Each layer was updated on a schedule.

“Our old index had several layers, some of which were refreshed at a faster rate than others; the main layer would update every couple of weeks. To refresh a layer of the old index, we would analyze the entire web, which meant there was a significant delay between when we found a page and made it available to you,” Carrie Grimes, a Google software engineer, explained on the Official Google Blog.

Basically, Google was sometimes missing the latest, breaking search results - tweets, recently updated news items, etc.

“Content on the web is blossoming. It's growing not just in size and numbers but with the advent of video, images, news and real-time updates, the average webpage is richer and more complex. In addition, people's expectations for search are higher than they used to be. Searchers want to find the latest relevant content and publishers expect to be found the instant they publish,” said Grimes.

That's where Caffeine comes in. The new index is constantly updating, and it searches all sites at once.

Grimes offers some numbers to illustrate Caffeine's effectiveness.

“Every second, Caffeine processes hundreds of thousands of pages in parallel. If this were a pile of paper it would grow three miles taller every second. Caffeine takes up nearly 100 million gigabytes of storage in one database and adds new information at a rate of hundreds of thousands of gigabytes per day. You would need 625,000 of the largest iPods to store that much information; if these were stacked end-to-end they would go for more than 40 miles.”