Friday, September 02, 2005

Technorati cosmos search

For those of you that don’t know, a Technorati cosmos search shows blogs that are linking to a particular blog address.  I have posted recently about Technorati sucking, and so I was interested to see this post by Joy Ito that links through to this post by Dave Silfry with Technorati.  From Dave’s post:

The blogosphere has been growing at an explosive rate - Technorati is now indexing over 16 million blogs, with about 100,000 new blogs created every day. And there's over 1.4 Million new posts every day, and about 22% of those posts are from spam or fake blogs, which means that even after we pull out the spam and fake blogs from the indexes, we are dealing with about 1.2 Million posts each day.

We just weren't expecting that kind of sudden growth, both on the posting side and also on the search side, and frankly we didn't plan well enough to handle the load. We've been adding new machines to our datacenter, - over 400 now - and more coming each week, and we've been fixing bugs and making performance enhancements on the web site as well.

. . . Cosmos search (or URL search) is still being worked on, and is often timing out under the increased load. Unfortunately this is also one of the searches that bloggers find most compelling, as it helps you to all know who is linking to your blog, and it is the very first type of search that Technorati made available, so it is near and dear to our hearts. Everyone here also uses it every day, so it really sucks when it isn't working right.

Once we got our keyword search infrastructure back on track, our infrastructure team has been working 100% on fixing Cosmos search. Our current plan is to have Cosmos search back up and running by the end of September, and you'll see incremental improvement throughout the coming month. I'll keep you informed on progress of this critical project. As the project progresses throughout the month, you'll be able to see progress because you'll see fewer and fewer error messages when you do a URL search as September progresses.  

I guess I’ve been waiting this long for a Cosmos search that I can wait through September, but I do have to say that I haven’t had a cosmos search return results for the last 3 days.

No comments: