I just found out we’ve been selected to speak on a panel at SxSWi! The panel is called “Big Data, Big Dreams” and is headed by Juan Sequeda of Semantic Web Austin. The purpose of the panel is to explore how to scale applications that must handle very large amounts of data. From the panel description:
How can we have applications that can scale with large amounts of data? Are relational databases sufficient? What other technologies are out there that can scale? This panel will talk about existing technologies that manage large amounts of data.
The 80legs crawler uses its own, custom-built data storage system, which has the very original name “Crawler Data Store”, or CDS. We actually explored various general-purpose data storage systems before deciding to build our own. At the same time, we do use relational databases to handle other parts of our system. We really do have a mash-up of data storage technologies, all working together.
By the way – vote up the panel if you haven’t already!