Selected as a panelist for SxSWi

I just found out we’ve been selected to speak on a panel at SxSWi!  The panel is called “Big Data, Big Dreams” and is headed by Juan Sequeda of Semantic Web Austin.  The purpose of the panel is to explore how to scale applications that must handle very large amounts of data.  From the panel description:

How can we have applications that can scale with large amounts of data? Are relational databases sufficient? What other technologies are out there that can scale? This panel will talk about existing technologies that manage large amounts of data.

The 80legs crawler uses its own, custom-built data storage system, which has the very original name “Crawler Data Store”, or CDS.  We actually explored various general-purpose data storage systems before deciding to build our own.  At the same time, we do use relational databases to handle other parts of our system.  We really do have a mash-up of data storage technologies, all working together.

By the way – vote up the panel if you haven’t already!

0 Responses to “Selected as a panelist for SxSWi”

  1. Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Twitter Updates

%d bloggers like this: