Crawl Packages: Aggregate Website Data in a Few Clicks

We’re excited to announce a new service at 80legs: Crawl Packages.

What crawl packages are:

Crawl packages are pre-configured crawls that you can access and run in just a few clicks.

For a specific website or group of websites, we’ve designed and setup an 80legs crawl, along with custom data extractors, to crawl that site and extract all the interesting information from it.  These are crawls you could have setup yourself, but we’ve gone ahead and done all the work for you.

Types of crawl packages available:

We’re currently offering crawl packages for social networks, retail/shopping sites and business directories.  We’ll be expanding our offerings to include other websites as well.  Initial plans include crawling blogs (and their comments), semantic annotation feeds of various websites, and so on.

Results & Pricing:

Most crawl packages will cost $350 per month and produce 10 – 20 million records per month.  The type of records produced depend on the crawl package.  Social network packages produce publicly-available profiles, Retail packages produce product listings, etc.

Open Data:

We realize that the availability of crawl packages will raise some concerns over what data should be crawled and shouldn’t.  We only crawl publicly-available Web data.  We don’t crawl private data and have no interest in that.

What we are interested in is what our users can do with Web data that is more accessible.  Since our launch, we’ve seen many startups come to us asking for large amounts of Web data so that they can create additional value on top of that data.  They want to do interesting things like provide new insight into how people connect with one another, create CPIs of online product invetory, and more.  We want to make that possible, and crawl packages are a step in that direction.

13 Responses to “Crawl Packages: Aggregate Website Data in a Few Clicks”

  1. 1 RAJAT KUMAR MUND April 25, 2010 at 11:09 pm

    I am a student of class VIII in O.F.School,dist:-Bolangir (767770)
    and also I am a athletic champion.

  2. 2 satish May 5, 2010 at 1:28 am


  3. 3 Richard Bennett August 3, 2010 at 2:43 pm

    I am intersted in purchasing your pre configured crawl

    I gather data from association web site business & Trade , I am interested in directories of members eg name,address,tel
    e maail,URL

    I would like to try program as soon as possible
    Please e mail me where I can call

    my cell is 201 245 1600

    I am in NJ

  4. 4 Richard Bennett August 3, 2010 at 3:01 pm

    contact me
    I am very interested in an immediate purchase
    Richard Bennett
    cell 201 245 1600

  5. 5 Roger Hadden September 5, 2012 at 10:42 am

    Please stop crawling our site – you are causing a virtual DoS and it’s affecting our business.

  6. 6 online business marketing services August 6, 2013 at 5:36 am

    By doing online discussion forums linked to their particular business and
    contributing valuable content and insights, entrepreneurs can increase
    exposure for their brand and business. Perform
    an internet search with the organization name and the words
    “rip off,” “scam,” or “complaints” to get more information.
    Most from the sites and services are exceptionally user-friendly if additional support is necessary, online tutorials and assistance are bountiful.

  7. 7 search engine optimization an hour a day 3rd edition pdf August 30, 2013 at 1:24 pm

    If attracting visitors to your internet site is not
    just a cause of major concern when you already get them regularly, find
    out what number of of them are becoming your clients.
    Not all people possess the same idea in terms of using of SEO software.
    Which is besides like handing over an auto mechanic to take care of your automobile.

  8. 8 online marketing September 5, 2013 at 2:17 am

    Have you ever considered about adding a little bit more than just youur articles?

    I mean, what you say is valuable and everything.

    But just imagine if you added some great images or videos to give your posts more, “pop”!
    Yoour content iis excellent but with images and video
    clips, this site could definitely be one off the
    very best in its niche. Excelkent blog!

  9. 9 Law offcie August 4, 2014 at 9:49 am

    Internet marketing has become a complicated
    discipline that involves a great deal of theoretical knowledge in combination with applied techniques.
    From customer satisfaction to the importance of certain aspects of your company, this is a
    very versatile option to use in your online marketing questionnaires.
    A skilled SEO campaigner knows to ease out a complex site
    and re-structure it to suit real users as well as
    search engines.

  10. 10 Francesco December 5, 2014 at 12:11 am

    You may distinguish infection by a noticeable discoloration of the nails (which may
    vary from white, yellow, to black), irritation and
    redness of the surrounding skin, foul odor, thickening and formation of cracks on the outer layer.
    With proper foot care Windsor our feet will not only look good, but they will feel
    great. Surgery can range from ingrown toenails, to bunion and hammertoe correction, to amputations of infected parts of the foot.

  11. 11 Sanford January 13, 2015 at 11:34 pm

    The place has all the facilities for you to have a rocking time and that.
    I’ve read other articles that claim waiting until you arrive
    at your destination will net you the lowest rate.
    Secure Your Valuables Never leave your valuables ‘ money, passport,
    license, jewelries’ unattended.

  12. 12 Mikel March 5, 2015 at 11:24 pm

    A good incentive for a finance web site is a free
    calculator for tax returns. Since all profitable businesses need to develop
    an effective plan for promoting and delivering their goods this is another area you need to consider.
    Whatever the challenge, there is always a way to achieve
    the goal.

  13. 13 Consuelo April 2, 2015 at 10:06 am

    I am truly glad to read this website posts which contains tons
    of useful data, thanks for providing such information.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Twitter Updates

%d bloggers like this: