80legs Subscription Plans and Free Web-Crawling

We have just updated 80legs with some exciting new changes.  Starting today, the 80legs service will be divided into 3 tiers: Basic, Plus and Premium.  Since the time we’ve launched, we’ve noticed that our customer base can be classified into 3 major groups – light, medium and heavy users.  Each of these plans is targeted to each group and designed to fulfill their specific needs.

Here are details on each plan:

Basic Plan:

  • Free to use
  • Normal crawling speed (up to 1 request/second/domain)
  • Access to 80legs Web Portal
  • 1 job running at a time
  • Up to 100K crawled pages per job
  • Low priority in 80legs job queue
  • No recurring jobs allowed

Plus Plan:

  • $99/month + crawling fees
  • Fast crawling speed (up to 5 requests/second/domain)
  • Access to 80legs Web Portal and API
  • Up to 3 jobs running at a time
  • Up to 1M crawled pages per job
  • Normal priority in 80legs job queue
  • Recurring jobs allowed

Premium Plan:

  • $299/month + crawling fees
  • Ultra-fast crawling speed (up to 10 requests/second/domain)
  • Access to 80legs Web Portal and API
  • Up to 5 jobs running at a time
  • Up to 10M crawled pages per job
  • Preferred priority in 80legs job queue
  • Recurring jobs allowed

Existing users can sign up for a plan by going to the new Subscription section in the 80legs Web Portal, where there are complete details and instructions on signing up for a plan.

We’re really excited about these changes.  Of course, the Basic Plan now enables completely free web-crawling, which until today has been completely unheard of.  The Plus and Premium Plans give heavier users the ability to set up and run more intensive crawls.

If any of our users have questions about the changes, please contact us or submit a tickets.  We’re always happy to hear from you!

Advertisements

30 Responses to “80legs Subscription Plans and Free Web-Crawling”


  1. 1 API-user December 20, 2009 at 7:48 am

    “We’re really excited about these changes.”
    Users, that want to try out your API and then decide, if they want to pay 99$/month, are of cause excited too! Now they do not have this question at all, because they simply can’t access your API without paying 99$! Great changes!

  2. 2 API. December 21, 2009 at 11:57 am

    It basically is 99 Dollar more expensive for API users and there is a limit on crawls per job now. Why punish your most important users, and reward the clients you don’t make anything from with a Free plan? Without wanting to be offensive, honestly this is a much less attractive offering than before.

    Wasn’t the goal of 80legs to make crawling accessible? With a 99 Dollar subscription for using the API, this is not the case anymore.

  3. 3 Shion Deysarkar December 21, 2009 at 1:25 pm

    The Plus and Premium plans include more benefits than just API access, such as increased crawling rates, ability to return more data, etc.

    That said, we have been getting some feedback on the API issue, and are looking into making a limited version of the API available to Basic users.

  4. 4 john December 23, 2009 at 1:19 pm

    I am trying to contact you (80legs) through your contact page, as I have a question about the terms of use agreement. But in order to leave a comment, I have to sign in, and in order to sign in I have to agree to the terms of use agreement. Kind of a chicken and egg thing.

    Can you email me via the address supplied with this comment?
    Thanks,
    John

  5. 5 venkat January 1, 2010 at 2:15 am

    How do you arrive at crawling fee?

  6. 6 Shion Deysarkar January 3, 2010 at 7:37 pm

    If you’re asking how we came up with the numbers for the prices, it’s mostly based on our analysis of demand for the crawling and the value we’ve placed on the tools the plans offer.

    If you’re asking about what the price for the crawling fees are, they are $2.00 per million pages crawled and $0.03 per CPU-hour used.

  7. 7 James Simmons January 19, 2010 at 12:50 pm

    Oh wow. I just did a 10 million page crawl for like what, $25? Now if I want to do that exact same thing again I’ll have to pay $300/mo ON TOP of the crawling fees? This is terrible, you’ve sank my battleship. I learned Java to use you guys for Pete’s sake. Do you know what that’s like to a Python developer?

    Madness.

  8. 8 CondesendingButRight January 24, 2010 at 4:33 pm

    @James Simmons Jesus you’re whiney. If learning Java was hard you cant be that good as a developer. I had to learn both Java and Python – it was not hard.

  9. 9 maroc January 25, 2010 at 3:08 pm

    Existing users can sign up for a plan by going to the new Subscription section in the 80legs Web Portal, where there are complete details and instructions on signing up for a plan.

  10. 10 James Simmons February 10, 2010 at 3:42 am

    @CondesendingButRight: I find it hard to believe that anyone not working for 80legs would be so upset over my comment to anonymously insult me. Or are you really that pissed that I despise Java? Hard to believe, then again, you’re a Java developer.

    Comments on this blog are moderated and do not appear until an 80legs employee approves them. Perhaps they need to learn discretion, or perhaps they think future customers will see this blog post and say, “wow I really want to use their service.”

    PS: I’m posting this comment with full details visible, and I’m certainly not afraid to do so.

  11. 11 Shion Deysarkar February 10, 2010 at 10:04 am

    1. Comments are automatically posted but we can delete them if we want.

    2. We generally believe in letting people express what they want.

    @James: We realize that our new pricing is not suited to everyone, but we’re sticking with it as it’s helped our operations and business in a number of ways.

  12. 12 James Simmons February 11, 2010 at 2:17 am

    Thanks for the quick response Shion. I can certainly understand that, and I look forward to continue using your service.

  13. 13 NotHappy February 25, 2010 at 11:13 pm

    Is there a plan to stop your spider from attacking my web site with hundreds of sessions per day from hundreds of different IP address?

    You’re the small web site owner’s worst nightmare.. you’re the plague!

    I have 57 sessions on my site now, 10 from real users and 47 from your spider… thats not cool!

  14. 14 Shion February 25, 2010 at 11:29 pm

    We provide several options for a webmaster to stop their site from crawling their site: http://80legs.pbworks.com/FAQ#Webmasters

  15. 15 Mindy Franklin March 9, 2010 at 12:27 pm

    Please forward to the President or VP of Marketing:

    I am writing to introduce myself and SS | PR.

    We have a deep Rolodex of editorial contacts with trades and national magazines in the mainstream media. My firm has been in business for over 25 years and has strong editorial relationships. Our agency can be instrumental in generating features, industry trend stories, etc.

    If you would like to talk or schedule a meeting please let me know.

    Regards,
    Mindy

  16. 16 GWIZIT April 3, 2010 at 10:05 am

    Reply Too:NotHappy

    Yes it’s the plague!

    Shion says they provide options to stop their spider: http://80legs.pbworks.com/FAQ#Webmasters

    Yet – I have done exactly what above link says, and it has encouraged even more spider activity.

    Getting attacked by 008/0.83; http://www.80legs.com/spider.html from all angles – What a JOKE!

  17. 17 Shion Deysarkar April 3, 2010 at 1:11 pm

    Gwizit,

    It takes some time for the crawl requests to go down due to the distributed nature of our system. We have internally blocked crawl requests as well.

    – Shion

  18. 18 algdfa April 3, 2010 at 8:32 pm

    Is there a plan to stop your spider from attacking my web site with hundreds of sessions per day from hundreds of different IP address?

    You’re the small web site owner’s worst nightmare.. you’re the plague!

    I have 57 sessions on my site now, 10 from real users and 47 from your spider… thats not cool!

  19. 19 GWIZIT April 6, 2010 at 1:04 am

    Appreciate the reply & placing internal block crawl requests.

    I probably wouldn’t mind having more spider activity if my midget server could handle it.

    O-Well … Beats paying for hosting & alot more challenging this way.

    Thanks Again

  20. 20 edithann April 12, 2010 at 8:29 am

    I don’t want you anywhere near my website. We have you blocked in our robots.txt file but you choose to ignore that. Get off my website and stay off. http://www.northcountrygazette.org

    Host: 96.246.238.92
    /robots.txt
    Http Code: 200 Date: Apr 12 09:15:47 Http Version: HTTP/1.1 Size in Bytes: 598
    Referer: –
    Agent: Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/spider.html;) Gecko/2008032620

    /2008/08/14/security_blackout/
    Http Code: 200 Date: Apr 12 09:15:47 Http Version: HTTP/1.1 Size in Bytes: 73463
    Referer: –
    Agent: Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/spider.html;) Gecko/2008032620

  21. 21 Shion Deysarkar April 12, 2010 at 9:51 am

    There was a problem with our block for your site; it should be fixed now.

  22. 22 Marsha Friedman April 22, 2010 at 8:35 am

    Interesting back and forth. Think we will skip this one.

  23. 23 Expart September 24, 2011 at 5:54 am

    I am trying to contact you (80legs) through your contact page, as I have a question about the terms of use agreement. But in order to leave a comment, I have to sign in, and in order to sign in I have to agree to the terms of use agreement. Kind of a chicken and egg thing.Escort Services Chelsea

  24. 25 tuputamadre May 22, 2012 at 7:33 am

    hijos de puta, vais a spammear a tu puta madre

  25. 26 Juan November 13, 2012 at 1:16 pm

    Hello, for all time i used to check blog
    posts here early in the morning, for the reason that i enjoy to learn more and more.

  26. 27 crear facebook May 9, 2013 at 3:54 am

    What’s up i am kavin, its my first occasion to commenting anyplace, when i read this paragraph i thought i could also make comment due to this brilliant piece of writing.

  27. 28 Steven August 7, 2013 at 7:38 pm

    Aw, this was a very good post. Spending some time and actual effort
    to create a superb article but what can I say I procrastinate a whole
    lot and don’t seem to get nearly anything done.


  1. 1 80Legs, 50k Computers and a Web Crawler | Tech Startups Trackback on December 22, 2009 at 5:01 pm
  2. 2 80Legs Inc. offers free crawling service as part of the Subscription Plans « Trackback on December 23, 2009 at 6:05 am

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s





%d bloggers like this: