0.9 released! 10M page crawls, API, easy-to-use interface and more!

We just pushed out version 0.9, which is a big, big update to the system.  This release includes several upgrades to our back-end architecture (allowing larger jobs), a Java API (allowing programmatic access), an easy-to-use job form (allowing easier access), and a bunch of other cool things!

Here’s a list of the specific features:

  • Large crawls are now supported.  Crawl up to 10 million pages per job!
  • The API is officially released.  Submit jobs, download results and much more using Java.
  • A much easier-to-use job form.  We realized the old job form was a bit clunky.  The new one is much easier to understand.
  • To go along with the new job form, we’ve updated the entire portal to be easier to navigate and use.
  • You can now load in external JARs into your 80Apps.  This lets developers use third-party code more easily.
  • Several improvements to the crawler, including:
    • Options to select your type of crawl.  Choose among fast, comprehensive, and breadth-first.
    • Crawler now crawls https:// pages.
    • Crawler tries to fetch a page more than once before giving up.

Since we just released 0.9, I suppose that technically makes us 0.1 from a beta exit!  Some of the upcoming features are:

  • Finalizing the payment system in preparation for beta exit and charging actual money.
  • Providing useful default 80Apps for all users (this is also in preparation for the app store model we’ll be pursuing).

See full release log details at http://80legs.pbworks.com/Release-Log.

1 Response to “0.9 released! 10M page crawls, API, easy-to-use interface and more!”

  1. 1 Phoenix October 7, 2013 at 4:05 pm

    Any chance of getting a php based API? Also your blog doesn’t work on blog.80legs.com giving a 404.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Twitter Updates

%d bloggers like this: