Testing out some improvements to our crawling back-end

Some of our users may have noticed we recently lowered the limit on # of pages crawled per job to just 10,000.  This is a temporary measure while we test out some major improvements to the crawling back-end.  If the tests go well, the limit will be pushed back up to 1 million (probably 10 million, actually) by the end of this week or early next week.  We think this will be the last major upgrade to our back-end.  It should allow 80legs to more-easily scale into the billions of pages crawled per job.

So please bear with us as we continue to work on the service – thanks so much!!

1 Response to “Testing out some improvements to our crawling back-end”

  1. 1 reseller iptv May 29, 2019 at 8:59 pm

    When I initially left a comment I seem to have
    clicked the -Notify me when new comments are added- checkbox and from
    now on every time a comment is added I recieve 4 emails with the same comment.
    There has to be an easy method you can remove me
    from that service? Cheers!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Twitter Updates

%d bloggers like this: