Please see the website for the complete list of features and improvements (http://80legs.com/using.html#releases).
We have bumped up the maximum number of pages to crawl in a single crawl to 1,000,000 (still free for our current beta users). For a very broad crawl, you should expect 1M pages to take about 10-20 minutes. If your crawl is restricted or is not very broad, it can take much longer that that because of the way we throttle ourselves to prevent hitting single domains and servers too hard.
We are expecting this to be our last release before we push the first beta version containing our processDocument() functionality in 0.8.
Advertisements
0 Responses to “New Beta Release 0.76”