This approach also added 15.6 times more embedded resources than Heritrix to the crawl frontier, but at a crawl rate that was 38.9 times slower than simply ...

This work proposes a method of discovering and archiving deferred representations and their descendants (representation states) that are only reachable ...

Jun 19, 2017 · This approach also added 15.6 times more embedded resources than Heritrix to the crawl frontier, but at a crawl rate that was 38.9 times slower ...

Archival Crawlers and JavaScript: Discover More Stuff but Crawl More Slowly ... but at a crawl rate that was 38.9 times slower than simply using Heritrix.

ABSTRACT. The web is today's primary publication medium, making web archiving an important activity for historical and ana- lytical purposes.

Archival Crawlers and JavaScript: Discover More Stuff but Crawl More Slowly ... Most crawlers evade Javascript™ links, implying that Web pages using forms ...

Conclusions · Crawling all descendants is 38.9 times slower than crawling with only Heritrix, but adds 15.60 times more data to the crawl frontier than Heritrix ...

This architecture would mean that the overall crawl would proceed more slowly, particularly at first. However, based on our experience so far, we do not ...

Nelson, “Archival Crawlers and JavaScript: Discover More Stuff but Crawl More Slowly,” In Proceedings of the ACM/IEEE Joint Conference on Digital Libraries ( ...

The archival difficulty is based on the use of client-side technologies (e.g., JavaScript) to change the client-side state of a representation after it has ...