Find HTML and file links within sites.
Find HTML and file links within one or many specified URLs. Save results as text files.
Ключевые слова:
crawl my site to create a sitemap,
Web crawling or spidering,
Web crawler,
ants,
automatic indexers,
bots,
and worms,
robot,
downloaded pages to provide fast searches,
Web crawler is one type of bot,
or software agent,
list of URLs to visit,
seeds