Soft Maximum - best free downloads
dirs

RafaBot
... website from a starting URL, search engine results or web dirs. RafaBot is a high-speed, multi-threading, large scale web spidering ... website from a starting URL, search engine results or web dirs and able to follow external links. It can process either a single website ...

1