Simple and extensible Java crawler.

Configuration

The best way to learn how to tweak every single configuration is taking a look at the class net.vidageek.crawler.config.CrawlerConfiguration .