Simple and extensible Java crawler.
ContentVisitor
The only code you'll usually need to write is a implementation of net.vidageek.crawler.ContentVisitor . This interface provides two methods:- void visit(Page page) : This method is called on each page found on the site.
- void onError(Url errorUrl, Status statusError) : This method is called on pages that failed to respond with a Status.OK or Status.REDIRECTION for any reason.