The only code you'll usually need to write is a implementation of net.vidageek.crawler.ContentVisitor .
This interface provides two methods:
- void visit(Page page) : This method is called on each page found on the site.
- void onError(Url errorUrl, Status statusError) : This method is called on pages that failed to
respond with a Status.OK or Status.REDIRECTION for any reason.