• https://github.com/scumola/crawler-fast A multi-threadded web crawler written in C that doesn't care about robots.txt