What it is
Colly is a fast web scraping framework for Go with callback-based parsing, automatic rate limiting, caching, cookie handling and parallel collectors.
A Collector visits URLs and fires callbacks on HTML elements matching CSS selectors. Limits control parallelism and delay per domain.
Installation
go get -u github.com/gocolly/colly/v2Getting started
The smallest useful thing you can do with it, and what each part means.
go
c := colly.NewCollector(
colly.AllowedDomains("example.com"),
colly.MaxDepth(2),
colly.CacheDir("./cache"), // avoid re-fetching during development
)
c.Limit(&colly.LimitRule{
DomainGlob: "*",
Parallelism: 2,
Delay: time.Second, // be a good citizen
})
c.OnHTML("a.book-link[href]", func(e *colly.HTMLElement) {
e.Request.Visit(e.Attr("href")) // resolves relative URLs
})
c.OnHTML("h1.title", func(e *colly.HTMLElement) {
fmt.Println(e.Text, e.Request.URL)
})
c.OnError(func(r *colly.Response, err error) {
log.Printf("failed %s: %v", r.Request.URL, err)
})
c.Visit("https://example.com/books")
c.Wait()Advanced usage
Where the library earns its place over a simpler alternative.
go
type Book struct {
Title string `selector:"h1.title"`
Author string `selector:".author"`
Tags []string `selector:".tag"`
}
c.OnHTML(".book-detail", func(e *colly.HTMLElement) {
var book Book
if err := e.Unmarshal(&book); err != nil {
return
}
results = append(results, book)
})Errors and fixes
The failures you are most likely to hit, and what actually resolves them.
- Callbacks never fire
- The domain is outside AllowedDomains, or the selector does not match. Print the response body to confirm what was actually returned.
- Content is missing from the HTML
- The page renders it with JavaScript. Colly only fetches HTML — use a headless browser such as chromedp instead.
Best practices
- Always set a LimitRule with a delay — unthrottled scraping gets blocked and is antisocial.
- Respect robots.txt and the site's terms; Colly can parse it for you.
- Use CacheDir during development so you are not refetching on every run.
- Set a descriptive UserAgent so site owners can identify and contact you.
Background
Why it exists, and what it was reacting to.
Colly gave Go an equivalent to Python's Scrapy: a scraper that handles concurrency, politeness and retries rather than leaving them to the caller.