What it is
Sarama is a pure-Go client for Apache Kafka, supporting producers, consumer groups, offset management and the admin API with no cgo dependency.
Sarama offers a synchronous producer, an asynchronous producer, and consumer groups that handle partition rebalancing. Configuration is explicit and extensive.
Installation
go get github.com/IBM/saramaGetting started
The smallest useful thing you can do with it, and what each part means.
go
config := sarama.NewConfig()
config.Producer.RequiredAcks = sarama.WaitForAll // durability over latency
config.Producer.Retry.Max = 5
config.Producer.Return.Successes = true // required for SyncProducer
config.Producer.Idempotent = true // no duplicates on retry
config.Net.MaxOpenRequests = 1 // required with Idempotent
producer, err := sarama.NewSyncProducer(brokers, config)
defer producer.Close()
partition, offset, err := producer.SendMessage(&sarama.ProducerMessage{
Topic: "orders",
Key: sarama.StringEncoder(orderID), // same key -> same partition -> ordered
Value: sarama.ByteEncoder(payload),
})Advanced usage
Where the library earns its place over a simpler alternative.
go
type handler struct{}
func (handler) Setup(sarama.ConsumerGroupSession) error { return nil }
func (handler) Cleanup(sarama.ConsumerGroupSession) error { return nil }
func (handler) ConsumeClaim(sess sarama.ConsumerGroupSession,
claim sarama.ConsumerGroupClaim) error {
for msg := range claim.Messages() {
if err := process(msg.Value); err != nil {
return err // triggers a rebalance; the offset is not marked
}
sess.MarkMessage(msg, "") // commit only after successful processing
}
return nil
}
for {
if err := group.Consume(ctx, []string{"orders"}, handler{}); err != nil {
return err
}
if ctx.Err() != nil { return nil } // Consume returns on every rebalance
}Errors and fixes
The failures you are most likely to hit, and what actually resolves them.
- kafka: invalid configuration
- Call config.Validate() to see which field is wrong — idempotent producers in particular constrain acks and MaxOpenRequests.
- Consumer stops after a rebalance
- Consume returned normally. Re-enter it in a loop and only exit when the context is cancelled.
Best practices
- Mark offsets after processing, not before, unless you can tolerate message loss.
- Always wrap group.Consume in a loop; it returns on every rebalance.
- Use a message key when ordering within an entity matters.
- Set Producer.Return.Successes for SyncProducer or it will block forever.
Background
Why it exists, and what it was reacting to.
Originally from Shopify and now community maintained, Sarama was the first production-grade Kafka client for Go and remains the most widely deployed.