Go Kafka, Cloud Karafka and Heroku

A sample for networking kafka in docker using targeted builds.

Deploying to heroku free stack!!

Getting Started

First off, what is all this stuff? Apache kafka is a messaging system with Producers, Consumers and Brokers that publish and subscribe to Topics. There's a little more complexity to it, but you can read a lot more here https://kafka.apache.org/intro

What are we Building

This is just an example of building and crucially, deploying a simple consumer and producer. They are so simple in fact, that were just copied from the official go kafka examples. But this tutorial is more about getting your Kafka deployed, in a free testing environment, and doing that with docker too!

Sample Code

The sample code is available here: https://github.com/simonireilly/go-kafka-docker

The code includes some security settings for rdkafka. It implements the SASL security method here:

	p, err := kafka.NewProducer(&kafka.ConfigMap{
		"bootstrap.servers": os.Getenv("KAFKA_BROKERS"),
		"sasl.username":     os.Getenv("KAFKA_USERNAME"),
		"sasl.password":     os.Getenv("KAFKA_PASSWORD"),
		"security.protocol": "SASL_SSL",
		"sasl.mechanisms":   "SCRAM-SHA-256",
	})

Docker File(s)

So there are multiple dockerfiles, but they are all the same ¯\_(ツ)_/¯ There is a limitation in the heroku.yml (see below) that the dockerfile must be in the same directory as the code.

FROM golang:1.13-alpine3.10 AS builder
# Build with Go
WORKDIR /go/src/github.com/simonireilly/go-kafka-tester
RUN apk add librdkafka-dev build-base
COPY . .# Tell go, to build binaries, as a file called ./main
RUN go build -a -o ./main
# Run main at entry in the builder
ENTRYPOINT ["./main"]
# Production is cleaner, smaller image, no build deps
FROM alpine:edge
# We'll just need librdkafka to actually use kafka
WORKDIR /rootCOPY --from=builder /go/src/github.com/simonireilly/go-kafka-tester/main .
RUN apk --no-cache librdkafka
# Enter at main again
ENTRYPOINT [ "./main" ]

Heroku YAML

The heroku yaml file takes care of prvisioning an instance of Kafka from CloudKarafka. This file will also handle building our consumer and producer, publishing them to a docker registry, handy!

setup:
  addons:
    - plan: cloudkarafka:ducky
    as: KAFKAbuild:
  docker:
    consumer: ./consumer/Dockerfile
    producer: ./producer/Dockerfile

Deploy

To deploy with heroku run:

heroku create your-app-name --manifest

git push heroku master

That's it!

Kafka is included in the heroku.yml and everything is configured to work out of the box.