Multi-port Containers Behind Google's L7 Load Balancer

I know, that is the worst title for a blog post, but it's exactly what I spent hours searching for recently.

If you want to skip over the reasons as to why this is a problem and just want to get to the solution, skip down to The Solution

I spent my Christmas break taking some time, as most engineers do, looking at some cool new tools on GitHub, primarily Cog. Cog is chatbot built on Elixir that shifts away from some of the existing chatbots in the world. Rather than a conversation approach to being a bot, like Hubot and Lita, Cog feels more like talking to a *nix command line.

One of the oddities about setting up Cog is that it runs as a docker container with 3 APIs on 3 different ports (4000, 4001, 4002). This is all well and good, except that if using Kubernetes with an Ingress (which utilizes Google Cloud's L7 Load Balancer), there is a serious issue. To start let's look at a standard deployment of Cog to Kubernetes.

Cog on Kubernetes

The Cog container has 3 ports which are exposed through a service (using NodePorts). Traffic from the outside world is fed to the service through a single ingress. All is good and happy, right? Wrong. As I said previously, Google Container Engine uses the L7 Load Balancer for ingress'. The GCE LB uses forwarding rules (of which we have 3) to route requests from the outside world to one of the 3 backend (services) that Cog exposes. This is where the trouble starts.

GCE LB's will not serve traffic to backends which are found to be unhealthy. How does the LB determine if a backend is unhealth? Easy, a health check pings the root path of the backend once a minute. Spot the problem? What if a backend doesn't respond with an HTTP 200 on the root path? The unfortunate answer when using Google Container Engine is: don't use an ingress. Deployments can define a readiness probe and a liveness probe, but you can only define 1 per container in a deployment. This means that for Cog, which has 3 ports and thus 3 different backends, we can't define the probes for all 3.

There is a secondary issue that we need to resolve which is this. Let's say that our ingress uses a forwarding rule to forward all traffic to to the triggers API on port 4001. When a request comes in the GCE LB forwards the request, but it leaves the request path as /triggers/v1/trigger/.... The trigger API's router expects paths to be /v1/trigger/..., so all requests end up as a 404.

The solution

As with most things, there's more than one way to shake a carbuncle. I used Nginx to solve my issue, because I know and love Nginx. I've read about other solutions using Linkerd and other tools.

Nginx is more than a webserver. In fact, where it shines the most is when it simply acts as a reverse proxy; taking in requests and proxying them to backends based on forwarding rules. Sound really familiar? What if we routed all traffic from the outside world to Nginx and then let Nginx decide where it needs to go next? The Nginx config for all of this is pretty simply actually:

upstream cog_base     { cog:4000; }  
upstream cog_triggers { cog:4001; }  
upstream cog_service  { cog:4002; }

server {  
  listen 80 default_server;

  proxy_set_header X-Real_IP $remote_addr;
  proxy_set_header X-Forwarded-Host $host;
  proxy_set_header X-Forwarded-Server $host;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

  location / {
    proxy_pass http://cog_base;
  location /trigger {
    proxy_pass http://cog_triggers/;
  location /service {
    proxy_pass http://cog_service/;
  location /healthz {
    add_header Content-Type text/plain;
    return 200 "All good";

The upstream blocks define where our 3 services live. According to this, I named the service for Cog "cog", so the host is cog along with the port for each API. The proxy_set_header commands are just adding information to the request so that the Cog API knows where the original request came from and what was requested, since Nginx is going to make some changes to it. The proxy_pass commands actually pass the request to the upstream backends that will handle the request. If you notice, I don't even need to rewrite the request path, because Nginx strips out the forwarding rule when it sees a URI (the last /) in the address passed to proxy_pass.

How do we use the Nginx config? Rather than creating our own Nginx image that we bundle the config into, we can use a Kubernetes config map and inject the config into the public Nginx image from the Docker Hub. Creating the config map is pretty straightforward using the kubectl command. Copy the Nginx config into a file and import into a config map with:

$ kubectl create cm nginx-config --from-file ./cog.conf

Config maps can be mounted into a deployment as a volume. Our Nginx deployment with the config map mounts looks like this:

apiVersion: extensions/v1beta1  
kind: Deployment  
  name: cog-nginx
    app: cog
  replicas: 1
        app: cog-nginx
      - name: nginx
        image: "nginx:1.11"
        imagePullPolicy: IfNotPresent
        - containerPort: 80
        - mountPath: /etc/nginx/conf.d
          name: config
      - name: config
          name: nginx-config
          - key: cog.conf
            path: cog.conf

What the deployment does is mount the cog.conf file into /etc/nginx/conf.d which is automatically loaded when Nginx starts up. Because there's only 1 port on the deployment, and if we don't need SSL, publishing the deployment to the public is easy with a single kubectl command:

$ kubectl expose deploy nginx --port 80 --target-port 80 --type LoadBalancer

If you want to put an SSL in front of your application you can either load it into Nginx with the config map and Kubernetes secrets, or you can create a Kubernetes Ingress to route traffic into Nginx using.

If you want to skip over the hassle of getting Cog running from scratch on Kubernetes, use @ohaiwalt's Cog Helm chart to build the environment with 1 command. Note: Currently there is a PR to add the Nginx deployment to handle inbound traffic. You can use my fork of the project until the PR is merged.

If you want to see the power of Cog and chatops in your organization, drop me a line.

Dave Long

Read more posts by this author.

Subscribe to Dave Long

Get the latest posts delivered right to your inbox.

or subscribe via RSS with Feedly!