Overview

This is Part 2 of the Rails on Kubernetes series.

Updates

  • 2017-11-13: Updated all docker images to pull from Alpine.

Code

You can skip straight to the completed deployment here Rails app on Github.

Rails Docker Image

If you followed my previous post you should have a Rails application working with Docker Compose.

In order to prep for the Kubernetes deploy we need to package the Rails Docker image and push to a Docker registry. I'll use a public Docker Hub repo for simplicity, but you should always store your images in a private registry if your source code lives in the image.

bin/rails g task docker push_image

We create a rake task for this. It's very simple, we're grabbing the latest git revision hash and using that as an image tag. We're also attaching a latest. Note that it's recommended you use the actual commit hash in your Kube deploys to increase visibility into what's running on each pod - I'm just tagging latest for simplicity.

namespace :docker do
  desc "Push docker images to DockerHub"
  task :push_image do
    TAG = `git rev-parse --short HEAD`.strip

    puts "Building Docker image"
    sh "docker build -t tzumby/rails-app:#{TAG} ."

    IMAGE_ID = `docker images | grep tzumby\/rails-app | head -n1 | awk '{print $3}'`.strip

    puts "Tagging latest image"
    sh "docker tag #{IMAGE_ID} tzumby/rails-app:latest"

    puts "Pushing Docker image"
    sh "docker push tzumby/rails-app:#{TAG}"
    sh "docker push tzumby/rails-app:latest"

    puts "Done"
  end

end

Now we can run rake docker:push_image every time we want to push a new image version.

Kubernetes

We are ready to translate the docker-compose.yml config into Kubernetes resources. For our local development we accessed the app under the classic port 3000 but with Kubernetes we will setup an Ingress resource running Nginx and proxy requests to a Rails service.

Postgres

We'll start with our Postgres server. Here are the Kube resources we will use to create the DB:

  • Service
    • The service maps traffic from the Ingress to our pods. There are a several ways to do this via the type option: NodePort, ClusterIP, LoadBalancer or ExternalName. If we don't specify anything, Kube will create the service as ClusterIP - meaning it will only be accessible from within the cluster. This is what we want for Postgres as we will only be connecting to it from the Rails pods.
  • Secret
    • This is an object used for storing sensitive information such as passwords or TLS certificates. We will create one for our Postgres username and password.
  • Persistent Volume (PV).
    • Pod storage is ephemeral just like the container file system. Using Kube's API we can allocate space using a number of different file systems (local, EBS, CephFS, iSCSI etc.)
  • Persistent Volume Claim (PVC)
    • If the PV allocates the space, the PVC binds that resource to our Pods.
  • Replication Controller (RC)
    • The RC controls the life cycle of our Pods. We specify what image to pull, how to mount the persistent volumes, what commands to run and define ENV variables to be used by our app.

Let's create the secret to store our username and password first:

$ kubectl create secret generic db-user-pass --from-literal=password=mysecretpass
$ kubectl create secret generic db-user --from-literal=username=postgres

Here is the yaml file containing the Service, PV, PVC and RC objects:

apiVersion: v1
kind: Service
metadata:
  name: postgres
  labels:
    app: rails-kube-app
spec:
  ports:
    - port: 5432
  selector:
    app: rails-kube-app
    tier: postgres
---
kind: PersistentVolume
apiVersion: v1
metadata:
  name: postgres-pv
  labels:
    type: local
spec:
  capacity:
    storage: 4Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/tmp/data"
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: postgres-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 4Gi
---
apiVersion: v1
kind: ReplicationController
metadata:
  name: postgres
  labels:
    app: rails-kube-app
spec:
  replicas: 1
  selector:
    app: rails-kube-app
    tier: postgres
  template:
    metadata:
      name: postgres
      labels:
        app: rails-kube-app
        tier: postgres
    spec:
      volumes:
      - name: postgres-pv
        persistentVolumeClaim:
          claimName: postgres-pvc
      containers:
      - name: postgres
        image: postgres:9.6-alpine
        env:
        - name: POSTGRES_USER
          valueFrom:
            secretKeyRef:
              name: db-user
              key: username
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-user-pass
              key: password
        - name: POSTGRES_DB
          value: rails-kube-demo_production
        - name: PGDATA
          value: /var/lib/postgresql/data
        ports:
        - containerPort: 5432
        volumeMounts:
        - mountPath: "/var/lib/postgresql/data"
          name: postgres-pv

Names and labels

name - uniquely identifies the object we are creating. A Service or a Replaication Controller will use the value directly because Kube will create one one resource for each. The Pods created by the RC will append a random string to the name because the number of pods is dynamic and depends on the number of replicas we specify in a Replication Controller. If name the Postgres RC postgres this is what we'll also use in our database.yml for example - Kube's DNS will resolve that to the proper resource.

labels - are key-value pairs used to organize and group resources. For example: environment (dev, staging, production) or tier (frontend, backend, database, cache). Given those examples we could run queries against our system such as:

kubectl get pods -l environment=production,tier=frontend

Back to our Postgres deploy, let's run this to create the Kube resources:

kubectl create -f postgres.yaml

We can verify that the pod ran successfully:

kubectl get pods -w
NAME             READY     STATUS    RESTARTS   AGE
postgres-k3mqv   1/1       Running   0          1m

Redis Deployment

Next is the Redis deployment. We're not using volumes here for simplicity's sake. This could be a problem in a production environment: we will loose the memory stored Redis data if we re-deploy. If there are unprocessed jobs store in there that could be a big problem. I would recommend looking a setting up a Redis cluster with Kube for a production environment.

apiVersion: v1
kind: Service
metadata:
  name: redis
  labels:
    app: rails-kube-app
spec:
  ports:
    - port: 6379
  selector:
    app: rails-kube-app
    tier: redis
---
apiVersion: v1
kind: ReplicationController
metadata:
  name: redis
spec:
  replicas: 1
  selector:
    app: rails-kube-app
    tier: redis
  template:
    metadata:
      name: redis
      labels:
        app: rails-kube-app
        tier: redis
    spec:
      containers:
      - name: redis
        image: redis:3.2-alpine
        ports:
        - containerPort: 6379

We'll run this yaml file as well:

kubectl create -f redis.yaml

Now we should have both Postgres and Redis running:

NAME             READY     STATUS    RESTARTS   AGE
postgres-k3mqv   1/1       Running   0          5m
redis-dz20q      1/1       Running   0          1m

Rails Migrations

In my previous post I ran the migrations using a separate docker-compose service and passing it a bash script that waited for the Postgres server to go up and then ran the migrations.

Kubernetes provides a special Job resource for this.

This is pretty neat, Kube will bring this Pod up, run the command and then shut it down. Note that we're running the db:create and db:migrate with a single Job. Normally you would create a separate job for the db creation and another one for ongoing jobs like running migrations.

We already have the kube secrets for the Postgres DB. Let's create another one for the secret key base that Rails will use in production as well:

$ kubectl create secret generic secret-key-base --from-literal=secret-key-base=50dae16d7d1403e175ceb2461605b527cf87a5b18479740508395cb3f1947b12b63bad049d7d1545af4dcafa17a329be4d29c18bd63b421515e37b43ea43df64

And this is our Kube Job:

apiVersion: batch/v1
kind: Job
metadata:
  name: setup
spec:
  template:
    metadata:
      name: setup
    spec:
      containers:
      - name: setup
        image: tzumby/rails-app:latest
        args: ["rake db:create && rake db:migrate"]
        env:
        - name: DATABASE_NAME
          value: "rails-kube-demo_production"
        - name: DATABASE_URL
          value: "postgres"
        - name: DATABASE_PORT
          value: 5432
        - name: DATABASE_USER
          valueFrom:
            secretKeyRef:
              name: "db-user"
              key: "username"
        - name: DATABASE_PASSWORD
          valueFrom:
            secretKeyRef:
              name: "db-user-pass"
              key: "password"       
        - name: RAILS_APP
          value: "production"
        - name: REDIS_URL
          value: "redis"
        - name: REDIS_PORT
          value: "6379"
        - name: SECRET_KEY_BASE
          valueFrom:
            secretKeyRef:
              name: "secret-key-base"
              key: "secret-key-base"          
      restartPolicy: Never

Running the job is predictable:

kubectl create -f setup.yaml

Let's check if everything ran successfully:

kubectl get jobs
NAME      DESIRED   SUCCESSFUL   AGE
setup     1         1            1m

Rails app

We are now ready to deploy the Rails app. It's a pretty standard config with a Service and a Replication Controller. We're using the handy RAILS_LOG_TO_STDOUT environment flag to trigger Rails logging to stdout. Since the Nginx server runs on a separate server we'll have to serve the static assets from Rails. There are a few ways to go around this if we wanted Nginx to serve the assets without hitting the Rails server:

  • We could run an Nginx instance on the Rails pods and configure it to serve the assets.
  • We could customize the Nginx Ingress Controller and copy the assets there on each deploy (this one doesn't seem feasable).

For now let's configure Rails to server its own assets via RAILS_SERVE_STATIC_FILES. I think the best compromise for a production setup would be to configure the Pagespeed module in the Nginx Intress Controller - but that's a topic for another post.

apiVersion: v1
kind: Service
metadata:
  name: rails
  labels:
    app: rails-kube-app
spec:
  ports:
    - port: 3000
  selector:
    app: rails-kube-app
    tier: rails
---
apiVersion: v1
kind: ReplicationController
metadata:
  name: rails
spec:
  replicas: 1
  selector:
    app: rails-kube-app
    tier: rails
  template:
    metadata:
      name: rails
      labels:
        app: rails-kube-app
        tier: rails
    spec:
      containers:
      - name: rails
        image: tzumby/rails-app:latest
        args: ["rails s -p 3000 -b 0.0.0.0"]
        env:
        - name: DATABASE_URL
          value: "postgres"
        - name: DATABASE_NAME
          value: "rails-kube-demo_production"
        - name: DATABASE_PORT
          value: "5432"
        - name: DATABASE_USER
          valueFrom:
            secretKeyRef:
              name: "db-user"
              key: "username"
        - name: DATABASE_PASSWORD
          valueFrom:
            secretKeyRef:
              name: "db-user-pass"
              key: "password"         
        - name: RAILS_APP
          value: "production"
        - name: RAILS_LOG_TO_STDOUT
          value: "true"
        - name: RAILS_SERVE_STATIC_ASSETS
          value: "true"
        - name: REDIS_URL
          value: "redis"
        - name: REDIS_PORT
          value: "6379"
        - name: SECRET_KEY_BASE
          valueFrom:
            secretKeyRef:
              name: "secret-key-base"
              key: "secret-key-base"
        ports:
        - containerPort: 3000

The args parameter is equivalent to a Dockerfile's CMD and it will add that command as an argument to our ENTRYPOINT bash script we defined earlier.

Let's run this one as well:

kubectl create -f rails.yaml

Sidekiq

The Sidekiq deploy is very similar to Rails. The only exception is the args we pass to the ENTRYPOINT:

apiVersion: v1
kind: ReplicationController
metadata:
  name: sidekiq
spec:
  replicas: 1
  selector:
    app: rails-kube-app
    tier: sidekiq
  template:
    metadata:
      name: sidekiq
      labels:
        app: rails-kube-app
        tier: sidekiq
    spec:
      containers:
      - name: sidekiq
        image: tzumby/rails-app:latest
        args: ["sidekiq -C config/sidekiq.yml"]
        env:
        - name: DATABASE_NAME
          value: "rails-kube-demo_production"
        - name: DATABASE_PORT
          value: "5432"
        - name: DATABASE_URL
          value: "postgres"
        - name: DATABASE_USER
          valueFrom:
            secretKeyRef:
              name: "db-user"
              key: "username"
        - name: DATABASE_PASSWORD
          valueFrom:
            secretKeyRef:
              name: "db-user-pass"
              key: "password"         
        - name: RAILS_APP
          value: "production"
        - name: REDIS_URL
          value: "redis"
        - name: REDIS_PORT
          value: "6379"
        - name: SECRET_KEY_BASE
          valueFrom:
            secretKeyRef:
              name: "secret-key-base"
              key: "secret-key-base"

Note that I'm also not specifying a port number. We will not connect to this Pod directly: the sidekiq worker connects to the Redis service and listens for jobs.

Ingress Controller

We're almost ready to access our newly deployed Rails application. I created a simple Ingress resource that listens for rails.local HOST header and targets a service called rails on port 3000. This matches what I defined in my Rails kube deployment:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: rails-demo-ing
spec:
  rules:
    - host: rails.local
      http:
        paths:
          - backend:
              serviceName: rails
              servicePort: 3000
            path: /

We use the handy create command to spin this one:

kubectl create -f ingress.yaml

Now we can access our app at http://rails.local. I added an entry in my /etc/hosts file that points the minikube IP to the rails.local domain:

~ minikube ip
192.168.64.2

And my hosts file:

192.168.64.2 rails.local

If you deploy this on AWS or GCE you can have the option to spin a Load Balancer when you create an Ingress. You would then take the LB's Address and create the proper DNS records in your domain.

If you are running minikube locally, make sure you enable the nginx ingress addon by running:

minikube addons enable ingress

Then check if it's running in your kube-system namespace:

kubectl -n kube-system get po -w

You should see the ingress controller and the default backend running:

NAME                             READY     STATUS    RESTARTS   AGE
default-http-backend-cgf5r       1/1       Running   0          2m
nginx-ingress-controller-gprm2   1/1       Running   0          2m

Now you can point your browser to http://rails.local and access your newly deployed app.

What's next

In the next post we will look at a few deployments and rolling updates scenarios. We'll also use apache bench to load test our setup and see how quickly we can respond to increase in load.