Deploy application in Kubernetes cluster using GoCD with helm

This blog will explain the deployment of the micro-service applications using the GoCD with Helm deployment manager in the K8s cluster.

Blog consider that user is having some background knowledge of below technology

1. Kubernetes [https://www.talentica.com/blogs/kubernetes-introduction-architecture-overview/]
2. Micro-services
3. CI/CD deployments

GOCD:

GoCD is an opensource tool used in the software development process. It supports continuous integration and continuous delivery (CI/CD) software lifecycle.

GoCD Architecture

• GoCD follows the server and agent approach.

• GoCD server runs 3 components to achieve CI/CD of an application:

– material update sub-system

– scheduling sub-system

– work assignment sub-system.

• The material update sub-system and scheduling sub-system work to initiate the build/deployment if there is a new commit or at a fixed time interval respectively.
• Work assignment sub-system gets activated when agent connects to server
• The agent pulls work from the server instead of server sending work to an agent, and it’s responsible for sending the status back to server.
• Agent gets a job allocated to it if it matches the constraints
• The server will assign jobs to agents based on agent status. The agent is responsible for sharing the agent ping and agent state to the server. If the server finds the agent in idle state it assigns the job to it.

Concepts in GoCD

Tasks:

This is a single entity consider as a single command.

Eg. sh –c mvn install

Fig. Task

Job:

Job consist of multiple tasks. Bended to run sequentially. The tasks are run independently will not get affected by other tasks definitions. Job will stop execution if previous tasks failed tasks queue.

Fig. Job

Stage:

stage consist of multiple jobs. The jobs are executed in parallel. if a single job failed then also the other job will complete its tasks independently.

Fig. Stage

Pipeline:

pipeline consist of multiple stages. The stages are run in sequence.

Fig. Pipeline

Materials:

GoCD supports Git, SVN, Mercurial repository, artifact publish by the GoCD pipeline.

Trigger:

GoCD pipeline can be triggered based on manual, commit, time schedule and based on previous pipeline status – failed/passed

Fig. Materials and trigger

Value Stream Map:

Visual map for pipelines and its connectivity.

Artifacts:

Every job can publish its own material so that other jobs or stage can access in execution.

Agent:

GoCD agents are the workers in GoCD, they will pick-up the tasks from the pipeline and execute them. The agents are in ideal state until the pipeline gets a trigger.

Resources:

Resource is the tagging for an agent. Using tagging jobs can decide to run the tasks on the specific agents. Generally, the tags are assigned with specific software installed on agent. For example, the maven job we can have the agent running with a maven package installed on it.

Environments:

Environment is to isolate the pipeline and agent. The pipeline will only run its job on an agent within the same environment. Else if no environment is assigned pipeline will run on an agent with no environment assigned

Installation of  GoCD

1. Login to server and install using root permission
2. Install GoCD server [https://docs.gocd.org/current/installation/installing_go_server.html]
3. Start the GoCD server.

systemctl start go-server

4. Open the GoCD console from <PUBLIC_IP>:8153

In GoCD practical mention the lab setup we are using.

Configure kubectl in GoCD server

1. Login to GoCD server and install using root permission
2. Install kubectl [https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-using-native-package-management]
3. Download the kubectl config file for K8s cluster.
4. Log in to the GoCD server with ‘go’ username.
5. Copy the config content from k8s config file and paste it in ~/.kube/config.
6. Check the connectivity with k8s cluster by running

kubectl get pod

Install docker and enable the docker access to `go` system user.

1. Login to GoCD server and install using root permission
2. Install docker [https://docs.docker.com/install/linux/docker-ce/ubuntu/]
3. Enable docker access to `go` user [https://docs.docker.com/install/linux/linux-postinstall/]

Helm:

Helm tool helps to manage the Kubernetes application. Just store the Kubernetes template using helm chart and use helm with install, update. Helm allows creating a configuration package so we can use the same k8s template with multiple environments.

In our scenario, we are going to use the Helm atop of Kubernetes orchestration scripts.

Configure the helm package

1. Login to GoCD server and install using root permission
2. Install helm package [https://helm.sh/docs/using_helm/#installing-the-helm-client]
3. Initialize the helm package. It will install the helm tiller which will run in K8s cluster.

helm init

Create the helm chart.

1. Helm uses the packaging called chart. The package is a directory structure that contains the Kubernetes resources. [https://helm.sh/docs/developing_charts/#charts]

Install maven on GoCD server

1. Login to GoCD server and install using root permission
2. Install maven plugin [https://maven.apache.org/install.html]
3. Verify mvn installation

mvn -v

Quick check for what we have setup.

1. GoCD server setup
2. Install docker on GoCD server.
3. Configure the kubectl from go user to connect k8s cluster.
4. Setup the helm client and helm tiller.
5. Store the Kubernetes template in helm chart directory structure.
6. Install maven package on GoCD

Let’s start with the practical

In the following tutorial, we are going to create a maven build, build a docker image from build and deploy the image in Kubernetes cluster using the helm package.

Pre-requisite

1. Running Kubernetes cluster. You need to have admin role to deploy pods in cluster.
2. Docker repository(private/public).
3. Dockerfile placed in code to build in GoCD pipeline.

Deploy Kubernetes application using GoCD pipelines

1. Create pipeline. Select Admin –> Pipeline

• Create pipeline group. Pipelines tab –> Add new Pipeline Group
• Give the name for pipeline group, for example, config and save.
• Create the pipeline in pipeline group.

Step 1. In basic setting pass the pipeline name and pipeline group name. –> Next

Step 2.  Provide the materials. Materials type URL is mandatory. Also, check the connection before processing. If a connection does not work, follow this step to Connecting to GitHub with SSH [ https://help.github.com/en/articles/connecting-to-github-with-ssh ]

Once the step is done. Run the below command by replacing your repo name. Confirm the connection request.
git ls-remote git@github.com:<repo_name>  refs/heads/dev

Once, this step is done again try to run the connection test from the GoCD console.

Step 3. Stage/Job definition, add the name for stage. Configure the trigger.
           Initial job details. Save the settings.

• Open the pipeline and add the remaining steps for mvn build, docker build and docker push steps

• Create local artifact for helm package so that we can fetch same in deployment stage.

• Create the deployment pipeline in the same pipeline group. From top left to go Admin –> Pipelines. You will see the option to add new pipeline in config pipeline group

• Let’s create the deployment pipeline

Step 1. Add the pipeline name and add the pipeline group name

Step 2. Add the material as build pipeline. So, the deployment will get trigger when build is done. Select the build pipeline name from dropdown.

Step 3: Add the stage and job for the deployment as below. Adding first step as docker pull

• Open the pipeline and add the remaining steps for deployment to k8s cluster

• On the Dashboard you will see the 2 pipeline setup under config pipeline group.

It’s done.