Deploy Elasticsearch with Kubernetes on AWS in 10 steps


Prince Raj

Kubernetes aka k8s is an open-source system for automating deployment, scaling and management of containerized applications. In this tutorial, I will show how to setup a Kubernetes cluster and deploy an Elasticsearch cluster on it in AWS. A similar setup should also work for GCE and Azure.

Before we get started, you should have administrative access to the following AWS services: S3, EC2, Route53, IAM, and VPC.

1.Installation: I will be showing the installation setup for a Linux based CLI. If you are on a different machine, follow the below tool links to get installation instructions for your OS.

First, we will get the AWS CLI to access AWS via CLI. If you already have Python and pip in your system, you can run this command:

pip install awscli –upgrade –user

Next, we will be using a tool called Kops, a command line tool that offers an opinionated approach for setting up a production grade K8S cluster.
We will install the Kops binaries directly from github.

wget -O kops https://github.com/kubernetes/kops/releases/download/$(curl -s https://api.github.com/repos/kubernetes/kops/releases/latest | grep tag_name | cut -d ‘”‘ -f 4)/kops-linux-amd64
chmod +x ./kops
sudo mv ./kops /usr/local/bin/

And finally, we will use kubectl — a CLI for managing a K8S cluster (if you have used docker, this is similar to the docker CLI). We can install the latest stable release with the following command:

wget -O kubectl https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl

Note: You can also run the Kubernetes cluster and the setup we are going through in this tutorial on your own machine with minikube.

2. IAM User Creation: In order to build clusters within AWS, we’ll create a dedicated IAM user for kops. This user requires API credentials in order to use kops. You can create the user and assign credentials using the AWS console UI. kops user will require the following IAM permissions.
* AmazonEC2FullAccess
* AmazonRoute53FullAccess
* AmazonS3FullAccess
* IAMFullAccess
* AmazonVPCFullAccess

Image: Showing user creation from AWS dashboard

Alternatively, you can do the same from the CLI as well. Apply the below commands.

aws iam create-group –group-name kops

aws iam attach-group-policy –policy-arn arn:aws:iam::aws:policy/AmazonEC2FullAccess –group-name kops
aws iam attach-group-policy –policy-arn arn:aws:iam::aws:policy/AmazonRoute53FullAccess –group-name kops
aws iam attach-group-policy –policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess –group-name kops
aws iam attach-group-policy –policy-arn arn:aws:iam::aws:policy/IAMFullAccess –group-name kops
aws iam attach-group-policy –policy-arn arn:aws:iam::aws:policy/AmazonVPCFullAccess –group-name kops

aws iam create-user –user-name kops

aws iam add-user-to-group –user-name kops –group-name kops

aws iam create-access-key –user-name kops

Note the SecretAccessKey and AccessKeyID of kops.

Now, let’s configure the AWS CLI to use these credentials using
aws configure.

Confirm that the user you just created is in the list with the following
aws iam list-users.

We will also need to export the AWS credentials as the following environment variables for the kops CLI to be able to use these.

export AWS_ACCESS_KEY_ID=$(aws configure get aws_access_key_id)
export AWS_SECRET_ACCESS_KEY=$(aws configure get aws_secret_access_key)

If you are using Kops 1.6.2 or later, then DNS configuration is optional. Instead, a gossip-based cluster can be easily created. The only requirement to trigger this is to have the cluster name end with .k8s.local.

Configuring DNS
If you have already hosted your domain via AWS and plan to use the domain, you do not need do anything. Alternatively, if you plan to use a sub-domain of your hosted domain, you will have to create a second public hosted zone for this sub-domain. For this tutorial we are going with a private hosted zone. You need to setup a private hosted zone by any name. Use this name for creating Kubernetes clusters. For details on configuring DNS check this out.

3.Create S3 bucket: In order to store the state and representation of our K8S cluster, we need to create a dedicated S3 bucket for kops to use. This bucket will become the source of truth for our cluster configuration.

aws s3api create-bucket
–bucket
–region us-east-1

Note: If you are provisioning your bucket in region other us-east-1, in addition to setting the–region switch to your desired region, you will need to add a LocationConstraint to the same region. As an example, the following will be the command for creating the bucket in the us-west-1 region.

aws s3api create-bucket
–bucket
–region us-west-1
–create-bucket-configuration LocationConstraint=us-west-1

You might also want to version your S3 bucket with the following command in case you ever need to revert or recover a previous state store.

aws s3api put-bucket-versioning
–bucket
–versioning-configuration Status=Enabled

4. Creating our first Kubernetes cluster: Now we’re ready to start creating our first cluster! Let’s first set up a few environment variables to make this process easier. In case you skipped the DNS configuration (after step 2.), suffix .k8s.local to the value of NAME.

export NAME=myfirstcluster.example.com
export KOPS_STATE_STORE=s3://your-bucket-name

We will also need to note which availability zones are available to us. In this example we will be deploying our cluster to the us-east-2 region.

aws ec2 describe-availability-zones –region us-east-2

Create the cluster with the following command, if you are using a public hosted zone:

kops create cluster
–zones us-east-2c
–node-count 3
${NAME}

If you are using a private hosted zone, run:

kops create cluster
–zones us-east-2c
–node-count 3
–dns private ${NAME}

This command should give you the log of setting up of your K8S cluster. You need to give some time for the cluster to be up as it creates new EC2 machines for the master and minion nodes.

[ec2-user@ip-172-31-35-145 test]$ kops create cluster
> –dns private
> –zones us-east-2c
> –node-count 3
> ${NAME} –yes
I0306 09:45:29.636834 20628 create_cluster.go:439] Inferred –cloud=aws from zone “us-east-2c”
I0306 09:45:29.637030 20628 create_cluster.go:971] Using SSH public key: /home/ec2-user/.ssh/id_rsa.pub
I0306 09:45:29.850021 20628 subnets.go:184] Assigned CIDR 172.20.32.0/19 to subnet us-east-2c
I0306 09:45:31.118837 20628 dns.go:92] Private DNS: skipping DNS validation
I0306 09:45:46.986963 20628 executor.go:91] Tasks: 73 done / 73 total; 0 can run
I0306 09:45:46.987101 20628 dns.go:153] Pre-creating DNS records
I0306 09:45:47.668392 20628 update_cluster.go:248] Exporting kubecfg for cluster
kops has set your kubectl context to k8s.appbaseCluster is starting. It should be ready in a few minutes.

Voila! Our K8s cluster should be now up and running.

5. Cluster Validation: All instances created by kops will be built within ASG (Auto Scaling Groups). ASG instances are automatically rebuilt and monitored if they suffer a failure.

If you want to edit any of the cluster’s configurations, you can run the following command:

kops edit cluster ${NAME}

Every time you make a change in the cluster configuration, you will also need to build the cluster by running the following command:

kops update cluster ${NAME} –yes

You should see something similar to this.

[ec2-user@ip-172-31-35-145 examples]$ kops update cluster –yes
Using cluster from kubectl context: k8s.appbaseI0216 05:09:06.074467 2158 dns.go:92] Private DNS: skipping DNS validation
I0216 05:09:07.699380 2158 executor.go:91] Tasks: 73 done / 73 total; 0 can run
I0216 05:09:07.699486 2158 dns.go:153] Pre-creating DNS records
I0216 05:09:07.961703 2158 update_cluster.go:248] Exporting kubecfg for cluster
kops has set your kubectl context to k8s.appbaseCluster changes have been applied to the cloud.

Let’s validate our cluster to make sure that everything is working correctly.

kops validate cluster

You should see that your cluster is up and ready.

Using cluster from kubectl context: k8s.appbaseValidating cluster k8s.appbaseINSTANCE GROUPS
NAME ROLE MACHINETYPE MIN MAX SUBNETS
master-us-east-2c Master t2.large 1 1 us-east-2c
nodes Node t2.medium 3 3 us-east-2cNODE STATUS
NAME ROLE READY
ip-172-20-44-33.us-east-2.compute.internal master True
ip-172-20-52-48.us-east-2.compute.internal node True
ip-172-20-62-30.us-east-2.compute.internal node True
ip-172-20-64-53.us-east-2.compute.internal node TrueYour cluster k8s.appbase is ready

Go checkout your new k8s!

A simple Kubernetes API call can be used to check if the API is online and listening. Let’s use kubectl to check the nodes.

kubectl get nodes

This will give the details of your nodes and their current status.

[ec2-user@ip-172-31-35-145 elasticsearch]$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-172-20-44-33.us-east-2.compute.internal Ready master 1m v1.8.6
ip-172-20-52-48.us-east-2.compute.internal Ready node 3m v1.8.6
ip-172-20-62-30.us-east-2.compute.internal Ready node 2m v1.8.6
ip-172-20-64-53.us-east-2.compute.internal Ready node 4m v1.8.6

A Pod in Kubernetes is an abstraction that represents a group of one or more application containers (such as Docker), and some shared resources for those containers. A pod is deployed on a node. As you need to scale out your application, you can add nodes to your K8S deployment.

To get the available pods, run:
kubectl get pods

This command will list the available pods in our cluster.

[ec2-user@ip-172-31-35-145 ~]$ kubectl get pods
NAME READY STATUS RESTARTS AGE
es-5967f5d99c-5vcpb 1/1 Running 0 3h
es-5967f5d99c-cqk88 1/1 Running 0 3h
es-5967f5d99c-lp789 1/1 Running 0 3h

Leave a Comment

Your email address will not be published. Required fields are marked *