That Kubernetes And Friends Stack
I wanted to write a post about Continuous Delivery pipelines but you will end up with a Kubernetes Stack anyway so I will write this post oriented around Kubernetes and friends. The hardest part of Kubernetes is deciding on your stack so I am here to give you a good starting point.
Before starting this post make sure that your Continues Integration Pipeline is in place and your images are pushed to a docker repository and they are semantically versioned.
Our aim is to have a common, mostly provider-independent easy to use infrastructure that everyone can understand.
There may be variations based on your cloud provider but the main points of our stack will be like this:
Docker Hub + Kubernetes + Weave Net+ Nginx + Prometheus+Grafana + Fluentd+ ElasticSearch +Kibana+ ArgoCD + Helm 3 + Github + CircleCI
Docker Hub is our docker image registry. You can use AWS ECR or GCP Container Registry based on your cloud provider. I urge you to share your public images in Docker Hub which is default and easy to reach by developers. Some private container registries have tricks to make it work with a Kubernetes like periodically updating the image pull secret.
Kubernetes is a production-grade container orchestration platform. It is a battle-tested and commonly used platform currently. Please use a managed Kubernetes cluster if you are not a Kubernetes provider or a large company that has enough employees to keep people on-call, all the time. There are very good providers right now. Almost all big providers (AWS, GCP, Azure) are in a good state and provide a decent solution (EKS, GKE, AKS,…). In my experience, GCP is the best for Kubernetes but most people are AWS fanatics. Also, to keep costs low, you can use spot instances in AWS and reserved instances in GCP so your costs will be lower compared to a similar solution with EC2.
Weave Net from Weave works as a networking driver. Weave Net also has support to encrypt your network so you will be safer while developing applications on your network. This is a serious issue, depending on your business there are regulations about it. Encrypt your network and data on rest.
Nginx is a reverse proxy and it has a pretty commonly used ingress controller and API Gateway for Kubernetes. It is really easy to use if you don’t know what to use Nginx. Depending on your requirements you can move to HaProxy, Traefik, ISTIO, Ambassador, Envoy, or Kong. You can start with Nginx in a couple of minutes and there are enough documentation and support in the public domain. ISTIO has the most complete solution and also supports Mutual TLS for internal communication but it is a bit harder to work with. I have been working with ISTIO since version 0.5, it has evolved to be a nice solution but you may need an expert to make it work for you.
Prometheus is for metrics gathering. It has a standard API and supported by many programming languages. You can view metrics with Grafana but as a software developer, I use Prometheus API to build other products over it. You can build A/B testing, performance optimization, canary releasing, anomaly detection based on metrics stored in Prometheus.
ElasticSearch is for storing logs. ElasticSearch can do lots of things but it is a text search engine. I see many companies trying to use it as a metrics engine and failing miserably. ElasticSearch can be installed in your cluster with Fluentd or FluentBit to gather logs of your containers. It is a very common solution. You can use Kibana to visualize the data stored in ElasticSearch. Also, you may need to write a job to delete old indexes periodically, the free version still doesn’t support the index lifecycle. I recommend you to use a hosted ElasticSearch if you are not insane. When you need the see the logs, it is because something went wrong with your cluster and the same problem may affect your ElasticSearch if your ElasticSearch is running in the same cluster.
ArgoCD is very new on my list. It is used for Infrastructure as Code pattern. Where you install ArgoCD controller in our cluster and keep cluster definition in Git and it automatically syncs our container state with the Git. There are two popular solutions FluxCD and ArgoCD. FluxCD is a more simple solution targeting an automated environment. ArgoCD has UI and targeting more complex companies. Probably you will use Github as your Git solution.
Helm 3 is for templating your YAML definitions and it is very common. And use helm 3, not helm 2.
CircleCI is for the CI pipeline. It is really easy to use but you may go with other solutions but it has enough features and works well. You can use a CI flow to update the Infrastructure Code as the last stage of your CI pipeline which will start the CD pipeline. You can also write some periodic control tasks for your clusters.
There are other components and controllers that you may need like Cert Manager, External DNS. They are easy to integrate after adding these main components.
I hope this will give a good starting point for your Kubernetes Cluster.
Go run your own!
