A Fullstack developer working with Kubernetes

Part I - Bring your cluster services to your local machineBy vicjicama


This time I will write about how I work as a frontend/backend with kubernetes (specially minikube), the evolution of the current approach, the tools to help me and use cases during the development/testing of services. In my experience I learned that...

If you are developing on your localhost you can use all your scripts, your terminal, your tools, attach your debuggers and favorite IDE as always, that would be great!... thinking about it... what would we need to accomplish this? I saw the next is needed:

That's it! Following the previous points you will have all the service dependencies from the cluster on your localhost ready to be called by the service that you are developing locally. Also you will have this local service replacing the cluster service, this means that if any other service on the cluster calls the replaced service they will be calling the service from your localhost.

I will split this into two parts cluster->localhost and localhost->cluster this time I am going to write only about the cluster to localhost, I will write about the other direction (replace your cluster services with your localhost ports) in another post.

Reaching the cluster...

At the beginning I had a minikube instance running in my laptop, but my laptop only have 8GB I notice that between the browser, atom and minikube there was not much memory left for me and my services. Then we moved the minikube instance to an old pc here in the office, it's an old 12GB and 4CPU machine but this was enough to a couple minikube instances, each one with a couple namespaces for test, CI and development.

The issue now was the access to the minikube instance... since it is running inside virtualbox in another machine it was not as straightforward to access the cluster as if the minikube instance were in our localhost. I saw that doing to forward services approach was very convenient to access minikube or any other cluster, no matter if I am on the office or from home, all I need to do it to forward the services to my localhost and then forward my localhost services back to the cluster.

From the cluster to your localhost

The next is a list of the components that I forward from the cluster to my localhost, we are going to check in detail why each one is helpful during the development of the services in my localhost.

  • Ingress
  • Your App services
  • Resource services (Cache, Databases, Queues)
  • Minikube SSH
  • Kubernetes API & Dashboard

I am working on a couple systems that are running on Kubernetes, one in particular is our lab/blog, it's a cluster that is up and running that you can visit, I version everything that goes into this cluster and it is open-source, you can find the code and the details here: microservices and you can run it yourself in your minikube instance. I will use this cluster/app for the examples during this post.

As we were working with Kubernetes we gradually took all those scripts and ssh functionality to automate the steps and details to bring the cluster to our localhost in a straightforward way (just click the start button), this is specially useful for the team members that does not have much experience with ssh tunnels, don't have the kubectl installed or want to focus on the development of their ui or apis, we are going to use the GUI tool for the use cases examples.


For example... if you are working with a SPA chances are that you need to make api request to the backend once the page is initially loaded in the browser on your machine, in my case I do graphql requests to https://blog.repoflow.com/blog/backend/graphql, I have the ingress configured for the host blog.repoflow.com inside minikube (same as in production), if I forward the ingress and modify the /etc/hosts file then all the services that you are developing locally and your browser will resolve blog.repoflow.com to your minikube instance, now we have real ssl with the same domain but in a test env.. one thing less to change in my dev/test env!

Your app services (GraphQL and Web SSR in my case)

If you are developing services for the backend then it's almost guaranteed that you will need to call another service from the cluster from time to time. In my case I am using SSR (server side render) for my SPA in React, this means that for the SSR functionality I am on the cluster context and if I wanted to access a service I need to use one of the service discover mechanism of the cluster, in my case I use the DNS based service discovery.

Following the example of the ingress, the url https://blog.repoflow.com/blog/backend/graphql was used to access the graphql outside the cluster for any user, but for the SSR functionality we are inside the cluster and the way you reach that service changes to: http://blog-graph-service.repoflow-blog-stage:4000/blog/backend/graphql (This is why I have an external and internal env variable for the graphql url, you can check this code here web.yaml )

Here is an example of the blog graph and web services being forwarded to my localhost, Now I can test my backend and SSR services locally as within the cluster

I have a separate graph and web app for each mayor functionality of this lab/blog system (auth, home, blog and resources), there is a lot of communication between the services and having the ability to only forward the ingress or the services that I need is really useful.

Resource services (Cache, Databases, Queues)

One frequent need while working within the cluster was to connect to the stateful services like the database to debug or manipulate the data directly during development.

For example in my case the blog graph service is a busy service, it access the next services within the cluster, take a look to its deployment file graph.yaml
  • resources-data-service
  • resources-queue-service
  • resources-cache-service
  • auth-account-graph-service
  • auth-cache-service

I bring all those services to my localhost while I am doing any change to the blog graphql service, as an added benefit I can connect utility tools to those services to help me during the debugging and testing.

Here I am forwarding the resources database and the cache for the auth and the resources to my localhost, we are using the same connection url than within the cluster. if you are developing a service locally you can connect to the dependency services needed without modify the dns or port, you just need to forward them and modify the /etc/hosts file.

Did you notice the Link button in each service's footer? this is for replace a service on the cluster with a port running on your localhost. Beside replacing the port of the services that you are developing, another cool case for this is to replace stateful services, you could have a database running locally using a docker-compose and share it with the team in the cluster, this is really useful when you are triaging/debugging a bug. This way your development cluster acts as a vessel for all the team's local services. I will cover the local to cluster part in detail in the next post.

SSH from the cluster node

If besides your Fullstack activities you need to perform cluster configuration then the next use cases will be very helpful for you. If you have used minikube you probably have connected to the minikube node via ssh with the command

minikube ssh

This is very helpful for rights configuration of folders, seed test cases or copy debug result files, etc... but if you install the minikube instance in another machine you need to forward the ssh port to your localhost or if you are out of the office you need to forward the port to a pivot machine and then to your localhost.

The next is an example of a command that I use to copy test data to the minikube node from my localhost.

rsync -ra -e "ssh -p 2265 -i /home/victor/cluster/gn2/minikube/minikube.key" /home/victor/temp/* docker@localhost:/home/docker/volumes/blog-stage/blog-stage-blog-repoflow-metric-volume

Kubernetes API & Dashboard

A very useful tool that I use while working on the cluster is the Kubernetes Dashboard, minikube have a command to enable and forward proxy the dashboard to the host of the minikube virtual machines, but you will need to perform additional forward operations if you want to access it from another machine or if you want to access the dashboard out of the office or cloud.

Last but not least the Kubernetes API, this is specially useful when you are out of the office and you wanted to have access to the minikube's kubernetes api with kubectl.

Here is the configuration file that I am using:

apiVersion: v1
- cluster:
    certificate-authority: /home/victor/cluster/gn2/minikube/ca.crt
    server: https://localhost:8443
  name: minikube
- context:
    cluster: minikube
    user: minikube
  name: minikube
current-context: minikube
kind: Config
preferences: {}
users:- name: minikube

    client-certificate: /home/victor/cluster/gn2/minikube/client.crt
    client-key: /home/victor/cluster/gn2/minikube/client.key

You need to copy the files client.crt, client.key and ca.crt from the minikube host to your localhost


The forward approach lets us develop our services in our localhost in a very useful and simple way, you can use your debuggers, your terminal and build tools that you already have up and working. After all the examples our connected endpoints looks like this with all the services we need to develop locally:

This is a very convenient way to share my services with the team or to perform integration test, other can access your development/patched services in a transparent way. Under the hood everything that we do to archive the previous use cases is ssh tunnel the services, you can access the cluster services in your localhost even if the cluster is inside a virtual machine, in the cloud or if you are out of the office.

I think that a key aspect of the Kubernetes adoption in a team/project is to make as easy as possible the interaction of the development team with the cluster.

I realized that the presented approach was convenient, easy and generic while working on stateless frontend/backend tasks, just forward the services that I need to my localhost and then forward the local services to replace the kubernetes services. A key added benefit of this apporach is that you don't have to touch you code at all, you don't need to add any config file to you repositories. I know that this approach does not cover all the cases out there, but for me this has been working incredibly well for stateless Fullstack activities.

This is half of the story of our approach testing/developing stateless services in Kubernetes, next time we will look in detail the local to cluster phase to replace cluster services with you local ports and talk about the way this service replacement greatly simplify/reduce cost for our CI activities. If you want to save time to your team forwarding the services give it a try to the tool that we use during the use cases of this post: linker-tool

If you have any feedback, any question, any use case to discuss or if you want to reach out don't hesitate to contact me my email is vic@repoflow.com