Article
Kubectl-r[ex]ec: A kubectl plugin for auditing kubectl exec commands
By Marton Natko, Staff Engineer - Infrastructure
We always move fast at Adyen. One of our core principles is “Launch fast and iterate.” And now that we’re moving more toward containerized workloads, we’re able to get things done even faster. Containers offer a great way to do fast release cycles.
But speed can’t be everything: first and foremost, we need to be sure we comply with regulations. The Monetary Authority of Singapore’s Technology Risk Management (MAS TRM) Guidelines are especially strict in this regard: any local banks or fintechs hoping to clear MAS regulatory audits need to have strong tools at their disposal. Here’s what we recommend.
For many of our greenfield projects, we use Kubernetes as a container orchestrator. Our engineers have direct access to interact with it, which gives them proper engineering autonomy. This is very handy when it comes to debugging, as our engineering teams are responsible for their own applications.
Enter kubectl exec
If you’re familiar with the Kubernetes ecosystem, then you’ve probably used the kubectl exec command to debug a running application.
If you’re not familiar with it, here’s the simple explanation: it allows a user to execute commands inside a container running on Kubernetes. Going further, you could even use a teletype (TTY) in that container.
In our industry, it is a must to have audit logs of events like these. Sadly, this feature is absent from a lot of the debugging solutions engineers use.
Let’s go deeper
To solve this problem, let’s take a closer look at the data flow. We’ll only cover the part between the user and the apiserver, which is the most convenient part of figuring out a solution.
When a user executes a kubectl exec command, we can easily catch the command parameter from the request. But suppose the user also requests a TTY while passing their STDIN to the container. They will end up with a TTY inside the container, and any command executed will go through an upgraded protocol.
For Kubernetes versions below v1.30, this upgraded protocol was SPDY, but that changed to Websocket from v1.30 onwards. At this point, things get a little complicated because auditing what is executed inside the container through these sessions is not so easy to observe.
What’s our solution? kubectl-rexec
We decided to keep all the features that kubectl exec provides while considering possible solutions. We came up with the following problem statement: what if we have the kubectl exec command as a kubectl plugin, but call a different API endpoint that ends up in a component we control?
Kubernetes has the APIService resource that allows us to extend the Kubernetes API server with the new endpoints.
We introduced an APIService shown in the code block below. It will make requests with the path /apis/audit.adyen.internal/v1beta1/... to be proxied to the service called rexec in the namespace kube-system.
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
name: v1beta1.audit.adyen.internal
spec:
group: audit.adyen.internal
groupPriorityMinimum: 100
caBundle: caCertAsBase64…
service:
name: rexec
namespace: kube-system
port: 8443
version: v1beta1
versionPriority: 100
When the user does not request TTY, we log the command parameters from the request as an audit event and proxy the request directly to the Kubernetes API server.
When TTY is requested, the proxy will not forward the request directly to the Kubernetes API server. Instead, it will proxy it onto itself via a Transmission Control Protocol (TCP) listener on a Unix socket that is being started for each exec request. Each of these TCP listeners act as TCP proxies; proxying requests to the Kubernetes API server. By having a TCP proxy, we can investigate the raw TCP traffic, parse the web socket frames and capture each keystroke the user’s terminal sends into the container’s TTY. Additionally, having one TCP session per user is an easy way to keep track of the user’s identity and session while remaining on the TCP level.
But this only solves half of the problem: we have audit logging. Now we need to make sure that users do not use the native command. To do this, we added a ValidatingWebhookConfiguration.
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
name: deny-pod-exec
webhooks:
- name: deny-pod-exec.k8s.io
clientConfig:
service:
name: rexec
namespace: kube-system
port: 8443
path: /validate-exec
caBundle: caCertAsBase64...
rules:
- apiGroups: [""]
apiVersions: ["v1"]
operations: ["CONNECT"]
resources: ["pods/exec"]
admissionReviewVersions: ["v1"]
sideEffects: None
failurePolicy: Fail
This configuration will do one thing: every time the Kubernetes API server receives a request targeting api/v1/namespaces/{{ namespace }}/pods/{{ pod }}/exec, the api will call the rexec proxy with the path /validate-exec.
This lets us gatekeep the native exec requests, so we can deny everything not coming through rexec proxy itself. To tackle this, on exec requests coming through our proxy, we add a header with a value acting as a shared key between the two endpoints we provide. If the header is present and the value is the same, the request can go forward.
Conclusion
With this minimalistic application, we can easily audit exec commands by installing a few manifests on the Kubernetes side and distributing our plugin to our engineers.
Want to try it out for yourself? Check it out on Github.