Creating Validating Admission Controller in Kubernetes using Go
Admission Controllers
Admission controllers provide a mechanism to validate or modify Kubernetes object creation requests before the object is actually created. This definition might not make complete sense right now but we will be able to understand what this means as the tutorial progresses, so don’t worry about it :).
There are two kinds of admission controllers in Kubernetes.
- Validating admission controller
- Mutating admission controller
Let’s try to understand each of them by means of an example.
Validating admission controller
Let’s say we are creating a deployment in kubernetes using kubectl. The following yaml file is used to create the deployment.
Save this yaml with the name sample-depl.yaml and create the deployment using the command kubectl create -f sample-depl.yaml
1apiVersion: apps/v1
2kind: Deployment
3metadata:
4 labels:
5 app: sample-dep
6 name: sample-dep
7spec:
8 replicas: 1
9 selector:
10 matchLabels:
11 app: sample-dep
12 template:
13 metadata:
14 labels:
15 app: sample-dep
16 spec:
17 containers:
18 - image: busybox
19 name: busybox
20 command: [ "sleep" ]
21 args: [ "infinity" ]
The deployment will be created successfully.
What if we have a requirement that each container in the deployment must have requests and limits set for memory. The following yaml
shows how memory requests and limits can be set.
1apiVersion: apps/v1
2kind: Deployment
3metadata:
4 labels:
5 app: sample-dep
6 name: sample-dep
7spec:
8 replicas: 1
9 selector:
10 matchLabels:
11 app: sample-dep
12 template:
13 metadata:
14 labels:
15 app: sample-dep
16 spec:
17 containers:
18 - image: busybox
19 name: busybox
20 command: [ "sleep" ]
21 args: [ "infinity" ]
22 resources:
23 requests:
24 memory: "100Mi"
25 limits:
26 memory: "200Mi"
The above requirement of validating whether the memory requests and limits are set before creating creating the deployment can be accomplished with the help
of a validating admission controller. The validating admission controller will check the deployment for the presence of the memory limits and requests fields.
Deployments which do not have memory request and limits specified will error out and they will not be created. We will learn how to achieve this by creating a validating
admission controller in this tutorial.
Mutating admission controller
The second type of admission controller supported by Kubernetes is a mutating admission controller. Let’s consider the example of the memory requests and limits again. Instead of failing the deployment if memory requests and limits are not set, what if we want to add a default memory request and limit for the container. This can be done using a mutating admission controller. Using the mutating admission controller, it’s possible to specify a default memory request and limit if it’s absent before the deployment is created.
We will only be learning about the validating admission controller in this tutorial.
Where admission controllers fit in the Kubernetes request life cycle?
Before we write the code for the validating admission controller, let’s take a step back and understand where both the mutating and validating admission controller fit within the lifecycle of an object creation in Kubernetes. This will help us better understand the architecture of Kubernetes which will greatly assist in the code writing of the validating admission controller.
The lifecycle of a kubectl request is captured in the above image. Let’s take the example of a user creating a deployment using the command kubectl create -f sample-depl.yaml.
When a kubectl command is executed, the request first reaches the Kube API server for authentication and authorization checks.
The Kube API server first authenticates the request to ensure that the user making the request is a valid user.
After authentication, the Kube API server performs authorization checks on the request to ensure that the user making the request has the necessary permission to perform the request. In our case, the Kube API server validates whether the user is allowed to create a deployment.
After the authentication and authorization checks are over, the Kube API server makes a JSON REST API call to the mutating admission controller with the sample-depl deployment creation request in the request body. A part of the JSON request sent to the mutating admission controller is provided below.
1{
2 "kind": "AdmissionReview",
3 "apiVersion": "admission.k8s.io/v1",
4 "request": {
5 "uid": "8b0ad8c2-c657-49a9-bb01-b402c9326dbe",
6 "kind": {
7 "group": "apps",
8 "version": "v1",
9 "kind": "Deployment"
10 },
11 "resource": {
12 "group": "apps",
13 "version": "v1",
14 "resource": "deployments"
15 },
16 "name": "sample-dep",
17 "namespace": "webhooktest",
18 "operation": "CREATE",
19 "userInfo": {
20 "username": "kubernetes-admin",
21 "groups": [
22 "kubeadm:cluster-admins",
23 "system:authenticated"
24 ]
25 },
26 "object": {
27 "kind": "Deployment",
28 "apiVersion": "apps/v1",
29 "metadata": {
30 "name": "sample-dep",
31 "namespace": "webhooktest",
32 "uid": "be512fcd-ee50-49f1-8782-93a167405fcb",
33 "generation": 1,
34 "creationTimestamp": "2024-07-16T10:08:15Z",
35 "labels": {
36 "app": "sample-dep"
37 }
38 },
39 "spec": {
40 "replicas": 1,
41 "template": {
42 "metadata": {
43 "creationTimestamp": null,
44 "labels": {
45 "app": "sample-dep"
46 }
47 },
48 "spec": {
49 "containers": [
50 {
51 "name": "busybox",
52 "image": "busybox",
53 "command": [
54 "sleep"
55 ],
56 "args": [
57 "infinity"
58 ],
59 "resources": {},
60 "terminationMessagePath": "/dev/termination-log",
61 "terminationMessagePolicy": "File",
62 "imagePullPolicy": "Always"
63 }
64 ],
65 "restartPolicy": "Always",
66 "terminationGracePeriodSeconds": 30,
67 "dnsPolicy": "ClusterFirst",
68 "securityContext": {},
69 "schedulerName": "default-scheduler"
70 }
71 }
72 },
73 "status": {}
74 },
75 ...
The mutating admission controller can make changes to the above request and patch it when it sends the response. For example, memory request and limits can be added to the request if they are missing and a appropriate JSON response can be sent.
After sending the request to the mutating admission controller, the Kube API server next makes a JSON REST API call to the validating admission controller.
This is the controller that we will be developing in this tutorial. The validating admission controller that we are going to develop will check for the presence of memory request and limits and it will fail the deployment creation if the memory requests and limits are missing.
A sample JSON response which the validating admission control will send after processing the request is provided below.
1{
2 "kind": "AdmissionReview",
3 "apiVersion": "admission.k8s.io/v1",
4 "response": {
5 "uid": "761170a6-9e89-4268-a262-c1e87f3fa07c",
6 "allowed": false,
7 "status": {
8 "metadata": {},
9 "status": "Failure",
10 "message": "Memory request not specified for container busybox"
11 }
12 }
13}
The allowed boolean field in the above JSON response determines whether a particular k8s object creation request is allowed or not. We will be setting the allowed field in the admission review response sent from the validating admission controller based on the deployment request. If memory requests and limits are present in the deployment, the allowed field will be set to true and the corresponding JSON response will be returned by the validating admission controller to the Kube API server. If either the memory requests or limits is not present, the allowed field will be set to false
If the request is successfully validated by the validating admission controller, i.e in our case if the memory requests and limits are present, the Kube API server will persist the request in etcd which will be processed by the controller manager and the Kube scheduler finally schedules the pod to a node.
The reason why the mutating admission controller is invoked first by the Kube API server before the validating admission controller is because, the request can be validated by the validating admission controller after any changes are made to it by the mutating admission controller.
Now that we have understood how a request is processed by the Kube API server, let’s start developing our validating admission controller which will check for the presence of memory requests and limits
Boiler plate code for encoding and decoding AdmissionReview requests
The core logic for the validating admission controller is to check for the presence of memory requests and limits in all the containers in the deployment. Part of a sample deployment containing memory requests and limits is provided below.
1spec:
2 containers:
3 - image: busybox
4 name: busybox
5 command: [ "sleep" ]
6 args: [ "infinity" ]
7 resources:
8 requests:
9 memory: "100Mi"
10 limits:
11 memory: "200Mi"
Before the core logic is written, a little bit of boiler plate code is needed to encode/decode k8s AdmissionReview objects to Go structs.
The boiler plate code to perform the encoding/decoding of AdmissionReview objects is provided below.
1package main
2
3import (
4 "log/slog"
5 "net/http"
6 "os"
7
8 admission "k8s.io/api/admission/v1"
9 "k8s.io/apimachinery/pkg/runtime"
10 "k8s.io/apimachinery/pkg/runtime/serializer"
11)
12
13type admissionValidationHandler struct {
14 schemeDecoder runtime.Decoder
15}
16
17func (avh admissionValidationHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
18}
19
20func main() {
21 runtimeScheme := runtime.NewScheme()
22 if err := admission.AddToScheme(runtimeScheme); err != nil {
23 slog.Error("error adding AdmissionReview to scheme", "error", err)
24 os.Exit(1)
25 }
26 codecFactory := serializer.NewCodecFactory(runtimeScheme)
27 deserializer := codecFactory.UniversalDeserializer()
28 admissionValidationHandler := admissionValidationHandler{
29 schemeDecoder: deserializer,
30 }
31
32 http.Handle("/validate", admissionValidationHandler)
33 slog.Info("Server started ...")
34 if err := http.ListenAndServeTLS(":7443", "/etc/ssl/certs/tls.crt", "/etc/ssl/certs/tls.key", nil); err != nil {
35 slog.Error("error starting admission webhook", "error", err)
36 os.Exit(1)
37 }
38}
A scheme is used to convert Kubernetes group, version and kind information to Go structs and vice versa. In order to deal with the encoding and decoding of AdmissionReview requests, the version and kind information of AdmissionReview objects should be registered to the scheme. This is done by line nos. 21 to 25 in the above code.
The next step is to create the deserializer which is used to decode the JSON AdmissionReview requests to Go struct. This is done by lines 26 to 30 in the above code.
The web server is then started in line no. 34. Kube API server will only make https calls to our validating admission controller web server and hence SSL cert and key is needed. We will learn how to create a self signed cert using our own root CA in the upcoming sections of this tutorial but for now let’s assume that the cert and key will be available at /etc/ssl/certs/tls.crt and /etc/ssl/certs/tls.key.
Create a folder named validating-admission-controller and save the above code to a file named main.go. Also initialize the go module running the following commands.
1go mod init github.com/golangbot/validating-admission-controller
2go mod tidy
3go mod vendor
If you want to know more about go modules, I recommend reading https://golangbot.com/go-packages/
If the program is run, it will return the following error since the certs are not created yet. This error will be fixed as the tutorial progresses.
error starting admission webhook error="open /etc/ssl/certs/tls.crt: no such file or directory"
Decoding AdmissionReview and Deployment requests
Now that the boiler plate code is ready, the next step is to decode the deployment in the AdmissionReview request and find out whether it has memory requests and limits. As discussed earlier, the Kube API server will make a JSON REST API to the validating admission controller with AdmissionReview request in the body of the request.
We will use the schemeDecoder we created above to decode this AdmissionReview JSON request to a Go struct.
1func (avh admissionValidationHandler) decodeRequest(request []byte, expectedGVK schema.GroupVersionKind, into runtime.Object) (schema.GroupVersionKind, error) {
2 _, requestGVK, err := avh.schemeDecoder.Decode(request, nil, into)
3 if err != nil {
4 return schema.GroupVersionKind{}, err
5 }
6 if requestGVK == nil {
7 return schema.GroupVersionKind{}, errors.New("unable to find schema group, version and kind from request")
8 }
9
10 if *requestGVK != expectedGVK {
11 errMsg := fmt.Sprintf(`Expected admission review with group: %s version: %s kind: %s
12but got group: %s version: %s kind: %s`, expectedGVK.Group, expectedGVK.Version, expectedGVK.Kind, requestGVK.Group, requestGVK.Version, requestGVK.Kind)
13 return schema.GroupVersionKind{}, errors.New(errMsg)
14 }
15 return *requestGVK, nil
16}
17
18func (avh admissionValidationHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
19 requestBody, err := io.ReadAll(r.Body)
20 if err != nil {
21 errMsg := fmt.Sprintf("error %s reading request body", err)
22 slog.Error(errMsg)
23 http.Error(w, errMsg, http.StatusInternalServerError)
24 return
25 }
26 slog.Debug("request body read successfully", "request body", requestBody)
27 admissionReview := new(admission.AdmissionReview)
28 expectedAdmissionReviewGVK := schema.GroupVersionKind{
29 Group: "admission.k8s.io",
30 Version: "v1",
31 Kind: "AdmissionReview",
32 }
33 admissionReviewGVK, err := avh.decodeRequest(requestBody, expectedAdmissionReviewGVK, admissionReview)
34 if err != nil {
35 http.Error(w, err.Error(), http.StatusInternalServerError)
36 return
37 }
38...
The first step is to read the body of the incoming http request. This is done in line no. 19 of the above code. After reading the request body, the next step is to decode the body into a AdmissionReview struct. The struct that the request body will be decoded into is created in line no. 27. The actual decoding is done by the decodeRequest method. It takes the request body, expected group version and kind of the object to be decoded and the struct into which the request has to be decoded.
Our Validating Admission Controller will only handle requests with version v1. The GroupVersionKind struct representing AdmissionReview v1 is created in line no. 28 of the above code. The decoded AdmissionReview request after calling decodeRequest method in line no. 33 is stored in admissionReview struct. Please note that the ServeHTTP method in the above code snippet is incomplete. We will finish this method as the tutorial progresses.
The next step is to decode the K8s Deployment from the AdmissionReview request. The deployment is present in the Request field of the AdmissionReview struct.
1 admissionReviewRequest := admissionReview.Request
2 if admissionReviewRequest == nil {
3 errMsg := "Expected admission review request but did not get one"
4 slog.Error(errMsg)
5 http.Error(w, errMsg, http.StatusInternalServerError)
6 return
7 }
8 deploymentGVR := metav1.GroupVersionResource{
9 Group: "apps",
10 Version: "v1",
11 Resource: "deployments",
12 }
13 if admissionReviewRequest.Resource != deploymentGVR {
14 errMsg := fmt.Sprintf("Expected apps/v1/deployments resource but got %+v", admissionReviewRequest)
15 slog.Error(errMsg)
16 http.Error(w, errMsg, http.StatusBadRequest)
17 return
18 }
19
20 deploymentRequest := new(appsv1.Deployment)
21 expectedDeploymentGVK := schema.GroupVersionKind{
22 Group: "apps",
23 Version: "v1",
24 Kind: "Deployment",
25 }
26 if _, err = avh.decodeRequest(admissionReviewRequest.Object.Raw, expectedDeploymentGVK, deploymentRequest); err != nil {
27 http.Error(w, err.Error(), http.StatusInternalServerError)
28 return
29 }
The above code decodes the Deployment object from the AdmissionReview into the deploymentRequest struct in line no. 20. This code is self explanatory and similar to how AdmissionReview was decoded earlier.
Checking for presence of memory request and limit
Now that we have the deployment decoded, the final step the validation admission controller is supposed to do is to check for the presence of memory request and limits and respond with error if there is no memory request or limit present. The code that does this is provided below.
1 var errorMessage string
2 for _, container := range deploymentRequest.Spec.Template.Spec.Containers {
3 if _, ok := container.Resources.Requests[corev1.ResourceMemory]; !ok {
4 errorMessage = fmt.Sprintf("Memory request not specified for container %s", container.Name)
5 break
6 }
7 if _, ok := container.Resources.Limits[corev1.ResourceMemory]; !ok {
8 errorMessage = fmt.Sprintf("Memory limit not specified for container %s", container.Name)
9 break
10 }
11 }
12 admissionReviewResponse := &admission.AdmissionReview{
13 Response: &admission.AdmissionResponse{
14 UID: admissionReviewRequest.UID,
15 },
16 }
17 admissionReviewResponse.SetGroupVersionKind(admissionReviewGVK)
18 if errorMessage != "" {
19 admissionReviewResponse.Response.Allowed = false
20 admissionReviewResponse.Response.Result = &metav1.Status{
21 Status: "Failure",
22 Message: errorMessage,
23 }
24 } else {
25 admissionReviewResponse.Response.Allowed = true
26 }
27 respBytes, err := json.Marshal(admissionReviewResponse)
28 if err != nil {
29 slog.Error("error marshaling response for admission review", "error", err)
30 http.Error(w, err.Error(), http.StatusInternalServerError)
31 return
32 }
33 slog.Info("admission review response json marshalled", "json", respBytes)
34 w.Header().Set("Content-Type", "application/json")
35 if _, err := w.Write(respBytes); err != nil {
36 slog.Error("error %s writing admission response", err)
37 http.Error(w, err.Error(), http.StatusInternalServerError)
38 }
Line nos. 1 to 11 in the above code check whether each and every container in the deployment have memory request and limit specified. If it is not specified, errorMessage is populated with the appropriate error.
The final step is to return a JSON response with the allowed boolean field in the AdmissionResponse set to true or false.
If allowed is true, the deployment will allowed to be created. Otherwise the error message will be displayed and the deployment creation will fail.This field will be set to true only if all the containers have memory request and limit set. If request and limit are not set for any of the containers, allowed will be set to false and the name of the container which doesn’t have the memory requests and limits set will be present in the error message.
The admissionReviewResponse is json marshalled in line no. 27 and sent as the http response.
Complete code of the validating admission controller web server
The complete code of the validating admission controller web server is provided below.
1package main
2
3import (
4 "errors"
5 "fmt"
6 "io"
7 "log/slog"
8 "net/http"
9 "os"
10
11 "encoding/json"
12
13 admission "k8s.io/api/admission/v1"
14 appsv1 "k8s.io/api/apps/v1"
15 corev1 "k8s.io/api/core/v1"
16 metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
17 "k8s.io/apimachinery/pkg/runtime"
18 "k8s.io/apimachinery/pkg/runtime/schema"
19 "k8s.io/apimachinery/pkg/runtime/serializer"
20)
21
22type admissionValidationHandler struct {
23 schemeDecoder runtime.Decoder
24}
25
26func (avh admissionValidationHandler) decodeRequest(request []byte, expectedGVK schema.GroupVersionKind, into runtime.Object) (schema.GroupVersionKind, error) {
27 _, requestGVK, err := avh.schemeDecoder.Decode(request, nil, into)
28 if err != nil {
29 return schema.GroupVersionKind{}, err
30 }
31 if requestGVK == nil {
32 return schema.GroupVersionKind{}, errors.New("unable to find schema group, version and kind from request")
33 }
34
35 if *requestGVK != expectedGVK {
36 errMsg := fmt.Sprintf(`Expected admission review with group: %s version: %s kind: %s
37but got group: %s version: %s kind: %s`, expectedGVK.Group, expectedGVK.Version, expectedGVK.Kind, requestGVK.Group, requestGVK.Version, requestGVK.Kind)
38 return schema.GroupVersionKind{}, errors.New(errMsg)
39 }
40 return *requestGVK, nil
41}
42
43func (avh admissionValidationHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
44 requestBody, err := io.ReadAll(r.Body)
45 if err != nil {
46 errMsg := fmt.Sprintf("error %s reading request body", err)
47 slog.Error(errMsg)
48 http.Error(w, errMsg, http.StatusInternalServerError)
49 return
50 }
51 slog.Debug("request body read successfully", "request body", requestBody)
52 admissionReview := new(admission.AdmissionReview)
53 expectedAdmissionReviewGVK := schema.GroupVersionKind{
54 Group: "admission.k8s.io",
55 Version: "v1",
56 Kind: "AdmissionReview",
57 }
58 admissionReviewGVK, err := avh.decodeRequest(requestBody, expectedAdmissionReviewGVK, admissionReview)
59 if err != nil {
60 http.Error(w, err.Error(), http.StatusInternalServerError)
61 return
62 }
63
64 slog.Info("Successfully decoded AdmissionReview")
65
66 admissionReviewRequest := admissionReview.Request
67 if admissionReviewRequest == nil {
68 errMsg := "Expected admission review request but did not get one"
69 slog.Error(errMsg)
70 http.Error(w, errMsg, http.StatusInternalServerError)
71 return
72 }
73 deploymentGVR := metav1.GroupVersionResource{
74 Group: "apps",
75 Version: "v1",
76 Resource: "deployments",
77 }
78 if admissionReviewRequest.Resource != deploymentGVR {
79 errMsg := fmt.Sprintf("Expected apps/v1/deployments resource but got %+v", admissionReviewRequest)
80 slog.Error(errMsg)
81 http.Error(w, errMsg, http.StatusBadRequest)
82 return
83 }
84
85 deploymentRequest := new(appsv1.Deployment)
86 expectedDeploymentGVK := schema.GroupVersionKind{
87 Group: "apps",
88 Version: "v1",
89 Kind: "Deployment",
90 }
91 if _, err = avh.decodeRequest(admissionReviewRequest.Object.Raw, expectedDeploymentGVK, deploymentRequest); err != nil {
92 http.Error(w, err.Error(), http.StatusInternalServerError)
93 return
94 }
95
96 slog.Debug("Deployment request decoded successfully", "decoded request", deploymentRequest)
97 var errorMessage string
98 for _, container := range deploymentRequest.Spec.Template.Spec.Containers {
99 if _, ok := container.Resources.Requests[corev1.ResourceMemory]; !ok {
100 errorMessage = fmt.Sprintf("Memory request not specified for container %s", container.Name)
101 break
102 }
103 if _, ok := container.Resources.Limits[corev1.ResourceMemory]; !ok {
104 errorMessage = fmt.Sprintf("Memory limit not specified for container %s", container.Name)
105 break
106 }
107 }
108 admissionReviewResponse := &admission.AdmissionReview{
109 Response: &admission.AdmissionResponse{
110 UID: admissionReviewRequest.UID,
111 },
112 }
113 admissionReviewResponse.SetGroupVersionKind(admissionReviewGVK)
114 if errorMessage != "" {
115 admissionReviewResponse.Response.Allowed = false
116 admissionReviewResponse.Response.Result = &metav1.Status{
117 Status: "Failure",
118 Message: errorMessage,
119 }
120 } else {
121 admissionReviewResponse.Response.Allowed = true
122 }
123 respBytes, err := json.Marshal(admissionReviewResponse)
124 if err != nil {
125 slog.Error("error marshaling response for admission review", "error", err)
126 http.Error(w, err.Error(), http.StatusInternalServerError)
127 return
128 }
129 slog.Info("admission review response json marshalled", "json", respBytes)
130 w.Header().Set("Content-Type", "application/json")
131 if _, err := w.Write(respBytes); err != nil {
132 slog.Error("error %s writing admission response", err)
133 http.Error(w, err.Error(), http.StatusInternalServerError)
134 }
135}
136
137func main() {
138 runtimeScheme := runtime.NewScheme()
139 if err := admission.AddToScheme(runtimeScheme); err != nil {
140 slog.Error("error adding AdmissionReview to scheme", "error", err)
141 os.Exit(1)
142 }
143 codecFactory := serializer.NewCodecFactory(runtimeScheme)
144 deserializer := codecFactory.UniversalDeserializer()
145 admissionValidationHandler := admissionValidationHandler{
146 schemeDecoder: deserializer,
147 }
148
149 http.Handle("/validate", admissionValidationHandler)
150 slog.Info("Server started ...")
151 if err := http.ListenAndServeTLS(":7443", "/etc/ssl/certs/tls.crt", "/etc/ssl/certs/tls.key", nil); err != nil {
152 slog.Error("error starting admission webhook", "error", err)
153 os.Exit(1)
154 }
155}
Creating self signed SSL certs
Kube API server will only make https calls to the validating admission controller server. This means a valid SSL cert, key and CA cert is needed for the REST API call from the KubeAPI server to the validating admission controller to be successful. We will create a self signed cert and key using our own root CA.
Generate private key
The first step is to create the private key for our validating admission controller server. This can be done using the following
openssl command. Please run this command from a folder named certs to ensure that all the needed certs are created in a separate folder.
mkdir certs
cd certs
openssl genpkey -algorithm RSA -out server.pem -pkeyopt rsa_keygen_bits:2048
The above command will generate a private key in a file named server.pem
Generate CSR
The next step is to create a certificate signing request (CSR) for our validating admission controller server. We will be creating a K8s service named validating-webhook-svc in a namespace called webhooktest in the next step. So the subject alternative name of our server will be validating-webhook-svc.webhooktest.svc.
openssl req -new -key server.pem -out server.csr -subj "/CN=validating-webhook-svc.webhooktest.svc" -addext "subjectAltName = DNS:validating-webhook-svc.webhooktest.svc"
The above command takes the private key server.pem as input and creates a csr named server.csr.
Create root CA cert and key
The next step is to create a root CA cert and a root CA key which will be used to sign our server.csr
openssl req -x509 -sha256 -newkey rsa:2048 -keyout rootCA.pem -out rootCA.crt -days 650 -subj "/CN=RootCA" -nodes
The above command creates a rootCA key named rootCA.pem and root CA cert named rootCA.crt.
Signing the server’s CSR with the root CA
The final step is to sign the admission controller server’s csr with the root CA’s key and cert we created. The following command
will do that
openssl x509 -copy_extensions copy -req -CA rootCA.crt -CAkey rootCA.pem -in server.csr -out server.crt -days 650 -CAcreateserial
The following files will be present in the certs folder after all the commands are executed successfully.
1rootCA.crt
2rootCA.pem
3server.crt
4server.csr
5server.pem
Dockerfile
Dockerfile is needed to build the image for the validating admission controller to deploy it to Kubernetes. A multistage Dockerfile which builds the validating admission controller web server and runs it on alpine is provided below.
1FROM golang:1.22-alpine AS builder
2COPY . /src/admissioncontroller/
3WORKDIR /src/admissioncontroller
4RUN go build -o server
5
6FROM alpine:3.19
7WORKDIR /bin/admissioncontroller
8COPY --from=builder /src/admissioncontroller/server /bin/admissioncontroller/server
9CMD ["./server"]
Save the above file as Dockerfile.
The Docker image can be build and pushed to the docker registry using the command
docker build -t msgtonaveen/validating-admission-controller:v0.1.0 .
docker push msgtonaveen/validating-admission-controller:v0.1.0
This docker image will be used in the validating admission controller Kubernetes deployment which will be created in the next section. Please note that msgtonaveen/validating-admission-controller is the path for my docker registry and you will not be able to push to it. Please replace this with the path for your docker repository.
Creating Admission Controller Kubernetes Deployment
If we recollect, our main.go contains the following line, which needs the cert and key.
1if err := http.ListenAndServeTLS(":443", "/etc/ssl/certs/tls.crt", "/etc/ssl/certs/tls.key", nil); err != nil {
2 slog.Error("error starting admission webhook")
3 os.Exit(1)
4 }
We will be mounting server.crt and server.pem we just generated to the location /etc/ssl/certs/tls.crt and /etc/ssl/certs/tls.key using a Kubernetes secret in the validating admission controller deployment.
The following command will create a secret named validating-webhook-tls with the server.crt and the server.pem
in the webhooktest namespace.
kubectl create ns webhooktest
kubectl create secret tls validating-webhook-tls --cert certs/server.crt --key certs/server.pem -n webhooktest
Let’s describe the validating-webhook-tls secret we just created. Run kubectl describe secret -n webhooktest to describe the secret
we just created.
1Name: validating-webhook-tls
2Namespace: webhooktest
3Labels: <none>
4Annotations: <none>
5
6Type: kubernetes.io/tls
7
8Data
9====
10tls.crt: 1025 bytes
11tls.key: 1704 bytes
From the above output, we can find that the cert and key is available in data fields tls.crt and tls.key.
The following manifest will create a deployment using the validating-admission-controller docker image we built earlier. The tls.crt and tls.key fields from the validating-webhook-tls secret are mounted in location /etc/ssl/certs. These files are now available in the /etc/ssl/certs/tls.crt and /etc/ssl/certs/tls.key which is referenced in main.go. A service named validating-webhook-svc is also created which makes the validating controller deployment available on port 443 using a ClusterIP service. This service will be used by the Kube API server to call our validation webhook server.
1apiVersion: apps/v1
2kind: Deployment
3metadata:
4 name: validating-webhook
5 namespace: webhooktest
6 labels:
7 app: validating-webhook
8spec:
9 replicas: 1
10 selector:
11 matchLabels:
12 app: validating-webhook
13 template:
14 metadata:
15 labels:
16 app: validating-webhook
17 spec:
18 containers:
19 - name: validating-webhook
20 image: msgtonaveen/validating-admission-controller:v0.1.0
21 imagePullPolicy: Always
22 volumeMounts:
23 - name: tls-certs
24 mountPath: /etc/ssl/certs
25 readOnly: true
26 volumes:
27 - name: tls-certs
28 secret:
29 secretName: validating-webhook-tls
30---
31apiVersion: v1
32kind: Service
33metadata:
34 name: validating-webhook-svc
35 namespace: webhooktest
36spec:
37 type: ClusterIP
38 selector:
39 app: validating-webhook
40 ports:
41 - port: 443
42 targetPort: 7443
Save the above filed as validating-webhook-deploy.yaml and create it using the following command.
kubectl apply -f validating-webhook-deploy.yaml
The following will be output of running the above command.
deployment.apps/validating-webhook created
service/validating-webhook-svc created`
Run kubectl logs deploy/validating-webhook -n webhooktest -f to display the logs of the validating webhook pod.
The following logs will be displayed in the validation webhook pod in the webhooktest namespace.
12024/08/03 10:51:53 INFO Server started ...
The above logs confirm that the validation webhook server has been started successfully.
Validating Webhook Configuration
The final step in the creation of the validating admission controller is to register our validating admission controller
so that the Kube API server calls it. This is done by creating a ValidatingWebhookConfiguration.
1apiVersion: admissionregistration.k8s.io/v1
2kind: ValidatingWebhookConfiguration
3metadata:
4 name: deployment-resource-validation
5webhooks:
6 - name: "memoryvalidation.webhook.local"
7 namespaceSelector:
8 matchExpressions:
9 - key: kubernetes.io/metadata.name
10 operator: In
11 values: ["webhooktest"]
12 rules:
13 - operations: ["CREATE","UPDATE"]
14 apiGroups: ["apps"]
15 apiVersions: ["v1"]
16 resources: ["deployments"]
17 clientConfig:
18 service:
19 namespace: webhooktest
20 name: validating-webhook-svc
21 path: "/validate"
22 caBundle: $CA_CERT_BASE64
23 admissionReviewVersions: ["v1"]
24 timeoutSeconds: 5
25 sideEffects: None
26 failurePolicy: Fail
Save the above file as validating-webhook.yaml. In the above manifest, namespaceSelector specifies that this webhook must only be called when a deployment is created or updated in the webhooktest namespace. The clientConfig.service field specifies the namespace and the name of the validating webhook service validating-webhook-svc which we created earlier. This will be the service the Kube API server calls to reach our validating webhook controller. The ${CA_CERT_BASE64} must be replaced with the base64 encoded rootCA.crt. We will write a command which does this shortly.
The timeoutSeconds specifies the timeout for the validating controller API call. If the request times out, the failurePolicy fail specifies the webhook call will be considered as failure and the deployment creation request will fail.
The following commands set the environment variable $CA_CERT_BASE64 to the base64 encoded rootCA.crt and replaces the value of $CA_CERT_BASE64 in validating-webhook.yaml with the base64 encoded cert and creates the ValidatingWebhookConfiguration.
1export CA_CERT_BASE64=$(cat certs/rootCA.crt | base64 -w 0)
2envsubst < validating-webhook.yaml | kubectl create -f -
Testing the validating admission controller
Let’s first try to create a deployment without memory requests and limits and test whether the validating admission controller works as expected. Save the following yaml to a file named test-depl.yaml
1apiVersion: apps/v1
2kind: Deployment
3metadata:
4 labels:
5 app: test-dep
6 name: test-dep
7 namespace: webhooktest
8spec:
9 replicas: 1
10 selector:
11 matchLabels:
12 app: test-dep
13 template:
14 metadata:
15 labels:
16 app: test-dep
17 spec:
18 containers:
19 - image: busybox
20 name: busybox
21 command: [ "sleep" ]
22 args: [ "infinity" ]
The above deployment doesn’t have both memory requests and limits. Try creating the deployment using the command
kubectl apply -f test-depl.yaml
The above command will fail to execute with the following error
Error from server: error when creating "test-depl.yaml": admission webhook "memoryvalidation.webhook.local" denied the request: Memory request not specified for container busybox
The above message confirms that our validating admission controller is working successfully. Let’s go ahead and check the logs of our validating admission controller server. The following log message will be displayed in the validating webhook pod logs
12024/08/04 17:54:11 INFO admission review response json marshalled json="{\"kind\":\"AdmissionReview\",\"apiVersion\":\"admission.k8s.io/v1\",\"response\":{\"uid\":\"a0d7b6a8-99d1-44f6-a5f4-3381aaaa6e05\",\"allowed\":false,\"status\":{\"metadata\":{},\"status\":\"Failure\",\"message\":\"Memory request not specified for container busybox\"}}}"
The above logs show that "allowed":false which led to the failure of the deployment creation.
Now let’s try to add only the memory request and not the limit to the deployment.
1apiVersion: apps/v1
2kind: Deployment
3metadata:
4 labels:
5 app: test-dep
6 name: test-dep
7 namespace: webhooktest
8spec:
9 replicas: 1
10 selector:
11 matchLabels:
12 app: test-dep
13 template:
14 metadata:
15 labels:
16 app: test-dep
17 spec:
18 containers:
19 - image: busybox
20 name: busybox
21 command: [ "sleep" ]
22 args: [ "infinity" ]
23 resources:
24 requests:
25 memory: 20Mi
Creating the above deployment will fail with the following error since memory limit is not specified for the container.
Error from server: error when creating "test-depl.yaml": admission webhook "memoryvalidation.webhook.local" denied the request: Memory limit not specified for container busybox
Finally we can add both the requests and limits.
1apiVersion: apps/v1
2kind: Deployment
3metadata:
4 labels:
5 app: test-dep
6 name: test-dep
7 namespace: webhooktest
8spec:
9 replicas: 1
10 selector:
11 matchLabels:
12 app: test-dep
13 template:
14 metadata:
15 labels:
16 app: test-dep
17 spec:
18 containers:
19 - image: busybox
20 name: busybox
21 command: [ "sleep" ]
22 args: [ "infinity" ]
23 resources:
24 requests:
25 memory: 20Mi
26 limits:
27 memory: 50Mi
The above deployment will be created successfully.
Running kubectl apply -f test-depl.yaml will display the following response
deployment.apps/test-dep created
The above response confirms that the deployment has been created successfully.
The following message will be displayed in the pod logs
2024/08/04 17:56:56 INFO admission review response json marshalled json="{\"kind\":\"AdmissionReview\",\"apiVersion\":\"admission.k8s.io/v1\",\"response\":{\"uid\":\"986250e8-ec38-4833-b1e7-3893271ffb2e\",\"allowed\":true}}"
In the above logs, "allowed":true confirms that the deployment has both memory requests and limits and hence it is allowed to be created successfully.
The source for the tutorial is available at https://github.com/golangbot/validating-admission-controller
This brings us to and end of this tutorial. I hope you liked it.
Please leave your feedback and comments. Please consider sharing this tutorial on twitter and LinkedIn. Have a good day.
