Skip to content

编写一个禁止删除namespace的webhook 原创

260c724b4ebd962e7ac173a61d40ce52 MD5

昨天收到一个朋友的信息,说不小心把集群的业务 namespace 干掉了,导致整个业务都停滞了,问我有没有禁止删除 namespace 的方案。

在我的记忆里,Kubernetes 的准入里并没有这个控制器,所以我就给他说需要自己开发一个准入控制器来实现自己的目标。

作为人,何为正确!我不能只脱裤子,不放屁。所以这里也整理了一下如何自定义 Kubernetes 的准入控制器。

理论介绍

准入控制器(Admission Controller)位于 API Server 中,在对象被持久化之前,准入控制器拦截对 API Server 的请求,一般用来做身份验证和授权。其中包含两个特殊的控制器:MutatingAdmissionWebhook 和 ValidatingAdmissionWebhook。

  • MutatingAdmissionWebhook :用于变更请求对象,比如 istio 为每个 Pod 注入 sidecar,就是通过它实现。
  • ValidatingAdmissionWebhook:用于验证请求对象

整个准入控制器的流程如下:

e7f4b5bdc4137d97f21c52ec32368db0 MD5

当 API 请求进入时,mutating 和 validating 控制器使用配置中的外部 webhooks 列表并发调用,规则如下:

  • 如果所有的 webhooks 批准请求,准入控制链继续流转。
  • 如果有任意一个 webhooks 阻止请求,那么准入控制请求终止,并返回第一个 webhook 阻止的原因。其中,多个 webhooks 阻止也只会返回第一个 webhook 阻止的原因。
  • 如果在调用 webhook 过程中发生错误,那么请求会被终止或者忽略 webhook。

准入控制器是在 API Server 的启动参数重配置的。一个准入控制器可能属于以上两者中的一种,也可能两者都属于。

我们在部署 Kubernetes 集群的时候都会默认开启一系列准入控制器,如果没有设置这些准入控制器的话可以说你的 Kubernetes 集群就是在裸奔,应该叫管理员为集群添加准入控制器。

代码实现

实现逻辑

在开发之前先大致了解一下准入控制器的 Webhook 的大致实现逻辑:

  • Webhook 是一个标准的 HTTP 服务,接收 HTTP 请求
  • 接收到的请求是一个 AdmissionReview 对象
  • 然后我们自定义的 Hook 会处理这个 AdmissionReview 对象
  • 处理完过后再返回一个 AdmissionReview 对象,这里面会包含处理结果

AdmissionReview 的结构体如下:

go

// AdmissionReview describes an admission review request/response.

type AdmissionReview struct {

  metav1.TypeMeta `json:",inline"`

  // Request describes the attributes for the admission request.

  // +optional

  Request *AdmissionRequest `json:"request,omitempty" protobuf:"bytes,1,opt,name=request"`

  // Response describes the attributes for the admission response.

  // +optional

  Response *AdmissionResponse `json:"response,omitempty" protobuf:"bytes,2,opt,name=response"`

}

从代码的命名中可以很清晰的看出,在请求发送到 WebHook 时我们只需要关注内部的 AdmissionRequest(实际入参),在我们编写的 WebHook 处理完成后只需要返回包含有 AdmissionResponse(实际返回体) 的 AdmissionReview 对象即可;总的来说 AdmissionReview 对象是个套壳,请求是里面的 AdmissionRequest,响应是里面的 AdmissionResponse。

7254d6fc73eff15f7e4f0513461ab2e1 MD5

具体实现

(1)首先创建一个 HTTP Server,监听端口,接收请求

go

package main



import (

    "context"

    "flag"

    "github.com/joker-bai/validate-namespace/http"

    log "k8s.io/klog/v2"

    "os"

    "os/signal"

    "syscall"

)



var (

    tlscert, tlskey, port string

)



func main() {

    flag.StringVar(&tlscert, "tlscert", "/etc/certs/cert.pem", "Path to the TLS certificate")

    flag.StringVar(&tlskey, "tlskey", "/etc/certs/key.pem", "Path to the TLS key")

    flag.StringVar(&port, "port", "8443", "The port to listen")

    flag.Parse()

    server := http.NewServer(port)

    go func() {

        if err := server.ListenAndServeTLS(tlscert, tlskey); err != nil {

            log.Errorf("Failed to listen and serve: %v", err)

        }

    }()

    log.Infof("Server running in port: %s", port)

    // listen shutdown signal

    signalChan := make(chan os.Signal, 1)

    signal.Notify(signalChan, syscall.SIGINT, syscall.SIGTERM)

    <-signalChan

    log.Info("Shutdown gracefully...")

    if err := server.Shutdown(context.Background()); err != nil {

        log.Error(err)

    }

}

由于准入控制器和 Webhook 之间需要使用 TLS 进行通信,所以上面监听的端口是 TLS 端口,通过server.ListenAndServeTLS实现,后续在部署服务的时候需要把证书挂到相应的目录中。

(2)定义 Handler,将请求分发到具体的处理方法

go

package http



import (

  "fmt"

  "github.com/joker-bai/validate-namespace/namespace"

  "net/http"

)



// NewServer creates and return a http.Server

func NewServer(port string) *http.Server {

  // Instances hooks

  nsValidation := namespace.NewValidationHook()



  // Routers

  ah := newAdmissionHandler()

  mux := http.NewServeMux()

  mux.Handle("/healthz", healthz())

  mux.Handle("/validate/delete-namespace", ah.Serve(nsValidation))



  return &http.Server{

    Addr:    fmt.Sprintf(":%s", port),

    Handler: mux,

  }

}

实现 admissionHandler,主要作用是将 http body 的内容解析成 AdmissionReview 对象,然后调用具体的 Hook 处理,再将结果放到 AdmissionReview 中,返回给客户端。

go

package http



import (

  "encoding/json"

  "fmt"

  "io"

  "net/http"



  "github.com/douglasmakey/admissioncontroller"



  "k8s.io/api/admission/v1beta1"

  admission "k8s.io/api/admission/v1beta1"

  meta "k8s.io/apimachinery/pkg/apis/meta/v1"

  "k8s.io/apimachinery/pkg/runtime"

  "k8s.io/apimachinery/pkg/runtime/serializer"

  log "k8s.io/klog/v2"

)



// admissionHandler represents the HTTP handler for an admission webhook

type admissionHandler struct {

  decoder runtime.Decoder

}



// newAdmissionHandler returns an instance of AdmissionHandler

func newAdmissionHandler() *admissionHandler {

  return &admissionHandler{

    decoder: serializer.NewCodecFactory(runtime.NewScheme()).UniversalDeserializer(),

  }

}



// Serve returns a http.HandlerFunc for an admission webhook

func (h *admissionHandler) Serve(hook admissioncontroller.Hook) http.HandlerFunc {

  return func(w http.ResponseWriter, r *http.Request) {

    w.Header().Set("Content-Type", "application/json")

    if r.Method != http.MethodPost {

      http.Error(w, fmt.Sprint("invalid method only POST requests are allowed"), http.StatusMethodNotAllowed)

      return

    }



    if contentType := r.Header.Get("Content-Type"); contentType != "application/json" {

      http.Error(w, fmt.Sprint("only content type 'application/json' is supported"), http.StatusBadRequest)

      return

    }



    body, err := io.ReadAll(r.Body)

    if err != nil {

      http.Error(w, fmt.Sprintf("could not read request body: %v", err), http.StatusBadRequest)

      return

    }



    var review admission.AdmissionReview

    if _, _, err := h.decoder.Decode(body, nil, &review); err != nil {

      http.Error(w, fmt.Sprintf("could not deserialize request: %v", err), http.StatusBadRequest)

      return

    }



    if review.Request == nil {

      http.Error(w, "malformed admission review: request is nil", http.StatusBadRequest)

      return

    }



    result, err := hook.Execute(review.Request)

    if err != nil {

      log.Error(err)

      w.WriteHeader(http.StatusInternalServerError)

      return

    }



    admissionResponse := v1beta1.AdmissionReview{

      Response: &v1beta1.AdmissionResponse{

        UID:     review.Request.UID,

        Allowed: result.Allowed,

        Result:  &meta.Status{Message: result.Msg},

      },

    }



    res, err := json.Marshal(admissionResponse)

    if err != nil {

      log.Error(err)

      http.Error(w, fmt.Sprintf("could not marshal response: %v", err), http.StatusInternalServerError)

      return

    }



    log.Infof("Webhook [%s - %s] - Allowed: %t", r.URL.Path, review.Request.Operation, result.Allowed)

    w.WriteHeader(http.StatusOK)

    w.Write(res)

  }

}



func healthz() http.HandlerFunc {

  return func(w http.ResponseWriter, r *http.Request) {

    w.WriteHeader(http.StatusOK)

    w.Write([]byte("ok"))

  }

}

上面处理是通过hook.Execute来处理请求,这是 admissionController 内部实现的一个结构体,它为每个操作定义了一个方法,如下:

go

// AdmitFunc defines how to process an admission request

type AdmitFunc func(request *admission.AdmissionRequest) (*Result, error)



// Hook represents the set of functions for each operation in an admission webhook.

type Hook struct {

  Create  AdmitFunc

  Delete  AdmitFunc

  Update  AdmitFunc

  Connect AdmitFunc

}

我们就需要实现具体的 AdmitFunc,并注册。

(3)将自己实现的方法注册到 Hook 中。

go

package namespace



import (

  "github.com/douglasmakey/admissioncontroller"

)



// NewValidationHook delete namespace validation hook

func NewValidationHook() admissioncontroller.Hook {

  return admissioncontroller.Hook{

    Delete: validateDelete(),

  }

}

(4)实现具体的 AdmitFunc

go

package namespace



import (

  "github.com/douglasmakey/admissioncontroller"

  log "k8s.io/klog/v2"



  "k8s.io/api/admission/v1beta1"

)



func validateDelete() admissioncontroller.AdmitFunc {

  return func(r *v1beta1.AdmissionRequest) (*admissioncontroller.Result, error) {

    if r.Kind.Kind == "Namespace" {

      log.Info("You cannot delete namespace: ", r.Name)

      return &admissioncontroller.Result{Allowed: false}, nil

    } else {

      return &admissioncontroller.Result{Allowed: true}, nil

    }

  }

}

这里实现很简单,如果 Kind 为 Namespace,就拒绝操作。

部署测试

上面完成了业务逻辑开发,下面就把它部署到 Kubernetes 集群测试一番。

部署

(1)编写 Dockerfile,将应用打包成镜像

dockerfile

FROM golang:1.17.5 AS build-env

ENV GOPROXY https://goproxy.cn

ADD . /go/src/app

WORKDIR /go/src/app

RUN go mod tidy

RUN cd cmd && GOOS=linux GOARCH=amd64 go build -v -a -ldflags '-extldflags "-static"' -o /go/src/app/app-server /go/src/app/cmd/main.go



FROM registry.cn-hangzhou.aliyuncs.com/coolops/ubuntu:22.04

ENV TZ=Asia/Shanghai

COPY --from=build-env /go/src/app/app-server /opt/app-server

WORKDIR /opt

EXPOSE 80

CMD [ "./app-server" ]

(2)创建 TLS 证书,使用脚本进行创建

shell

#!/bin/bash



set -e



usage() {

    cat <<EOF

Generate certificate suitable for use with an sidecar-injector webhook service.



This script uses k8s' CertificateSigningRequest API to a generate a

certificate signed by k8s CA suitable for use with sidecar-injector webhook

services. This requires permissions to create and approve CSR. See

https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster for

detailed explantion and additional instructions.



The server key/cert k8s CA cert are stored in a k8s secret.



usage: ${0} [OPTIONS]



The following flags are required.



       --service          Service name of webhook.

       --namespace        Namespace where webhook service and secret reside.

       --secret           Secret name for CA certificate and server certificate/key pair.

EOF

    exit 1

}



while [[ $## -gt 0 ]]; do

    case ${1} in

        --service)

            service="$2"

            shift

            ;;

        --secret)

            secret="$2"

            shift

            ;;

        --namespace)

            namespace="$2"

            shift

            ;;

        *)

            usage

            ;;

    esac

    shift

done



[ -z ${service} ] && service=validate-delete-namespace

[ -z ${secret} ] && secret=validate-delete-namespace-tls

[ -z ${namespace} ] && namespace=default



if [ ! -x "$(command -v openssl)" ]; then

    echo "openssl not found"

    exit 1

fi



csrName=${service}.${namespace}

tmpdir=$(mktemp -d)

echo "creating certs in tmpdir ${tmpdir} "



cat <<EOF >> ${tmpdir}/csr.conf

[req]

req_extensions = v3_req

distinguished_name = req_distinguished_name

[req_distinguished_name]

[ v3_req ]

basicConstraints = CA:FALSE

keyUsage = nonRepudiation, digitalSignature, keyEncipherment

extendedKeyUsage = serverAuth

subjectAltName = @alt_names

[alt_names]

DNS.1 = ${service}

DNS.2 = ${service}.${namespace}

DNS.3 = ${service}.${namespace}.svc

EOF



openssl genrsa -out ${tmpdir}/server-key.pem 2048

openssl req -new -key ${tmpdir}/server-key.pem -subj "/CN=${service}.${namespace}.svc" -out ${tmpdir}/server.csr -config ${tmpdir}/csr.conf



## clean-up any previously created CSR for our service. Ignore errors if not present.

kubectl delete csr ${csrName} 2>/dev/null || true



## create  server cert/key CSR and  send to k8s API

cat <<EOF | kubectl create -f -

apiVersion: certificates.k8s.io/v1beta1

kind: CertificateSigningRequest

metadata:

  name: ${csrName}

spec:

  groups:

  - system:authenticated

  request: $(cat ${tmpdir}/server.csr | base64 | tr -d '\n')

  usages:

  - digital signature

  - key encipherment

  - server auth

EOF



## verify CSR has been created

while true; do

    kubectl get csr ${csrName}

    if [ "$?" -eq 0 ]; then

        break

    fi

done



## approve and fetch the signed certificate

kubectl certificate approve ${csrName}

## verify certificate has been signed

for x in $(seq 10); do

    serverCert=$(kubectl get csr ${csrName} -o jsonpath='{.status.certificate}')

    if [[ ${serverCert} != '' ]]; then

        break

    fi

    sleep 1

done

if [[ ${serverCert} == '' ]]; then

    echo "ERROR: After approving csr ${csrName}, the signed certificate did not appear on the resource. Giving up after 10 attempts." >&2

    exit 1

fi

echo ${serverCert} | openssl base64 -d -A -out ${tmpdir}/server-cert.pem




## create the secret with CA cert and server cert/key

kubectl create secret generic ${secret} \

        --from-file=key.pem=${tmpdir}/server-key.pem \

        --from-file=cert.pem=${tmpdir}/server-cert.pem \

        --dry-run -o yaml |

    kubectl -n ${namespace} apply -f -

(3)编写 Deployment 部署服务

yaml

apiVersion: apps/v1

kind: Deployment

metadata:

  name: validate-delete-namespace

  labels:

    app: validate-delete-namespace

spec:

  replicas: 1

  selector:

    matchLabels:

      app: validate-delete-namespace

  template:

    metadata:

      labels:

        app: validate-delete-namespace

    spec:

      containers:

        - name: server

          image: registry.cn-hangzhou.aliyuncs.com/coolops/validate-delete-namespace:latest

          imagePullPolicy: Always

          livenessProbe:

            httpGet:

              path: /healthz

              port: 8443

              scheme: HTTPS

          ports:

            - containerPort: 8443

          volumeMounts:

            - name: tls-certs

              mountPath: /etc/certs

              readOnly: true

      volumes:

        - name: tls-certs

          secret:

            secretName: validate-delete-namespace-tls

---

apiVersion: v1

kind: Service

metadata:

  name: validate-delete-namespace

spec:

  selector:

    app: validate-delete-namespace

  ports:

    - port: 443

      targetPort: 8443

(4)部署 Webhook

go

apiVersion: admissionregistration.k8s.io/v1beta1

kind: ValidatingWebhookConfiguration

metadata:

  name: validate-delete-namespace

webhooks:

  - name: validate-delete-namespace.default.svc.cluster.local

    clientConfig:

      service:

        namespace: default

        name: validate-delete-namespace

        path: "/validate/delete-namespace"

      caBundle: "${CA_BUNDLE}"

    rules:

      - operations:

          - DELETE

        apiGroups:

          - ""

        apiVersions:

          - "v1"

        resources:

          - namespaces

    failurePolicy: Ignore

这里有一个${CA_BUNDLE}占位符,在创建 Webhook 的时候要将起替换掉,使用如下命令:

shell

cat ./validate-delete-namespace.yaml | sh ./patch-webhook-ca.sh > ./webhook.yaml

然后创建 webhook.yaml 即可。

shell

kubectl apply -f webhook.yaml

上面的所有文件都在代码库里,可以直接使用脚本进行部署。

shell

## sh deploy.sh

creating certs in tmpdir /tmp/tmp.SvMHWcPI6x

Generating RSA private key, 2048 bit long modulus

..........................................+++

.............................................................+++

e is 65537 (0x10001)

certificatesigningrequest.certificates.k8s.io/validate-delete-namespace.default created

NAME                                AGE   REQUESTOR          CONDITION

validate-delete-namespace.default   0s    kubernetes-admin   Pending

certificatesigningrequest.certificates.k8s.io/validate-delete-namespace.default approved

secret/validate-delete-namespace-tls created

Creating k8s admission deployment

deployment.apps/validate-delete-namespace created

service/validate-delete-namespace created

validatingwebhookconfiguration.admissionregistration.k8s.io/validate-delete-namespace created

执行完成过后,可以查看具体的信息。

shell

## kubectl get po

NAME                                         READY   STATUS    RESTARTS   AGE

validate-delete-namespace-74c9b8b7bd-5g9zv   1/1     Running   0          3s

## kubectl get secret

NAME                            TYPE                                  DATA   AGE

default-token-kx5wf             kubernetes.io/service-account-token   3      72d

validate-delete-namespace-tls   Opaque                                2      53s

## kubectl get ValidatingWebhookConfiguration

NAME                                  CREATED AT

validate-delete-namespace             2022-06-24T09:39:26Z

测试

(1)首先打开 webhook 的 pod 日志

shell

## kubectl logs validate-delete-namespace-74c9b8b7bd-5g9zv -f

I0624 17:39:27.858753       1 main.go:30] Server running in port: 8443

(2)创建一个 namespace 并删除

shell

## kubectl create ns joker

## kubectl get ns | grep joker

joker                             Active   4h5m

## kubectl delete ns joker

Error from server: admission webhook "validate-delete-namespace.default.svc.cluster.local" denied the request without explanation

## kubectl get ns | grep joker

joker                             Active   4h5m

可以发现我们的删除操作被拒绝了,并且查看 namespace 还存在。

我们也可以到日志中查看,如下:

shell

## kubectl logs validate-delete-namespace-74c9b8b7bd-5g9zv -f

I0624 17:39:27.858753       1 main.go:30] Server running in port: 8443

2022/06/24 17:43:34 You cannot delete namespace:  joker

I0624 17:43:34.664945       1 handler.go:94] Webhook [/validate/delete-namespace - DELETE] - Allowed: false

2022/06/24 17:43:34 You cannot delete namespace:  joker

I0624 17:43:34.667043       1 handler.go:94] Webhook [/validate/delete-namespace - DELETE] - Allowed: false

上面就是简单的实现了一个准入控制器,只要思想不滑坡,办法总比困难多。

感谢万能的百度,感谢牛逼的网友。

参考

最近更新