乔克
乔克
Published on 2024-11-15 / 35 Visits
1
0

修改Kubernetes证书时间到100年(kubeadm版)

(1)、查看当前的证书时间

# kubeadm alpha certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Jun 20, 2021 11:21 UTC   364d                                    no      
apiserver                  Jun 20, 2021 11:21 UTC   364d            ca                      no      
apiserver-etcd-client      Jun 20, 2021 11:21 UTC   364d            etcd-ca                 no      
apiserver-kubelet-client   Jun 20, 2021 11:21 UTC   364d            ca                      no      
controller-manager.conf    Jun 20, 2021 11:21 UTC   364d                                    no      
etcd-healthcheck-client    Jun 20, 2021 11:21 UTC   364d            etcd-ca                 no      
etcd-peer                  Jun 20, 2021 11:21 UTC   364d            etcd-ca                 no      
etcd-server                Jun 20, 2021 11:21 UTC   364d            etcd-ca                 no      
front-proxy-client         Jun 20, 2021 11:21 UTC   364d            front-proxy-ca          no      
scheduler.conf             Jun 20, 2021 11:21 UTC   364d                                    no      

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Jun 18, 2030 11:21 UTC   9y              no      
etcd-ca                 Jun 18, 2030 11:21 UTC   9y              no      
front-proxy-ca          Jun 18, 2030 11:21 UTC   9y              no      

(2)、下载源码

git clone https://github.com/kubernetes/kubernetes.git

(3)、切换到自己的版本,修改源码,比如我的是 v1.17.2 版本

cd kubernetes
git checkout v1.17.2

vim cmd/kubeadm/app/constants/constants.go,找到 CertificateValidity,修改如下

....
const (
        // KubernetesDir is the directory Kubernetes owns for storing various configuration files
        KubernetesDir = "/etc/kubernetes"
        // ManifestsSubDirName defines directory name to store manifests
        ManifestsSubDirName = "manifests"
        // TempDirForKubeadm defines temporary directory for kubeadm
        // should be joined with KubernetesDir.
        TempDirForKubeadm = "tmp"

        // CertificateValidity defines the validity for all the signed certificates generated by kubeadm
        CertificateValidity = time.Hour * 24 * 365 * 100
....

(4)、编译 kubeadm

make WHAT=cmd/kubeadm

编译完生成如下目录和二进制文件

# ll _output/bin/
total 76172
-rwxr-xr-x 1 root root  6799360 Jun 20 21:08 conversion-gen
-rwxr-xr-x 1 root root  6778880 Jun 20 21:08 deepcopy-gen
-rwxr-xr-x 1 root root  6750208 Jun 20 21:08 defaulter-gen
-rwxr-xr-x 1 root root  4883629 Jun 20 21:08 go2make
-rwxr-xr-x 1 root root  2109440 Jun 20 21:09 go-bindata
-rwxr-xr-x 1 root root 39256064 Jun 20 21:11 kubeadm
-rwxr-xr-x 1 root root 11419648 Jun 20 21:09 openapi-gen

(5)、备份原 kubeadm 和证书文件

cp /usr/bin/kubeadm{,.bak20200620}
cp -r /etc/kubernetes/pki{,.bak20200620}

(7)、将新生成的 kubeadm 进行替换

cp _output/bin/kubeadm /usr/bin/kubeadm

(8)、生成新的证书

cd /etc/kubernetes/pki
kubeadm alpha certs renew all

输出如下

[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'

certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed

(9)、验证结果

kubeadm alpha certs check-expiration

输出如下

[root@k8s-master pki]#  kubeadm alpha certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 May 27, 2120 13:25 UTC   99y                                     no      
apiserver                  May 27, 2120 13:25 UTC   99y             ca                      no      
apiserver-etcd-client      May 27, 2120 13:25 UTC   99y             etcd-ca                 no      
apiserver-kubelet-client   May 27, 2120 13:25 UTC   99y             ca                      no      
controller-manager.conf    May 27, 2120 13:25 UTC   99y                                     no      
etcd-healthcheck-client    May 27, 2120 13:25 UTC   99y             etcd-ca                 no      
etcd-peer                  May 27, 2120 13:25 UTC   99y             etcd-ca                 no      
etcd-server                May 27, 2120 13:25 UTC   99y             etcd-ca                 no      
front-proxy-client         May 27, 2120 13:25 UTC   99y             front-proxy-ca          no      
scheduler.conf             May 27, 2120 13:25 UTC   99y                                     no      

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Jun 18, 2030 11:21 UTC   9y              no      
etcd-ca                 Jun 18, 2030 11:21 UTC   9y              no      
front-proxy-ca          Jun 18, 2030 11:21 UTC   9y              no      

查看集群状态是否 OK。

[root@k8s-master pki]# kubectl get node
NAME         STATUS   ROLES    AGE    VERSION
k8s-master   Ready    master   127m   v1.17.2
k8s-node01   Ready    <none>   94m    v1.17.2
k8s-node02   Ready    <none>   95m    v1.17.2
[root@k8s-master pki]# kubectl get pod -n kube-system 
NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-589b5f594b-76vwr   1/1     Running   0          93m
calico-node-4qvfj                          1/1     Running   0          93m
calico-node-cn79s                          1/1     Running   0          93m
calico-node-sppn9                          1/1     Running   0          93m
coredns-7f9c544f75-hc5q5                   1/1     Running   0          127m
coredns-7f9c544f75-z77s8                   1/1     Running   0          127m
etcd-k8s-master                            1/1     Running   0          114m
kube-apiserver-k8s-master                  1/1     Running   0          115m
kube-controller-manager-k8s-master         1/1     Running   0          114m
kube-proxy-6kckk                           1/1     Running   0          94m
kube-proxy-r7mn2                           1/1     Running   0          127m
kube-proxy-zf48c                           1/1     Running   0          95m
kube-scheduler-k8s-master                  1/1     Running   0          114m

更新 kubeconfig

kubeadm init phase kubeconfig all --config kubeadm.yaml
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/admin.conf"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/scheduler.conf"

将新生成的 admin 配置文件覆盖掉原本的 admin 文件:

mv $HOME/.kube/config $HOME/.kube/config.old
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

完成后重启 kube-apiserver、kube-controller、kube-scheduler、etcd 这 4 个容器即可,我们可以查看 apiserver 的证书的有效期来验证是否更新成功:

$ echo | openssl s_client -showcerts -connect 127.0.0.1:6443 -servername api 2>/dev/null | openssl x509 -noout -enddate
notAfter=Aug 26 03:47:23 2021 GMT

到此证书修改完成。

如果 github 上下载很慢的话可以到 gitee 上下载,地址:https://gitee.com/mirrors/Kubernetes/tree/master/

不过证书修改虽然完成了,但是 kubelet 的证书并没有更新,这时候我们可以开启证书自动轮转。
(1)增加 kubelet 参数
修改 /usr/lib/systemd/system/kubelet.service

--feature-gates=RotateKubeletServerCertificate=true

(2)增加 controller-manager 参数
修改 controller-manager 的 yaml 文件

--experimental-cluster-signing-duration=87600h0m0s
--feature-gates=RotateKubeletServerCertificate=true

(3)创建 rbac 对象
创建 rbac 对象,允许节点轮换 kubelet server 证书:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver
rules:
- apiGroups:
  - certificates.k8s.io
  resources:
  - certificatesigningrequests/selfnodeserver
  verbs:
  - create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubeadm:node-autoapprove-certificate-server
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:nodes

查看证书时间

openssl x509 -in ca.crt -noout -text | grep "Not"

Comment