NGINX容器热更新
Nginx 作为 WEB 服务器被广泛使用。其自身支持热更新，在修改配置文件后，使用nginx -s reload命令可以不停服务重新加载配置。然而对于 Dockerize 的 Nginx 来说，如果每次都进到容器里执行对应命令去实现配置重载，这个过程是很痛苦的。本文介绍了一种 kubernetes 集群下 nginx 的热更新方案。
首先我们创建正常的一个 nginx 资源，资源清单如下：
yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: nginx-config
data:
default.conf: |-
server {
server_name localhost;
listen 80 default_server;
location = /healthz {
add_header Content-Type text/plain;
return 200 'ok';
}
location / {
root /usr/share/nginx/html;
index index.html index.htm;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
spec:
replicas: 1
selector:
matchLabels:
app: my-app
template:
metadata:
labels:
app: my-app
spec:
containers:
- name: my-app
image: nginx
imagePullPolicy: IfNotPresent
volumeMounts:
- name: nginx-config
mountPath: /etc/nginx/conf.d
volumes:
- name: nginx-config
configMap:
name: nginx-config然后创建资源对象。
yaml
# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
my-app-9bdd6cbbc-x9gnt 1/1 Running 0 112s 192.168.58.197 k8s-node02 <none> <none>然后我们访问 pod 资源，如下：
yaml
# curl -I 192.168.58.197
HTTP/1.1 200 OK
Server: nginx/1.17.10
Date: Tue, 26 May 2020 06:18:18 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 14 Apr 2020 14:19:26 GMT
Connection: keep-alive
ETag: "5e95c66e-264"
Accept-Ranges: bytes现在我们来更新一下 ConfigMap，也就是更改配置文件如下：
yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: nginx-config
data:
default.conf: |-
server {
server_name localhost;
listen 8080 default_server;
location = /healthz {
add_header Content-Type text/plain;
return 200 'ok';
}
location / {
root /usr/share/nginx/html;
index index.html index.htm;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
}等待数秒...
然后我们可以看到 nginx pod 里的配置信息已经更改为如下：
bash
# kubectl exec -it my-app-9bdd6cbbc-x9gnt -- /bin/bash
root@my-app-9bdd6cbbc-x9gnt:/# cat /etc/nginx/conf.d/default.conf
server {
server_name localhost;
listen 8080 default_server;
location = /healthz {
add_header Content-Type text/plain;
return 200 'ok';
}
location / {
root /usr/share/nginx/html;
index index.html index.htm;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
}
root@my-app-9bdd6cbbc-x9gnt:/#这时候我们访问 8080 是不通的，访问 80 是没问题的，如下：
bash
[root@k8s-master nginx]# curl -I 192.168.58.197
HTTP/1.1 200 OK
Server: nginx/1.17.10
Date: Tue, 26 May 2020 06:21:05 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 14 Apr 2020 14:19:26 GMT
Connection: keep-alive
ETag: "5e95c66e-264"
Accept-Ranges: bytes
[root@k8s-master nginx]# curl -I 192.168.58.197:8080
curl: (7) Failed connect to 192.168.58.197:8080; Connection refused我们可以看到，我们需要的配置文件已经更新的，但是并没有使用上，pod 里的 nginx 也没有重载配置文件，这时候如果我们重新部署 Pod，资源对象肯定就生效了。
但是这并不我们想要的效果，我们希望配置文件更改了，服务也跟着 reload，并不需要我们手动的去干预。
目前有三种方法：
- 应用本身可以检测配置文件，然后自动 reload
- 给 Pod 增加一个 sidecar，用它来检测配置文件
- 第三方组件 reloader，在 deployment 的 annotations 增加字段
reloader.stakater.com/auto: "true",即可检测 configmap 的更改来重启 pod
应用本身检测的话这里就不做介绍了。这里主要来实验一下第 2，3 种方法
一、以 sidecar 形式
1.1、方法
- Kubernetes 集群中部署 Nginx Pod。该 Pod 包含两个 Container，一个是 nginx container，实现 nginx 自身的功能；另一个是 nginx-reloader container，负责实时监测目标 configmap 的变化，当发现 configmap 更新以后，会主动向 nginx 的 master 进程发送 HUP 信号，实现配置的热加载。
- 配置文件是通过 ConfigMap 的形式挂载到 Nginx Pod 上，两个 Container 共享该 ConfigMap。
- 依赖 K8s 集群的 shareProcessNamespace 特性（版本需在 1.12 之后），两个 Container 需要在 Pod 中共享进程名字空间。
1.2、实现
1.2.1、镜像制作
（1）、主容器使用官方 NG 容器即可
（2）、sidecar 容器制作
Dockerfile 如下：
dockerfile
FROM golang:1.12.0 as build
RUN go get github.com/fsnotify/fsnotify
RUN go get github.com/shirou/gopsutil/process
RUN mkdir -p /go/src/app
ADD main.go /go/src/app/
WORKDIR /go/src/app
RUN CGO_ENABLED=0 GOOS=linux go build -a -o nginx-reloader .
# main image
FROM nginx:1.14.2-alpine
COPY --from=build /go/src/app/nginx-reloader /
CMD ["/nginx-reloader"]main.go 脚本如下：
go
package main
import (
"log"
"os"
"path/filepath"
"syscall"
"github.com/fsnotify/fsnotify"
proc "github.com/shirou/gopsutil/process"
)
const (
nginxProcessName = "nginx"
defaultNginxConfPath = "/etc/nginx"
watchPathEnvVarName = "WATCH_NGINX_CONF_PATH"
)
var stderrLogger = log.New(os.Stderr, "error: ", log.Lshortfile)
var stdoutLogger = log.New(os.Stdout, "", log.Lshortfile)
func getMasterNginxPid() (int, error) {
processes, processesErr := proc.Processes()
if processesErr != nil {
return 0, processesErr
}
nginxProcesses := map[int32]int32{}
for _, process := range processes {
processName, processNameErr := process.Name()
if processNameErr != nil {
return 0, processNameErr
}
if processName == nginxProcessName {
ppid, ppidErr := process.Ppid()
if ppidErr != nil {
return 0, ppidErr
}
nginxProcesses[process.Pid] = ppid
}
}
var masterNginxPid int32
for pid, ppid := range nginxProcesses {
if ppid == 0 {
masterNginxPid = pid
break
}
}
stdoutLogger.Println("found master nginx pid:", masterNginxPid)
return int(masterNginxPid), nil
}
func signalNginxReload(pid int) error {
stdoutLogger.Printf("signaling master nginx process (pid: %d) -> SIGHUP\n", pid)
nginxProcess, nginxProcessErr := os.FindProcess(pid)
if nginxProcessErr != nil {
return nginxProcessErr
}
return nginxProcess.Signal(syscall.SIGHUP)
}
func main() {
watcher, watcherErr := fsnotify.NewWatcher()
if watcherErr != nil {
stderrLogger.Fatal(watcherErr)
}
defer watcher.Close()
done := make(chan bool)
go func() {
for {
select {
case event, ok := <-watcher.Events:
if !ok {
return
}
if event.Op&fsnotify.Create == fsnotify.Create {
if filepath.Base(event.Name) == "..data" {
stdoutLogger.Println("config map updated")
nginxPid, nginxPidErr := getMasterNginxPid()
if nginxPidErr != nil {
stderrLogger.Printf("getting master nginx pid failed: %s", nginxPidErr.Error())
continue
}
if err := signalNginxReload(nginxPid); err != nil {
stderrLogger.Printf("signaling master nginx process failed: %s", err)
}
}
}
case err, ok := <-watcher.Errors:
if !ok {
return
}
stderrLogger.Printf("received watcher.Error: %s", err)
}
}
}()
pathToWatch, ok := os.LookupEnv(watchPathEnvVarName)
if !ok {
pathToWatch = defaultNginxConfPath
}
stdoutLogger.Printf("adding path: `%s` to watch\n", pathToWatch)
if err := watcher.Add(pathToWatch); err != nil {
stderrLogger.Fatal(err)
}
<-done
}1.2.2、部署 NG
（1）、NG 的配置以 configMap 进行部署：
nginx-config.yaml
yaml
// nginx-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: nginx-config
data:
default.conf: |-
server {
server_name localhost;
listen 80 default_server;
location = /healthz {
add_header Content-Type text/plain;
return 200 'ok';
}
location / {
root /usr/share/nginx/html;
index index.html index.htm;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
}（2）、NG 的 Deployment 清单(需打开共享进程命名空间特性：shareProcessNamespace: true)：
nginx-deploy.yaml
yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
name: nginx
labels:
app: nginx
spec:
shareProcessNamespace: true
containers:
- name: nginx
image: nginx
imagePullPolicy: IfNotPresent
volumeMounts:
- name: nginx-config
mountPath: /etc/nginx/conf.d
readOnly: true
- name: nginx-reloader
image: registry.cn-hangzhou.aliyuncs.com/rookieops/nginx-reloader:v1
imagePullPolicy: IfNotPresent
env:
- name: WATCH_NGINX_CONF_PATH
value: /etc/nginx/conf.d
volumeMounts:
- name: nginx-config
mountPath: /etc/nginx/conf.d
readOnly: true
volumes:
- name: nginx-config
configMap:
name: nginx-config手动修改 configmap 后，reloader 监测到 configmap 变化，会主动向 nginx 主进程发起 HUP 信号，实现配置热更新。
二、第三方插件 reloader
项目地址：https://github.com/stakater/Reloader
资源清单如下，我修改了镜像地址：
yaml
---
# Source: reloader/templates/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
labels:
app: reloader-reloader
chart: "reloader-v0.0.58"
release: "reloader"
heritage: "Tiller"
name: reloader-reloader-role
namespace: default
rules:
- apiGroups:
- ""
resources:
- secrets
- configmaps
verbs:
- list
- get
- watch
- apiGroups:
- "apps"
resources:
- deployments
- daemonsets
- statefulsets
verbs:
- list
- get
- update
- patch
- apiGroups:
- "extensions"
resources:
- deployments
- daemonsets
verbs:
- list
- get
- update
- patch
---
# Source: reloader/templates/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
labels:
app: reloader-reloader
chart: "reloader-v0.0.58"
release: "reloader"
heritage: "Tiller"
name: reloader-reloader-role-binding
namespace: default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: reloader-reloader-role
subjects:
- kind: ServiceAccount
name: reloader-reloader
namespace: default
---
# Source: reloader/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: reloader-reloader
chart: "reloader-v0.0.58"
release: "reloader"
heritage: "Tiller"
group: com.stakater.platform
provider: stakater
version: v0.0.58
name: reloader-reloader
spec:
replicas: 1
revisionHistoryLimit: 2
selector:
matchLabels:
app: reloader-reloader
release: "reloader"
template:
metadata:
labels:
app: reloader-reloader
chart: "reloader-v0.0.58"
release: "reloader"
heritage: "Tiller"
group: com.stakater.platform
provider: stakater
version: v0.0.58
spec:
containers:
- env:
image: "registry.cn-hangzhou.aliyuncs.com/rookieops/stakater-reloader:v0.0.58"
imagePullPolicy: IfNotPresent
name: reloader-reloader
args:
serviceAccountName: reloader-reloader
---
# Source: reloader/templates/role.yaml
---
# Source: reloader/templates/rolebinding.yaml
---
# Source: reloader/templates/service.yaml
---
# Source: reloader/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
app: reloader-reloader
chart: "reloader-v0.0.58"
release: "reloader"
heritage: "Tiller"
name: reloader-reloader然后部署资源，结果如下：
yaml
kubectl get pod
NAME READY STATUS RESTARTS AGE
my-app-9bdd6cbbc-x9gnt 1/1 Running 0 38m
reloader-reloader-ff767bb8-cpzgz 1/1 Running 0 56s然后给 deployment 增加一个 annotations。如下：
yaml
kubectl patch deployments.apps my-app -p '{"metadata": {"annotations": {"reloader.stakater.com/auto": "true"}}}'然后我们更改 configMap 清单，重新 apply 过后，我们可以看到 pod 会删除重启，如下：
yaml
kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
my-app-7c4fc77f5f-w4mbn 1/1 Running 0 3s 192.168.58.202 k8s-node02 <none> <none>
my-app-df6fbdb67-bnftb 1/1 Terminating 0 35s 192.168.58.201 k8s-node02 <none> <none>
reloader-reloader-ff767bb8-cpzgz 1/1 Running 0 3m47s 192.168.85.195 k8s-node01 <none> <none>然后我们 curl pod 也可以通了，如下：
yaml
# curl 192.168.58.202:8080 -I
HTTP/1.1 200 OK
Server: nginx/1.17.10
Date: Tue, 26 May 2020 06:58:38 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 14 Apr 2020 14:19:26 GMT
Connection: keep-alive
ETag: "5e95c66e-264"
Accept-Ranges: bytes三、附加
附加一个 sidecar 形式的 python 脚本
python
#!/usr/bin/env python
# -*- encoding: utf8 -*-
"""
需求：nginx配置文件变化，自动更新配置文件，类似nginx -s reload
实现：
1、用pyinotify实时监控nginx配置文件变化
2、如果配置文件变化，给系统发送HUP来reload nginx
"""
import os
import re
import pyinotify
import logging
from threading import Timer
# Param
LOG_PATH = "/root/python/log"
CONF_PATHS = [
"/etc/nginx",
]
DELAY = 5
SUDO = False
RELOAD_COMMAND = "nginx -s reload"
if SUDO:
RELOAD_COMMAND = "sudo " + RELOAD_COMMAND
# Log
logger = logging.getLogger(__name__)
logger.setLevel(level = logging.INFO)
log_handler = logging.FileHandler(LOG_PATH)
log_handler.setLevel(logging.INFO)
log_formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
log_handler.setFormatter(log_formatter)
logger.addHandler(log_handler)
# Reloader
def reload_nginx():
os.system(RELOAD_COMMAND)
logger.info("nginx is reloaded")
t = Timer(DELAY, reload_nginx)
def trigger_reload_nginx(pathname, action):
logger.info("nginx monitor is triggered because %s is %s" % (pathname, action))
global t
if t.is_alive():
t.cancel()
t = Timer(DELAY, reload_nginx)
t.start()
else:
t = Timer(DELAY, reload_nginx)
t.start()
events = pyinotify.IN_MODIFY | pyinotify.IN_CREATE | pyinotify.IN_DELETE
watcher = pyinotify.WatchManager()
watcher.add_watch(CONF_PATHS, events, rec=True, auto_add=True)
class EventHandler(pyinotify.ProcessEvent):
def process_default(self, event):
if event.name.endswith(".conf"):
if event.mask == pyinotify.IN_CREATE:
action = "created"
if event.mask == pyinotify.IN_MODIFY:
action = "modified"
if event.mask == pyinotify.IN_DELETE:
action = "deleted"
trigger_reload_nginx(event.pathname, action)
handler = EventHandler()
notifier = pyinotify.Notifier(watcher, handler)
# Start
logger.info("Start Monitoring")
notifier.loop()