K8s 通过 keepalive+nginx 实现 nginx-ingress-controller 高可用
xnh888 2024-10-14 19:07:00 技术教程 130 ℃ 0 评论
- 通过 keepalive+nginx 实现 nginx-ingress-controller 高可用
- 给 node 节点增加标签
- kubectl label node k8s-01 kubernetes.io/ingress=nginx
kubectl label node k8s-02 kubernetes.io/ingress=nginx
- 下载 yaml 文件
- wget https://ghproxy.com/https://github.com/kubernetes/ingress-nginx/blob/main/deploy/static/provider/baremetal/deploy.yaml -O ingress-deploy.yaml
- 更新 yaml 文件
- 在 k8s-01 和 k8s-02 上分别安装 keepalive 和 nginx
- yum install nginx keepalived nginx-mod-stream -y修改 nginx 配置文件。主备一样user nginx;worker_processes auto;error_log /var/log/nginx/error.log;pid /run/nginx.pid;include /usr/share/nginx/modules/*.conf;events {worker_connections 1024;}stream {log_format main 'remoteaddrremoteaddrupstream_addr - [timelocal]timelocal]status $upstream_bytes_sent';access_log /var/log/nginx/k8s-access.log main;upstream k8s-apiserver { server 192.168.2.20:80; #后端的服务器IP地址，根据实际情况填写
server 192.168.2.21:80; } server {proxy_pass k8s-apiserver;}}http {log_format main 'remoteaddr?remoteaddr?remote_user [timelocal]"timelocal]"request" ''statusstatusbody_bytes_sent "$http_referer" ''"httpuseragent""httpuseragent""http_x_forwarded_for"';access_log /var/log/nginx/access.log main;sendfile on;tcp_nopush on;tcp_nodelay on;keepalive_timeout 65;types_hash_max_size 2048;include /etc/nginx/mime.types;default_type application/octet-stream;}
scp一份到备用服务器后，分别启动 nginxsystemctl enable nginx.service --now - 配置 keepalive
- 主 keepalivedvim keepalived.confglobal_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id NGINX_MASTER
}vrrp_script check_nginx {
script "/etc/keepalived/check_nginx.sh"
}vrrp_instance VI_1 {state MASTERinterface ens33 # 修改为实际网卡名virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的priority 100# 优先级，备服务器设置 90advert_int 1 # 指定VRRP 心跳包通告间隔时间，默认1秒authentication {auth_type PASSauth_pass 1111}# 虚拟IPvirtual_ipaddress { 192.168.2.168/24}track_script {check_nginx}}#vrrp_script：指定检查nginx工作状态脚本（根据nginx状态判断是否故障转移）拷贝一份配置文件到备用服务器scp /etc/keepalived/keepalived.conf root@k8s-02:/etc/keepalived/global_defs {notification_email { acassen@firewall.loc failover@firewall.loc sysadmin@firewall.loc } notification_email_from Alexandre.Cassen@firewall.loc smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id NGINX_MASTER} vrrp_script check_nginx {script "/etc/keepalived/check_nginx.sh"}vrrp_instance VI_1 {state BACKUPinterface ens33 # 修改为实际网卡名virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的 priority 90 # 优先级，备服务器设置 90advert_int 1 # 指定VRRP 心跳包通告间隔时间，默认1秒 authentication { auth_type PASS auth_pass 1111 } # 虚拟IPvirtual_ipaddress { 192.168.2.168/24} track_script {check_nginx} }#vrrp_script：指定检查nginx工作状态脚本（根据nginx状态判断是否故障转移）#virtual_ipaddress：虚拟IP（VIP） - nginx 检测脚本vim /etc/keepalived/check_nginx.sh#!/bin/bash#1、判断 Nginx 是否存活counter=ps?Cng∈x??no?header∣wc?lps-Cng∈x--no-header∣wc-lif [ $counter -eq 0 ]; then#2、如果不存活则尝试启动 Nginxservice nginx startsleep 2#3、等待 2 秒后再次获取一次 Nginx 状态counter=ps?Cng∈x??no?header∣wc?lps-Cng∈x--no-header∣wc-l#4、再次进行判断，如 Nginx 还不存活则停止 Keepalived，让地址进行漂移if [ $counter -eq 0 ]; thenservice keepalived stopfifi
- chmod +x /etc/keepalived/check_nginx.sh
- 分别启动 keepalived测试 keepalived：停掉 k8s-01 上的 keepalived。Vip 会漂移到 k8s-02
- 测试 Ingress HTTP 代理 k8s 内部站点部署后端 tomcat 服务vim ingress-demo.yamlapiVersion: v1kind: Servicemetadata:tomcatnamespace: defaultspec:selector:app: tomcatrelease: canaryports:- name: httptargetPort: 8080port: 8080- name: ajptargetPort: 8009port: 8009---apiVersion: apps/v1kind: Deploymentmetadata:name: tomcat-deploynamespace: defaultspec:replicas: 2selector:matchLabels:app: tomcatrelease: canarytemplate:metadata:labels:app: tomcatrelease: canaryspec:containers:- name: tomcatimage: tomcat:8.5.34-jre8-alpine imagePullPolicy: IfNotPresent ports:- name: httpcontainerPort: 8080name: ajp
- 编写 ingress 规则
- #编写 ingress 的配置清单
vim ingress-myapp.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-myapp
namespace: default
annotations:
kubernetes.io/ingress.class: "nginx"
spec:
rules:
- host: tomcat.lucky.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: tomcat
port:
number: 8080 - rules: #定义后端转发的规则
- host: tomcat.lucky.com #通过域名进行转发
http:
paths:
- path: / #配置访问路径，如果通过 url 进行转发，需要修改；空默认为访问的路径为"/"
pathType: Prefix
backend: #配置后端服务
service :
name: tomcat #转发到前面定义的serviceIngress 的路径类型 - ImplementationSpecific (默认): 对于这种类型，匹配取决于 IngressClass。 具体实现可以将其作为单独的 pathType 处理或者与 Prefix 或 Exact 类型作相同处理。
- Exact ：精确匹配 URL 路径，且对大小写敏感。
- Prefix ：基于以 / 分隔的 URL 路径前缀匹配。匹配对大小写敏感，并且对路径中的元素逐个完成。 路径元素指的是由 / 分隔符分隔的路径中的标签列表。 如果每个 p 都是请求路径 p 的元素前缀，则请求与路径 p 匹配。
- #修改电脑本地的 host 文件，增加如下一行，下面的 ip 是 keepalived 的 vip
192.168.2.168 tomcat.lucky.com - 查看 pod 调度spec:hostNetwork: true #表示容器使用和宿主机一样的网络affinity: #设置亲和性podAntiAffinity: #设置 pod 的反亲和性preferredDuringSchedulingIgnoredDuringExecution: #软亲和性- weight: 100podAffinityTerm:labelSelector:matchLabels:app.kubernetes.io/name: ingress-nginx #选择标签
- 测试
- 添加 nginx 默认站点 访问 vip http://192.168.2.168:8080发现没有经过 ingress 代理，访问 http://tomcat.lucky.com:16800/
- 另外注意 ingress 代理的 nginx 负载均衡的端口最好大于 30000
本文暂时没有评论，来添加一个吧(●'◡'●)