Kubernetes（K8s）HA高可用集群部署完整指南（2026 最新版）踩坑 n 次总结出的最佳实践

一 集群架构规划1.1 节点规划IPhostname角色组件192.168.10.1master1mastermaster组件、haproxy、keepalived192.168.10.2master2mastermaster组件、haproxy、keepalived192.168.10.3master3mastermaster组件、haproxy、keepalived192.168.10.4woker1wokerwoker组件192.168.10.100VIP--1.2 整体架构图Client │ │ kubectl │ kubelet │ 192.168.10.100:6443 VIP │ Keepalived HAProxy ┌──────────┼──────────┐ │ │ │ ▼ ▼ ▼ Master01 Master02 Master03 Control Control Control Plane Plane Plane etcd etcd etcd │ │ │ └──────────┼──────────┘ │ Etcd Raft │ Kubernetes API │ Calico Network │ Worker01(Node) │ Business Pods二 环境初始化所有节点2.1系统配置2.1.1 查看系统版本安装工具包$ cat /etc/redhat-release $ yum install -y wget yum-utils device-mapper-persistent-data lvm2 ipvsadm bridge-utils iptables-services2.1.2 配置yum 源国外docker源 $ yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo # 阿里云docker源 $ yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo # 阿里云base源 $ curl -o /etc/yum.repos.d/CentOS-Base.repo \ https://mirrors.aliyun.com/repo/Centos-7.repo $ yum makecache fast # 配置 Kubernetes yum仓库 $ cat /etc/yum.repos.d/kubernetes.repo EOF [kubernetes] nameKubernetes baseurlhttps://pkgs.k8s.io/core:/stable:/v1.29/rpm/ enabled1 gpgcheck1 gpgkeyhttps://pkgs.k8s.io/core:/stable:/v1.29/rpm/repodata/repomd.xml.key EOF yum clean all yum makecache2.1.3 时间同步略2.1.4 关闭 SELinux swap分区# 禁用firewalld $ systemctl disable firewalld $ systemctl stop firewalld # 开启iptables $ systemctl start iptables $ iptables -F $ systemctl enable iptables $ service iptables save # 禁用selinux $ sed -i 7s/enforcing/disabled/ /etc/selinux/config $ setenforce 0 # Disabled $ getenforce # 关闭swap分区 记得重启机器 $ swapoff -a2.1.5 配置内核参数# 所有经过网桥的流量必须要被防火墙处理 $ modprobe br_netfilter # 开机自动加载 $ cat /etc/modules-load.d/bridge.conf EOF br_netfilter EOF # Docker/K8S 网络参数 $ cat /etc/sysctl.d/kubernetes.conf EOF net.bridge.bridge-nf-call-iptables 1 net.bridge.bridge-nf-call-ip6tables 1 net.ipv4.ip_forward 1 net.ipv4.tcp_tw_reuse 1 net.ipv4.tcp_fin_timeout 15 net.core.somaxconn 32768 net.core.netdev_max_backlog 32768 vm.swappiness 0 fs.inotify.max_user_instances 8192 fs.inotify.max_user_watches 1048576 EOF # 生效 加载所有 sysctl 配置目录 $ sysctl --system2.2 Docker 配置2.2.1 安装 Docker# 彻底卸载旧版本【选做 慎用】 docker rm $(docker ps -aq) systemctl stop docker systemctl stop docker.socket rpm -qa | grep -i docker yum remove ... $ yum remove docker \ docker-client \ docker-client-latest \ docker-common \ docker-latest \ docker-latest-logrotate \ docker-logrotate \ docker-engine # 安装docker $ yum list docker-ce --showduplicates | sort -r # 安装 默认最新 可以指定版本 20.10.24:主版本.次版本.补丁版本 $ yum install docker-ce docker-ce-cli containerd.io docker-compose-plugin $ docker --version2.2.2 Docker 配置优化# 配置daemon.json $ mkdir -p /etc/docker $ vim /etc/docker/daemon.json { data-root: /data/docker, exec-opts: [native.cgroupdriversystemd], insecure-registries: [信任仓库地址], registry-mirrors: # 加速源 [ https://docker.rainbond.cc, https://docker.mirrors.sjtug.sjtu.edu.cn, https://docker.m.daocloud.io, https://docker.itelyou.cf, https://noohub.ru, https://docker.fxxk.dedyn.io, https://huecker.io, https://dockerhub.timeweb.cloud, https://registry.cn-hangzhou.aliyuncs.com, https://dockerproxy.com, https://docker.mirrors.ustc.edu.cn, https://docker.nju.edu.cn, https://xx4bwyg2.mirror.aliyuncs.com, https://f1361db2.m.daocloud.io, https://registry.docker-cn.com, https://hub-mirror.c.163.com ] } 备注># 安装cri-docker $ wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.9/cri-dockerd-0.3.9.amd64.tgz $ tar -xf cri-dockerd-0.3.9.amd64.tgz $ cp cri-dockerd/cri-dockerd /usr/bin $ chmod x /usr/bin/cri-dockerd # 配置cri-docker service $ cat /usr/lib/systemd/system/cri-docker.service EOF [Unit] DescriptionCRI Interface for Docker Application Container Engine Documentationhttps://docs.mirantis.com Afternetwork-online.target firewalld.service docker.service Wantsnetwork-online.target Requirescri-docker.socket [Service] Typenotify ExecStart/usr/bin/cri-dockerd --network-plugincni --pod-infra-container-imageregistry.aliyuncs.com/google_containers/pause:3.8 ExecReload/bin/kill -s HUP $MAINPID TimeoutSec0 RestartSec2 Restartalways StartLimitBurst3 StartLimitInterval60s LimitNOFILEinfinity LimitNPROCinfinity LimitCOREinfinity TasksMaxinfinity Delegateyes KillModeprocess [Install] WantedBymulti-user.target EOF # 添加cri-docker套接字 cat /usr/lib/systemd/system/cri-docker.socket EOF [Unit] DescriptionCRI Docker Socket for the API PartOfcri-docker.service [Socket] ListenStream%t/cri-dockerd.sock SocketMode0660 SocketUserroot SocketGroupdocker [Install] WantedBysockets.target EOF # 启动 cri-docker 服务 systemctl daemon-reload systemctl enable cri-docker systemctl start cri-docker systemctl is-active cri-docker2.3.2 安装 kubeadm/kubelet/kubectl# 安装 kubeadm、kubelet、kubectl $ yum install -y kubelet-1.29.0 kubectl-1.29.0 kubeadm-1.29.0 # kubelet 必须绑定 cri-dockerd $ cat /etc/sysconfig/kubelet EOF KUBELET_EXTRA_ARGS--container-runtime-endpointunix:///var/run/cri-dockerd.sock EOF $ systemctl daemon-reload systemctl restart cri-docker $ systemctl enable --now kubelet 这里不必在意kubelet是否起来三 master节点配置3.1 安装 HAProxy和 Keepalived3.1.1 架构kubectl │ 192.168.10.100 │ Keepalived管理 VIP │ HAProxy:6443 负载均衡、健康检查、故障摘除 ┌────────┼─────────┐ │ │ │ Master01 Master02 Master03 API Server API Server API Server3.1.2 安装HAproxy# 安装软件 $ yum install -y haproxy keepalived # 验证安装 haproxy -v keepalived -v # 配置HAproxy $ cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.bak $ vim /etc/haproxy/haproxy.cfg ——————————————————————————————————————————————————————————————————- global log 127.0.0.1 local2 chroot /var/lib/haproxy pidfile /run/haproxy.pid maxconn 4000 user haproxy group haproxy daemon defaults log global mode tcp option tcplog option dontlognull retries 3 timeout connect 10s timeout client 1m timeout server 1m frontend kube-apiserver bind *:16443 default_backend kube-apiserver backend kube-apiserver balance roundrobin option tcp-check default-server inter 3s fall 3 rise 2 server master1 192.168.10.1:6443 check server master2 192.168.10.2:6443 check server master3 192.168.10.3:6443 check ———————————————————————————————————————————————————————————————————— # 检查配置 haproxy -c -f /etc/haproxy/haproxy.cfg #输出Configuration file is valid # 启动HAproxy systemctl enable haproxy systemctl restart haproxy systemctl status haproxy ss -lntp | grep 164433.1.3 安装Keepalived# 安装软件 $ yum install -y haproxy keepalived # 验证安装 haproxy -v keepalived -v # 配置Keepalived cp -a /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak # master1 节点配置 vim /etc/keepalived/keepalived.conf global_defs { router_id K8S_01 enable_script_security } vrrp_script chk_haproxy { script pidof haproxy interval 2 weight -20 } vrrp_instance VI_1 { state BACKUP interface eth0 virtual_router_id 66 priority 120 advert_int 1 unicast_src_ip 192.168.10.1 unicast_peer { 192.168.10.2 192.168.10.3 } authentication { auth_type PASS auth_pass 888666 } virtual_ipaddress { 192.168.10.100/19 } track_script { chk_haproxy } } # master2 节点配置 vim /etc/keepalived/keepalived.conf global_defs { router_id K8S_02 } vrrp_script chk_haproxy { script pidof haproxy interval 2 weight -20 } vrrp_instance VI_1 { state BACKUP interface eth0 virtual_router_id 66 priority 110 advert_int 1 unicast_src_ip 192.168.10.2 unicast_peer { 192.168.10.1 192.168.10.3 authentication { auth_type PASS auth_pass 888666 } virtual_ipaddress { 192.168.10.100/19 } track_script { chk_haproxy } # master3 节点配置 vim /etc/keepalived/keepalived.conf global_defs { router_id K8S_03 } vrrp_script chk_haproxy { script pidof haproxy interval 2 weight -20 } vrrp_instance VI_1 { state BACKUP interface eth0 virtual_router_id 66 priority 100 advert_int 1 unicast_src_ip 192.168.10.3 unicast_peer { 192.168.10.1 192.168.10.2 } authentication { auth_type PASS auth_pass 888666 } virtual_ipaddress { 192.168.10.100/19 } track_script { chk_haproxy } } 注意⚠️virtual_router_id 和 auth_pass 三个节点要保持一致 注意⚠️真正决定谁成为 Master 的是 priority优先级 # 启动 Keepalived systemctl enable keepalived systemctl restart keepalived systemctl status keepalived # 验证VIP ipaddr #只能在权重高的机器上看见VIP systemctl stop keepalived # VIP 漂移到其他机器3.2 master1初始化集群要在 keepalived master上初始化$ kubeadm config print init-defaults kubeadm.yaml apiVersion: kubeadm.k8s.io/v1beta3 kind: InitConfiguration bootstrapTokens: - token: abcdef.0123456789abcdef ttl: 24h usages: - signing - authentication groups: - system:bootstrappers:kubeadm:default-node-token localAPIEndpoint: advertiseAddress: 192.168.10.1 bindPort: 6443 nodeRegistration: name: ops-wbzhongdan-02.dev.kwaidc.com criSocket: unix:///var/run/cri-dockerd.sock imagePullPolicy: IfNotPresent --- apiVersion: kubeadm.k8s.io/v1beta3 kind: ClusterConfiguration clusterName: kubernetes-test kubernetesVersion: v1.29.0 imageRepository: registry.aliyuncs.com/google_containers controlPlaneEndpoint: 192.168.10.100:16443 networking: dnsDomain: cluster.local serviceSubnet: 10.96.0.0/12 podSubnet: 10.244.0.0/16 apiServer: timeoutForControlPlane: 4m0s certSANs: - 192.168.10.100 - 192.168.10.1 - 192.168.10.2 - 192.168.10.3 - master1 - master2 - master3 controllerManager: {} scheduler: {} dns: {} etcd: local: dataDir: /var/lib/etcd # 拉镜像 $ kubeadm config --configkubeadm.yaml images pull # 初始化 $ kubeadm init --configkubeadm.yaml --upload-certs # 不加--upload-certs 在master1初始化完需要手动把证书拷贝到其他节点 ———————————————————————————————————————————————————————————————— 成功输出 Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Alternatively, if you are the root user, you can run: export KUBECONFIG/etc/kubernetes/admin.conf You should now deploy a pod network to the cluster. Run kubectl apply -f [podnetwork].yaml with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ You can now join any number of the control-plane node running the following command on each as root: kubeadm join 192.168.10.100:16443 --token abcdef.zzz789abcdef \ --discovery-token-ca-cert-hash sha256:b7c1acd922a75d7a572b04d6504a0950e39130c9f87405f556e0bbe6782e0285 \ --control-plane --certificate-key 54b6c6486ab7109d7a34afbea23d8065aa1013a338f80c050dbfa85507521171 Please note that the certificate-key gives access to cluster sensitive data, keep it secret! As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use kubeadm init phase upload-certs --upload-certs to reload certs afterward. Then you can join any number of worker nodes by running the following on each as root: kubeadm join 192.168.10.100:16443 --token abcdef.zzzz3456789abcdef \ --discovery-token-ca-cert-hash sha256:b7c1acd922a75d7a572b04d6504a0950e39130c9f87405f556e0bbe6782e0285 —————————————————————————————————————————————————————————————— $ mkdir -p $HOME/.kube $ cp -i /etc/kubernetes/admin.conf $HOME/.kube/config $ chown $(id -u):$(id -g) $HOME/.kube/config # 安装网络插件(需翻墙) $ curl https://raw.githubusercontent.com/projectcalico/calico/v3.26.3/manifests/calico-typha.yaml -o calico.yaml 修改calico.yaml # 修改为 BGP 模式 关闭IPIP模式value: Off # Enable IPIP # 修改pod网段 value: 10.244.0.0/16 # CALICO_IPV4POOL_CIDR 注意格式对齐 $ kubectl create -f calico.yaml $ kubectl get pods -A -w # 观察集群状态 $ kubectl get nodes # 查看节点3.3 master2 master3 加入集群$ kubeadm join 192.168.10.100:16443 --token abcdef.xxx6789abcdef \ --discovery-token-ca-cert-hash sha256:b7c1acd922a75d7a572b04d6504a0950e39130c9f87405f556e0bbe6782e0285 \ --control-plane --certificate-key 54b6c6486ab7109d7a34afbea23d8065aa1013a338f80c050dbfa85507521171 \ --cri-socket unix:///var/run/cri-dockerd.sock $ mkdir -p $HOME/.kube $ cp -i /etc/kubernetes/admin.conf $HOME/.kube/config $ chown $(id -u):$(id -g) $HOME/.kube/config3.4 踩过的坑踩坑一 $ tail -f /var/log/messages $ journalctl -f -u kubelet # kubelet 启动失败 报错misconfiguration: kubelet cgroup driver: systemd is different from docker cgroup driver: cgroupfs 原因kubelet 的 cgroup 驱动 是 systemd而 Docker 的 cgroup 驱动 是 cgroupfs两者不一致 解决(注意,问题) $ vim /etc/docker/daemon.json exec-opts: [native.cgroupdriversystemd] # kubeadm init 和 kubeadm join 都只能执行一遍,可以删除集群重新初始化 $ kubeadm reset -f --cri-socket unix:///var/run/cri-dockerd.sock 踩坑二 报错[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory /etc/kubernetes/manifests. This can take up to 4m0s [kubelet-check] Initial timeout of 40s passed. 原因 1 kubelet 和 Docker cgroup driver 不一致 $ docker info | grep -A 5 Swarm 显示Swarm: inactive $ docker swarm init $ vim /etc/docker/daemon.json exec-opts: [native.cgroupdriversystemd] 2 Docker 开启了HTTP/HTTPS 代理但未配置NO_PROXY内网必踩坑关闭代理 3 CentOS7 内核参数未开启 bridge-nf-call-iptables $ vim /etc/sysctl.d/kubernetes.conf net.bridge.bridge-nf-call-ip6tables 1 net.bridge.bridge-nf-call-iptables 1 net.ipv4.ip_forward 1 4 之前初始化失败的残留文件未彻底清理 # 强制重置kubeadm集群 $ kubeadm reset -f --cri-socket unix:///var/run/cri-dockerd.sock # 删除kubectl配置文件 $ rm -rf $HOME/.kube/config # 删除CNI网络插件配置 $ rm -rf /etc/cni/net.d # 删除kubelet核心数据目录脏数据重灾区必须删 $ rm -rf /var/lib/kubelet # 删除etcd数据目录 $ rm -rf /var/lib/etcd # 删除kubeadm生成的静态pod清单目录 $ rm -rf /etc/kubernetes/manifests 踩坑三 $ systemctl stop kubelet 报错Authorization not available. Check if polkit service is running or see debug message for more information. 原因主要是因为 PolkitPolicyKit授权服务未运行 解决 $ systemctl start dbus $ systemctl start polkit 踩坑四 用 cri-docker 作为crisocket 注意事项 1 kubeadm config 里 必须写 “criSocket: unix:///var/run/cri-dockerd.sock” 2 kubelet 也必须绑定 cri-dockerd $ cat /etc/sysconfig/kubelet EOF KUBELET_EXTRA_ARGS--container-runtime-endpointunix:///var/run/cri-dockerd.sock EOF四 woker 节点配置$ kubeadm join 192.168.10.100:16443 --token abcdef.0ddd56789abcdef --discovery-token-ca-cert-hash sha256:b7c1acd922a75d7a572b04d6504a0950e39130c9f87405f556e0bbe6782e0285 --cri-socket unix:///var/run/cri-dockerd.sock五 安装helm#基本安装 wget https://get.helm.sh/helm-v3.17.3-linux-amd64.tar.gz tar -zxvf helm-v3.17.3-linux-amd64.tar.gz mv linux-amd64/helm /usr/bin/ chmod x /usr/local/bin/helm helm version # 添加仓库 $ wget https://github.com/kubernetes/dashboard/releases/download/kubernetes-dashboard-7.11.1/kubernetes-dashboard-7.11.1.tgz $ helm repo list # 查看添加的 Repository 列表 $ helm repo update 更新仓库六 部署dashboard# 创建namespace kubectl create namespace kubernetes-dashboard # 安装 $ export 翻墙代理 $ wget https://github.com/kubernetes/dashboard/releases/download/kubernetes-dashboard-7.11.1/kubernetes-dashboard-7.11.1.tgz $ unset https_proxy http_proxy HTTPS_PROXY HTTP_PROXY $ helm upgrade --install kubernetes-dashboard ./kubernetes-dashboard-7.11.1.tgz \ --namespace kubernetes-dashboard # 验证 $ helm list -A $ kubectl get pod -n kubernetes-dashboard $ kubectl get svc -n kubernetes-dashboard # 创建管理员账户 apiVersion: v1 kind: ServiceAccount metadata: name: admin-user namespace: kubernetes-dashboard --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: admin-user roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin subjects: - kind: ServiceAccount name: admin-user namespace: kubernetes-dashboard $ kubectl apply -f dashboard-admin.yaml # 获取token $ kubectl -n kubernetes-dashboard create token admin-user # 端口转发 $ kubectl -n kubernetes-dashboard edit svc kubernetes-dashboard-kong-proxy spec: type: NodePort # ClusterIP改为NodePort nodePort: 30443 # ports: 中增加 # 申请域名 绑定8443端口➕ 蛋蛋不迷路 有疑问随时留言哦 不定时更新