1. 从零开始Kubernetes集群部署前的关键准备在真正动手部署Kubernetes集群之前我们需要做好充分的准备工作。就像建造房屋需要打地基一样合理的准备工作能避免后续80%的部署问题。根据我多年部署Kubernetes集群的经验以下这些环节最容易被忽视但至关重要。1.1 硬件资源规划与节点角色分配一个生产可用的Kubernetes集群至少需要3个节点1个Master 2个Worker。但实际部署中我建议采用3个Master节点实现高可用Worker节点根据业务需求动态扩展。以下是硬件配置的黄金法则Master节点至少2核CPU/4GB内存/50GB磁盘etcd数据目录单独挂载SSDWorker节点根据容器负载调整建议4核CPU起/8GB内存/100GB磁盘网络要求节点间延迟5ms带宽≥1Gbps特别注意etcd对磁盘IOPS要求极高使用云盘时务必选择高性能SSD。我曾遇到etcd因磁盘性能不足导致集群不可用的情况更换为NVMe SSD后问题立即解决。1.2 操作系统与内核参数调优推荐使用CentOS 7.9/Ubuntu 20.04 LTS作为宿主机系统。部署前必须完成以下系统配置# 关闭SwapKubernetes 1.8强制要求 sudo swapoff -a sed -i / swap / s/^/#/ /etc/fstab # 加载内核模块 cat EOF | sudo tee /etc/modules-load.d/k8s.conf br_netfilter ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh nf_conntrack EOF # 设置内核参数 cat EOF | sudo tee /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables 1 net.bridge.bridge-nf-call-iptables 1 net.ipv4.ip_forward 1 vm.swappiness 0 EOF sudo sysctl --system1.3 容器运行时选择与配置虽然Docker仍是常见选择但根据Kubernetes官方建议我更推荐使用containerd作为运行时# 安装containerd sudo apt-get update sudo apt-get install -y containerd sudo mkdir -p /etc/containerd containerd config default | sudo tee /etc/containerd/config.toml sudo systemctl restart containerd # 配置cgroup驱动为systemd必须与kubelet一致 sudo sed -i s/SystemdCgroup false/SystemdCgroup true/ /etc/containerd/config.toml2. 集群部署实战kubeadm最佳实践2.1 使用kubeadm初始化控制平面以下命令将在第一个Master节点上初始化集群sudo kubeadm init \ --control-plane-endpoint CLUSTER_API_ENDPOINT:6443 \ --upload-certs \ --pod-network-cidr10.244.0.0/16 \ --service-cidr10.96.0.0/12 \ --image-repository registry.aliyuncs.com/google_containers关键参数说明--control-plane-endpoint负载均衡器地址生产环境必须配置--upload-certs自动轮转证书--image-repository使用国内镜像源加速初始化成功后按照提示配置kubectlmkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config2.2 高可用控制平面部署对于生产环境需要添加额外的Master节点实现高可用kubeadm join 192.168.1.100:6443 \ --token token \ --discovery-token-ca-cert-hash sha256:hash \ --control-plane \ --certificate-key key高可用架构建议使用Keepalived HAProxy实现API Server负载均衡etcd集群配置为独立节点非堆叠式提升稳定性每个Master节点部署在不同可用区实现跨AZ容灾2.3 工作节点加入集群Worker节点加入命令与Master类似去掉--control-plane参数kubeadm join 192.168.1.100:6443 \ --token token \ --discovery-token-ca-cert-hash sha256:hash3. 网络插件选型与部署3.1 Calico网络方案实践Calico是生产环境最常用的CNI插件提供高性能网络策略kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml关键配置调优启用IPIP模式跨子网通信配置IP地址池匹配pod-network-cidr启用eBPF加速模式内核4.183.2 网络策略实战示例以下策略只允许frontend Pod访问backend服务的80端口apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: backend-allow-frontend spec: podSelector: matchLabels: app: backend ingress: - from: - podSelector: matchLabels: app: frontend ports: - protocol: TCP port: 804. 存储方案设计与实现4.1 动态存储供应配置以NFS为例创建StorageClassapiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: nfs-storage provisioner: example.com/nfs parameters: server: nfs-server.example.com path: /share4.2 CSI驱动集成实践以AWS EBS为例kubectl apply -k github.com/kubernetes-sigs/aws-ebs-csi-driver/deploy/kubernetes/overlays/stable/?refmaster创建StorageClassapiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: ebs-sc provisioner: ebs.csi.aws.com volumeBindingMode: WaitForFirstConsumer5. DevOps流水线集成5.1 GitOps工作流实现使用Argo CD部署应用kubectl create namespace argocd kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml配置自动同步apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: myapp spec: destination: server: https://kubernetes.default.svc namespace: default source: repoURL: https://github.com/example/myapp.git targetRevision: HEAD path: k8s syncPolicy: automated: selfHeal: true prune: true5.2 Tekton CI/CD流水线示例定义构建任务apiVersion: tekton.dev/v1beta1 kind: Task metadata: name: build-push spec: steps: - name: build image: maven:3.6.0-jdk-11 command: [mvn, clean, package] - name: build-image image: gcr.io/kaniko-project/executor:latest args: [ --dockerfileDockerfile, --context., --destinationgcr.io/myproject/myimage:latest ]6. 监控与日志方案6.1 Prometheus-Operator部署使用kube-prometheus-stackhelm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \ --namespace monitoring \ --create-namespace6.2 EFK日志收集方案部署ElasticsearchFluentdKibanahelm install elasticsearch elastic/elasticsearch \ --namespace logging \ --create-namespace helm install fluentd fluent/fluentd \ --namespace logging \ --set elasticsearch.hostelasticsearch-master.logging.svc.cluster.local helm install kibana elastic/kibana \ --namespace logging \ --set service.typeNodePort7. 安全加固实践7.1 RBAC权限控制创建最小权限ServiceAccountapiVersion: v1 kind: ServiceAccount metadata: name: ci-deployer --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: deployer-role rules: - apiGroups: [apps] resources: [deployments] verbs: [get, list, watch, create, update, patch] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: deployer-binding subjects: - kind: ServiceAccount name: ci-deployer roleRef: kind: Role name: deployer-role7.2 Pod安全策略使用PodSecurityPolicy限制特权容器apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: restricted spec: privileged: false allowPrivilegeEscalation: false requiredDropCapabilities: - ALL volumes: - configMap - emptyDir - secret hostNetwork: false hostIPC: false hostPID: false runAsUser: rule: MustRunAsNonRoot8. 生产环境运维技巧8.1 集群升级策略使用kubeadm进行原地升级# 升级控制平面 sudo apt-get update sudo apt-get install -y kubeadm1.22.5-00 sudo kubeadm upgrade plan sudo kubeadm upgrade apply v1.22.5 # 升级节点 sudo apt-get update sudo apt-get install -y kubelet1.22.5-00 kubectl1.22.5-00 sudo systemctl daemon-reload sudo systemctl restart kubelet8.2 故障排查命令大全常用诊断命令# 查看节点状态 kubectl get nodes -o wide # 检查Pod事件 kubectl describe pod pod-name # 查看容器日志 kubectl logs -f pod-name -c container-name # 网络连通性测试 kubectl run -it --rm --restartNever test-pod --imagebusybox -- sh # API健康检查 curl -k https://localhost:6443/healthz在实际生产运维中我发现80%的问题可以通过检查kubelet日志和Pod事件解决。建议将以下命令加入日常检查清单journalctl -u kubelet -f kubectl get events --sort-by.metadata.creationTimestamp Kubernetes集群部署与运维实战指南
1. 从零开始Kubernetes集群部署前的关键准备在真正动手部署Kubernetes集群之前我们需要做好充分的准备工作。就像建造房屋需要打地基一样合理的准备工作能避免后续80%的部署问题。根据我多年部署Kubernetes集群的经验以下这些环节最容易被忽视但至关重要。1.1 硬件资源规划与节点角色分配一个生产可用的Kubernetes集群至少需要3个节点1个Master 2个Worker。但实际部署中我建议采用3个Master节点实现高可用Worker节点根据业务需求动态扩展。以下是硬件配置的黄金法则Master节点至少2核CPU/4GB内存/50GB磁盘etcd数据目录单独挂载SSDWorker节点根据容器负载调整建议4核CPU起/8GB内存/100GB磁盘网络要求节点间延迟5ms带宽≥1Gbps特别注意etcd对磁盘IOPS要求极高使用云盘时务必选择高性能SSD。我曾遇到etcd因磁盘性能不足导致集群不可用的情况更换为NVMe SSD后问题立即解决。1.2 操作系统与内核参数调优推荐使用CentOS 7.9/Ubuntu 20.04 LTS作为宿主机系统。部署前必须完成以下系统配置# 关闭SwapKubernetes 1.8强制要求 sudo swapoff -a sed -i / swap / s/^/#/ /etc/fstab # 加载内核模块 cat EOF | sudo tee /etc/modules-load.d/k8s.conf br_netfilter ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh nf_conntrack EOF # 设置内核参数 cat EOF | sudo tee /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables 1 net.bridge.bridge-nf-call-iptables 1 net.ipv4.ip_forward 1 vm.swappiness 0 EOF sudo sysctl --system1.3 容器运行时选择与配置虽然Docker仍是常见选择但根据Kubernetes官方建议我更推荐使用containerd作为运行时# 安装containerd sudo apt-get update sudo apt-get install -y containerd sudo mkdir -p /etc/containerd containerd config default | sudo tee /etc/containerd/config.toml sudo systemctl restart containerd # 配置cgroup驱动为systemd必须与kubelet一致 sudo sed -i s/SystemdCgroup false/SystemdCgroup true/ /etc/containerd/config.toml2. 集群部署实战kubeadm最佳实践2.1 使用kubeadm初始化控制平面以下命令将在第一个Master节点上初始化集群sudo kubeadm init \ --control-plane-endpoint CLUSTER_API_ENDPOINT:6443 \ --upload-certs \ --pod-network-cidr10.244.0.0/16 \ --service-cidr10.96.0.0/12 \ --image-repository registry.aliyuncs.com/google_containers关键参数说明--control-plane-endpoint负载均衡器地址生产环境必须配置--upload-certs自动轮转证书--image-repository使用国内镜像源加速初始化成功后按照提示配置kubectlmkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config2.2 高可用控制平面部署对于生产环境需要添加额外的Master节点实现高可用kubeadm join 192.168.1.100:6443 \ --token token \ --discovery-token-ca-cert-hash sha256:hash \ --control-plane \ --certificate-key key高可用架构建议使用Keepalived HAProxy实现API Server负载均衡etcd集群配置为独立节点非堆叠式提升稳定性每个Master节点部署在不同可用区实现跨AZ容灾2.3 工作节点加入集群Worker节点加入命令与Master类似去掉--control-plane参数kubeadm join 192.168.1.100:6443 \ --token token \ --discovery-token-ca-cert-hash sha256:hash3. 网络插件选型与部署3.1 Calico网络方案实践Calico是生产环境最常用的CNI插件提供高性能网络策略kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml关键配置调优启用IPIP模式跨子网通信配置IP地址池匹配pod-network-cidr启用eBPF加速模式内核4.183.2 网络策略实战示例以下策略只允许frontend Pod访问backend服务的80端口apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: backend-allow-frontend spec: podSelector: matchLabels: app: backend ingress: - from: - podSelector: matchLabels: app: frontend ports: - protocol: TCP port: 804. 存储方案设计与实现4.1 动态存储供应配置以NFS为例创建StorageClassapiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: nfs-storage provisioner: example.com/nfs parameters: server: nfs-server.example.com path: /share4.2 CSI驱动集成实践以AWS EBS为例kubectl apply -k github.com/kubernetes-sigs/aws-ebs-csi-driver/deploy/kubernetes/overlays/stable/?refmaster创建StorageClassapiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: ebs-sc provisioner: ebs.csi.aws.com volumeBindingMode: WaitForFirstConsumer5. DevOps流水线集成5.1 GitOps工作流实现使用Argo CD部署应用kubectl create namespace argocd kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml配置自动同步apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: myapp spec: destination: server: https://kubernetes.default.svc namespace: default source: repoURL: https://github.com/example/myapp.git targetRevision: HEAD path: k8s syncPolicy: automated: selfHeal: true prune: true5.2 Tekton CI/CD流水线示例定义构建任务apiVersion: tekton.dev/v1beta1 kind: Task metadata: name: build-push spec: steps: - name: build image: maven:3.6.0-jdk-11 command: [mvn, clean, package] - name: build-image image: gcr.io/kaniko-project/executor:latest args: [ --dockerfileDockerfile, --context., --destinationgcr.io/myproject/myimage:latest ]6. 监控与日志方案6.1 Prometheus-Operator部署使用kube-prometheus-stackhelm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \ --namespace monitoring \ --create-namespace6.2 EFK日志收集方案部署ElasticsearchFluentdKibanahelm install elasticsearch elastic/elasticsearch \ --namespace logging \ --create-namespace helm install fluentd fluent/fluentd \ --namespace logging \ --set elasticsearch.hostelasticsearch-master.logging.svc.cluster.local helm install kibana elastic/kibana \ --namespace logging \ --set service.typeNodePort7. 安全加固实践7.1 RBAC权限控制创建最小权限ServiceAccountapiVersion: v1 kind: ServiceAccount metadata: name: ci-deployer --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: deployer-role rules: - apiGroups: [apps] resources: [deployments] verbs: [get, list, watch, create, update, patch] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: deployer-binding subjects: - kind: ServiceAccount name: ci-deployer roleRef: kind: Role name: deployer-role7.2 Pod安全策略使用PodSecurityPolicy限制特权容器apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: restricted spec: privileged: false allowPrivilegeEscalation: false requiredDropCapabilities: - ALL volumes: - configMap - emptyDir - secret hostNetwork: false hostIPC: false hostPID: false runAsUser: rule: MustRunAsNonRoot8. 生产环境运维技巧8.1 集群升级策略使用kubeadm进行原地升级# 升级控制平面 sudo apt-get update sudo apt-get install -y kubeadm1.22.5-00 sudo kubeadm upgrade plan sudo kubeadm upgrade apply v1.22.5 # 升级节点 sudo apt-get update sudo apt-get install -y kubelet1.22.5-00 kubectl1.22.5-00 sudo systemctl daemon-reload sudo systemctl restart kubelet8.2 故障排查命令大全常用诊断命令# 查看节点状态 kubectl get nodes -o wide # 检查Pod事件 kubectl describe pod pod-name # 查看容器日志 kubectl logs -f pod-name -c container-name # 网络连通性测试 kubectl run -it --rm --restartNever test-pod --imagebusybox -- sh # API健康检查 curl -k https://localhost:6443/healthz在实际生产运维中我发现80%的问题可以通过检查kubelet日志和Pod事件解决。建议将以下命令加入日常检查清单journalctl -u kubelet -f kubectl get events --sort-by.metadata.creationTimestamp