高可用kubernetes丛集搭建最新消息

架构说明

Kubernetes丛集元件:

etcd 一个高可用的K/V键值对储存和服务发现系统flannel 实现夸主机的容器网络的通讯kube-apiserver 提供kubernetes丛集的API呼叫kube-controller-manager 确保丛集服务kube-scheduler 排程容器，分配到Nodekube-proxy 提供网络代理服务kubeadm:用来初始化丛集的指令。kubectl: 用来与丛集通讯的命令列工具。通过kubectl可以部署和管理应用，检视各种资源，建立、删除和更新各种元件。kubelet 执行在丛集中的每个节点上，用来启动 pod 和 container 等。k8s丛集高可用，一般是etcd，kube-apiserver，kube-controller-manager，kube-scheduler服务元件的高可用。

规划：

3个master节点，2个worker节点，使用keepalived+haproxy做高可用

一、前期环境准备

修改所有主机的hosts档案cat >/etc/hosts130.252.10.235 node1

130.252.10.236 node2

130.252.10.237 node3

130.252.10.238 node4

130.252.10.239 node5

EOF

同步时间ntpdate time.windows.com

hwclock --systohc

禁用selinux，需要重启setenforce 0

sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/sysconfig/selinux

sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config

核心引数修改cat /etc/sysctl.d/k8s.conf

net.bridge.bridge-nf-call-ip6tables = 1

net.bridge.bridge-nf-call-iptables = 1

net.ipv4.ip_nonlocal_bind = 1

net.ipv4.ip_forward = 1

vm.swappiness = 0

EOF

sysctl --system

关闭系统swapKubernetes 1.8开始要求关闭系统的Swap，如果不关闭，预设配置下kubelet将无法启动。

方法一通过kubelet的启动引数–fail-swap-on=false更改这个限制。

方法二关闭系统的Swap， swapoff -a

修改/etc/fstab档案，注释掉SWAP的自动挂载，使用free -m确认swap已经关闭。

swapoff -a

sed -i '/ swap / s/^(.*)$/#/g' /etc/fstab

关闭防火墙，生产环境不建议关闭防火墙systemctl disable firewalld.service

systemctl stop firewalld.service

systemctl disable postfix

载入ipvs核心模组cat > /etc/sysconfig/modules/ipvs.modules #!/bin/bash

modprobe -- ip_vs

modprobe -- ip_vs_rr

modprobe -- ip_vs_wrr

modprobe -- ip_vs_sh

modprobe -- nf_conntrack_ipv4

EOF

chmod 755 /etc/sysconfig/modules/ipvs.modules

bash /etc/sysconfig/modules/ipvs.modules

lsmod | grep -e ip_vs -e nf_conntrack_ipv4

二、docker安装

安装docker-ce，所有节点都要安装

yum-utils 提供yum-config-manager工具，devicemapper储存需要device-mapper-persistent-data和lvm2

yum install -y yum-utils device-mapper-persistent-data lvm2

新增yum源仓库

官方仓库

yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

安装docker

yum -y install docker-ce docker-ce-cli containerd.io

启动docker，设定开机启动docker

systemctl start docker

systemctl enable docker

systemctl list-unit-files |grep docker

配置docker中国映象加速器，修改cgroup driver为systemd，k8s建议使用systemd，配置后重启docker

cat > /etc/docker/daemon.json {

"registry-mirrors": ["https://registry.docker-cn.com", "https://docker.mirrors.ustc.edu.cn"],

"exec-opts": ["native.cgroupdriver=systemd"],

"log-driver": "json-file",

"log-opts": {

"max-size": "100m"

"storage-driver": "overlay2",

"storage-opts": [

"overlay2.override_kernel_check=true"

]

}

EOF

systemctl restart docker

三、安装haproxy，keepalived

安装haproxy和keepalived，实现kube-apiserver高可用

psmisc提供killall命令

yum install keepalived haproxy psmisc ipvsadm -y

配置haproxy

cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.bak

cat > /etc/haproxy/haproxy.cfg global

log 127.0.0.1 local2

chroot /var/lib/haproxy

pidfile /var/run/haproxy.pid

maxconn 4000

user haproxy

group haproxy

daemon

stats socket /var/lib/haproxy/stats

defaults

mode tcp

log global

option tcplog

option dontlognull

option redispatch

retries 3

timeout http-request 10s

timeout queue 1m

timeout connect 10s

timeout client 1m

timeout server 1m

timeout http-keep-alive 10s

timeout check 10s

maxconn 3000

listen stats

mode http

bind :10086

stats enable

stats uri /admin?stats

stats auth admin:admin

stats admin if TRUE

frontend k8s_https *:8443

mode tcp

maxconn 2000

default_backend https_sri

backend https_sri

balance roundrobin

server s1 130.252.10.235:6443 check inter 2000 fall 3 rise 3 weight 1

server s2 130.252.10.236:6443 check inter 2000 fall 3 rise 3 weight 1

server s3 130.252.10.237:6443 check inter 2000 fall 3 rise 3 weight 1

EOF

haproxy 在 10086 埠输出 status 资讯；haproxy 监听所有界面的 8443 埠，该埠与环境变数 ${KUBE_APISERVER} 指定的埠必须一致；server 字段列出所有 kube-apiserver 监听的 IP 和埠；配置keepalived

cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak

cat > /etc/keepalived/keepalived.conf ! Configuration File for keepalived

global_defs {

script_user root

enable_script_security

router_id LVS_MASTER

}

vrrp_script check-haproxy {

script "/usr/bin/killall -0 haproxy"

interval 3

weight -30

}

vrrp_instance VI_kube-master {

state MASTER

interface ens32

virtual_router_id 60

dont_track_primary

priority 120

advert_int 3

track_script {

check-haproxy

}

authentication {

auth_type PASS

auth_pass 1111

}

virtual_ipaddress {

130.252.10.233

}

EOF

注意：在另外两个节点，设定state为BACKUP，priority设定为110,100

启动所有节点的haproxy和keepalived

systemctl restart haproxy && systemctl enable haproxy && systemctl status haproxy

systemctl restart keepalived && systemctl enable keepalived && systemctl status keepalived

检视VIP

注意，使用ifconfig命令是看不到的，必须使用ip addr命令

# ip addr

1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

inet 127.0.0.1/8 scope host lo