magnum部署k8s

introduction

使用 openstack magnum 项目来创建稳定的 k8s 集群。

版本信息

name	version
magnum	ocata
k8s	v1.5.3
kubectl	v1.5.3

部署需要的所有镜像

• google_containers/kube-ui:v4
• google_containers/pause-amd64:3.0
• google_containers/hyperkube:v1.5.3
• google_containers/pause:0.8.0
• google_containers/kubedns-amd64:1.9
• google_containers/dnsmasq-metrics-amd64:1.0
• google_containers/kube-dnsmasq-amd64:1.4
• google_containers/exechealthz-amd64:1.2
• google_containers/defaultbackend:1.0
• google_containers/nginx-ingress-controller:0.9.0-beta.11

上述所有 docker 镜像打包为 k8s-ocata.tar.gz ，使用 local registry 时将其 push 即可

环境设置

将 k8s 部署的需要的租户、网络都固定下来，约定如下

• 租户: admin
• 网络: kycloud-net
• 子网: magnum-subnet, 172.16.0.0/24

创建并连接路由

heat 配置

# openstack role add –project admin –user admin heat_stack_owner

上述操作保证 heat stack-create 操作成功

上传镜像

镜像地址

https://fedorapeople.org/groups/magnum/fedora-atomic-ocata.qcow2

转换为 raw 格式

# qemu-img convert -f qcow2 -O raw fedora-atomic-ocata.qcow2 fedora-atomic-ocata.raw

上传镜像

# glance image-create –name fedora-atomic-ocata-raw –disk-format raw –container-format bare –file fedora-atomic-ocata.raw –os-distro fedora-atomic –progress

创建网络

创建一个独立的网络及其子网用于 k8s 集群

配置本地 registry

使用 magnum 创建 k8s 需要从 gcr.io 拉取 docker images，可以使用两种办法解决

• 配置翻墙
• 使用 local registry

我们采用使用 local registry 的方式实现。

部署

这里使用 k8s 集群的网络创建一个虚拟机，并安装 local registry，具体的步骤是

安装 docker

# curl -sSL https://get.docker.io | bash
# curl -sSL https://get.daocloud.io/daotools/set_mirror.sh | sh -s http://db9b6d5c.m.daocloud.io

部署 local registry

# docker run -d -p 5000:5000 –restart=always –name registry \
 -v pwd/data:/var/lib/registry \
 registry:2

注意需要打开安全组 5000 端口

将需要的 images 导入到本地

# docker load -i k8s-ocata.tar.gz

假设 local registry url 是 192.168.21.10:5000, 给这些镜像打上 tag

# registry_url=192.168.21.10:5000
# docker tag gcr.io/google_containers/kube-ui:v4 $registry_url/google_containers/kube-ui:v4
# docker tag gcr.io/google_containers/pause-amd64:3.0 $registry_url/google_containers/pause-amd64:3.0
# docker tag gcr.io/google-containers/hyperkube:v1.5.3 $registry_url/google_containers/hyperkube:v1.5.3
# docker tag gcr.io/google_containers/pause:0.8.0 $registry_url/google_containers/pause:0.8.0

编辑 /etc/docker/daemon.json，配置 registry client(这里的 client 和 registry 都在同个虚拟机中)

{ 
 "registry-mirrors": ["http://db9b6d5c.m.daocloud.io"], 
 "insecure-registries":["192.168.21.10:5000"]
}

重启 docker

systemctl restart docker

push 到 registry

# docker push $registry_url/google_containers/kube-ui:v4
# docker push $registry_url/google_containers/pause-amd64:3.0
# docker push $registry_url/google_containers/hyperkube:v1.5.3
# docker push $registry_url/google_containers/pause:0.8.0

验证

验证的过程是先删除本地的一个 images，在 pull 一次看是否成功

# docker rmi $registry_url/google_containers/kube-ui:v4
# docker pull $registry_url/google_containers/kube-ui:v4

创建 k8s

template

创建 template

# magnum cluster-template-create k8s-cluster \
–image fedora-atomic-ocata-raw \
–keypair kolla1 \
–external-network-id cbd41bb4-7e9c-4fb3-bcc8-3c23f7de0587 \
–dns-nameserver 8.8.8.8 \
–master-flavor m1.small \
–flavor m1.large \
–docker-volume-size 40 \
–network-driver flannel \
–coe kubernetes \
–volume-driver cinder \
–floating-ip-enabled \
–fixed-network 4ca571c6-a37c-4035-8746-44c02e9c63bc \
–fixed-subnet 3304e1d2-c3a4-48f3-9691-53072510ec70 \
–insecure-registry 192.168.21.10:5000

根据实际情况配置参数

• flavor 使用默认的 m1.large
• insecure-registry 配置 local registry
• keypair 用来登录到各个节点作自定义配置
• 3 个 net 配置使用 id 的方式，否则使用 loadbalancer 会出错

k8s

创建 k8s

# magnum cluster-create –node-count 1 –cluster-template k8s-cluster

配置 k8s 集群

为了能够使用现有 iaas 的 loadbalance 和 cinder as storage 等 feature, 需要对已经创建的 k8s 集群手动作配置改动

配置节点 hosts

在每个节点配置各个节点的 ip –> hostname，e.g.

$ sudo vi /etc/hosts
172.16.0.55 ph-xmmx4bjtbp-0-asladitc3bww-kube-master-gzhiyy5bnysn
172.16.0.52 ph-rnyzvwp3z5-0-n4ijjqf2cah4-kube-minion-lblxlwli5oj6

cloud-config

配置集群每个节点的 cloud-config， /etc/sysconfig/kube_openstack_config ，将 auth 用户换成 admin 用户

[Global]
auth-url=http://192.168.21.100:5000/v2.0/ 
username=admin 
password=CAk723w8VaEpOpCzSm0SHWg5LK8LI4bEu0sT3o2H 
tenant-name=admin 
region=RegionOne 
trust-id=

master 节点

k8s 的 controller-manager 使用 docker 启动，之需要修改对应的 manifests 文件 修改 /etc/kubernetes/manifests/kube-controller-manager.yaml

- controller-manager
- –cloud-provider=openstack
- –cloud-config=/etc/sysconfig/kube_openstack_config

注意一定要将上述两个参数紧接着放在 controller-manager 之后

k8s 的 apiserver、kubelet 使用 systemd 管理，作如下修改

• /etc/kubernetes/apiserver
• /etc/kubernetes/kubelet

在 KUBE_API_ARGS 后添加

–cloud-provider=openstack –cloud-config=/etc/sysconfig/kube_openstack_config

$ sudo systemctl restart kube-apiserver.service
$ sudo systemctl restart kubelet.service

查询服务状态，确保正确

$ sudo systemctl status kube-apiserver.service
$ sudo systemctl status kubelet.service

slave 节点

slave 只需要对 kubelet 作修改

• /etc/kubernetes/kubelet

  –cloud-provider=openstack –cloud-config=/etc/sysconfig/kube_openstack_config

  $ sudo systemctl restart kubelet.service

查询服务状态，确保正确

$ sudo systemctl status kubelet.service

验证

use cinder in k8s https://docs.openstack.org/magnum/latest/userguide.html#using-cinder-in-kubernetes

use lbaas in k8s https://github.com/openstack/magnum/blob/master/doc/source/dev/kubernetes-load-balancer.rst

k8s-dns

部署

需要的 docker 镜像

• gcr.io/google_containers/kubedns-amd64:1.9
• gcr.io/google_containers/dnsmasq-metrics-amd64:1.0
• gcr.io/google_containers/kube-dnsmasq-amd64:1.4
• gcr.io/google_containers/exechealthz-amd64:1.2

(参照上面部分的方法将其放到 local registry)

修改 k8s 集群中节点的 kubelet 启动参数

$ vi /etc/kubernetes/kubelet
KUBELET_ARGS="–cluster_dns=10.254.0.10 –cluster_domain=cluster.local"

$ sudo systemctl restart kubelet

上传 k8s 的 config 等文件(即 magnum config xx 得到的文件)到集群各个节点

# scp -r auth/ fedora@x.x.x.x:~/

另一种方法使用 secret 来存储这些文件并 mount 到 container 内也可实现，需要修改描述文件中相关部分，secret 创建如下

# kubectl create secret generic client-auth-files –from-file=config=./config –from-file=ca.pem=./ca.pem –from-file=cert.pem=./cert.pem –from-file=key.pem=./key.pem -n kube-system

部署 dns，注意配置 skydns-rc.yaml 中的 –kube-master-url=https://172.16.0.107:6443

# kubectl create -f skydns-deployment.yaml

yaml 参考 https://github.com/ly798/k8s-deploy

验证

创建 busybox pod

# kubectl create -f busybox.yaml

使用该 pod 验证

# kubectl exec busybox nslookup kubernetes

ingress

部署

需要的 docker 镜像

• gcr.io/google_containers/defaultbackend:1.0
• gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.11

(参照上面部分的方法将其放到 local registry)

部署 imgress

# kubectl create -f ingress-deployment.yaml

yaml 参考 https://github.com/ly798/k8s-deploy

通过kubectl -n kube-system describe svc ingress-nginx找到 loadbalance，在 openstack 中找到并为其添加一个 floatingip

验证

创建一个 nginx 和一个 ingress 进行验证

# kubectl create ./nginx-ingress-path/
# kubectl get ing –all-namespaces
NAMESPACE NAME HOSTS ADDRESS PORTS AGE
default nginx-ingress * 192.168.21.236 80 2m
# curl 192.168.21.236/nginx
<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx/1.13.3</center>
</body>
</html>

k8s-dashboard

部署

需要的 docker 镜像

• gcr.io/google_containers/kubernetes-dashboard-amd64:v1.5.1

(注意版本对应，参照上面部分的方法将其放到 local registry)

部署 imgress

# kubectl create -f kube-dashboard-deployment.yaml

yaml 参考 https://github.com/ly798/k8s-deploy

验证

使用 proxy 实现访问

# kubectl proxy

然后是使用浏览器访问 http://127.0.0.1:8001/ui 即可

还可以使用 ingress 的方式来实现访问，但有安全问题，ingress 如下

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
 name: kubernetes-dashboard-ingress
 namespace: kube-system
spec:
 rules:
 - host: ui.myk8s.com
 http:
 paths:
 - backend:
 serviceName: kubernetes-dashboard
 servicePort: 80

访问 http://ui.myk8s.com 即可

FAQ

正常集群的 docker images 和服务

master node:

[fedora@k2-tgqfxoqymo-0-2lgmxw53emmp-kube-master-tfqau4357qaz ~]$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
gcr.io/google_containers/hyperkube v1.5.3 08fdc4720263 5 months ago 557.5 MB
gcr.io/google_containers/pause-amd64 3.0 99e59f495ffa 14 months ago 746.9 kB
[fedora@k2-tgqfxoqymo-0-2lgmxw53emmp-kube-master-tfqau4357qaz ~]$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
9cd5a7a5b52c gcr.io/google_containers/hyperkube:v1.5.3 "/hyperkube controlle" 4 minutes ago Up 4 minutes k8s_kube-controller-manager.d1de1666_kube-controller-manager-k2-tgqfxoqymo-0-2lgmxw53emmp-kube-master-tfqau4357qaz_kube-system_94d16460f7db6bd63dd617ff8527deb6_fc4d74ee
3c7e9189f60c gcr.io/google_containers/hyperkube:v1.5.3 "/hyperkube scheduler" 4 minutes ago Up 4 minutes k8s_kube-scheduler.a3560f4b_kube-scheduler-k2-tgqfxoqymo-0-2lgmxw53emmp-kube-master-tfqau4357qaz_kube-system_cd57cab0d68c7d6b68604c1873b9fd23_fedadfaf
9e7f59b94ecf gcr.io/google_containers/hyperkube:v1.5.3 "/hyperkube proxy –m" 4 minutes ago Up 4 minutes k8s_kube-proxy.adc74569_kube-proxy-k2-tgqfxoqymo-0-2lgmxw53emmp-kube-master-tfqau4357qaz_kube-system_00bbe3f517c93071991098603d8809d7_e05e71a0
f03216cf985b gcr.io/google_containers/pause-amd64:3.0 "/pause" 6 minutes ago Up 6 minutes k8s_POD.d8dbe16c_kube-controller-manager-k2-tgqfxoqymo-0-2lgmxw53emmp-kube-master-tfqau4357qaz_kube-system_94d16460f7db6bd63dd617ff8527deb6_c3f76b6b
420045f9d8c7 gcr.io/google_containers/pause-amd64:3.0 "/pause" 7 minutes ago Up 6 minutes k8s_POD.d8dbe16c_kube-scheduler-k2-tgqfxoqymo-0-2lgmxw53emmp-kube-master-tfqau4357qaz_kube-system_cd57cab0d68c7d6b68604c1873b9fd23_d9333644
825df619192b gcr.io/google_containers/pause-amd64:3.0 "/pause" 7 minutes ago Up 7 minutes k8s_POD.d8dbe16c_kube-proxy-k2-tgqfxoqymo-0-2lgmxw53emmp-kube-master-tfqau4357qaz_kube-system_00bbe3f517c93071991098603d8809d7_be91ac8e

miniom node:

[fedora@k2-qbjizye63f-0-dopyfnyolgvh-kube-minion-ib42uvm3myve ~]$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
gcr.io/google_containers/hyperkube v1.5.3 08fdc4720263 5 months ago 557.5 MB
gcr.io/google_containers/pause-amd64 3.0 99e59f495ffa 14 months ago 746.9 kB
gcr.io/google_containers/kube-ui v4 762cc97d5aa1 19 months ago 5.677 MB
[fedora@k2-qbjizye63f-0-dopyfnyolgvh-kube-minion-ib42uvm3myve ~]$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1c5b7b9a90fd gcr.io/google_containers/kube-ui:v4 "/kube-ui" 2 minutes ago Up 2 minutes k8s_kube-ui.53edfc94_kube-ui-v4-8vczr_kube-system_de2938a2-7104-11e7-8a3f-fa163e74b5a1_25809512
49328e69c8b3 gcr.io/google_containers/hyperkube:v1.5.3 "/hyperkube proxy –m" 3 minutes ago Up 2 minutes k8s_kube-proxy.e8846ca1_kube-proxy-k2-qbjizye63f-0-dopyfnyolgvh-kube-minion-ib42uvm3myve_kube-system_dc6c88a4f0d6b3341f1b942f3dbf8885_dd7f1dc1
c00f5c8e8bd7 gcr.io/google_containers/pause-amd64:3.0 "/pause" 4 minutes ago Up 3 minutes k8s_POD.1e570369_kube-ui-v4-8vczr_kube-system_de2938a2-7104-11e7-8a3f-fa163e74b5a1_518ad58d
6cc5cfda187f gcr.io/google_containers/pause-amd64:3.0 "/pause" 5 minutes ago Up 4 minutes k8s_POD.d8dbe16c_kube-proxy-k2-qbjizye63f-0-dopyfnyolgvh-kube-minion-ib42uvm3myve_kube-system_dc6c88a4f0d6b3341f1b942f3dbf8885_0f31b601

k8s 集群的扩展

1. 直接使用 magnum update 方式扩展节点个数

2. 节点 docker pool 的扩展

   可直接给对应 node 添加 cinder 卷，对 docker-pool 使用 lvm 进行扩展

k8s 集群节点重启

重启后到各节点上启动 docker 实例

$ sudo docker start sudo docker ps -a -q

使用 LoadBalancer 时报错

在 master 节点报错

Aug 09 07:16:56 ph-xmmx4bjtbp-0-asladitc3bww-kube-master-gzhiyy5bnysn.novalocal dockerd-current[2295]: I0809 07:16:56.799446 1 event.go:217] Event(api.ObjectReference{Kind:"ReplicaSet", Namespace:"default", Name:"nginx-deployment-1045676374", UID:"ba01e2a2-7cd2-11e7-82c2-fa163ed8eebc", APIVersion:"extensions", ResourceVersion:"1484", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: nginx-deployment-1045676374-618mn
Aug 09 07:16:56 ph-xmmx4bjtbp-0-asladitc3bww-kube-master-gzhiyy5bnysn.novalocal dockerd-current[2295]: E0809 07:16:56.932792 1 servicecontroller.go:760] Failed to process service. Retrying in 5s: Failed to create load balancer for service default/nginxservice-loadbalancer: Error creating loadbalancer ab9e95e8d7cd211e782c2fa163ed8eeb: Error creating loadbalancer {ab9e95e8d7cd211e782c2fa163ed8eeb Kubernetes external service ab9e95e8d7cd211e782c2fa163ed8eeb magnum-subnet <nil> }: Expected HTTP response code [201 202] when accessing [POST http://192.168.21.100:9696/v2.0/lbaas/loadbalancers], but got 400 instead
Aug 09 07:16:56 ph-xmmx4bjtbp-0-asladitc3bww-kube-master-gzhiyy5bnysn.novalocal dockerd-current[2295]: {"NeutronError": {"message": "Invalid input for vip_subnet_id. Reason: 'magnum-subnet' is not a valid UUID.", "type": "HTTPBadRequest", "detail": ""}}

template 中的 net 相关配置不能使用 name，要使用 uuid

reference

• https://docs.openstack.org/magnum/latest/
• https://mritd.me/2017/03/04/how-to-use-nginx-ingress/
• https://github.com/kubernetes/dashboard/blob/v1.5.1/src/deploy/kubernetes-dashboard.yaml
• https://github.com/kubernetes/dashboard/blob/master/docs/user-guide/troubleshooting.md