作者：老Z，云原生爱好者，目前专注于云原生运维，KubeSphere Ambassador。

前言

知识点

  • 定级：入门级
  • RabbitMQ 单节点安装部署
  • RabbitMQ 集群安装部署
  • GitOps 运维思想

演示服务器配置

主机名IPCPU内存系统盘数据盘用途
zdeops-master192.168.9.92440200Ansible 运维控制节点
ks-k8s-master-0192.168.9.9141640200+200KubeSphere/k8s-master/k8s-worker/Ceph
ks-k8s-master-1192.168.9.9241640200+200KubeSphere/k8s-master/k8s-worker/Ceph
ks-k8s-master-2192.168.9.9341640200+200KubeSphere/k8s-master/k8s-worker/Ceph
storage-node-0192.168.9.952840200+200ElasticSearch/GlusterFS
storage-node-0192.168.9.962840200+200ElasticSearch/GlusterFS
storage-node-0192.168.9.972840200+200ElasticSearch/GlusterFS
harbor192.168.9.892840200Harbor
合计822843202800

演示环境涉及软件版本信息

  • 操作系统：CentOS-7.9-x86_64
  • Ansible：2.8.20
  • KubeSphere：3.3.0
  • Kubernetes：v1.24.1
  • Rook：v1.9.7
  • Ceph： v16.2.9
  • GlusterFS：9.5.1
  • ElasticSearch：7.17.5
  • Harbor：2.5.1
  • RabbitMQ：3.9.22
  • RabbitMQ Cluster Operator：1.14.0

简介

RabbitMQ 单节点如何在 K8s 集群上部署？RabbitMQ 集群如何在 K8s 集群上部署？60分钟带你实战入门。

单节点 RabbitMQ 部署思路梳理

  • StatefulSet
  • Headless Service：内部服务用
  • External Service：外部管理用
  • Secrets：管理用户名和密码
  • Image：DockerHub 官方提供的 rabbitmq:3.9.22-management(带 management 插件)

准备离线镜像

此过程为可选项，离线内网环境可用，如果不配置内网镜像，后续的资源配置清单中注意更改容器的 image 为默认值。

在一台能同时访问互联网和内网 Harbor 仓库的服务器上进行下面的操作。

  • 下载镜像
docker pull rabbitmq:3.9.22-management
  • 重新打 tag
docker tag rabbitmq:3.9.22-management registry.zdevops.com.cn/library/rabbitmq:3.9.22-management
  • 推送到私有镜像仓库
docker push registry.zdevops.com.cn/library/rabbitmq:3.9.22-management
  • 清理临时镜像
docker rmi rabbitmq:3.9.22-managementdocker rmi registry.zdevops.com.cn/library/rabbitmq:3.9.22-management

资源配置清单

  • rabbitmq-secret.yaml
---kind: SecretapiVersion: v1metadata:  name: rabbitmq-secret  namespace: zdevopsdata:  pass: UEA4OHcwcmQ=  user: YWRtaW4=type: Opaque
  • rabbitmq-sts.yaml
---apiVersion: apps/v1kind: StatefulSetmetadata:  name: rabbitmq  namespace: zdevops  labels:    app: rabbitmqspec:  replicas: 1  selector:    matchLabels:      app: rabbitmq  template:    metadata:      creationTimestamp: null      labels:        app: rabbitmq    spec:      volumes:        - name: host-time          hostPath:            path: /etc/localtime            type: ''      containers:        - name: rabbitmq          image: 'registry.zdevops.com.cn/library/rabbitmq:3.9.22-management'          ports:            - name: tcp-5672              containerPort: 5672              protocol: TCP            - name: http-15672              containerPort: 15672              protocol: TCP          env:            - name: RABBITMQ_DEFAULT_USER              valueFrom:                secretKeyRef:                  name: rabbitmq-secret                  key: user            - name: RABBITMQ_DEFAULT_PASS              valueFrom:                secretKeyRef:                  name: rabbitmq-secret                  key: pass          resources:            limits:              cpu: '2'              memory: 4000Mi            requests:              cpu: 100m              memory: 500Mi          volumeMounts:            - name: host-time              readOnly: true              mountPath: /etc/localtime  serviceName: rabbitmq-headless---apiVersion: v1kind: Servicemetadata:  name: rabbitmq-headless  namespace: zdevops  labels:    app: rabbitmqspec:  ports:    - name: tcp-rabbitmq-5672      protocol: TCP      port: 5672      targetPort: 5672  selector:    app: rabbitmq  clusterIP: None  type: ClusterIP
  • rabbitmq-external.yaml
---apiVersion: v1kind: Servicemetadata:  name: rabbitmq-external  namespace: zdevops  labels:    app: rabbitmq-externalspec:  ports:    - name: http-rabbitmq-external      protocol: TCP      port: 15672      targetPort: 15672      nodePort: 31672  selector:    app: rabbitmq  type: NodePort

GitOps

在运维开发服务器上操作

# 在已有代码仓库创建 rabbitmq/single 目录[root@zdevops-master k8s-yaml]# mkdir -p rabbitmq/single# 编辑资源配置清单[root@zdevops-master k8s-yaml]# vi rabbitmq/single/rabbitmq-secret.yaml[root@zdevops-master k8s-yaml]# vi rabbitmq/single/rabbitmq-sts.yaml[root@zdevops-master k8s-yaml]# vi rabbitmq/single/rabbitmq-external.yaml# 提交 Git[root@zdevops-master k8s-yaml]# git add rabbitmq[root@zdevops-master k8s-yaml]# git commit -am '添加rabbitmq 单节点资源配置清单'[root@zdevops-master k8s-yaml]# git push

部署资源

在运维管理服务器上操作

  • 更新镜像仓库代码
[root@zdevops-master k8s-yaml]# git pull
  • 部署资源
[root@zdevops-master k8s-yaml]# kubectl apply -f rabbitmq/single/

验证

  • Secret
[root@zdevops-master k8s-yaml]# kubectl get secret -n zdevopsNAME              TYPE     DATA   AGErabbitmq-secret   Opaque   2      8s
  • StatefulSet
[root@zdevops-master k8s-yaml]# kubectl get sts -o wide -n zdevopsNAME       READY   AGE   CONTAINERS   IMAGESrabbitmq   1/1     25s   rabbitmq     registry.zdevops.com.cn/library/rabbitmq:3.9.22-management
  • Pods
[root@zdevops-master k8s-yaml]# kubectl get pods -o wide -n zdevopsNAME         READY   STATUS    RESTARTS   AGE   IP             NODE              NOMINATED NODE   READINESS GATESrabbitmq-0   1/1     Running   0          26s   10.233.87.13   ks-k8s-master-1
  • Service
[root@zdevops-master k8s-yaml]# kubectl get svc -o wide -n zdevopsNAME                TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)           AGE   SELECTORrabbitmq-external   NodePort    10.233.4.224           15672:31672/TCP   36s   app=rabbitmqrabbitmq-headless   ClusterIP   None                   5672/TCP          36s   app=rabbitmq
  • 图形化管理界面

清理资源

  • 清理 StatefulSet
[root@zdevops-master k8s-yaml]# kubectl delete sts rabbitmq -n zdevops
  • 清理服务
[root@zdevops-master k8s-yaml]# kubectl delete svc rabbitmq-external rabbitmq-headless -n zdevops

集群模式 RabbitMQ 部署思路梳理

使用官方提供的 RabbitMQ Cluster Operator for Kubernetes。

Open source RabbitMQ Cluster Kubernetes Operator by VMware。

官方文档快速部署

官方提供了快速部署的例子，只需要两步，这里我们仅作参考，后面我们参考内网离线镜像部署的方案。

  • 部署 RabbitMQ Cluster Operator
kubectl apply -f https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml
  • 部署 RabbitMQ Cluster
kubectl apply -f https://raw.githubusercontent.com/rabbitmq/cluster-operator/main/docs/examples/hello-world/rabbitmq.yaml

准备离线镜像

此过程为可选项，离线内网环境可用，如果不配置内网镜像，后续的资源配置清单中注意更改容器的 image 为默认值。

在一台能同时访问互联网和内网 Harbor 仓库的服务器上进行下面的操作。

  • 在 Harbor 中创建项目
curl -u "admin:Harbor12345" -X POST -H "Content-Type: application/json" https://registry.zdevops.com.cn/api/v2.0/projects -d '{ "project_name": "rabbitmqoperator", "public": true}'
  • 下载镜像
docker pull rabbitmqoperator/cluster-operator:1.14.0
  • 重新打 tag
docker tag rabbitmqoperator/cluster-operator:1.14.0 registry.zdevops.com.cn/rabbitmqoperator/cluster-operator:1.14.0
  • 推送到私有镜像仓库
docker push registry.zdevops.com.cn/rabbitmqoperator/cluster-operator:1.14.0
  • 清理临时镜像
docker rmi rabbitmqoperator/cluster-operator:1.14.0docker rmi registry.zdevops.com.cn/rabbitmqoperator/cluster-operator:1.14.0

资源配置清单

  • 通过官网获取 RabbitMQ Cluster Operator 部署资源配置清单「cluster-operator.yml」
wget https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml
  • 修改 RabbitMQ Cluster Operator image 为内网镜像
sed -i 's#rabbitmqoperator#registry.zdevops.com.cn/rabbitmqoperator#g' cluster-operator.yml
  • RabbitMQ Cluster 部署资源清单「rabbitmq-cluster.yaml」
apiVersion: rabbitmq.com/v1beta1kind: RabbitmqClustermetadata:  namespace: zdevops  name: rabbitmq-cluster  labels:    app: rabbitmq-clusterspec:  replicas: 3  image: registry.zdevops.com.cn/library/rabbitmq:3.9.22-management  resources:    limits:      cpu: 2      memory: 4Gi    requests:      cpu: 100m      memory: 500Mi  rabbitmq:    additionalConfig: |      default_user=admin      default_pass=P@88w0rd

更多配置参数和配置示例，请参考 官方文档

  • 管理页面的外部访问服务 rabbitmq-cluster-external.yaml
---apiVersion: v1kind: Servicemetadata:  name: rabbitmq-cluster-external  namespace: zdevops  labels:    app: rabbitmq-cluster-externalspec:  ports:    - name: management      protocol: TCP      port: 15672      targetPort: 15672      nodePort: 31672  selector:    app.kubernetes.io/name: rabbitmq-cluster  type: NodePort

GitOps

在运维开发服务器上操作

# 在已有代码仓库创建 rabbitmq/cluster 目录[root@zdevops-master k8s-yaml]# mkdir -p rabbitmq/cluster# 编辑资源配置清单[root@zdevops-master k8s-yaml]# vi rabbitmq/cluster/cluster-operator.yml[root@zdevops-master k8s-yaml]# vi rabbitmq/cluster/rabbitmq-cluster.yaml[root@zdevops-master k8s-yaml]# vi rabbitmq/cluster/rabbitmq-cluster-external.yaml# 提交 Git[root@zdevops-master k8s-yaml]# git add rabbitmq/cluster[root@zdevops-master k8s-yaml]# git commit -am '添加 rabbitmq 集群模式部署资源配置清单'[root@zdevops-master k8s-yaml]# git push

部署资源

在运维管理服务器上操作

  • `更新镜像仓库代码
[root@zdevops-master k8s-yaml]# git pull
  • 部署 RabbitMQ Cluster Operator
[root@zdevops-master k8s-yaml]# kubectl apply -f rabbitmq/cluster/cluster-operator.yml
  • 部署 RabbitMQ Cluster
[root@zdevops-master k8s-yaml]# kubectl apply -f rabbitmq/cluster/rabbitmq-cluster.yaml
  • 部署管理页面外部访问服务
[root@zdevops-master k8s-yaml]# kubectl apply -f rabbitmq/cluster/rabbitmq-cluster-external.yaml

验证

  • RabbitMQ Cluster Operator Deployment
[root@zdevops-master k8s-yaml]# kubectl get deployments -n rabbitmq-system -o wideNAME                        READY   UP-TO-DATE   AVAILABLE   AGE    CONTAINERS   IMAGES                                                             SELECTORrabbitmq-cluster-operator   1/1     1            1           107m   operator     registry.zdevops.com.cn/rabbitmqoperator/cluster-operator:1.14.0   app.kubernetes.io/name=rabbitmq-cluster-operator
  • RabbitmqClusters
[root@zdevops-master k8s-yaml]# kubectl get rabbitmqclusters -n zdevopsNAME               ALLREPLICASREADY   RECONCILESUCCESS   AGErabbitmq-cluster   False              Unknown            23s
  • StatefulSet
[root@zdevops-master k8s-yaml]# kubectl get sts -o wide -n zdevopsNAME                      READY   AGE   CONTAINERS   IMAGESrabbitmq-cluster-server   3/3     74s   rabbitmq     registry.zdevops.com.cn/library/rabbitmq:3.9.22-management
  • Pods
[root@zdevops-master k8s-yaml]# kubectl get pods -o wide -n zdevopsNAME                        READY   STATUS    RESTARTS   AGE   IP              NODE              NOMINATED NODE   READINESS GATESrabbitmq-cluster-server-0   1/1     Running   0          84s   10.233.116.26   ks-k8s-master-2              rabbitmq-cluster-server-1   1/1     Running   0          83s   10.233.117.28   ks-k8s-master-0              rabbitmq-cluster-server-2   1/1     Running   0          82s   10.233.87.31    ks-k8s-master-1
  • Services
[root@zdevops-master k8s-yaml]# kubectl get svc -n zdevops -o wideNAME                        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                        AGE    SELECTORrabbitmq-cluster            ClusterIP   10.233.56.153           15692/TCP,5672/TCP,15672/TCP   107s   app.kubernetes.io/name=rabbitmq-clusterrabbitmq-cluster-external   NodePort    10.233.63.84            15672:31672/TCP                74m    app.kubernetes.io/name=rabbitmq-clusterrabbitmq-cluster-nodes      ClusterIP   None                    4369/TCP,25672/TCP             107s   app.kubernetes.io/name=rabbitmq-cluster
  • 图形化管理界面

从管理界面中可以看到一个三节点的集群

清理资源

  • 清理 RabbitmqClusters
[root@zdevops-master k8s-yaml]# kubectl delete rabbitmqclusters rabbitmq-cluster -n zdevops
  • 清理管理页面外部服务
[root@zdevops-master k8s-yaml]# kubectl delete svc rabbitmq-cluster-external -n zdevops

结束语

本系列文档是我在云原生技术领域的学习和运维实践的手记，用输出倒逼输入是一种高效的学习方法，能够快速积累经验和提高技术，只有把学到的知识写出来并能够让其他人理解，才能说明真正掌握了这项知识。

本系列文档内容涵盖 (但不限于) 以下技术领域：

  • KubeSphere
  • Kubernetes
  • Ansible
  • 自动化运维
  • CNCF 技术栈

如果你喜欢本文，请分享给你的小伙伴！

Get 文档

  • Github：https://github.com/devops/z-notes
  • Gitee：https://gitee.com/zdevops/z-notes
  • 知乎：https://www.zhihu.com/people/zdevops/

Get 代码

  • Github：https://github.com/devops/ansible-zdevops
  • Gitee：https://gitee.com/zdevops/ansible-zdevops

Get 视频 B 站

  • 老 Z 手记 https://space.bilibili.com/1039301316

本文由博客一文多发平台 OpenWrite 发布！