安装Zookeeper和Kafka集群

本文介绍如何安装Zookeeper和Kafka集群。为了方便,介绍的是在一台服务器上的安装,实际应该安装在多台服务器上,但步骤是一样的。

安装Zookeeper集群下载安装包

从官网上下载安装包:

curl https://dlcdn.apache.org/zookeeper/zookeeper-3.7.1/apache-zookeeper-3.7.1-bin.tar.gz -o apache-zookeeper-3.7.1-bin.tar.gz

解压:

tar xvf apache-zookeeper-3.7.1-bin.tar.gz 

配置

创建目录 zk1,然后添加如下配置:

zk1/myid:

1

zk1/zk.config:

tickTime=2000initLimit=10syncLimit=5dataDir=/Users/larry/IdeaProjects/pkslow-samples/other/install-kafka-cluster/src/main/zookeeper/zk1clientPort=2181server.1=127.0.0.1:2888:3888server.2=127.0.0.1:2889:3889server.3=127.0.0.1:2890:3890

对于zk2zk3也重复同样的步骤,并修改相应的配置:

zk2/myid:

2

zk2/zk.config:

tickTime=2000initLimit=10syncLimit=5dataDir=/Users/larry/IdeaProjects/pkslow-samples/other/install-kafka-cluster/src/main/zookeeper/zk2clientPort=2182server.1=127.0.0.1:2888:3888server.2=127.0.0.1:2889:3889server.3=127.0.0.1:2890:3890

zk3/myid:

3

zk3/zk.config:

tickTime=2000initLimit=10syncLimit=5dataDir=/Users/larry/IdeaProjects/pkslow-samples/other/install-kafka-cluster/src/main/zookeeper/zk3clientPort=2183server.1=127.0.0.1:2888:3888server.2=127.0.0.1:2889:3889server.3=127.0.0.1:2890:3890

启动集群

启动三个服务如下:

$ ./apache-zookeeper-3.7.1-bin/bin/zkServer.sh start ./zk1/zk.config ZooKeeper JMX enabled by defaultUsing config: ./zk1/zk.configStarting zookeeper ... STARTED$ ./apache-zookeeper-3.7.1-bin/bin/zkServer.sh start ./zk2/zk.config ZooKeeper JMX enabled by defaultUsing config: ./zk2/zk.configStarting zookeeper ... STARTED$ ./apache-zookeeper-3.7.1-bin/bin/zkServer.sh start ./zk3/zk.config ZooKeeper JMX enabled by defaultUsing config: ./zk3/zk.configStarting zookeeper ... STARTED

查看状态

通过status命令查看:

$ ./apache-zookeeper-3.7.1-bin/bin/zkServer.sh status ./zk1/zk.config ZooKeeper JMX enabled by defaultUsing config: ./zk1/zk.configClient port found: 2181. Client address: localhost. Client SSL: false.Mode: follower$ ./apache-zookeeper-3.7.1-bin/bin/zkServer.sh status ./zk2/zk.config ZooKeeper JMX enabled by defaultUsing config: ./zk2/zk.configClient port found: 2182. Client address: localhost. Client SSL: false.Mode: leader$ ./apache-zookeeper-3.7.1-bin/bin/zkServer.sh status ./zk3/zk.config ZooKeeper JMX enabled by defaultUsing config: ./zk3/zk.configClient port found: 2183. Client address: localhost. Client SSL: false.Mode: follower

连接其中一个服务并添加数据:

$ ./apache-zookeeper-3.7.1-bin/bin/zkCli.sh -server localhost:2181[zk: localhost:2181(CONNECTED) 0] create /pkslowCreated /pkslow[zk: localhost:2181(CONNECTED) 1] create /pkslow/website www.pkslow.comCreated /pkslow/website

连接另外一个服务,并查看数据,发现与之前创建的是一样的:

$ ./apache-zookeeper-3.7.1-bin/bin/zkCli.sh -server localhost:2182[zk: localhost:2182(CONNECTED) 1] get /pkslow/websitewww.pkslow.com

目录结构如下:

安装Kafka集群下载安装包

通过官网下载如下:

curl https://downloads.apache.org/kafka/3.4.0/kafka_2.13-3.4.0.tgz -o kafka_2.13-3.4.0.tgz

解压安装包:

tar -xzf kafka_2.13-3.4.0.tgz

配置

Broker 1的配置如下:

broker.id=1port=9091listeners=PLAINTEXT://:9091zookeeper.connect=127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183log.dirs=/Users/larry/IdeaProjects/pkslow-samples/other/install-kafka-cluster/src/main/kafka/kafka1/kafka-logs

Broker 2的配置如下:

broker.id=2port=9092listeners=PLAINTEXT://:9092zookeeper.connect=127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183log.dirs=/Users/larry/IdeaProjects/pkslow-samples/other/install-kafka-cluster/src/main/kafka/kafka2/kafka-logs

Broker 3的配置如下:

broker.id=3port=9093listeners=PLAINTEXT://:9093zookeeper.connect=127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183log.dirs=/Users/larry/IdeaProjects/pkslow-samples/other/install-kafka-cluster/src/main/kafka/kafka3/kafka-logs

目录结构如下:

启动集群

启动kafka服务如下:

./kafka_2.13-3.4.0/bin/kafka-server-start.sh ./kafka1/server.properties./kafka_2.13-3.4.0/bin/kafka-server-start.sh ./kafka2/server.properties./kafka_2.13-3.4.0/bin/kafka-server-start.sh ./kafka3/server.properties

检查如测试

创建topic:

$ kafka_2.13-3.4.0/bin/kafka-topics.sh --create --topic pkslow-topic --bootstrap-server localhost:9091,localhost:9092,localhost:9093 --partitions 3 --replication-factor 3Created topic pkslow-topic.

列出topic:

$ kafka_2.13-3.4.0/bin/kafka-topics.sh --list --bootstrap-server localhost:9091,localhost:9092,localhost:9093pkslow-topic

查看topic:

$ kafka_2.13-3.4.0/bin/kafka-topics.sh --describe --topic pkslow-topic --bootstrap-server localhost:9091,localhost:9092,localhost:9093Topic: pkslow-topic     TopicId: 7CLy7iZeRvm8rCrn8Dw_mA PartitionCount: 3       ReplicationFactor: 3    Configs:         Topic: pkslow-topic     Partition: 0    Leader: 3       Replicas: 3,1,2 Isr: 3,1,2        Topic: pkslow-topic     Partition: 1    Leader: 1       Replicas: 1,2,3 Isr: 1,2,3        Topic: pkslow-topic     Partition: 2    Leader: 2       Replicas: 2,3,1 Isr: 2,3,1

生产者发消息到brokers:

$ kafka_2.13-3.4.0/bin/kafka-console-producer.sh --broker-list localhost:9091,localhost:9092,localhost:9093 --topic pkslow-topic>My name is Larry Deng.>My website is www.pkslow.com.>

消费者从brokers收消息:

$ kafka_2.13-3.4.0/bin/kafka-console-consumer.sh --bootstrap-server localhost:9091,localhost:9092,localhost:9093 --topic pkslow-topic --from-beginningMy name is Larry Deng.My website is www.pkslow.com.

代码

配置可以参考 GitHub pkslow-samples