I have a Kafka cluster running with 2 partitions. I was looking for a way to increase the partition count to 3. However, I don’t want to lose existing messages in the topic. I tried stopping Kafka, modifying the
server.properties file to increase the number of partitions to 3 and restart Kafka. However, that does not seem to change anything. Using Kafka
ConsumerOffsetChecker, I still see it is using only 2 partitions. The Kafka version I am using is 0.8.2.2. In version 0.8.1, there used to be a script called
kafka-add-partitions.sh, which I guess might do the trick. However, I don’t see any such script in 0.8.2. Is there any way of accomplishing this? I did experiment with creating a whole new topic and for that one it does seem to use 3 partitions as per the change in the
server.properties file. However, for existing topics, it doesn’t seem to care.
Looks like you can use this script instead:
bin/kafka-topics.sh --zookeeper zk_host:port/chroot --alter --topic my_topic_name --partitions 40
In the code it looks like they do same thing:
AdminUtils.createOrUpdateTopicPartitionAssignmentPathInZK(topic, partitionReplicaList, zkClient, true)
However you have to be aware of re-partitioning when using key:
Be aware that one use case for partitions is to semantically partition
data, and adding partitions doesn’t change the partitioning of existing data so this may disturb consumers if they rely on that
partition. That is if data is partitioned by
hash(key) % number_of_partitionsthen this partitioning will potentially be
shuffled by adding partitions but Kafka will not attempt to
automatically redistribute data in any way.
In my case the value
zk_host:port/chroot for parameter
--zookeeper threw the following exception:
ERROR java.lang.IllegalArgumentException: Topic my_topic_name does not exist on ZK path zk_host:port/chroot.
So, I tried the following and it worked:
bin/kafka-topics.sh --alter --zookeeper zk_host:port --topic my_topic_name --partitions 10
For anyone who wants solution for newer Kafka versions.Please follow this method.
Kafka’s entire data retention and transfer policy depends on partitions so be careful about effects of increasing partitions. (Kafka’s newer versions display warning regarding this) Try to avoid configuration in which one broker has too many leader partitions.
There is simple 3 stage approach to this.
Step 1: Increase the partitions in topics
./bin/kafka-topics.sh --zookeeper localhost:9092 --alter --topic testKafka_5 --partitions 6
Step 2: Create a partitioning json file for given topic
Create file with newer partition and replicas. It’s better to expand replicas to different brokers but they should be present within same cluster. Take latency into consideration for distant replicas.
Transfer the given file to your Kafka.
Step 3: Reassign partitions and verify
./bin/kafka-reassign-partitions.sh --zookeeper localhost:9092 --reassignment-json-file bin/increase-replication-factor.json --execute ./bin/kafka-reassign-partitions.sh --zookeeper localhost:9092 --reassignment-json-file bin/increase-replication-factor.json --verify
You can check the effects of your change using –describe command.
If you are using Kafka in Windows
try this code for alter or add partition in topic
.\bin\windows\kafka-topics.bat --alter --zookeeper localhost:2181 --topic TopicName --partitions 20
.\bin\windows\kafka-topics.bat --alter --zookeeper localhost:2181 --topic TopicName --replica-assignment 0:1:2,0:1:2,0:1:2,2:1:0 --partitions 10