Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Creating Kafka topic does not create partition in kakfa-logs

Highlighted

Creating Kafka topic does not create partition in kakfa-logs

Rising Star

Hello, I am using the command line to create a kafka topic and when I do I dont see the partition directory for logs created in Kafka-logs directory... I am using this command:

./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic dataloss30-1

It then says that the topic was successfully created. I can see the topic when I run a command to list the topics such as :

./bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic dataloss30-1 --from-beginning

I can even see the data IN the topic itself with the command line but when I go to /kafka-logs I do not see a directory for the topic and this is an issue because when I use Nifi to try to ConsumeKafka with it, it cannot find the topic.

When I use Nifi to create a topic through auto-generated topics, it creates the partition directory.

Has anyone else experienced anything like this or has any ideas as to why its happening?

2 REPLIES 2

Re: Creating Kafka topic does not create partition in kakfa-logs

Rising Star

I restarted Kafka and now it seems to be working. Anyone know why this might be happening?

Re: Creating Kafka topic does not create partition in kakfa-logs

New Contributor

Hi All,

I am also having similar issue. I tried creating topic from command line and topic created successfully and able to list it. But when I navigate to /kafka-logs, I couldn't find any files listed over there.

# bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
Created topic "test".

# bin/kafka-topics.sh --list --zookeeper localhost:2181
test

Also, below is the log message I see in state_change.log

2018-04-06 00:53:58,214] TRACE Controller 1001 epoch 1 changed partition [test,0] state from NonExistentPartition to NewPartition with assigned replicas 1001 (state.change.logger)
[2018-04-06 00:53:58,223] TRACE Controller 1001 epoch 1 changed state of replica 1001 for partition [test,0] from NonExistentReplica to NewReplica (state.change.logger)
[2018-04-06 00:53:58,281] TRACE Controller 1001 epoch 1 changed partition [test,0] from NewPartition to OnlinePartition with leader 1001 (state.change.logger)
[2018-04-06 00:53:58,282] TRACE Controller 1001 epoch 1 sending become-leader LeaderAndIsr request (Leader:1001,ISR:1001,LeaderEpoch:0,ControllerEpoch:1) to broker 1001 for partition [test,0] (state.change.logger)
[2018-04-06 00:53:58,293] TRACE Controller 1001 epoch 1 sending UpdateMetadata request (Leader:1001,ISR:1001,LeaderEpoch:0,ControllerEpoch:1) to broker 1001 for partition test-0 (state.change.logger)
[2018-04-06 00:53:58,297] TRACE Controller 1001 epoch 1 changed state of replica 1001 for partition [test,0] from NewReplica to OnlineReplica (state.change.logger)
[2018-04-06 00:53:58,372] TRACE Controller 1001 epoch 1 received response {error_code=31,partitions=[{topic=test,partition=0,error_code=31}]} for a request sent to broker XX.XX.XX.XX:6667 (id: 1001 rack: null) (state.change.logger)
[2018-04-06 00:53:58,384] TRACE Controller 1001 epoch 1 received response {error_code=31} for a request sent to broker 35.197.50.244:6667 (id: 1001 rack: null) (state.change.logger)
~                                                                                 

My HDP Version 2.3.0 and this setup is done in GCP. Also, for kafka to be listened over public IP, I added below additional parameters

advertised.host.name

advertised.port

host.name

,

Hi,

I am also having similar issue and restarting kafka didn;t help. Itried creating topic using below command.

# bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
Created topic "test".

List Command below is listing the topic name
# bin/kafka-topics.sh --list --zookeeper localhost:2181
test

But when I go to /kafka-logs, I unable to fiind any files partitions created over there.

Also below is the message that I see in state-change.log

2018-04-06 00:53:58,214] TRACE Controller 1001 epoch 1 changed partition [test,0] state from NonExistentPartition to NewPartition with assigned replicas 1001 (state.change.logger)
[2018-04-06 00:53:58,223] TRACE Controller 1001 epoch 1 changed state of replica 1001 for partition [test,0] from NonExistentReplica to NewReplica (state.change.logger)
[2018-04-06 00:53:58,281] TRACE Controller 1001 epoch 1 changed partition [test,0] from NewPartition to OnlinePartition with leader 1001 (state.change.logger)
[2018-04-06 00:53:58,282] TRACE Controller 1001 epoch 1 sending become-leader LeaderAndIsr request (Leader:1001,ISR:1001,LeaderEpoch:0,ControllerEpoch:1) to broker 1001 for partition [test,0] (state.change.logger)
[2018-04-06 00:53:58,293] TRACE Controller 1001 epoch 1 sending UpdateMetadata request (Leader:1001,ISR:1001,LeaderEpoch:0,ControllerEpoch:1) to broker 1001 for partition test-0 (state.change.logger)
[2018-04-06 00:53:58,297] TRACE Controller 1001 epoch 1 changed state of replica 1001 for partition [test,0] from NewReplica to OnlineReplica (state.change.logger)
[2018-04-06 00:53:58,372] TRACE Controller 1001 epoch 1 received response {error_code=31,partitions=[{topic=test,partition=0,error_code=31}]} for a request sent to broker x.x.x.x:6667 (id: 1001 rack: null) (state.change.logger)
[2018-04-06 00:53:58,384] TRACE Controller 1001 epoch 1 received response {error_code=31} for a request sent to broker 35.197.50.244:6667 (id: 1001 rack: null) (state.change.logger)
~                                                                                 

Please help me. This is sitting on Google cloud and below configurations have been to be able to listen over public IP

advertised.host.name

host.name

advertised.port

Don't have an account?
Coming from Hortonworks? Activate your account here