Creating Kafka topic does not create partition in kakfa-logs

[#1]

Hello, I am using the command line to create a kafka topic and when I do I dont see the partition directory for logs created in Kafka-logs directory... I am using this command:

./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic dataloss30-1

It then says that the topic was successfully created. I can see the topic when I run a command to list the topics such as :

./bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic dataloss30-1 --from-beginning

I can even see the data IN the topic itself with the command line but when I go to /kafka-logs I do not see a directory for the topic and this is an issue because when I use Nifi to try to ConsumeKafka with it, it cannot find the topic.

When I use Nifi to create a topic through auto-generated topics, it creates the partition directory.

Has anyone else experienced anything like this or has any ideas as to why its happening?

[#2]

I restarted Kafka and now it seems to be working. Anyone know why this might be happening?

[#3]

Hi All,

I am also having similar issue. I tried creating topic from command line and topic created successfully and able to list it. But when I navigate to /kafka-logs, I couldn't find any files listed over there.

# bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
Created topic "test".

# bin/kafka-topics.sh --list --zookeeper localhost:2181
test

Also, below is the log message I see in state_change.log

2018-04-06 00:53:58,214] TRACE Controller 1001 epoch 1 changed partition [test,0] state from NonExistentPartition to NewPartition with assigned replicas 1001 (state.change.logger)
[2018-04-06 00:53:58,223] TRACE Controller 1001 epoch 1 changed state of replica 1001 for partition [test,0] from NonExistentReplica to NewReplica (state.change.logger)
[2018-04-06 00:53:58,281] TRACE Controller 1001 epoch 1 changed partition [test,0] from NewPartition to OnlinePartition with leader 1001 (state.change.logger)
[2018-04-06 00:53:58,282] TRACE Controller 1001 epoch 1 sending become-leader LeaderAndIsr request (Leader:1001,ISR:1001,LeaderEpoch:0,ControllerEpoch:1) to broker 1001 for partition [test,0] (state.change.logger)
[2018-04-06 00:53:58,293] TRACE Controller 1001 epoch 1 sending UpdateMetadata request (Leader:1001,ISR:1001,LeaderEpoch:0,ControllerEpoch:1) to broker 1001 for partition test-0 (state.change.logger)
[2018-04-06 00:53:58,297] TRACE Controller 1001 epoch 1 changed state of replica 1001 for partition [test,0] from NewReplica to OnlineReplica (state.change.logger)
[2018-04-06 00:53:58,372] TRACE Controller 1001 epoch 1 received response {error_code=31,partitions=[{topic=test,partition=0,error_code=31}]} for a request sent to broker XX.XX.XX.XX:6667 (id: 1001 rack: null) (state.change.logger)
[2018-04-06 00:53:58,384] TRACE Controller 1001 epoch 1 received response {error_code=31} for a request sent to broker 35.197.50.244:6667 (id: 1001 rack: null) (state.change.logger)
~                                                                                 

My HDP Version 2.3.0 and this setup is done in GCP. Also, for kafka to be listened over public IP, I added below additional parameters

advertised.host.name

advertised.port

host.name

,

Hi,

I am also having similar issue and restarting kafka didn;t help. Itried creating topic using below command.

# bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
Created topic "test".

List Command below is listing the topic name
# bin/kafka-topics.sh --list --zookeeper localhost:2181
test

But when I go to /kafka-logs, I unable to fiind any files partitions created over there.

Also below is the message that I see in state-change.log

2018-04-06 00:53:58,214] TRACE Controller 1001 epoch 1 changed partition [test,0] state from NonExistentPartition to NewPartition with assigned replicas 1001 (state.change.logger)
[2018-04-06 00:53:58,223] TRACE Controller 1001 epoch 1 changed state of replica 1001 for partition [test,0] from NonExistentReplica to NewReplica (state.change.logger)
[2018-04-06 00:53:58,281] TRACE Controller 1001 epoch 1 changed partition [test,0] from NewPartition to OnlinePartition with leader 1001 (state.change.logger)
[2018-04-06 00:53:58,282] TRACE Controller 1001 epoch 1 sending become-leader LeaderAndIsr request (Leader:1001,ISR:1001,LeaderEpoch:0,ControllerEpoch:1) to broker 1001 for partition [test,0] (state.change.logger)
[2018-04-06 00:53:58,293] TRACE Controller 1001 epoch 1 sending UpdateMetadata request (Leader:1001,ISR:1001,LeaderEpoch:0,ControllerEpoch:1) to broker 1001 for partition test-0 (state.change.logger)
[2018-04-06 00:53:58,297] TRACE Controller 1001 epoch 1 changed state of replica 1001 for partition [test,0] from NewReplica to OnlineReplica (state.change.logger)
[2018-04-06 00:53:58,372] TRACE Controller 1001 epoch 1 received response {error_code=31,partitions=[{topic=test,partition=0,error_code=31}]} for a request sent to broker x.x.x.x:6667 (id: 1001 rack: null) (state.change.logger)
[2018-04-06 00:53:58,384] TRACE Controller 1001 epoch 1 received response {error_code=31} for a request sent to broker 35.197.50.244:6667 (id: 1001 rack: null) (state.change.logger)
~                                                                                 

Please help me. This is sitting on Google cloud and below configurations have been to be able to listen over public IP

advertised.host.name

host.name

advertised.port