@Andy LoPresto I got below ouput after executing GET request to http://<hostname>:9090/nifi-api/provenance/query-id : {"provenanceEvents":[{"id":"a9d352d4-20e9-4fbb-851c-3191204d58ba4","eventId":4,"eventTime":"04/18/2017 09:30:14.775 UTC","lineageDuration":1,"eventType":"CREATE","flowFileUuid":"c87d810d-17b7-4c5f-b313-dd4653e81a8d","fileSize":"0 bytes","fileSizeBytes":0,"clusterNodeId":"a9d352d4-20e9-4fbb-851c-3191204d58ba","clusterNodeAddress":"amnifi1.openstacklocal:9090","groupId":"b90e8d65-015a-1000-22ce-e993c787469e","componentId":"61665eb8-015b-1000-0000-0000653a830f","componentType":"ListFTP","componentName":"AAAAAAAAAAAA","attributes":[{"name":"file.group","value":"0"},{"name":"file.lastModifiedTime","value":"2017-04-18T09:30:00+0000"},{"name":"file.owner","value":"0"},{"name":"file.permissions","value":"rw-r--r--"},{"name":"filename","value":"jj1.txt"},{"name":"ftp.listing.user","value":"akshay"},{"name":"ftp.remote.host","value":"amnifi3.openstacklocal"},{"name":"ftp.remote.port","value":"21"},{"name":"path","value":"ALL_DAY_WORK"},{"name":"uuid","value":"c87d810d-17b7-4c5f-b313-dd4653e81a8d"}],"parentUuids":[],"childUuids":[],"contentEqual":false,"inputContentAvailable":false,"outputContentAvailable":false,"outputContentClaimFileSize":"0 bytes","outputContentClaimFileSizeBytes":0,"replayAvailable":false,"replayExplanation":"Cannot replay data from Provenance Event because the event does not contain the required Content Claim”}, Now when I am trying to submit GET request to get more info about provenance event I am getting below error message : $ curl -i -X GET http://<hostname>:9090/nifi-api/provenance-events/{4} HTTP/1.1 400 Bad Request Date: Wed, 19 Apr 2017 07:21:13 GMT Content-Type: text/plain Transfer-Encoding: chunked Server: Jetty(9.3.9.v20160517) The cluster node identifier must be specified. Nifi is clustered. Not sure where I should mentioned cluster node id. Please help. ... View more

@Maeve Ryan What is the error you are getting ? How are you checking the size of hdfs ? ... View more

@Sumit Sharma - Are you using custom producer or kafka-console-producer script ? Post the exact command used for producing messages to topic. - Are you facing this issue for a specific topic or for all topics ? - As which user you are running producer ? - Please post the output of klist command as a user which is executing producer code/script. - Output of describe command. - What is the version of jdk you are running ? ... View more

@Param NC From command I can see that you are using --new-consumer to describe consumer group. When new consumer is used it tries to fetch consumer group info from consumer offset topics which gets created in kafka log directory. Try using --zookeeper instead of --new-consumer, for eg : $ /usr/bin/kafka-consumer-groups.sh --zookeeper <zookeeper-hostname>:2181 --describe --group <consumer-group> ... View more

@yjiang Creating topics with kafka user is a good practice. But If you want to create a topic as a non kafka user in a kerberized environment you need to workaround by following below steps : If you are not using Ranger : 1. Make sure "auto.create.topic.enable = true" 2. Give acl's for the user from which you want to create a topic, for ex : # bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=localhost:2181 --add --allow-principal User:Bob --producer --topic Test-topic 3. Do a kinit as a user from which you want to create topic. 4. Now try to produce messages to topic as that user : # ./kafka-console-producer.sh --broker-list <hostname-broker>:6667 --topic Test-topic --security-protocol PLAINTEXTSASL If you are using Ranger : Instead of point 2 in above steps you will need to add a policy for the topic in ranger. Allow permissions for that user to produce, create, consume. Restart kafka service. Then follow step 3 and 4 as mentioned above. ... View more

@yjiang If you try to create a topic as a non kafka user, it creates a topic but with no Leader and ISR. This is a known issue. According to me, the reason behind this could be the zookeeper acl's. Once topic is created in zookeeper, its acl's will not allow kafka to read details about it. If you want to create a topic as a non kafka user you need to workaround by following below steps : If you are not using Ranger : 1. Make sure "auto.create.topic.enable = true" 2. Give acl's for the user from which you want to create a topic, for ex : # bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=localhost:2181 --add --allow-principal User:Bob --producer --topic Test-topic 3. Do a kinit as a user from which you want to create topic. 4. Now try to produce messages to topic as that user : # ./kafka-console-producer.sh --broker-list <hostname-broker>:6667 --topic Test-topic --security-protocol PLAINTEXTSASL If you are using Ranger : Instead of point 2 in above steps you will need to add a policy for the topic in ranger. Allow permissions for that user to produce, create, consumer. Restart kafka service. Then follow step 3 and 4 as mentioned above. Hope this helps !! ... View more

@Jeeva Jeeva Could you please check in zookeeper if they are registered : Execute below command to login to zkcli : /usr/hdp/current/zookeeper-server/bin/zkCli.sh ls /brokers/ids/<id's> <-- here you should see two ids as there are two broker nodes get /brokers/ids/<id1> and <id2> <--- Please share the output Do you see any error is server.log which is located at /var/log/kafka/ ... View more

@Jeeva Jeeva Could you please confirm if all brokers/broker in your cluster is up and running ? If not please start your broker node and try to execute create topic command. ... View more