Hi @Lucky_Luke, the script "kafka-topics.sh" with parameter "--describe" is what you are looking for. To get details for a certain topic, e.g. "test-topic", you would call (adjust zookeeper connect string according to your env.): /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper sandbox.hortonworks.com:2181/kafka --describe --topic test-topic The output contains (amongst others) no. of partitions, leading broker for each partition, in-snyc-replicas The topic-level configuration properties are listed under "Configs:" . If this is blank, then the default (broker-wide) settings are applied and you should check your broker config file (or Ambari section) for property "log.retention.hours" ...... assuming you mean the retention time by mentioning "TTL" HTH, Gerd ... View more

Hello @Alexandru Anghel , many thanks....works brilliant ! ... View more

Hi @Robin Dong , if you try the standalone-mode, there is no configuration via REST at all, hence you do NOT need any curl command to provide the connector config to your worker. In standalone-mode you pass the connector config as a second commandline parameter to start your worker, see here for an example how to start the standalone stuff including the connector config. Maybe it is worth providing both configurations, the standalone worker as well the distributed one. If you start the distributed worker, at the end of the commandline output there you will find the URL to the REST-Interface. Can you paste that terminal output as well ? Do you execute the curl command from the same node where you started Connect Worker, or is it from a remote host and maybe the AWS Network/Security settings prevent you from talking to the REST Interface ? Regards, Gerd ... View more

Hi @Robin Dong , port 8083 is the default port for the KafkaConnect Worker, if started in distributed mode....which is the case in the URL you are referring to. You can set this port to another one in the properties file you provide as parameter to the connect-distributed.sh cmdline call (the property is called rest.port , see here ). In distributed mode you have to use the REST API to configure your Connectors, that's the only option. You can of course also start investigating into using Connect by starting with standalone mode. Then you do not need a REST call to configure your connector, you can just provide the connector.properties file as additional parameter at starting time of the ConnectWorker to the connect-standalone.sh script (ref. here) Please try to replace 'localhost' by the FQDN of the host , where the Connect worker was started, and of course check if this start was successfull by looking at the listening ports e.g. netstat -tulpn | grep 8083 HTH, Gerd --------- If you find this post useful, any vote/reward highly appreciated 😉 ... View more

Hi @mel mendoza , maybe it is worth checking Flume to ingest multiple files to Kafka. Alternatively you can use HDF (particularly NiFi) to do so. ... View more

Hello @mel mendoza , Kafka is basically not a file based systems, but event based. If you want to process files with Spark-Streaming via Kafka you have a 2-step approach. First is ingest to Kafka, then consume the events from Kafka by Spark-Streaming. To ingest into Kafka you can e.g. use Kafka-Connect with the file source (check /usr/hdp/current/kafka-broker/conf/connect-file-source.properties). It works like a "tail -f " on that file and streams any incoming data from that file to the Kafka topic. Afterwards you have to consume the events from that Kafka topic with your Spark-Streaming job. HTH, Gerd ... View more

Hi @Sebastian Carroll , why not copying over content of previous log.dirs to new ones, before restarting kafka ? this should reduce/remove the takeover of partitions ... View more