Yeah, you can use edge node however this is subject to matter of Cloudera Support terms between you and Cloudera.  ... View more

I used to work at Cloudera/Hortonworks, and now I am a Hashmap Inc. consultant. This solution worked perfectly, thank you. ... View more

Thank you @TonyStank . This helps me. ... View more

Hey @kaf,   Thanks for reaching out to the Cloudera community. You can use "tail" command and then pipeline it to Kafka console producer if you want to read the whole file and then continue tailing for subsequently appended lines.   $ tail -f -n +1 <filename> | kafka-console-producer --broker-list <Broker_Host>:9092 --topic <topic_name>   Let me know if this helps. ... View more

Hey @rishav1412,   Thanks for reaching out to the Cloudera community.   I don't think we have a single way/process/configuration in Kafka to stream data from all Social Media Platforms. Every social media platform have their own APIs/methods and policy on data streaming.   If you want to stream data from Twitter you can use any of the mentioned ways/processes/services to send data from Twitter to Kafka topics.    Twitter >> Kafka Connect(Kafka Connect Twitter) >> Kafka Topics Twitter >> Flume(org.apache.flume.source.twitter.TwitterSource) >> Kafka Topics Twitter >> NiFi(GetTwitter Processor) >> Kafka Topics   Let me know if this helps. ... View more

Hey @Manoj690,   Thanks for reaching out to the Cloudera community.   You can execute a PUT request using the mentioned path "/connectors/<Connector_name>/config" to update the configuration for an existing connector. Also, pass a JSON object with the update parameter/s in the PUT request.   Example request:   PUT /connectors/<Connector_name>/config   Accept: application/json   { "flush.size": "100", "rotate.interval.ms": "1000" }   Let me know if this helps. ... View more

Hey @saihadoop,   Thanks for reaching out to the Cloudera community.   After setting up the Cluster infra and installing CDH & CM, you can use Cloudera Manager API[1] to Backing Up and Restoring the Cloudera Manager Configuration of an existing Cluster to a New Cluster.   [1]https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/cm_intro_api.html#concept_dnn_cr5_mr   Let me know if this helps.   Cheers, ... View more

Hey @AndyTech,   Thanks for reaching out to the Cloudera community.   The commit-id mentioned here isn't related to any Kafka usage related terms such as 'commit offsets' or other terms. This commit id refers to the Kafka source from which it was built.   It is not an error but just an info message. This doesn't impact Kafka client's functionality in any way.   Let me know if this helps.   Cheers, ... View more

Hey @AndyTech,   Thanks for reaching out to the Cloudera community.   This issue is due to the missing "kafka-python" module in your Python installation. You have to manually install the "kafka-python" module using the mentioned command in the edge node and all the hosts on which Spark job executes.   $ pip install kafka-python ... View more

Hey @sharathkumar13,   Thanks for reaching out to the Cloudera community.   Can you clarify on this "Do we have options to do ?"? Are you looking to use  Prometheus and Graffana to monitor Kafka Service? ... View more

@TonyStank appreciate your help. Stay safe.  ... View more

Hey @ping_pong,   Thanks for reaching out to the Cloudera community.   Do you have TLS enabled for this CDH Cluster? What are the steps followed to add new Host to this CDH Cluster?   > After installing all the required parcels/packages, have you started the Cloudera Manager Agent using the mentioned command?   $ sudo service cloudera-scm-agent start     ... View more

Hey,   If you have an existing subscription for HDP products, try logging in using your existing HDP credentials. If not, try registering in the Cloudera portal.   For learning and development purpose you can try using Hortonworks Sandbox. ... View more

Thanks all. I went to an other way downloaded on machine and extracted done the profile setting now it is work fine. ... View more

Hey,   CSD Version: 2.3 & higher, I think.   Regards, Ankit. ... View more

Hey,   Thank you for sharing the outcome and the steps. Much appreciated.   Regards. ... View more

Hey,   Refer mentioned Cloudera Documentation[1] on "Configuring the Flume Properties File".   [1]https://docs.cloudera.com/documentation/enterprise/5-14-x/topics/cdh_ig_flume_config.html   Please let me know if this helps.   Regards, Ankit. ... View more

I didnt use FQDN, instead i just added ip in /etc/hosts file. i used the same host ip in the kafka config ... View more

Hi Tony, Thanks for your reply, appericate all the hep provided by you and Gzigldrum Regards Wert ... View more

Hello,   Kafka Connect is included with Cloudera Distribution of Apache Kafka 2.0.0 but is not supported at this time.   Cloudera recommends using Flume and Sqoop as proven solutions for batch and real-time data loading that complement Kafka's message broker capability[2].   Kindly refer mentioned link[1] for more information.   [1]https://www.cloudera.com/documentation/kafka/latest/topics/kafka_known_issues.html#xd_583c10bfdbd326ba-590cb1d1-149e9ca9886--6fcb__section_ens_4bf_55 [2]https://blog.cloudera.com/blog/2014/11/flafka-apache-flume-meets-apache-kafka-for-event-processing/ ... View more

Hello,   1. Writing Streaming Aggregation to File   In order to use append mode with aggregations, you need to set an event time watermark (using "withWatermark"). Otherwise, Spark doesn't know when to output an aggregation result as "final".   A watermark is a threshold to specify how long the system waits for late events.   For example: df2 = df1.filter("code > 300").select("agent").withWatermark("timestamp", "2 minutes").groupBy("agent").count()   2. Reading from Kafka (Consumer) using Streaming   You have to set SPARK_KAFKA_VERSION environment variable.   When running jobs that require the new Kafka integration, set SPARK_KAFKA_VERSION=0.10 in the shell before launching spark-submit.   # Set the environment variable for the duration of your shell session: export SPARK_KAFKA_VERSION=0.10 spark-submit arguments   https://www.cloudera.com/documentation/spark2/latest/topics/spark2_kafka.html ... View more

Hello,   Monitoring consumer group lag using Cloudera Manager seems unlikely as I tried configuring a chart to display the consumer group lag but couldn't generate the desired results.   However, on further research, I came around a few GitHub projects that provide additional monitoring functionality. One among them is "Kafka Manager(Yahoo, Apache 2.0 License)", I think with this tool you can monitor  consumer group lag.     Please refer mentioned link[1] for more information on Kafka Manager.   [1]https://github.com/yahoo/kafka-manager ... View more