Please try Nifi - Kakfa   https://community.cloudera.com/t5/Community-Articles/Apache-NiFi-1-10-Support-for-Parquet-RecordReader/ta-p/282390#:~:text=With%20the%20release%20of%20Apache,data%20as%20a%20single%20unit ... View more

Hello @BGabor    This error "TrustManager is not specified" can be thrown due to some of the following issues -    Missing cert files or missing/wrong values for the below configs  Make sure following properties are set:- xasecure.policymgr.clientssl.keystore.credential.file=jceks://file/{{credential_file}} xasecure.policymgr.clientssl.truststore.credential.file=jceks://file/{{credential_file}} xasecure.policymgr.clientssl.truststore=/path/to/truststore   Also came across these Ranger jiras which indicates that the truststore info not specified in cacert or cacert needs to be manually configured. Note - they are fixed in Ranger 2.0.1, so you may also want to check the ranger version. https://issues.apache.org/jira/browse/RANGER-2611 https://issues.apache.org/jira/browse/RANGER-2907 ... View more

Please try Kafka connect then, that seems to be the best option suited.  ... View more

Hello @sriven    Found this - https://community.cloudera.com/t5/Support-Questions/How-to-insert-parquet-file-to-Kafka-and-pass-them-to-HDFS/td-p/178340   Please let me know if it helps.   Thanks & Regards, Nandini ... View more

Hello,   What is the file format? Why is it that you say  HDFS source connector is also not the solution if files written by spark programming.? Spark - HDFS - Kafka is your entire flow correct? Spark to HDFS they have done now you are looking for HDFS - Kafka. If you can help me understand the file format that Spark saves it while I can find if HDFS Source connector should not be able to help your usecase.     ... View more

Hello @sriven ,   - As @Daming Xue mentioned Kafka Connect is one of the good options, the doc https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/kafka-connect/kafka-connect.pdf shares an example of HDFS as a sink connector. https://docs.cloudera.com/cdp-private-cloud-base/7.1.5/kafka-connect/topics/kafka-connect-connector-hdfs-example.html   - Flume (CDH)   https://docs.cloudera.com/documentation/kafka/latest/topics/kafka_flume.html#concept_rsb_tyb_kv__section_iwb_tyb_kv   - Nifi (https://blog.cloudera.com/adding-nifi-and-kafka-to-cloudera-data-platform/ https://community.cloudera.com/t5/Community-Articles/Integrating-Apache-NiFi-and-Apache-Kafka/ta-p/247433 )   - Kafka- Hive Integeration (https://docs.cloudera.com/cdp-private-cloud-base/7.1.5/integrating-hive-and-bi/topics/hive-kafka-integration.html)   - Custom Java app (https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/kafka-developing-applications/topics/kafka-develop-example-producer.html)   - To try out quickly (testing purpose), you can use the console producer  hadoop fs -cat file.txt |  kafka-console-producer --broker-list <host:port>   --topic <topic> https://docs.cloudera.com/cdp-private-cloud-base/7.1.5/kafka-managing/topics/kafka-manage-cli-producer.html   - Spark (which you do not want) https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/developing-spark-applications/topics/spark-using-spark-streaming.html   These are some I could quickly think of, there must be many more options.   Thanks & Regards, Nandini   P.S. If you found this answer useful please upvote/accept.     ... View more

Hello @Christ ,   Yes, it is possible.  You can write a custom script or java code(atlas-client) that would extract the metadata from your source file and use the below REST APIs to insert it into Atlas. http://atlas.incubator.apache.org/api/v2/index.html   For eg here, you can see a spark atlas connector -  https://github.com/hortonworks-spark/spark-atlas-connector/blob/master/README.md#to-use-it-in-secure-environment   Hope these documents help -  https://docs.cloudera.com/runtime/7.2.7/atlas-leveraging-business-metadata/topics/atlas-business-metadata-bulk-import.html   https://docs.cloudera.com/cdp-private-cloud-base/7.1.5/atlas-import-utility/topics/atlas-importing-hive-metadata-utility.html   Thanks, Nandini   P.S. As always, if you find this post helpful, don't forget to upvote/accept the answer. ... View more

As you mentioned  certain cli tools   are marked not to be supported, for your requirement being the kafka-server-startup and shutdowns scripts. It is not guaranteed that the manual setup initializes the context of Kafka the same way, as Cloudera Manager would. While it may work for the restart but there are environment scripts and some other process management that Cloudera Manager does. Hence, when you restart using Cloudera Manager there would be uncertainties on the state of the process.    As you said you have problems accessing the Cloudera Manager UI, you can also use the Cloudera Manager API for your requirement.    https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/cm_intro_api.html    ... View more

Hello @bhara,   Do you have Ranger enabled? What is the Data center version?  (I see a bug OPSAPS-58584 on 7.1.5 which was fixed in 7.2.4 , 7.3.0 )     ... View more

We have two filter options external.kafka.metrics.exclude.prefix and external.kafka.metrics.include.prefix. https://github.com/hortonworks/ambari/blob/AMBARI-2.7.0-maint/ambari-metrics/ambari-metrics-kafka-sink/src/main/java/org/apache/hadoop/metrics2/sink/kafka/KafkaTimelineMetricsReporter.java#L417   ... View more