Hello, What are the best practices to install Kafka/NiFi ? 1/ Should Kafka brokers be located within the same data nodes, or should they be on separate nodes? Which way is better in term of performance ? Is it possible to have Kafka on a datanode when Kafka is installed using HDF? 2/ Can Kafka and NiFi shares the same zookeeper or should Kafka have its own ZK used exclusively by Kafka? 3/ Does the installation of NiFi by HDF(ambari) apply the needed system requirements such as max files handles, max forked processes ...or should theses requirement be done before proceeding to the installation by ambari? 4/ Is it possible to have a node that belong to both a HDF and HDP clusters at the same time with same ambari agent running on the node ? Thanks in advance ! ... View more

Hello, Is it possible to create an internal ORC hive table from an avro hdfs file? I tried something like this: CREATE TABLE orc_table ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' STORED as orc LOCATION '/user/someuser/avro_folder/' TBLPROPERTIES ('avro.schema.url'='hdfs:///user/someuser/schema.avsc'); but I get this error: Failed with exception java.io.IOException:java.lang.RuntimeException: serious problem Beside a describe formatted command returns: # Storage Information SerDe Library: org.apache.hadoop.hive.serde2.avro.AvroSerDe InputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat OutputFormat: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat Thanks in advance. ... View more

Hello, Is there a way to add an option clause like 'OPTION(hash join)' while performing an incremental import using QueryDatabaseTable Nifi Processor ? The goal of this is to optimize the query that will be executed in sql server by Nifi. Thanks in advance. ... View more