This is a very broad topic, and might make sense to use a vendor supported tool like EMR or Qubole. Cloudbreak or Hortonworks itself doesn't offer very well-defined backup tools. For example, Hadoop DistCP and mysqldump/pgdump, Hive/HBase Export only get you so far. ... View more

Hive Streaming tables need to be ORC, right? Do the Avro records automatically get converted? ... View more

@Shobhna Dhami After "available connectors" it does not list it, so you have not setup the classpath correctly, as I linked to. In Kafka 0.10, you need to run $ export CLASSPATH=/path/to/extracted-debezium-folder/*.jar # Replace with the real address $ connect-distributed ... # Start Connect Server You can also perform a request to the /connector-plugins URL address before sending any configuration to verify the Debezium connector was correctly installed. ... View more

@Vamshi Reddy Yes, "Confluent" is not some custom version of Apache Kafka In fact, this process is very repeatable for all other Kafka Connect plugins. Download the code Build it against the Kafka version you run Move the package to the Connect server Extract the JAR files onto the Connect server CLASSPATH Run/Restart Connect ... View more

From a non-Hadoop machine, install Java+Maven+Git git clone https://github.com/confluentinc/kafka-connect-hdfs cd kafka-connect-hdfs git fetch --all --tags --prune git checkout tags/v4.1.2 # This is a Confluent Release number, which corresponds to a Kafka release number mvn clean install -DskipTests This should generate some files under the target folder in that directory. So, using the 4.1.2 example, I would ZIP up the "target/kafka-connect-hdfs-4.1.2-package/share/java/" folder that was built, then copy this file and extract it into all HDP servers that I want to run Kafka Connect on. For example, /opt/kafka-connect-hdfs/share/java From there, you would find your "connect-distributed.properties" file and add a line for plugin.path=/opt/kafka-connect-hdfs/share/java Now, run something like this (I don't know the full location of the property files) connect-distributed /usr/hdp/current/kafka/.../connect-distributed.properties Once that starts, you can attempt to hit http://connect-server:8083/connector-plugins , and you should see an item for "io.confluent.connect.hdfs.HdfsSinkConnector" If successful, continue to read the HDFS Connector documentation, then POST the JSON configuration body to the Connect Server endpoint. (or use Landoop's Connect UI tool) ... View more

If the end goal is to transfer Kafka to Postgres, you have access to NiFi. Otherwise, exposing the internal Postgres server that is used for Hive, Ambari, Oozie, and other services is probably not a good idea. It would be recommended to run a standalone Postgres server to minimize the blast-radius of failure and maintain service uptime. ... View more

@Michael Bronson - Well, the obvious; Kafka Leader election would fail if only one Zookeeper stops responding. Your consumers and producers wouldn't be able to determine which topic partition should serve any requests. Hardware fails for a variety of reasons, and it would be better if you converted two of the 160 available worker nodes to be dedicated Zookeeper servers. ... View more

Load balancers would help in the case where you want a more friendly name than some DNS records or the case where IP's are dynamic. Besides that, remembering one address is easier than a long list of 3-5 servers. ... View more

@Michael Bronson - The terms "master/worker" don't really mean anything in Kafka terms. 17 Kafka brokers seems like a lot (we have about that many brokers in AWS handling about 2million messages per day), but yes, a minimum of 5 ZKs is encouraged to account for maintenance and hardware failure, as mentioned. ... View more