These are spam accounts, by the way. Look at all the "answers" from the other users for every question, and they all link back to dataflair's website. ... View more

I think you are asking about adding directories to Datanodes. dfs.datanode.data.dir in the hdfs-site.xml file is a comma-delimited list of directories for where the DataNode will store blocks for HDFS. Plus, https://community.hortonworks.com/questions/89786/file-uri-required-for-dfsdatanodedatadir.html Property Default Description dfs.datanode.data.dir file://${hadoop.tmp.dir}/dfs/data Determines where on the local filesystem an DFS data node should store its blocks. If this is a comma-delimited list of directories, then data will be stored in all named directories, typically on different devices. Directories that do not exist are ignored Otherwise, I'm afraid your question doesn't make sense other than running mkdir HDFS command to "add a new directory in HDFS" ... View more

@Manish Tiwari, perhaps you can look at https://docs.hortonworks.com/HDPDocuments/Cloudbreak/Cloudbreak-2.7.1/content/data-lake/index.html Otherwise, you can search https://docs.hortonworks.com/ for the keywords you are looking for ... View more

You can enable JMX for metrics + Grafana for visualization, then Ambari Infra for log collection However, you will not have visibility into Consumer Lag like Confluent Control Center offers, and you will need to find some external tools to do that for you such as LinkedIn Burrow. If you are not satisfied with that, Confluent Control Center can be added to an HDP cluster with manual setup. https://docs.confluent.io/current/control-center/docs/installation/install-apache-kafka.html You will need to copy the Confluent Metrics Reporter JARs from the Confluent Enterprise download over onto your HDP Kafka nodes under /usr/hdp/current/kafka ... View more

@Shobhna Dhami After "available connectors" it does not list it, so you have not setup the classpath correctly, as I linked to. In Kafka 0.10, you need to run $ export CLASSPATH=/path/to/extracted-debezium-folder/*.jar # Replace with the real address $ connect-distributed ... # Start Connect Server You can also perform a request to the /connector-plugins URL address before sending any configuration to verify the Debezium connector was correctly installed. ... View more

@Vamshi Reddy Yes, "Confluent" is not some custom version of Apache Kafka In fact, this process is very repeatable for all other Kafka Connect plugins. Download the code Build it against the Kafka version you run Move the package to the Connect server Extract the JAR files onto the Connect server CLASSPATH Run/Restart Connect ... View more

From a non-Hadoop machine, install Java+Maven+Git git clone https://github.com/confluentinc/kafka-connect-hdfs cd kafka-connect-hdfs git fetch --all --tags --prune git checkout tags/v4.1.2 # This is a Confluent Release number, which corresponds to a Kafka release number mvn clean install -DskipTests This should generate some files under the target folder in that directory. So, using the 4.1.2 example, I would ZIP up the "target/kafka-connect-hdfs-4.1.2-package/share/java/" folder that was built, then copy this file and extract it into all HDP servers that I want to run Kafka Connect on. For example, /opt/kafka-connect-hdfs/share/java From there, you would find your "connect-distributed.properties" file and add a line for plugin.path=/opt/kafka-connect-hdfs/share/java Now, run something like this (I don't know the full location of the property files) connect-distributed /usr/hdp/current/kafka/.../connect-distributed.properties Once that starts, you can attempt to hit http://connect-server:8083/connector-plugins , and you should see an item for "io.confluent.connect.hdfs.HdfsSinkConnector" If successful, continue to read the HDFS Connector documentation, then POST the JSON configuration body to the Connect Server endpoint. (or use Landoop's Connect UI tool) ... View more