Using Yahoo Kafka Manager Git clone the project (you need Java 8 to build). Then use SBT to do a clean distribution. This will take a while as it downloads a lot of jars. <code>kafka-manager.zkhosts="sandbox.hortonworks.com:2181" The build will produce a Zip file, unzip it, update configuration file (conf/application.conf) and then you can run it. ../kafka-manager/target/universal/kafka-manager-1.3.0.8.zip unzip ../kafka-manager/target/universal/kafka-manager-1.3.0.8.zip kafka-manager-1.3.0.8 git:(master) ✗ vi conf/application.conf kafka-manager-1.3.0.8 git:(master) ✗ bin/kafka-manager -Dconfig.file=conf/application.conf Access the Kafka Manager from Chrome http://localhost:9000/ Running Kafka Manager Resources https://github.com/yahoo/kafka-manager http://edbaker.weebly.com/blog/install-and-evaluation-of-yahoos-kafka-manager http://chennaihug.org/knowledgebase/yahoo-kafka-manager/ https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem Tools For Testing Kafka with a command-line client producer/consumer: https://github.com/edenhill/kafkacat (brew install kafkacat) For External Access You may need to set advertised.host.name http://stackoverflow.com/questions/31476679/send-kafkaproducer-from-local-machine-to-hortonworks-sandbox-on-virtualbox ... View more

From the Sandbox as Root /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --create --zookeeper sandbox.hortonworks.com:2181 --replication-factor 1 --partitions 1 --topic people Test The Topic [root@sandbox kafka]# /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --topic people --zookeeper sandbox.hortonworks.com:2181 {metadata.broker.list=sandbox.hortonworks.com:6667, request.timeout.ms=30000, client.id=console-consumer-10628, security.protocol=PLAINTEXT} Resources: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_secure-kafka-ambari/content/ch_secure-kafka-create-topics.html https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_secure-kafka-ambari/content/ch_secure-kafka-produce-events.html ... View more

In HDP 2.5 Sandbox, I did a quick walk through. First thing I liked was the Visualization: The Data Explorer provided a nice query tool to view tables and graphs. The Data Visualization tab provides some nice graphing capabilities. After you run your queries you can look at the Tez results to see how it ran, it's a nice way to see what you may need to optimize. The Hive Ambari View is getting to be a very solid tool for working with Hive. From DDL (creating tables is easy) to viewing data, to updates and inserts. ... View more

This tutorial is great: https://github.com/hortonworks-gallery/ambari-vnc-service Eclipse Plugin https://github.com/winghc/hadoop2x-eclipse-plugin JDK 7 is best for most use case and Scala 2.10. Maven and SBT are necessary as well. Setup your Environment https://dzone.com/articles/spark-and-scala-resources https://dzone.com/articles/whats-on-your-laptop Lots of options: This is an eclipse project for Hbase Coprocessor https://github.com/tspannhw/hbasecoprocessor Artem has a great project for testing https://github.com/dbist/HBaseUnitTest Once all the ports are open and not firewalled it’s usually straight forward. Eclipse to Spark https://community.hortonworks.com/questions/36354/eclipse-to-sandbox-1.html https://community.hortonworks.com/questions/32567/scala-with-hive-in-ecplipse-scala.html Hadoop Eclipse Plugin https://community.hortonworks.com/questions/10404/hadoop-eclipse-plugin.html IntelliJ Project for Spark https://github.com/agilemobiledev/sparkworkshop https://community.hortonworks.com/questions/31077/how-to-setup-intellij-idea-16-to-run-hortonworks-s.html IntelliJ Settings https://community.hortonworks.com/questions/37410/recommended-idea-intellij-vmoptions-setting-for-de.html These configuration files must be in project or class path: core-site.xml hdfs-site.xml yarn-site.xml Add Jars for Access http://nivemaham.com/index.php/technical/22-java/hadoop/40-how-to-use-ide-for-hadoop-development-with-hortonworks-sandbox For Apache Kylin development http://kylin.apache.org/development/dev_env.html Remote Debugging Spark https://nicolasmaillard.com/2016/02/06/remote-debugging-201-spark/ Testing with Hadoop MiniClusters https://github.com/sakserv/hadoop-mini-clusters ... View more