The Spark Component Guide and Command Line Installation Guide were updated to reflect new Spark features. Here are links to several of the latest features: Support for Spark 2, documented in several topics including: Installing Spark Using Ambari Installing and Configuring Apache Spark 2 (manual installation) Running Spark Configuring Spark2 for Wire Encryption Automating Spark Jobs with Oozie Spark Action Using Livy with Spark Versions 1 and 2 Livy API information, in Submitting Spark Applications Through Livy Enabling Spark SQL user impersonation for the Spark Thrift Server (doAs support), in Configuring the Spark Thrift Server The Zeppelin Component Guide was updated with additional details and examples for configuring Zeppelin with LDAP/AD and Kerberos security; see Configuring Zeppelin Security. In addition, the documentation for interpreters and user impersonation was extended. Portions of this information that apply to HDP 2.5 were also added to the Security chapter in the HDP 2.5 Zeppelin Component Guide. In the messaging area, the Kafka Component Guide has additional information in Configuring Kafka for a Production Environment. ... View more

There are many ways to run Hadoop on virtual machines. Earlier this year I tried several approaches, and ended up using a helpful Quick Start Guide written by Yusaku Sako. The Quick Start uses VirtualBox, Vagrant, and predefined scripts to set up a multi-node HDP cluster. You can choose which version of Ambari to install, and then choose and install an associated version of the HDP stack. For anyone new to virtual machines, there is now a Quick Start for New VM Users. The extended version adds background information and additional details for installing Ambari and the HDP stack. Topics include: Terminology Prerequisites Installing VirtualBox and Vagrant Starting Linux Virtual Machines Accessing Virtual Machines Installing Ambari Installing the HDP Stack Troubleshooting Reference information for basic Vagrant commands ... View more

Our stream analytics documentation received substantial updates for HDP 2.5.0. The Apache Storm Component Guide was reorganized to focus on administration and development workflow: installing and configuring Storm, developing Storm applications, moving data into and out of Storm, and managing topologies. New content includes the following topics: Installing Storm using Ambari. Configuring Storm for a production environment. Implementing windowing computations on data streams. Implementing state management in core Storm. Configuring and using the HDFS spout to ingest data from HDFS. Monitoring and debugging Apache Storm topologies through the use of dynamic log levels, topology event logging, distributed log search, and dynamic worker profiling. The Storm application development chapter differentiates more clearly between core Storm and Trident, and the Kafka spout subsection describes performance settings and tradeoffs. The Storm Ambari view is documented in the Apache Ambari Views Guide, and the Apache Ambari Apache Storm Kerberos Configuration Guide (formerly Configuring Storm for Kerberos Over Ambari) has moved to the Security Guide. The Apache Kafka Component Guide was enhanced with the following content: Installing Kafka using Ambari. Configuring Kafka for a production environment. Sample code for a basic Kafka producer and consumer, with and without SSL enabled on the cluster. The Apache Ambari Apache Kafka Kerberos Configuration Guide (formerly Configuring Kafka for Kerberos Over Ambari) has moved to the Security Guide. If you have comments or suggestions, corrections or updates regarding our documentation, let us know on HCC. Help us continue to improve our documentation! ... View more

The new Apache Zeppelin Component Guide has instructions for installing, configuring, and using the web-based Zeppelin notebook for interactive data exploration and visualization. The Apache Spark Component Guide was reorganized to focus on administration and development workflows: installing and configuring Spark, developing Spark applications, using SparkR, and tuning Spark. For HDP 2.5, the book includes the following new topics: Installing the Spark 2.0 technical preview alongside Spark 1.6.2, using Ambari. Configuring Spark for wire encryption. The Apache Solr Search Installation Guide now contains instructions for installing HDP Search using Ambari. ... View more