HDP 2.5 documentation updates for Data Movement

[#1]

There’s a new Data Movement and Integration (DMI) guide for the HDP 2.5.0 release. The guide currently focuses on Falcon features and functionality around the definition and scheduling of jobs for the backup, mirror, transfer, and processing of data. It also includes some content related to Flume, Sqoop, and Oozie.

The existing Falcon content was moved out of the Data Governance Guide and into the DMI guide. New content was also added to the DMI guide to reflect changes to the Falcon UI and new Falcon features. In addition, the content has gone through an initial reorganization in an effort to more closely reflect the user’s experience.

Significant changes to Falcon content in the guide include:

• Rewrote Tasks based on an improved, redesigned UI.
• Added Content about new functionality, including:
  • RDBMS backup capability
  • Snapshot-based backups
  • Hive data mirroring
  • Mirroring data between on-premise and cloud data with Azure and AWS S3
  • Ability to edit cluster entities using safe mode
  • New entity dependencies view
  • Replacement of client-side recipes with more flexible server-side extensions

There is also a Falcon View available in Ambari. Instructions for installing the View are provided in the Apache Ambari Views guide.

Apache Sqoop and Oozie content was moved out of the Data Access guide and into the DMI guide, with some minor updates to that content.

Flume 1.5.2 content is still available in the Apache Flume Component Guide. A new introductory document has been added that includes a list of feature updates to Flume 1.5.2 for each HDP release. Most of the functionality in Flume 1.6 is available in the 1.5.2 version used with HDP. For HDP 2.5, Kafka Channel and TailDir Source were added.

We will continue to add new HDP 2.5.0 content to the guide over the coming weeks, including:

• Configuring Falcon for HiveDR
• Information about Data Source entities

We’ll let you know when updates to the document are available.

If you have comments or suggestions, corrections or updates regarding our documentation, let us know on HCC. Help us continue to improve our documentation!

Thanks!

Hortonworks Technical Documentation Team