Community Articles

Find and share helpful community-sourced technical articles.
Announcements
Celebrating as our community reaches 100,000 members! Thank you!
Labels (1)
avatar
Rising Star

Background

Starting with Ambari-2.6.1 release, a new repository has been introduced called HDP-GPL. This article covers FAQs related to the purpose of this repository, when it takes effect and the impact it can have on your cluster esp. when dealing with LZO compression libraries.

FAQs

What is the need for a new GPL repo?

LZO compression libraries use a GPL license, which differs from rest of the packages in Ambari/HDP which are Apache licensed. Hence, the need for a separate repo which is named ‘HDP-<version_number>-GPL’. The Version Definition File (VDF) starting HDP-2.6.4.0 release has been updated to include the new ‘HDP-2.6-GPL’ repo. Refer the sample VDF file for CentOS7 here. Here is a snippet from Ambari web UI for the new 'HDP-2.6-GPL' field:

62823-screen-shot-2018-03-11-at-104256-pm.png

What are the exact contents of HDP-GPL repo?

As of HDP-2.6.4.0 release, the repo contains hadooplzo package files as follows:

  • hadooplzo_2_6_4_0_91-0.6.0.2.6.4.0-91.x86_64.rpm
  • hadooplzo_2_6_4_0_91-native-0.6.0.2.6.4.0-91.x86_64.rpm

How is Ambari configured to download and install LZO libraries?

For Fresh installs:

While running ‘ambari-server setup’, you would be prompted as follows:

Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)?

Depending on your response, a new entry would be created in ambari.properties file on Ambari server host:

‘gpl.license.accepted=true|false’

For Ambari Upgrades:

If you are upgrading to Ambari-2.6.1 (or higher) and have LZO enabled, you would be prompted to accept GPL license during the ‘ambari server upgrade’ step

[root@myhost yum.repos.d]# ambari-server upgrade
Using python  /usr/bin/python
Upgrading ambari-server
INFO: Upgrade Ambari Server
INFO: Updating Ambari Server properties in ambari.properties ...
INFO: Updating Ambari Server properties in ambari-env.sh ...
WARNING: Original file ambari-env.sh kept
INFO: Fixing database objects owner
Ambari Server configured for Embedded Postgres. Confirm you have made a backup of the Ambari Server database [y/n] (y)? y
INFO: Upgrading database schema
GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html
Your cluster is configured to use LZO which is GPL software. You must agree to enable Ambari to continue downloading and installing LZO  [y/n] (n)? y
INFO: Return code from schema upgrade command, retcode = 0
INFO: Console output from schema upgrade command:
INFO: {"lzo_enabled":"true"}

INFO: Schema upgrade completed
Adjusting ambari-server permissions and ownership...
Ambari Server 'upgrade' completed successfully.

For both fresh install and upgrades, enabling LZO GPL license is optional

For Express/Rolling Upgrades:

Before starting Express/Rolling Upgrade, a pre-check would be performed by Ambari to detect the existence of LZO configs in core-site. In case the GPL license is not accepted and LZO is enabled, you would see the following warning prior to start of Upgrade:

You have LZO codec enabled in the core-site config of your cluster. You have to accept GPL license during ambari-server setup to have LZO installed automatically. If any hosts require LZO, it should be installed before starting the upgrade. Consult Ambari documentation for instructions on how to do this.

How can I enable LZO GPL license afterwards?

ambari-server setup  --enable-lzo-under-gpl-license

and then restart ambari server service

For details on LZO, refer official Ambari documentation here

How can I know if GPL license is accepted by Ambari?

On Ambari server host, execute below command

grep "gpl.license.accepted" /etc/ambari-server/conf/ambari.properties

An output of 'gpl.license.accepted=true' implies the license was accepted

I do not see the HDP-GPL repo in Manage versions page on Ambari web UI. Is that expected?

Yes, this would be due to non-usage of LZO compression libraries on your cluster. To confirm, verify if GPL license was denied by Ambari (gpl.license.accepted=false in ambari.properties file)

I use a local repository. Does GPL need additional configuration in such cases?

The configuration steps for HDP-GPL local repo are identical to that of HDP and HDP-UTILS repo.

For instance, you would need to first download the hdp.gpl.repo or the gpl repository tarball to the selected mirror server. Thereafter, synchronize the repository contents to your mirror server followed by generating the repository metadata:

reposync -r HDP-GPL-2.6.4.0 
createrepo <web.server.directory>/hdp/<OS>/HDP-GPL-2.6.4.0

For further details on using local repository, refer official Ambari documentation here

What happens when you try to configure core-site with LZO configs and GPL license is not accepted?

Ambari’s stackadvisor has a check to validate such configs. In the current scenario, you would get the following error message:

62821-gpl-sa.png

How about some Troubleshooting?

==== Troubleshooting ====

Scenario: Post HDP upgrade, you observe the following error during Pig/MapReduce service check:

ERROR 2118: Compression codec com.hadoop.compression.lzo.LzoCodec not found.

Cause: Your have LZO configured, but not accepted the GPL license during Ambari upgrade

Resolution: Enable GPL license using "ambari-server setup --enable-lzo-under-gpl-license" command

==== Bug Fixes ====

AMBARI-22906 - Fixed in Ambari-2.6.1.5 release

AMBARI-23055 - Fixed in the upcoming Ambari-2.6.2.0 release

7,528 Views
Version history
Last update:
‎08-17-2019 08:23 AM
Updated by:
Contributors