Background
Starting with Ambari-2.6.1 release, a new repository has been introduced called HDP-GPL. This article covers FAQs related to the purpose of this repository, when it takes effect and the impact it can have on your cluster esp. when dealing with LZO compression libraries.
FAQs
What is the need for a new GPL repo?
LZO compression libraries use a GPL license, which differs from rest of the packages in Ambari/HDP which are Apache licensed. Hence, the need for a separate repo which is named ‘HDP-<version_number>-GPL’. The Version Definition File (VDF) starting HDP-2.6.4.0 release has been updated to include the new ‘HDP-2.6-GPL’ repo. Refer the sample VDF file for CentOS7 here. Here is a snippet from Ambari web UI for the new 'HDP-2.6-GPL' field:
What are the exact contents of HDP-GPL repo?
As of HDP-2.6.4.0 release, the repo contains hadooplzo package files as follows:
- hadooplzo_2_6_4_0_91-0.6.0.2.6.4.0-91.x86_64.rpm
- hadooplzo_2_6_4_0_91-native-0.6.0.2.6.4.0-91.x86_64.rpm
How is Ambari configured to download and install LZO libraries?
For Fresh installs:
While running ‘ambari-server setup’, you would be prompted as follows:
Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)?
Depending on your response, a new entry would be created in ambari.properties file on Ambari server host:
‘gpl.license.accepted=true|false’
For Ambari Upgrades:
If you are upgrading to Ambari-2.6.1 (or higher) and have LZO enabled, you would be prompted to accept GPL license during the ‘ambari server upgrade’ step
[root@myhost yum.repos.d]# ambari-server upgrade Using python /usr/bin/python Upgrading ambari-server INFO: Upgrade Ambari Server INFO: Updating Ambari Server properties in ambari.properties ... INFO: Updating Ambari Server properties in ambari-env.sh ... WARNING: Original file ambari-env.sh kept INFO: Fixing database objects owner Ambari Server configured for Embedded Postgres. Confirm you have made a backup of the Ambari Server database [y/n] (y)? y INFO: Upgrading database schema GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html Your cluster is configured to use LZO which is GPL software. You must agree to enable Ambari to continue downloading and installing LZO [y/n] (n)? y INFO: Return code from schema upgrade command, retcode = 0 INFO: Console output from schema upgrade command: INFO: {"lzo_enabled":"true"} INFO: Schema upgrade completed Adjusting ambari-server permissions and ownership... Ambari Server 'upgrade' completed successfully.
For both fresh install and upgrades, enabling LZO GPL license is optional
For Express/Rolling Upgrades:
Before starting Express/Rolling Upgrade, a pre-check would be performed by Ambari to detect the existence of LZO configs in core-site. In case the GPL license is not accepted and LZO is enabled, you would see the following warning prior to start of Upgrade:
You have LZO codec enabled in the core-site config of your cluster. You have to accept GPL license during ambari-server setup to have LZO installed automatically. If any hosts require LZO, it should be installed before starting the upgrade. Consult Ambari documentation for instructions on how to do this.
How can I enable LZO GPL license afterwards?
ambari-server setup --enable-lzo-under-gpl-license
and then restart ambari server service
For details on LZO, refer official Ambari documentation here
How can I know if GPL license is accepted by Ambari?
On Ambari server host, execute below command
grep "gpl.license.accepted" /etc/ambari-server/conf/ambari.properties
An output of 'gpl.license.accepted=true' implies the license was accepted
I do not see the HDP-GPL repo in Manage versions page on Ambari web UI. Is that expected?
Yes, this would be due to non-usage of LZO compression libraries on your cluster. To confirm, verify if GPL license was denied by Ambari (gpl.license.accepted=false in ambari.properties file)
I use a local repository. Does GPL need additional configuration in such cases?
The configuration steps for HDP-GPL local repo are identical to that of HDP and HDP-UTILS repo.
For instance, you would need to first download the hdp.gpl.repo or the gpl repository tarball to the selected mirror server. Thereafter, synchronize the repository contents to your mirror server followed by generating the repository metadata:
reposync -r HDP-GPL-2.6.4.0 createrepo <web.server.directory>/hdp/<OS>/HDP-GPL-2.6.4.0
For further details on using local repository, refer official Ambari documentation here
What happens when you try to configure core-site with LZO configs and GPL license is not accepted?
Ambari’s stackadvisor has a check to validate such configs. In the current scenario, you would get the following error message:
How about some Troubleshooting?
==== Troubleshooting ====
Scenario: Post HDP upgrade, you observe the following error during Pig/MapReduce service check:
ERROR 2118: Compression codec com.hadoop.compression.lzo.LzoCodec not found.
Cause: Your have LZO configured, but not accepted the GPL license during Ambari upgrade
Resolution: Enable GPL license using "ambari-server setup --enable-lzo-under-gpl-license" command
==== Bug Fixes ====
AMBARI-22906 - Fixed in Ambari-2.6.1.5 release
AMBARI-23055 - Fixed in the upcoming Ambari-2.6.2.0 release
