How to install Hue 3.11 in HDP 3.1

[#1]

Working with Hive in HDP 3.x is a challenge for Administrators and Developers because there is no Hive UI. The old Hive View has been removed in HDP 3.x and since then Developers were forced to use external 3rd party tools to access Hive.  The only other option for Administrators and Developers was the command line for Hive which certainly works, but is not as friendly as the old Hive View.  Both Beeline and the Hive View are limited when compared to the entire capabilities in the WebUI for Hue (HDFS, Hive, Zookeeper, YARN, Oozie, Sqoop, RDBMS, etc.).

First a little back story on Hue Ambari Service.  First created by Kyle Joe as a Hue Service for Ambari 2.4 the repo did not have any updates in three years. At that time, Hortonworks and Hue removed and changed the location of the 3.11 files, making Kyle's repo not operational. In this article, I am going to outline how I created an HDP 3 test cluster, installed Kyle's 2.4 repo, and began to debug. Next with what I learned debugging, I created an HDP 2.4 cluster, installed Kyle's repo, and finished the installation. With both learning sessions complete, I was able to create my repo version, install a new HDP 3 Single Node Cluster with all the services, install, start, and use Hue for HDFS File Manager, to view an RDMBS Database, view existing Hive Tables, and Execute New HQL Queries.  

Since creating the original repo, I have installed it in two additional HDP clusters (1 in Production). I will continue to make small updates to the repository as necessary for future projects. I plan to further increase functionality to include adjustments necessary for SSL, High Availability, and clusters without all the required components. I also plan to work towards Hue 4.x and bundling up everything nicely into a Management Pack.

Assumptions For the installation:

The following are some assumptions for the installation: 

1. This is tested on Centos 7 and RHEL 7
2. Single Node Cluster has the following Components
   • HDFS
   • Yarn
   • Hbase
   • Hive
   • Oozie
   • Sqoop
   • Spark2
   • Zookeeper
3. No SSL or High Availability (Not Tested Yet)
4. Install includes installing all current Hue Dependencies. See documentation here.
5. Install time lengthy due to compile time during command "make apps".
6. There are conflicts with "Ambari Managing Users" for Custom Components
7. There are conflicts with Spark and Hbase Version Differences

The HDP 3 Hue Service Repo

https://github.com/steven-dfheinz/HDP3-Hue-Service

Creating an HDP 3 Cluster

The following commands are used to quickly install a new HDP 3 Cluster

[root@hdp3 ~]# history
    1  hostnamectl set-hostname hdp3.cloudera.com
    2  yum install nano git wget -y
    3  nano /etc/hosts
    4  nano /etc/selinux/config
    5  nano /etc/cloud/cloud.cfg
    6  reboot
    7  exit
    8  ssh-keygen
    9  cat ~/.ssh/id_rsa.pub
   10  cat ~/.ssh/id_rsa
   13  nano ~/.ssh/authorized_keys
   14  ssh root@hdp3.cloudera.com
   15  wget -O /etc/yum.repos.d/ambari.repo http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.3.0/ambari.repo && yum --enablerepo=extras install epel-release -y && yum install nano java java-devel mariadb mariadb-server mysql-connector-java ambari-server ambari-agent -y && ambari-server setup -s && ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar && ambari-server start && ambari-agent start && service mariadb start && chkconfig mariadb on
   16  mysql

Installing Ambari Hue Service Using GIT

 The following commands will deliver the service to Ambari using GIT

yum install git -y
sudo git clone https://github.com/steven-dfheinz/HDP3-Hue-Service.git /var/lib/ambari-server/resources/stacks/HDP/3.1/services/HUE
ambari-server restart

 

Installing Ambari Hue Service Using ZIP

The following commands will deliver the service to Ambari using Zip Download

wget https://github.com/steven-dfheinz/HDP3-Hue-Service/archive/master.zip
unzip master.zip
mv HDP3-Hue-Service HUE
mv HUE /var/lib/ambari-server/resources/stacks/HDP/3.1/services/

Installing Ambari Hue Service via Ambari

With everything above you should be able to login to Ambari Server and click on Add Service to choose to install Hue. This article assumes you have some experience Adding Custom Services via Ambari, so I won't go into detail here.  If you need more info please reach out. The install should work very well without providing any details to the Installation Wizard. If the service fails to install or start, you should be able to determine the issue and take action to retry install/start.  See below for more information about debugging this install process.

Lessons Learned From This Project

 I always love a good technical challenge, but even more, I love to learn new things I can use in the future.  Some of those things are below.

1. When a custom Ambari Service install fails, clicking into the Failure Modal, and inspecting the stdout and stderr will provide everything you need to debug.
2. While debugging, you can directly edit python files in the Agent Cache location found in the above modal, then retry the action. When a code change is effective, make the same change in the source repository file set.
3. While debugging python, you can print "some variable", config['clusterHostInfo'] to see the output of the object in the modal.  This is useful to fix conflicts in config objects.  Be sure to remove these prints when done or just do NOT make those print changes in the source repository file set.
4. If working in the context of Ambari becomes cumbersome,  it is possible to work directly in the node as root, making changes to components configs, stopping/starting component, etc to a point where things become more operational.  Be sure to be mindful of following:
   • Always make the same effective changes in the source file sets (Ambari-Server, Ambari-Agent, and Parent Repo).
   • Restart the service via Ambari to overwrite local changes. Be careful here.
   • After making any changes directly in the node, make sure all the files are owned to the correct user.  This includes the component files and logs.  The wrong ownership will break ambari start/stop/restart actions.
5. Work in small forward-moving test cycles.  Making too many changes at a time, will confuse you in what did or didn't work to create forward movement.  Make a small change, and move forward in successful increments only.  If something does not work, back out to the last successful increment and try again.

Full History Output

 I always find it useful to record all of my node history.  Here, you can see all the commands I executed while working on my HDP 3 Cluster.  The only commands I have removed are typos or misfires.

[root@hdp3 ~]# history
    1  hostnamectl set-hostname hdp3.cloudera.com
    2  yum install nano git wget -y
    3  nano /etc/hosts
    4  nano /etc/selinux/config
    5  nano /etc/cloud/cloud.cfg
    6  reboot
    7  exit
    8  ssh-keygen
    9  cat ~/.ssh/id_rsa.pub
   10  cat ~/.ssh/id_rsa
   11  nano ~/.ssh/authorized_keys
   13  nano ~/.ssh/authorized_keys
   14  ssh root@hdp3.cloudera.com
   15  wget -O /etc/yum.repos.d/ambari.repo http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.3.0/ambari.repo && yum --enablerepo=extras install epel-release -y && yum install nano java java-devel mariadb mariadb-server mysql-connector-java ambari-server ambari-agent -y && ambari-server setup -s && ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar && ambari-server start && ambari-agent start && service mariadb start && chkconfig mariadb on
   16  mysql
   17  sudo git clone https://github.com/steven-dfheinz/HDP3-Hue-Service.git /var/lib/ambari-server/resources/stacks/HDP/3.1/services/HUE
   18  ambari-server restart
   19  mysql
   20  useradd hue
   21  useradd -g hue hue
   22  python /var/lib/ambari-server/resources/scripts/configs.py -u admin -p admin -n HDP3 -l hdp3.cloudera.com -t 8080 -a set -c cluster-env -k  ignore_groupsusers_create -v true
   23  python /var/lib/ambari-server/resources/scripts/configs.py -u admin -p admin -n HDP3 -l hdp3.cloudera.com -t 8080 -a set -c cluster-env -k  ignore_groupsusers_create -v false
   24  top
   25  python /var/lib/ambari-server/resources/scripts/configs.py -u admin -p admin -n HDP3 -l hdp3.cloudera.com -t 8080 -a set -c cluster-env -k  ignore_groupsusers_create -v true
   26  nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/params.py 
   27  nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/hue_server.py
   28  nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/common.py
   29  nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/params.py 
   30  nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/common.py
   31  cd /usr/local
   32  ls
   33  ls -al
   34  rm -rf hue
   35  ls -al
   36  du -hs ./*
   37  nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/params.py 
   38  ls 0al
   39  ls -al
   40  nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/setup_hue.py
   41  nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/params.py 
   42  sudo su - hdfs
   44  chown -R hue:hue hue*
   45  cd hue
   46  ls
   47  cd desktop/
   48  ls
   49  ls -al
   50  cd ..
   51  ls
   52  ls -al
   53  nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/hue_server.py
   54  nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/common.py
   55  cd /var/log/hue
   56  ls
   57  ls -al
   58  history
   59  rm -r /var/lib/ambari-server/resources/stacks/HDP/3.1/services/HUE
   60  rm -R /var/lib/ambari-server/resources/stacks/HDP/3.1/services/HUE
   61  rm -rf /var/lib/ambari-server/resources/stacks/HDP/3.1/services/HUE
   62  sudo git clone https://github.com/steven-dfheinz/HDP3-Hue-Service.git /var/lib/ambari-server/resources/stacks/HDP/3.1/services/HU
   63  sudo git clone https://github.com/steven-dfheinz/HDP3-Hue-Service.git /var/lib/ambari-server/resources/stacks/HDP/3.1/services/HUE
   64  rm -rf /var/lib/ambari-server/resources/stacks/HDP/3.1/services/HU
   65  rm -rf /var/lib/ambari-server/resources/stacks/HDP/3.1/services/HUE
   66  rm -rf /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE
   67  sudo git clone https://github.com/steven-dfheinz/HDP3-Hue-Service.git /var/lib/ambari-server/resources/stacks/HDP/3.1/services/HUE
   68  nano /var/lib/ambari-server/resources/stacks/HDP/3.1/services/HUE/package/scripts/common.py 
   69  service ambari-server restart
   70  exit
   71  cd /usr/local/
   72  ls -al
   73  rm -rf /usr/local/hue*
   74  ls -al
   75  history
   76  history | grep params
   77  nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/params.py
   78  nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/common.py
   79  sudo su - hdfs
   80  mysql
   81  cd /var/log/hue
   82  ls
   83  ls -al
   84  uptime
   85  tail -f error.log 
   86  cd /etc/hive/conf
   87  ls
   88  cd /var/log/hue
   89  tail -f error.log 
   90  cd /usr/local/hue
   91  ls
   92  cd etc
   93  ls
   94  cd ext
   95  ls
   96  cd ..
   97  cd app
   98  cd apps
   99  ls
  100  cd ..
  102  cd /var/log/hue
  103  ls
  104  ls -al
  105  tail -f supervisor.log 
  106  tail -f runcpserver.log 
  107  cd /etc/yum.repos.d
  108  ls
  109  nano epel.repo 
  110  exit
  111  history
  112  nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/common.py
  113  nano /usr/local/hue-3.11.0/desktop/conf
  114  cd /usr/local/hue-3.11.0/desktop/conf
  115  ls
  116  nano hue.ini
  117  yum –y install cyrus-sasl-plain cyrus-sasl-gssapi
  118  yum install cyrus-sasl-plain cyrus-sasl-gssapi -y
  120  ls
  121  ls -al
  122  nano pseudo-distributed.ini 
  123  cat hue.ini | grep hiveserver2
  124  cat hue.ini | grep hive
  125  nano pseudo-distributed.ini 
  126  ls -al
  127  chown -R hue:hue /usr/local/hue*
  128  cd /var/log/hue
  129  ls
  130  ls -al
  131  nano audit.log 
  132  nano error.log 
  133  history
  134  nano /usr/local/hue-3.11.0/desktop/conf/pseudo-distributed.ini 
  135  history | grep params
  136  nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/params.py
  137  nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/hue_server.py
  138  nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/params.py
  139  nano /usr/local/hue-3.11.0/desktop/conf/hue.ini
  140  nano /usr/local/hue-3.11.0/desktop/conf/pseudo-distributed.ini 
  141  nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/params.py
  142  nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/hue_server.py
  143  nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/common.py
  144  /usr/local/hue/build/env/bin/supervisor
  145  nano /usr/local/hue-3.11.0/desktop/conf/pseudo-distributed.ini 
  146  /usr/local/hue/build/env/bin/supervisor
  147  nano /usr/local/hue-3.11.0/desktop/conf/pseudo-distributed.ini 
  148  /usr/local/hue/build/env/bin/supervisor
  149  nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/params.py
  150  nano /usr/local/hue-3.11.0/desktop/conf/pseudo-distributed.ini 
  151  cd /usr/local/
  152  ls -al
  153  cd hue
  154  ls
  155  ls -al
  156  rm -rf hue
  157  cd desktop/
  158  ls
  159  cd conf
  160  ls
  161  ls -al
  162  cd /var/log/
  163  ls -al
  164  cd hue
  165  ls -al
  166  chown -R hue:hue /var/log/hue
  167  ls -al
  168  nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/params.py
  169  nano /usr/local/hue-3.11.0/desktop/conf/pseudo-distributed.ini 
  170  nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/params.py
  171  nano /usr/local/hue-3.11.0/desktop/conf/pseudo-distributed.ini 
  172  /usr/local/hue/build/env/bin/supervisor
  173  history | grep chown
  174  chown -R hue:hue /usr/local/hue*
  175  chown -R hue:hue /var/log/hue
  176  nano /usr/local/hue-3.11.0/desktop/conf/pseudo-distributed.ini 
  177  /usr/local/hue/build/env/bin/supervisor
  178  nano /usr/local/hue-3.11.0/desktop/conf/pseudo-distributed.ini 
  179  chown -R hue:hue /var/log/hue
  180  chown -R hue:hue /usr/local/hue*
  181  nano /usr/local/hue-3.11.0/desktop/conf/pseudo-distributed.ini 
  182  nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/params.py
  183  history

[#2]

Repo Branch created for Hue 4.6.0.  I am working on this right now but should have it operating here shortly.  If you want Hue 4.6.0 be sure to download Branch Hue.4.6.0.

[#3]

Update for Hue 4.x:

 I have finished initial required changes to for branch Hue.4.6.0 and successfully installed Hue 4.6.0 in HDP 3.1.4.    There ended up being some big differences in the hue.ini for 4.x.  Once these were resolved the only manual work needed for install in the node is creating the hue user and group and running the python to ignore_groupsusers_create.

User Commands:

useradd -g hue hue
usermod -a -G wheel hue
chown -R hue:hue /home/hue

Reset/Remove Commands:

rm -rf /var/lib/ambari-server/resources/stacks/HDP/3.1/services/HUE
rm -rf /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/
rm -rf /usr/local/hue
rm -rf /usr/hdp/current/hue-server
rm -rf /usr/local/hue-4.6.0/

Once installed it is possible to update/manage changes to hue.ini via ambari to enable HDFS, Hive, RDBMS, enable/disable Hue Plugins, or further configure Hue as necessary.

[#4]

Some more updates here:

1. HDP 3.x with Hue 3.11.0 Custom Ambari Hue Service is operational.  Repo Here
2. HDP 3.x with Hue 4.6.0 Custom Ambari Hue Service is operational.  Repo Here
3. HDP 3.x with Hue 4.6.0 DFHz Management Pack is operational.  Repo Here
4. HDP 2.x with Hue 3.11.0 Custom Ambari Hue Service is operational.  Repo Here
5. HDP 2.x with Hue 4.6.0 Custom Ambari Hue Service is operational.  Repo Here

If anyone has questions or issues please open a new question and tag me in it.  You can also reach out via Private Message for additional assistance with any of the above repos.

[#5]

I have now completed the DFHz Hue Management Pack which you can now use to install Hue 3 or Hue 4 in HDP 2 or HDP3.   Just be careful in selecting your correct version(s) and getting the correct mpack install command.  They are each very specific.

Check out the repo:

https://github.com/steven-dfheinz/dfhz_hue_mpack

[#6]

Hi,

I love that you've made HUE available to people like me (first time creating a production cluster). I am planning to install HUE on my 3 node cluster and I will give a sign when it is completed and all working well. 

Only I have a question (maybe stupid). What is the difference between a Mpack and a custom Ambari service?

[#7]

@dewi  A management pack is 1 or more Custom Ambari Services which are easily added to many different ambari versions via the management pack install command.  A custom service is something you would manually add to a single ambari installation.   So the management pack just makes it easier, more applicable to multiple versions, and stacks.