Member since
07-19-2018
613
Posts
101
Kudos Received
117
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 5095 | 01-11-2021 05:54 AM | |
| 3421 | 01-11-2021 05:52 AM | |
| 8789 | 01-08-2021 05:23 AM | |
| 8385 | 01-04-2021 04:08 AM | |
| 36688 | 12-18-2020 05:42 AM |
03-24-2020
06:47 AM
Are you running hive/beeline as the hive user or root? To get to Hive as the hive user I do: sudo su - hive hive Then execute the queries.
... View more
03-23-2020
12:05 PM
1 Kudo
After you get all the attributes you want, you check them with RouteOnAttribute. In that proc click + and set update / inserts equal to matches you want. You use regex in the value for each. For example insert is ${operation.equals('add')}. After RouteOnAttribute is configured, drag update to the update query proc, and insert to the insert query proc.
... View more
03-18-2020
04:51 AM
@prakashpunj You need to add the admin user to the hdfs user group. Depending on your system, you may need to make some changes to hdfs proxy hosts. Make it *. You can confirm by checking HDFS Files View with the admin user. If admin can view hdfs there, you should be good in Hue too. In Hue, you can also add a user: hdfs with password: hdfs and login as that user to view HDFS. You can do the same for Hive. In one of my recent tests, admin could view HDFS Files, but could not execute queries on hive tables. So I just created a hive user in hue to validate it works. I assume you are working in a test environment so it should be no problem to make minor users/groups changes to satisfy these access issues. Normally you would want LDAP integration and use Ranger to manage permissions. For example, giving a specific user privileges to HDFS and HIVE.
... View more
03-17-2020
09:29 AM
@prakashpunj Please accept my first reply as the solution?
... View more
03-16-2020
03:33 AM
Working with Hive in HDP 3.x is a challenge for Administrators and Developers because there is no Hive UI. The old Hive View has been removed in HDP 3.x and since then Developers were forced to use external 3rd party tools to access Hive. The only other option for Administrators and Developers was the command line for Hive which certainly works, but is not as friendly as the old Hive View. Both Beeline and the Hive View are limited when compared to the entire capabilities in the WebUI for Hue (HDFS, Hive, Zookeeper, YARN, Oozie, Sqoop, RDBMS, etc.).
First a little back story on Hue Ambari Service. First created by Kyle Joe as a Hue Service for Ambari 2.4 the repo did not have any updates in three years. At that time, Hortonworks and Hue removed and changed the location of the 3.11 files, making Kyle's repo not operational. In this article, I am going to outline how I created an HDP 3 test cluster, installed Kyle's 2.4 repo, and began to debug. Next with what I learned debugging, I created an HDP 2.4 cluster, installed Kyle's repo, and finished the installation. With both learning sessions complete, I was able to create my repo version, install a new HDP 3 Single Node Cluster with all the services, install, start, and use Hue for HDFS File Manager, to view an RDMBS Database, view existing Hive Tables, and Execute New HQL Queries.
Since creating the original repo, I have installed it in two additional HDP clusters (1 in Production). I will continue to make small updates to the repository as necessary for future projects. I plan to further increase functionality to include adjustments necessary for SSL, High Availability, and clusters without all the required components. I also plan to work towards Hue 4.x and bundling up everything nicely into a Management Pack.
Assumptions For the installation:
The following are some assumptions for the installation:
This is tested on Centos 7 and RHEL 7
Single Node Cluster has the following Components
HDFS
Yarn
Hbase
Hive
Oozie
Sqoop
Spark2
Zookeeper
No SSL or High Availability (Not Tested Yet)
Install includes installing all current Hue Dependencies. See documentation here.
Install time lengthy due to compile time during command "make apps".
There are conflicts with "Ambari Managing Users" for Custom Components
There are conflicts with Spark and Hbase Version Differences
The HDP 3 Hue Service Repo
https://github.com/steven-dfheinz/HDP3-Hue-Service
Creating an HDP 3 Cluster
The following commands are used to quickly install a new HDP 3 Cluster
[root@hdp3 ~]# history
1 hostnamectl set-hostname hdp3.cloudera.com
2 yum install nano git wget -y
3 nano /etc/hosts
4 nano /etc/selinux/config
5 nano /etc/cloud/cloud.cfg
6 reboot
7 exit
8 ssh-keygen
9 cat ~/.ssh/id_rsa.pub
10 cat ~/.ssh/id_rsa
13 nano ~/.ssh/authorized_keys
14 ssh root@hdp3.cloudera.com
15 wget -O /etc/yum.repos.d/ambari.repo http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.3.0/ambari.repo && yum --enablerepo=extras install epel-release -y && yum install nano java java-devel mariadb mariadb-server mysql-connector-java ambari-server ambari-agent -y && ambari-server setup -s && ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar && ambari-server start && ambari-agent start && service mariadb start && chkconfig mariadb on
16 mysql
Installing Ambari Hue Service Using GIT
The following commands will deliver the service to Ambari using GIT
yum install git -y
sudo git clone https://github.com/steven-dfheinz/HDP3-Hue-Service.git /var/lib/ambari-server/resources/stacks/HDP/3.1/services/HUE
ambari-server restart
Installing Ambari Hue Service Using ZIP
The following commands will deliver the service to Ambari using Zip Download
wget https://github.com/steven-dfheinz/HDP3-Hue-Service/archive/master.zip
unzip master.zip
mv HDP3-Hue-Service HUE
mv HUE /var/lib/ambari-server/resources/stacks/HDP/3.1/services/
Installing Ambari Hue Service via Ambari
With everything above you should be able to login to Ambari Server and click on Add Service to choose to install Hue. This article assumes you have some experience Adding Custom Services via Ambari, so I won't go into detail here. If you need more info please reach out. The install should work very well without providing any details to the Installation Wizard. If the service fails to install or start, you should be able to determine the issue and take action to retry install/start. See below for more information about debugging this install process.
Lessons Learned From This Project
I always love a good technical challenge, but even more, I love to learn new things I can use in the future. Some of those things are below.
When a custom Ambari Service install fails, clicking into the Failure Modal, and inspecting the stdout and stderr will provide everything you need to debug.
While debugging, you can directly edit python files in the Agent Cache location found in the above modal, then retry the action. When a code change is effective, make the same change in the source repository file set.
While debugging python, you can print "some variable", config['clusterHostInfo'] to see the output of the object in the modal. This is useful to fix conflicts in config objects. Be sure to remove these prints when done or just do NOT make those print changes in the source repository file set.
If working in the context of Ambari becomes cumbersome, it is possible to work directly in the node as root, making changes to components configs, stopping/starting component, etc to a point where things become more operational. Be sure to be mindful of following:
Always make the same effective changes in the source file sets (Ambari-Server, Ambari-Agent, and Parent Repo).
Restart the service via Ambari to overwrite local changes. Be careful here.
After making any changes directly in the node, make sure all the files are owned to the correct user. This includes the component files and logs. The wrong ownership will break ambari start/stop/restart actions.
Work in small forward-moving test cycles. Making too many changes at a time, will confuse you in what did or didn't work to create forward movement. Make a small change, and move forward in successful increments only. If something does not work, back out to the last successful increment and try again.
Full History Output
I always find it useful to record all of my node history. Here, you can see all the commands I executed while working on my HDP 3 Cluster. The only commands I have removed are typos or misfires.
[root@hdp3 ~]# history
1 hostnamectl set-hostname hdp3.cloudera.com
2 yum install nano git wget -y
3 nano /etc/hosts
4 nano /etc/selinux/config
5 nano /etc/cloud/cloud.cfg
6 reboot
7 exit
8 ssh-keygen
9 cat ~/.ssh/id_rsa.pub
10 cat ~/.ssh/id_rsa
11 nano ~/.ssh/authorized_keys
13 nano ~/.ssh/authorized_keys
14 ssh root@hdp3.cloudera.com
15 wget -O /etc/yum.repos.d/ambari.repo http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.3.0/ambari.repo && yum --enablerepo=extras install epel-release -y && yum install nano java java-devel mariadb mariadb-server mysql-connector-java ambari-server ambari-agent -y && ambari-server setup -s && ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar && ambari-server start && ambari-agent start && service mariadb start && chkconfig mariadb on
16 mysql
17 sudo git clone https://github.com/steven-dfheinz/HDP3-Hue-Service.git /var/lib/ambari-server/resources/stacks/HDP/3.1/services/HUE
18 ambari-server restart
19 mysql
20 useradd hue
21 useradd -g hue hue
22 python /var/lib/ambari-server/resources/scripts/configs.py -u admin -p admin -n HDP3 -l hdp3.cloudera.com -t 8080 -a set -c cluster-env -k ignore_groupsusers_create -v true
23 python /var/lib/ambari-server/resources/scripts/configs.py -u admin -p admin -n HDP3 -l hdp3.cloudera.com -t 8080 -a set -c cluster-env -k ignore_groupsusers_create -v false
24 top
25 python /var/lib/ambari-server/resources/scripts/configs.py -u admin -p admin -n HDP3 -l hdp3.cloudera.com -t 8080 -a set -c cluster-env -k ignore_groupsusers_create -v true
26 nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/params.py
27 nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/hue_server.py
28 nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/common.py
29 nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/params.py
30 nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/common.py
31 cd /usr/local
32 ls
33 ls -al
34 rm -rf hue
35 ls -al
36 du -hs ./*
37 nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/params.py
38 ls 0al
39 ls -al
40 nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/setup_hue.py
41 nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/params.py
42 sudo su - hdfs
44 chown -R hue:hue hue*
45 cd hue
46 ls
47 cd desktop/
48 ls
49 ls -al
50 cd ..
51 ls
52 ls -al
53 nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/hue_server.py
54 nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/common.py
55 cd /var/log/hue
56 ls
57 ls -al
58 history
59 rm -r /var/lib/ambari-server/resources/stacks/HDP/3.1/services/HUE
60 rm -R /var/lib/ambari-server/resources/stacks/HDP/3.1/services/HUE
61 rm -rf /var/lib/ambari-server/resources/stacks/HDP/3.1/services/HUE
62 sudo git clone https://github.com/steven-dfheinz/HDP3-Hue-Service.git /var/lib/ambari-server/resources/stacks/HDP/3.1/services/HU
63 sudo git clone https://github.com/steven-dfheinz/HDP3-Hue-Service.git /var/lib/ambari-server/resources/stacks/HDP/3.1/services/HUE
64 rm -rf /var/lib/ambari-server/resources/stacks/HDP/3.1/services/HU
65 rm -rf /var/lib/ambari-server/resources/stacks/HDP/3.1/services/HUE
66 rm -rf /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE
67 sudo git clone https://github.com/steven-dfheinz/HDP3-Hue-Service.git /var/lib/ambari-server/resources/stacks/HDP/3.1/services/HUE
68 nano /var/lib/ambari-server/resources/stacks/HDP/3.1/services/HUE/package/scripts/common.py
69 service ambari-server restart
70 exit
71 cd /usr/local/
72 ls -al
73 rm -rf /usr/local/hue*
74 ls -al
75 history
76 history | grep params
77 nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/params.py
78 nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/common.py
79 sudo su - hdfs
80 mysql
81 cd /var/log/hue
82 ls
83 ls -al
84 uptime
85 tail -f error.log
86 cd /etc/hive/conf
87 ls
88 cd /var/log/hue
89 tail -f error.log
90 cd /usr/local/hue
91 ls
92 cd etc
93 ls
94 cd ext
95 ls
96 cd ..
97 cd app
98 cd apps
99 ls
100 cd ..
102 cd /var/log/hue
103 ls
104 ls -al
105 tail -f supervisor.log
106 tail -f runcpserver.log
107 cd /etc/yum.repos.d
108 ls
109 nano epel.repo
110 exit
111 history
112 nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/common.py
113 nano /usr/local/hue-3.11.0/desktop/conf
114 cd /usr/local/hue-3.11.0/desktop/conf
115 ls
116 nano hue.ini
117 yum –y install cyrus-sasl-plain cyrus-sasl-gssapi
118 yum install cyrus-sasl-plain cyrus-sasl-gssapi -y
120 ls
121 ls -al
122 nano pseudo-distributed.ini
123 cat hue.ini | grep hiveserver2
124 cat hue.ini | grep hive
125 nano pseudo-distributed.ini
126 ls -al
127 chown -R hue:hue /usr/local/hue*
128 cd /var/log/hue
129 ls
130 ls -al
131 nano audit.log
132 nano error.log
133 history
134 nano /usr/local/hue-3.11.0/desktop/conf/pseudo-distributed.ini
135 history | grep params
136 nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/params.py
137 nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/hue_server.py
138 nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/params.py
139 nano /usr/local/hue-3.11.0/desktop/conf/hue.ini
140 nano /usr/local/hue-3.11.0/desktop/conf/pseudo-distributed.ini
141 nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/params.py
142 nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/hue_server.py
143 nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/common.py
144 /usr/local/hue/build/env/bin/supervisor
145 nano /usr/local/hue-3.11.0/desktop/conf/pseudo-distributed.ini
146 /usr/local/hue/build/env/bin/supervisor
147 nano /usr/local/hue-3.11.0/desktop/conf/pseudo-distributed.ini
148 /usr/local/hue/build/env/bin/supervisor
149 nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/params.py
150 nano /usr/local/hue-3.11.0/desktop/conf/pseudo-distributed.ini
151 cd /usr/local/
152 ls -al
153 cd hue
154 ls
155 ls -al
156 rm -rf hue
157 cd desktop/
158 ls
159 cd conf
160 ls
161 ls -al
162 cd /var/log/
163 ls -al
164 cd hue
165 ls -al
166 chown -R hue:hue /var/log/hue
167 ls -al
168 nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/params.py
169 nano /usr/local/hue-3.11.0/desktop/conf/pseudo-distributed.ini
170 nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/params.py
171 nano /usr/local/hue-3.11.0/desktop/conf/pseudo-distributed.ini
172 /usr/local/hue/build/env/bin/supervisor
173 history | grep chown
174 chown -R hue:hue /usr/local/hue*
175 chown -R hue:hue /var/log/hue
176 nano /usr/local/hue-3.11.0/desktop/conf/pseudo-distributed.ini
177 /usr/local/hue/build/env/bin/supervisor
178 nano /usr/local/hue-3.11.0/desktop/conf/pseudo-distributed.ini
179 chown -R hue:hue /var/log/hue
180 chown -R hue:hue /usr/local/hue*
181 nano /usr/local/hue-3.11.0/desktop/conf/pseudo-distributed.ini
182 nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/params.py
183 history
... View more
03-12-2020
08:01 AM
OperationalError: attempt to write a readonly database Need to fix the write perms on the database file... not sure of the path you have now, but mine was in the desktop folder.
... View more
03-12-2020
06:27 AM
Your /etc/hosts file is a mapping of a hostname to an IP. What is in /etc/hosts will override any dns settings. That is the whole point of using /etc/hosts/. Also the hostname of each machine should be same as the hostname you use in /etc/hosts to map to the the IP. You should be setting the hostname, and /etc/hosts accordingly before install anything. This should be done on all hosts, and all hosts of cluster should be listed in /etc/hosts on each machine. Assuming you have more than one node in the cluster. You can set hostnames to be whatever FQDN you want, just make sure they persist after reboot. In some OS's the /etc/hosts file and hostname file(s) are managed (manual changes over-written during reboot). So make sure you set them both per node, reboot, and check. You need to get the machines working on a single hostname and a single ip. This is the problem and is a documented requirement for working in CDH/CDP/HDP/etc.
... View more
03-12-2020
06:17 AM
looks like you have it installed as /root/hue-3.8.1 try using /root/hue-3.8.1 instead of /usr/local/hue in the startup command? I would recommend to put it in /usr/local and symlink if not installed as just /hue/ folder: ln -s /usr/local/hue-3.8.1 /usr/local/hue also make sure you execute "make apps" in the folder to install everything as the hue user...
... View more
03-12-2020
06:05 AM
I know some other people with big time Regex Skills can map it differently but the method I use it to get single values using strategic wildcard like searches in ReplaceText like: timestamp : ([0-9]{4}-(0[1-9]|1[0-2])-(0[1-9]|[1-2][0-9]|3[0-1]) (2[0-3]|[01][0-9]):([0-5][0-9]):([0-9]{2}).([0-9]{6})) .* source_ip : .* (.*) -> .* dest_ip : .* -> (.*) <.* Mac : ^.*] TS\|[0-9]{10}\|([a-fA-F0-9:]{17}|[a-fA-F0-9]{12})\|.*$ dbm : ^.*\|(.*)$ I also test using a regex tester like this one: REGEX
... View more
03-12-2020
05:30 AM
Per documentation, the start command is: /usr/local/hue/build/env/bin/supervisor If it doesn't start, make sure you have completed the "make apps" command and installed all dependencies. Also depending on your system you want to chown -R hue:hue /usr/local/hue
... View more