Ambari Cluster setup is not proceeding further during installation

[#1]

Hi,

We are setting up Ambari cluster in Ubuntu 14.04 system.

We have installed Ambari server on namenode nad trying to connect other datanode.

After adding FQDN in Ambari cluster setup, it gets stuck in installation step.

Please find below logs :

02 May 2016 17:02:43,530 INFO [Thread-23] JobStoreTX:861 - Freed 0 triggers from 'acquired' / 'blocked' state. 02 May 2016 17:02:43,549 INFO [Thread-23] JobStoreTX:871 - Recovering 0 jobs that were in-progress at the time of the last shut-down. 02 May 2016 17:02:43,549 INFO [Thread-23] JobStoreTX:884 - Recovery complete. 02 May 2016 17:02:43,550 INFO [Thread-23] JobStoreTX:891 - Removed 0 'complete' triggers. 02 May 2016 17:02:43,550 INFO [Thread-23] JobStoreTX:896 - Removed 0 stale fired job entries. 02 May 2016 17:02:43,583 INFO [Thread-23] QuartzScheduler:575 - Scheduler ExecutionScheduler_$_NON_CLUSTERED started. 02 May 2016 17:10:40,106 INFO [qtp-ambari-client-20] BootStrapImpl:108 - BootStrapping hosts hostname.domainname: 02 May 2016 17:10:40,135 INFO [Thread-40] BSRunner:160 - Kicking off the scheduler for polling on logs in /var/run/ambari-server/bootstrap/1 02 May 2016 17:10:40,135 INFO [Thread-40] BSRunner:335 - Error executing bootstrap Cannot create /var/run/ambari-server/bootstrap 02 May 2016 17:10:40,140 ERROR [Thread-40] BSRunner:397 - java.io.FileNotFoundException: /var/run/ambari-server/bootstrap/1/hostname.domainname.done (No such file or directory) 02 May 2016 17:10:40,141 WARN [Thread-40] BSRunner:364 - File does not exist: /var/run/ambari-server/bootstrap/1/sshKey 02 May 2016 17:10:40,142 INFO [pool-9-thread-1] BSHostStatusCollector:55 - Request directory /var/run/ambari-server/bootstrap/1 02 May 2016 17:10:40,142 INFO [pool-9-thread-1] BSHostStatusCollector:62 - HostList for polling on [hostname.domainname] 02 May 2016 17:10:50,142 INFO [pool-9-thread-1] BSHostStatusCollector:55 - Request directory /var/run/ambari-server/bootstrap/1 02 May 2016 17:10:50,143 INFO [pool-9-thread-1] BSHostStatusCollector:62 - HostList for polling on [hostname.domainname] 02 May 2016 17:11:00,143 INFO [pool-9-thread-1] BSHostStatusCollector:55 - Request directory /var/run/ambari-server/bootstrap/1 02 May 2016 17:11:00,143 INFO [pool-9-thread-1] BSHostStatusCollector:62 - HostList for polling on [hostname.domainname] 02 May 2016 17:11:10,144 INFO [pool-9-thread-1] BSHostStatusCollector:55 - Request directory /var/run/ambari-server/bootstrap/1 02 May 2016 17:11:10,144 INFO [pool-9-thread-1] BSHostStatusCollector:62 - HostList for polling on [hostname.domainname]

We are using root user for the setup

Please let us know what could be the issue

Regards

Pritam

[#2]

both hostname and hostname -f come back with the same FQDN. I think I found my problem to be that /var/run/ambari-server was missing both the ./bootstrap and ./stack_recommedations dirs.

/var/run/ambari-server was owned by root and it appears to have needed to be writable by the ambari user. After making these changes, these errors stopped. I have no idea why this suddenly happened though.

[#3 · Accepted solution]

Hi,

Ensure your network is unrstricted. Also follow Ambari guide for each and every steps.

Below are the steps which we applied for our 4 node Ubuntu systems cluster :

Ambari Cluster Environment setup commands before installation in all systems:

1. To install Ubuntu desktop packages :apt-get install ubuntu-desktop

2.Installing ssh

apt-get install openssh-server

ssh root@localhost

sudo passwd

cd /etc/ssh/sshd_config

>> PermitRootLogin yes

service ssh restart

3. Installing latest Jdk

java -version

apt-get install openjdk-7-jdk

java -version

mkdir /usr/jdk64

cd /home/hadoopuser/Downloads/

cp -r jdk-8u60-linux-x64.tar.gz /usr/jdk64/

cd /usr/jdk64/

tar -xzvf jdk-8u60-linux-x64.tar.gz

javac -version

update-alternatives --install "/usr/bin/java" "java" "/usr/jdk64/jdk1.8.0_60/bin/java" 1

update-alternatives --install "/usr/bin/javac" "javac" "/usr/jdk64/jdk1.8.0_60/bin/javac" 1

update-alternatives --config java

update-alternatives --config javac

java -version

4. To change hostname of system to FQDN:

cat /etc/hostname

nano /etc/hostname

e.g hostname.domainname.com

service hostname restart

hostname

hostname -f

reboot

5. To generate SSH keys for passwordless authentication

ssh-keygen -f "/root/.ssh/known_hosts" -R localhost

ssh root@localhost

cat /root/.ssh/known_hosts

cat /tmp/id_rsa.pub>> .ssh/authorized_keys

6 To enable NTP Services

sysv-rc-conf ntp on

reboot

nano /etc/ntp.conf

server 0.in.pool.ntp.org

server 1.in.pool.ntp.org

server 2.in.pool.ntp.org

server 3.in.pool.ntp.org

install NTP from software centre

service ntp restart

7. To disable Firewall

apt-get install ufw

cat /sys/kernel/mm/transparent_hugepage/enabled

It should be : always madvise [never]

else do following steps

nano /etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash transparent_hugepage=never"

update-grub

reboot

8. Remove SE-Linux

apparmor_status

/etc/init.d/apparmor stop

update-rc.d -f apparmor remove

9.Ambari installation

Log in to your host as root.

Download the Ambari repository file to a directory on your installation host.

wget -nv http://public-repo-1.hortonworks.com/ambari/ubuntu14/2.x/updates/2.2.1.0/ambari.list -O /etc/apt/sources.list.d/ambari.list

apt-key adv --recv-keys --keyserver keyserver.ubuntu.com B9733A7A07513CAD

apt-get update

apt-cache showpkg ambari-server

apt-cache showpkg ambari-agent

apt-cache showpkg ambari-metrics-assembly

apt-get install ambari-server

ambari-server setup

ambari-server start

ambari-server status

follow the remaining steps mentioned in the link mentioned below

Link: https://docs.hortonworks.com/HDPDocuments/Ambari-2.2.1.0/bk_Installing_HDP_AMB/content/ch_Installing...

10.Using Hive with MySQL

http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.1.0/bk_ambari_reference_guide/content/_using_hiv...

11. Using Oozie with MySQL

http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.1.0/bk_ambari_reference_guide/content/_using_ooz...

Hope this is helpful. If any error persists after doing all these things better try for fresh installation after formatting all the systems.

Regards

Pritam

[#4]

What's the status of your /var/run/ambari-server/bootstrap and /var/run/ambari-server/stack-recommendations directories? Do they exist?

I've seen this problem when those directories are missing for some reason.

Manually creating these as empty directories, owned by the user ambari fixed the problem.