Member since
01-16-2018
8
Posts
0
Kudos Received
0
Solutions
02-04-2018
02:59 AM
@Jay Kumar SenSharma is this an expected behaviour for ambari non root installations ?
... View more
02-04-2018
02:58 AM
@Jay Kumar SenSharma - is this an expected behaviour for ambari non root installations ?
... View more
01-30-2018
12:00 PM
@Scott Shaw - Does cluster creation take more time if ambari is configured for non root user ? Cluster creation with ambari configured for root took 21 minutes to install and 15 minutes to start all services. Cluster creation with ambari configured for non root took 29 minutes to install and 25 minutes to start all services. Post cluster creation i see 'stop all' and 'start all' services is also taking a lil more time non root user clusters. From the logs i see that /var/lib/ambari-agent/tmp/changeUid.sh and ambari-python-wrap executions are taking a few more milliseconds when compared with the cluster with root user.
... View more
01-22-2018
05:10 PM
hdp cluster creation is taking more time when compared with a cluster with ambari configured for root. Below is a comparison of hdp cluster creation with ambari root and ambari non root. I had installed the same services on both the clusters. Note - These clusters were created by following the steps as mentioned in hdp documentation. https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-installation/content/ch_Deploy_and_Configure_a_HDP_Cluster.html Cluster creation with ambari configured for root took 21 minutes to install and 15 minutes to start all services. Cluster creation with ambari configured for non root took 29 minutes to intall and 25 minutes to start all services. Checked the installation logs, found changeUid.sh script is taking more time in ambari non root cluster creations. App Timeline Server Install with Ambari (root) - changeUid.sh has taken 12 milli secs to complete - 2018-01-22 00:59:38,066 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2018-01-22 00:59:38,069 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'} 2018-01-22 00:59:38,081 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] due to not_if 2018-01-22 00:59:38,081 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'create_parents': True, 'mode': 0775, 'cd_access': 'a'} App Timeline Server Install with Ambari (non root) - changeUid.sh has taken 56 milli secs to complete - 2018-01-22 03:08:29,850 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2018-01-22 03:08:30,285 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'} 2018-01-22 03:08:30,341 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] due to not_if 2018-01-22 03:08:30,342 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'create_parents': True, 'mode': 0775, 'cd_access': 'a'} NameNode Install with Ambari (root) - changeUid.sh has taken 11 milli secs to complete - 2018-01-22 01:11:05,315 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2018-01-22 01:11:05,317 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'} 2018-01-22 01:11:05,328 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] due to not_if NameNode Install with Ambari (non root) - changeUid.sh has taken 54 milli secs to complete 2018-01-22 03:23:19,647 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2018-01-22 03:23:20,082 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'} 2018-01-22 03:23:20,136 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] due to not_if Oozie Server Start with Ambari (root) - changeUid.sh has taken 2 milli secs to complete 2018-01-22 01:26:17,614 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2018-01-22 01:26:17,616 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'} 2018-01-22 01:26:17,628 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] due to not_if 2018-01-22 01:26:17,629 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'create_parents': True, 'mode': 0775, 'cd_access': 'a'} 2018-01-22 01:26:17,630 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2018-01-22 01:26:17,632 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2018-01-22 01:26:17,634 - call['/var/lib/ambari-agent/tmp/changeUid.sh hbase'] {} 2018-01-22 01:26:17,651 - call returned (0, '1014') 2018-01-22 01:26:17,652 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase 1014'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'} 2018-01-22 01:26:17,663 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase 1014'] due to not_if Oozie Server Start with Ambari (non root) - changeUid.sh has taken 432 milli secs to complete 2018-01-22 04:04:02,249 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2018-01-22 04:04:02,681 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'} 2018-01-22 04:04:02,733 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] due to not_if 2018-01-22 04:04:02,734 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'create_parents': True, 'mode': 0775, 'cd_access': 'a'} 2018-01-22 04:04:03,223 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2018-01-22 04:04:03,665 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2018-01-22 04:04:04,098 - call['/var/lib/ambari-agent/tmp/changeUid.sh hbase'] {} 2018-01-22 04:04:04,154 - call returned (0, '1016') 2018-01-22 04:04:04,154 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase 1016'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'} 2018-01-22 04:04:04,211 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase 1016'] due to not_if The same is observance while installing and starting all the services.
... View more
Labels:
01-22-2018
05:03 PM
Looks like ambari with non root user cluster creation does take more time when compared to ambari root user cluster creation...
... View more
01-22-2018
11:22 AM
@Jay Kumar SenSharma - Thanks for the detailed info.
I tried the interactive ambari-server setup for non root user, even then i found that the hdp cluster creation is taking more time when compared with a cluster with ambari configured for root.
Below is a comparison of hdp cluster creation with ambari root and ambari non root. I had installed the same services on both the clusters.
Note - These clusters were created by following the steps as mentioned in hdp documentation. https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-installation/content/ch_Deploy_and_Configure_a_HDP_Cluster.html Cluster creation with ambari configured for root took 21 minutes to install and 15 minutes to start all services. Cluster creation with ambari configured for non root took 29 minutes to intall and 25 minutes to start all services.
Checked the installation logs, found changeUid.sh script is taking more time in ambari non root cluster creations.
App Timeline Server Install with Ambari (root) - changeUid.sh has taken 12 secs to complete - 2018-01-22 00:59:38,066 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2018-01-22 00:59:38,069 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'} 2018-01-22 00:59:38,081 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] due to not_if 2018-01-22 00:59:38,081 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'create_parents': True, 'mode': 0775, 'cd_access': 'a'} App Timeline Server Install with Ambari (non root) - changeUid.sh has taken 56 secs to complete - 2018-01-22 03:08:29,850 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2018-01-22 03:08:30,285 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'} 2018-01-22 03:08:30,341 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] due to not_if 2018-01-22 03:08:30,342 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'create_parents': True, 'mode': 0775, 'cd_access': 'a'} NameNode Install with Ambari (root) - changeUid.sh has taken 11 secs to complete - 2018-01-22 01:11:05,315 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2018-01-22 01:11:05,317 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'} 2018-01-22 01:11:05,328 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] due to not_if NameNode Install with Ambari (non root) - changeUid.sh has taken 54 secs to complete 2018-01-22 03:23:19,647 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2018-01-22 03:23:20,082 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2018-01-22 03:23:20,136 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] due to not_if Oozie Server Start with Ambari (root) - changeUid.sh has taken 2 secs to complete 2018-01-22 01:26:17,614 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2018-01-22 01:26:17,616 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2018-01-22 01:26:17,628 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] due to not_if
2018-01-22 01:26:17,629 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'create_parents': True, 'mode': 0775, 'cd_access': 'a'} 2018-01-22 01:26:17,630 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2018-01-22 01:26:17,632 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2018-01-22 01:26:17,634 - call['/var/lib/ambari-agent/tmp/changeUid.sh hbase'] {}
2018-01-22 01:26:17,651 - call returned (0, '1014')
2018-01-22 01:26:17,652 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase 1014'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'} 2018-01-22 01:26:17,663 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase 1014'] due to not_if Oozie Server Start with Ambari (non root) - changeUid.sh has taken 432 secs to complete 2018-01-22 04:04:02,249 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2018-01-22 04:04:02,681 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'} 2018-01-22 04:04:02,733 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] due to not_if
2018-01-22 04:04:02,734 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'create_parents': True, 'mode': 0775, 'cd_access': 'a'} 2018-01-22 04:04:03,223 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2018-01-22 04:04:03,665 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2018-01-22 04:04:04,098 - call['/var/lib/ambari-agent/tmp/changeUid.sh hbase'] {}
2018-01-22 04:04:04,154 - call returned (0, '1016')
2018-01-22 04:04:04,154 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase 1016'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'} 2018-01-22 04:04:04,211 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase 1016'] due to not_if The same is observance while installing and starting all the services.
... View more
01-17-2018
05:30 AM
Ok.. but was trying to understand if we can modify the property ambari-server.user in the config file /etc/ambari-server/conf/ambari.properties and give appropriate file permission for the logs and config directories. I am able to modify the property ambari-server.user in the config file /etc/ambari-server/conf/ambari.properties and do a silent ambari setup. ambari-server is started and i am able install the HDP stack, but the installation of the HDP stack components is longer than the expected time.
... View more
01-16-2018
12:18 PM
I am trying to automate the process of ambari-server setup and cluster creation. I want to configure ambari-server for non-root user in a silent mode. Please suggest the available options.
... View more
- Tags:
- non-kerberized