Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Cluster Installation is taking too long

Highlighted

Cluster Installation is taking too long

New Contributor

From the logs, it appears that each step is taking about 2-3 minutes. There is something wrong.

2017-05-28 16:34:06,847 - Stack Feature Version Info: stack_version=2.6, version=None, current_cluster_version=None -> 2.6

2017-05-28 16:34:06,854 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf User Group mapping (user_group) is missing in the hostLevelParams

2017-05-28 16:34:06,855 - Group['hadoop'] {}

2017-05-28 16:34:06,856 - Group['users'] {}

2017-05-28 16:34:06,856 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}

2017-05-28 16:36:58,098 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}

2017-05-28 16:39:48,629 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users']}

2017-05-28 16:42:38,065 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}

2017-05-28 16:45:31,929 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}

2017-05-28 16:45:31,931 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}

2017-05-28 16:45:31,940 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] due to not_if

2017-05-28 16:45:31,941 - Group['hdfs'] {}

2017-05-28 16:45:31,941 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': [u'hadoop', u'hdfs']}

2017-05-28 16:48:23,572 - FS Type:

2017-05-28 16:48:23,572 - Directory['/etc/hadoop'] {'mode': 0755}

10 REPLIES 10

Re: Cluster Installation is taking too long

Super Mentor

Looks like the "fetch_nonlocal_groups" is taking time. Please turn it off and then see the performance.

# /var/lib/ambari-server/resources/scripts/configs.sh set $AMBARI_HOST $CLUSTER_NAME cluster-env fetch_nonlocal_groups false 

.

Re: Cluster Installation is taking too long

New Contributor

I checked. It seems to be already OFF.

/var/lib/ambari-server/resources/scripts/configs.sh -u admin -p admin get xxx.net MY_HDP cluster-env USERID=admin

PASSWORD=admin

########## Performing 'GET' on (Site:cluster-env, Tag:version1495990248470687046)

"properties" :

{ "agent_mounts_ignore_list" : "", "alerts_repeat_tolerance" : "1", "enable_external_ranger" : "false", "fetch_nonlocal_groups" : "false", "hide_yarn_memory_widget" : "false.

.....

....

Re: Cluster Installation is taking too long

New Contributor

Do I have to do this for different configurations (e.g. hadoop-env etc)?

Re: Cluster Installation is taking too long

Super Mentor

It is strange ... I see in your output that is it "True" and taking time exactly there: (16:34 to 16:36) 3 minutes.

2017-05-28 16:34:06,856 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']} 

2017-05-28 16:36:58,098 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']} 

.

Re: Cluster Installation is taking too long

Super Mentor

Have you configured LDAP/AD user sync on Ambari?

Re: Cluster Installation is taking too long

Super Mentor

I will suggest you to try running the config.sh command that i mentioned explicitly.

As i see the 'fetch_nonlocal_groups' is set to 'true' in your output. Here is a similar thread.

https://community.hortonworks.com/questions/100078/starting-services-takes-a-long-time.html

See "ignore_groupsusers_create" setting to "true" here.

# /var/lib/ambari-server/resources/scripts/configs.sh set  <AMBARI_FQDN> <CLUSTER_NAME> cluster-env  "ignore_groupsusers_create"  "true" 

Re: Cluster Installation is taking too long

New Contributor

I will have to check this. I haven't done anything explicitly.

Re: Cluster Installation is taking too long

New Contributor

Yes. But the current configuration queried gives me "fetch_nonlocal_groups" to false.

Appears, that I have to do a clean install again.

Problem is, since ambari overwrites the configuration after Step 8 (Review), I am not sure if I will ever get out of this mess.

Re: Cluster Installation is taking too long

New Contributor

I queried "version1" of cluster-env (original). It shows "fetch_nonlocal_groups" : "true"

I modified cluster-env after step 8. It gave me "version1495990248470687046". This has fetch_nonlocal_groups : false.

Basically, it appears that configuration is all mixed up. Somethings are taken from old, some from new.

It would be great if there is someway of setting configuration which doesn't get overwritten.