question Re: ambari 2.2.1.0 start service failed with error in Support Questions

ambari 2.2.1.0 start service failed with error

darkz1984 — Tue, 29 Mar 2016 11:31:14 GMT

I use ambari 2.2.1.0 and use nonroot user running ther server and the agent.

Install hdp 2.3.4.0-3485 is ok,when I try to start the service such as hdfs ,it was failed with errors,the logs content is blow...

stderr: /var/lib/ambari-agent/data/errors-69.txt

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py", line 39, in <module>
    BeforeStartHook().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py", line 28, in hook
    import params
  File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/params.py", line 141, in <module>
    ambari_db_rca_password = config['hostLevelParams']['ambari_db_rca_password'][0]
TypeError: 'int' object is unsubscriptable

stdout:   /var/lib/ambari-agent/data/output-69.txt
2016-03-29 12:21:52,902 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.3.4.0-3485
2016-03-29 12:21:52,902 - Checking if need to create versioned conf dir /etc/hadoop/2.3.4.0-3485/0
2016-03-29 12:21:52,903 - call['conf-select create-conf-dir --package hadoop --stack-version 2.3.4.0-3485 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1}
2016-03-29 12:21:52,952 - call returned (1, '/etc/hadoop/2.3.4.0-3485/0 exist already', '')
2016-03-29 12:21:52,952 - checked_call['conf-select set-conf-dir --package hadoop --stack-version 2.3.4.0-3485 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False}
2016-03-29 12:21:52,996 - checked_call returned (0, '/usr/hdp/2.3.4.0-3485/hadoop/conf -> /etc/hadoop/2.3.4.0-3485/0')
2016-03-29 12:21:52,997 - Ensuring that hadoop has the correct symlink structure
2016-03-29 12:21:52,997 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf

I had google it for times but no result about.

Re: ambari 2.2.1.0 start service failed with error

sshimpi — Tue, 29 Mar 2016 11:36:52 GMT

@darkz yu

Was there any errors while installation ?

Can you run the below command and paste the output here -

#ls -ltr /etc/hadoop

#ls -ltr /usr/hdp/current/

#hdp-select

Thanks

Sagar

Re: ambari 2.2.1.0 start service failed with error

darkz1984 — Tue, 29 Mar 2016 12:49:44 GMT

@Sagar Shimpi

the informations is there:

ls -ltr /etc/hadoop
total 8
drwxr-xr-x 2 root root 4096 Mar 28 16:07 conf.backup
drwxr-xr-x 3 root root 4096 Mar 28 16:07 2.3.4.0-3485
lrwxrwxrwx 1 root root   35 Mar 28 16:07 conf -> /usr/hdp/current/hadoop-client/conf

ls -ltr /usr/hdp/current/
total 0
lrwxrwxrwx 1 root root 27 Mar 29 11:27 atlas-server -> /usr/hdp/2.3.4.0-3485/atlas
lrwxrwxrwx 1 root root 30 Mar 29 11:27 accumulo-tracer -> /usr/hdp/2.3.4.0-3485/accumulo
lrwxrwxrwx 1 root root 30 Mar 29 11:27 accumulo-tablet -> /usr/hdp/2.3.4.0-3485/accumulo
lrwxrwxrwx 1 root root 30 Mar 29 11:27 accumulo-monitor -> /usr/hdp/2.3.4.0-3485/accumulo
lrwxrwxrwx 1 root root 30 Mar 29 11:27 accumulo-master -> /usr/hdp/2.3.4.0-3485/accumulo
lrwxrwxrwx 1 root root 30 Mar 29 11:27 accumulo-gc -> /usr/hdp/2.3.4.0-3485/accumulo
lrwxrwxrwx 1 root root 30 Mar 29 11:27 accumulo-client -> /usr/hdp/2.3.4.0-3485/accumulo
lrwxrwxrwx 1 root root 31 Mar 29 11:27 zookeeper-server -> /usr/hdp/2.3.4.0-3485/zookeeper
lrwxrwxrwx 1 root root 31 Mar 29 11:27 zookeeper-client -> /usr/hdp/2.3.4.0-3485/zookeeper
lrwxrwxrwx 1 root root 25 Mar 29 11:27 tez-client -> /usr/hdp/2.3.4.0-3485/tez
lrwxrwxrwx 1 root root 27 Mar 29 11:27 storm-supervisor -> /usr/hdp/2.3.4.0-3485/storm
lrwxrwxrwx 1 root root 41 Mar 29 11:27 storm-slider-client -> /usr/hdp/2.3.4.0-3485/storm-slider-client
lrwxrwxrwx 1 root root 27 Mar 29 11:27 storm-nimbus -> /usr/hdp/2.3.4.0-3485/storm
lrwxrwxrwx 1 root root 27 Mar 29 11:27 storm-client -> /usr/hdp/2.3.4.0-3485/storm
lrwxrwxrwx 1 root root 27 Mar 29 11:27 sqoop-server -> /usr/hdp/2.3.4.0-3485/sqoop
lrwxrwxrwx 1 root root 27 Mar 29 11:27 sqoop-client -> /usr/hdp/2.3.4.0-3485/sqoop
lrwxrwxrwx 1 root root 27 Mar 29 11:27 spark-thriftserver -> /usr/hdp/2.3.4.0-3485/spark
lrwxrwxrwx 1 root root 27 Mar 29 11:27 spark-historyserver -> /usr/hdp/2.3.4.0-3485/spark
lrwxrwxrwx 1 root root 27 Mar 29 11:27 spark-client -> /usr/hdp/2.3.4.0-3485/spark
lrwxrwxrwx 1 root root 28 Mar 29 11:27 slider-client -> /usr/hdp/2.3.4.0-3485/slider
lrwxrwxrwx 1 root root 37 Mar 29 11:27 ranger-usersync -> /usr/hdp/2.3.4.0-3485/ranger-usersync
lrwxrwxrwx 1 root root 32 Mar 29 11:27 ranger-kms -> /usr/hdp/2.3.4.0-3485/ranger-kms
lrwxrwxrwx 1 root root 34 Mar 29 11:27 ranger-admin -> /usr/hdp/2.3.4.0-3485/ranger-admin
lrwxrwxrwx 1 root root 25 Mar 29 11:27 pig-client -> /usr/hdp/2.3.4.0-3485/pig
lrwxrwxrwx 1 root root 29 Mar 29 11:27 phoenix-server -> /usr/hdp/2.3.4.0-3485/phoenix
lrwxrwxrwx 1 root root 29 Mar 29 11:27 phoenix-client -> /usr/hdp/2.3.4.0-3485/phoenix
lrwxrwxrwx 1 root root 27 Mar 29 11:27 oozie-server -> /usr/hdp/2.3.4.0-3485/oozie
lrwxrwxrwx 1 root root 27 Mar 29 11:27 oozie-client -> /usr/hdp/2.3.4.0-3485/oozie
lrwxrwxrwx 1 root root 28 Mar 29 11:27 mahout-client -> /usr/hdp/2.3.4.0-3485/mahout
lrwxrwxrwx 1 root root 26 Mar 29 11:27 knox-server -> /usr/hdp/2.3.4.0-3485/knox
lrwxrwxrwx 1 root root 27 Mar 29 11:27 kafka-broker -> /usr/hdp/2.3.4.0-3485/kafka
lrwxrwxrwx 1 root root 35 Mar 29 11:27 hive-webhcat -> /usr/hdp/2.3.4.0-3485/hive-hcatalog
lrwxrwxrwx 1 root root 26 Mar 29 11:27 hive-server2 -> /usr/hdp/2.3.4.0-3485/hive
lrwxrwxrwx 1 root root 26 Mar 29 11:27 hive-metastore -> /usr/hdp/2.3.4.0-3485/hive
lrwxrwxrwx 1 root root 26 Mar 29 11:27 hive-client -> /usr/hdp/2.3.4.0-3485/hive
lrwxrwxrwx 1 root root 27 Mar 29 11:27 hbase-regionserver -> /usr/hdp/2.3.4.0-3485/hbase
lrwxrwxrwx 1 root root 27 Mar 29 11:27 hbase-master -> /usr/hdp/2.3.4.0-3485/hbase
lrwxrwxrwx 1 root root 27 Mar 29 11:27 hbase-client -> /usr/hdp/2.3.4.0-3485/hbase
lrwxrwxrwx 1 root root 33 Mar 29 11:27 hadoop-yarn-timelineserver -> /usr/hdp/2.3.4.0-3485/hadoop-yarn
lrwxrwxrwx 1 root root 33 Mar 29 11:27 hadoop-yarn-resourcemanager -> /usr/hdp/2.3.4.0-3485/hadoop-yarn
lrwxrwxrwx 1 root root 33 Mar 29 11:27 hadoop-yarn-nodemanager -> /usr/hdp/2.3.4.0-3485/hadoop-yarn
lrwxrwxrwx 1 root root 33 Mar 29 11:27 hadoop-yarn-client -> /usr/hdp/2.3.4.0-3485/hadoop-yarn
lrwxrwxrwx 1 root root 38 Mar 29 11:27 hadoop-mapreduce-historyserver -> /usr/hdp/2.3.4.0-3485/hadoop-mapreduce
lrwxrwxrwx 1 root root 38 Mar 29 11:27 hadoop-mapreduce-client -> /usr/hdp/2.3.4.0-3485/hadoop-mapreduce
lrwxrwxrwx 1 root root 35 Mar 29 11:27 hadoop-httpfs -> /usr/hdp/2.3.4.0-3485/hadoop-httpfs
lrwxrwxrwx 1 root root 33 Mar 29 11:27 hadoop-hdfs-secondarynamenode -> /usr/hdp/2.3.4.0-3485/hadoop-hdfs
lrwxrwxrwx 1 root root 33 Mar 29 11:27 hadoop-hdfs-portmap -> /usr/hdp/2.3.4.0-3485/hadoop-hdfs
lrwxrwxrwx 1 root root 33 Mar 29 11:27 hadoop-hdfs-nfs3 -> /usr/hdp/2.3.4.0-3485/hadoop-hdfs
lrwxrwxrwx 1 root root 33 Mar 29 11:27 hadoop-hdfs-namenode -> /usr/hdp/2.3.4.0-3485/hadoop-hdfs
lrwxrwxrwx 1 root root 33 Mar 29 11:27 hadoop-hdfs-journalnode -> /usr/hdp/2.3.4.0-3485/hadoop-hdfs
lrwxrwxrwx 1 root root 33 Mar 29 11:27 hadoop-hdfs-datanode -> /usr/hdp/2.3.4.0-3485/hadoop-hdfs
lrwxrwxrwx 1 root root 33 Mar 29 11:27 hadoop-hdfs-client -> /usr/hdp/2.3.4.0-3485/hadoop-hdfs
lrwxrwxrwx 1 root root 28 Mar 29 11:27 hadoop-client -> /usr/hdp/2.3.4.0-3485/hadoop
lrwxrwxrwx 1 root root 27 Mar 29 11:27 flume-server -> /usr/hdp/2.3.4.0-3485/flume
lrwxrwxrwx 1 root root 28 Mar 29 11:27 falcon-server -> /usr/hdp/2.3.4.0-3485/falcon
lrwxrwxrwx 1 root root 28 Mar 29 11:27 falcon-client -> /usr/hdp/2.3.4.0-3485/falcon

hdp-select
accumulo-client - None
accumulo-gc - None
accumulo-master - None
accumulo-monitor - None
accumulo-tablet - None
accumulo-tracer - None
atlas-server - None
falcon-client - None
falcon-server - None
flume-server - 2.3.4.0-3485
hadoop-client - 2.3.4.0-3485
hadoop-hdfs-datanode - 2.3.4.0-3485
hadoop-hdfs-journalnode - 2.3.4.0-3485
hadoop-hdfs-namenode - 2.3.4.0-3485
hadoop-hdfs-nfs3 - 2.3.4.0-3485
hadoop-hdfs-portmap - 2.3.4.0-3485
hadoop-hdfs-secondarynamenode - 2.3.4.0-3485
hadoop-httpfs - None
hadoop-mapreduce-historyserver - 2.3.4.0-3485
hadoop-yarn-nodemanager - 2.3.4.0-3485
hadoop-yarn-resourcemanager - 2.3.4.0-3485
hadoop-yarn-timelineserver - 2.3.4.0-3485
hbase-client - None
hbase-master - None
hbase-regionserver - None
hive-metastore - None
hive-server2 - None
hive-webhcat - None
kafka-broker - None
knox-server - None
mahout-client - None
oozie-client - None
oozie-server - None
phoenix-client - None
phoenix-server - None
ranger-admin - None
ranger-kms - None
ranger-usersync - None
slider-client - None
spark-client - None
spark-historyserver - None
spark-thriftserver - None
sqoop-client - None
sqoop-server - None
storm-client - None
storm-nimbus - None
storm-slider-client - None
storm-supervisor - None
zookeeper-client - 2.3.4.0-3485
zookeeper-server - 2.3.4.0-3485

Re: ambari 2.2.1.0 start service failed with error

divakarreddy_a — Tue, 29 Mar 2016 12:57:05 GMT

@darkz yu

Did you upgrade your HDP from 2.0.6 to HDP2.3.4?

Why your logs are shwoing "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/"

Re: ambari 2.2.1.0 start service failed with error

darkz1984 — Tue, 29 Mar 2016 13:04:12 GMT

@Divakar Annapureddy

No,I am not upgrade HDP,just test upgrade ambari,when I met the error:

https://community.hortonworks.com/questions/24208/help-ambari-agent-registe-fail-with-netutilpy77-er.html#answer-24705

and the problem is resolved now.

Then I install hdp,then met the error as I show in this topic

Re: ambari 2.2.1.0 start service failed with error

dgaodelong — Tue, 29 Mar 2016 14:07:33 GMT

do you resolved this problem ? i also has this problem

Re: ambari 2.2.1.0 start service failed with error

darkz1984 — Tue, 29 Mar 2016 14:55:20 GMT

@gor don

I have google it and find out some people have the same problem ,but no body resloved it yet.

Just wait some to tell us and I will have some try.

Re: ambari 2.2.1.0 start service failed with error

darkz1984 — Tue, 29 Mar 2016 15:13:12 GMT

@Divakar Annapuredd

@Sagar Shimpi

How to see the real command line ambari-agent execute? So I can debug the python yo find out what problem it is??

Re: ambari 2.2.1.0 start service failed with error

darkz1984 — Tue, 29 Mar 2016 15:50:15 GMT

@Sagar Shimpi @Divakar Annapureddy @gor don

I modifythe code of

/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/params.py

and find line 141:

ambari_db_rca_url = config['hostLevelParams']['ambari_db_rca_url'][0]

I add

import pprint
print "xxxxxxxxxxxxxxx"
pprint.pprint(config['hostLevelParams'])
print "xxxxxxxxxxxxxxx"

to debug the dictionary content:

{'agentCacheDir': '/var/lib/ambari-agent/cache',
 'ambari_db_rca_driver': 'com.mysql.jdbc.Driver',
 'ambari_db_rca_password': '123456',
 'ambari_db_rca_url': 'jdbc:mysql://127.0.0.1:3306/ambari',
 'ambari_db_rca_username': 'ambari',
...

So I think it call wrong,so I change

ambari_db_rca_url = config['hostLevelParams']['ambari_db_rca_url'][0]

ambari_db_rca_url = config['hostLevelParams']['ambari_db_rca_url']

then start the service is OK.

so I think is a bug of ambari-agent ??

Re: ambari 2.2.1.0 start service failed with error

darkz1984 — Tue, 29 Mar 2016 18:20:06 GMT

I have try to use an new virtual machine install the new ambari-agent version 2.2.1,but when I start server on that virtual machine,the ,og still told me :

  File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/params.py", line 141, in <module>
    ambari_db_rca_password = config['hostLevelParams']['ambari_db_rca_password'][0]
TypeError: 'int' object is unsubscriptable

So I think there must be something wrong about the ambari-agent code.

May be the basic ambari-agent python lib change lead the problem??

PS: I use the local ambari and hdp repo.

Re: ambari 2.2.1.0 start service failed with error

jeff1 — Tue, 29 Mar 2016 19:10:28 GMT

So to confirm: this is a clean install of Ambari 2.2.1.0, not an upgrade from a previous Ambari?

What OS are you using?

cc @mahadev

Re: ambari 2.2.1.0 start service failed with error

dgaodelong — Tue, 29 Mar 2016 20:23:14 GMT

My OS is centos6.5 and I not an upgrade Ambari

Re: ambari 2.2.1.0 start service failed with error

vpoornalingam — Tue, 29 Mar 2016 20:58:27 GMT

@darkz yu

Are you able to start the concerned service manually on the given node? If so, please go ahead and file a technical support case / bug. Upgrade to 2.2.1.1 if possible.

Re: ambari 2.2.1.0 start service failed with error

darkz1984 — Tue, 29 Mar 2016 21:57:21 GMT

@jeff

I once think it was the upgrade cause this problem，so I use clean virtual machine to try thi s，but problem is still the same。

My OS is centos 6.6 64 bit in virtual box，and I use my local ambari repo ，repo name is ambari 2.2.1 upgrade。When I try to install，I use yum install，an show that 2.2.1 is installed，is this the problem？

Re: ambari 2.2.1.0 start service failed with error

darkz1984 — Wed, 30 Mar 2016 15:09:52 GMT

This is the contents of my local yum repo file content :

/etc/yum.repos.d/hdp.repo 
[Ambari-2.1.0]
name=Ambari-2.1.0
baseurl=http://repo.test.yunwei/repo/ambari/centos6/
gpgcheck=0
enabled=1
#VERSION_NUMBER=2.2.1.0-161
[Updates-ambari-2.2.1.0]
name=ambari-2.2.1.0 - Updates
baseurl=http://repo.test.yunwei/repo/AMBARI-2.2.1.0/centos6/2.2.1.0-161/
gpgcheck=1
gpgkey=http://repo.test.yunwei/repo/AMBARI-2.2.1.0/centos6/2.2.1.0-161/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
[HDP-2.1]
name=HDP-2.1
baseurl=http://repo.test.yunwei/repo/HDP/centos6/2.x/GA/2.1.1.0/
path=/
enabled=1
gpgcheck=0
[HDP-UTILS-1.1.0.19]
name=HDP-UTILS-1.1.0.19
baseurl=http://repo.test.yunwei/repo/HDP-UTILS-1.1.0.17/repos/centos6/
path=/
enabled=1
#VERSION_NUMBER=2.3.4.0-3485
[HDP-2.3.4.0]
name=HDP Version - HDP-2.3.4.0
baseurl=http://repo.test.yunwei/repo/HDP/centos6/2.x/updates/2.3.4.0
#gpgcheck=1
#gpgkey=http://repo.test.yunwei/repo/HDP/centos6/2.x/updates/2.3.4.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
#priority=1
[HDP-UTILS-1.1.0.20]
name=HDP Utils Version - HDP-UTILS-1.1.0.20
baseurl=http://repo.test.yunwei/repo/HDP-UTILS-1.1.0.20/repos/centos6
#gpgcheck=1
gpgkey=http://repo.test.yunwei/repo/HDP-UTILS-1.1.0.20/repos/centos6/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
#priority=1

Re: ambari 2.2.1.0 start service failed with error

jeff1 — Wed, 30 Mar 2016 20:09:36 GMT

Would you mind posting the content of your /etc/ambari-server/conf/ambari.properties file? Also provide the output of "ambari-server --version" and "ambari-server --hash". Thx

Re: ambari 2.2.1.0 start service failed with error

darkz1984 — Wed, 30 Mar 2016 20:22:17 GMT

After I change the ambari-agent side python code of file:

"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/params.py"

as the contents I post above,I can start service on the host which the agent is installed through the ambari-server web gui.

So I think it must be a bug,but I check the same file of the agent at version 2.1,the code is the same and the dumped data structure in the memory is the same between two versions.So I don't know what differents lead the problem.

Re: ambari 2.2.1.0 start service failed with error

darkz1984 — Thu, 31 Mar 2016 09:20:22 GMT

# ambari-agent --version

2.2.1.0

# ambari-agent --hash

6a8abfa65789b87da764549c27ca0f1440b91297

Re: ambari 2.2.1.0 start service failed with error

dgaodelong — Tue, 05 Apr 2016 10:10:08 GMT

/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/params.py

How to modify params.py file ? you can start service

Re: ambari 2.2.1.0 start service failed with error

darkz1984 — Wed, 06 Apr 2016 11:26:07 GMT

I post it above