Support Questions
Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Innovation Accelerator group hub.

ambari 2.2.1.0 start service failed with error

Expert Contributor

I use ambari 2.2.1.0 and use nonroot user running ther server and the agent.

Install hdp 2.3.4.0-3485 is ok,when I try to start the service such as hdfs ,it was failed with errors,the logs content is blow...

stderr: /var/lib/ambari-agent/data/errors-69.txt
Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py", line 39, in <module>
    BeforeStartHook().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py", line 28, in hook
    import params
  File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/params.py", line 141, in <module>
    ambari_db_rca_password = config['hostLevelParams']['ambari_db_rca_password'][0]
TypeError: 'int' object is unsubscriptable

stdout:   /var/lib/ambari-agent/data/output-69.txt
2016-03-29 12:21:52,902 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.3.4.0-3485
2016-03-29 12:21:52,902 - Checking if need to create versioned conf dir /etc/hadoop/2.3.4.0-3485/0
2016-03-29 12:21:52,903 - call['conf-select create-conf-dir --package hadoop --stack-version 2.3.4.0-3485 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1}
2016-03-29 12:21:52,952 - call returned (1, '/etc/hadoop/2.3.4.0-3485/0 exist already', '')
2016-03-29 12:21:52,952 - checked_call['conf-select set-conf-dir --package hadoop --stack-version 2.3.4.0-3485 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False}
2016-03-29 12:21:52,996 - checked_call returned (0, '/usr/hdp/2.3.4.0-3485/hadoop/conf -> /etc/hadoop/2.3.4.0-3485/0')
2016-03-29 12:21:52,997 - Ensuring that hadoop has the correct symlink structure
2016-03-29 12:21:52,997 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf

I had google it for times but no result about.

1 ACCEPTED SOLUTION

Expert Contributor

I have reslove the problem.

as I posrt above ,I print the data struct

'ambari_db_rca_password':'123456',

I think it is a sting ,but when I print its type,it is a int type.

So I change the password to 'ambari',it is a sting,then every thing is goto OK!!!

Is still a problem of ambari

View solution in original post

22 REPLIES 22

@darkz yu

Was there any errors while installation ?

Can you run the below command and paste the output here -

#ls -ltr /etc/hadoop

#ls -ltr /usr/hdp/current/

#hdp-select

Thanks

Sagar

Expert Contributor

@Sagar Shimpi

the informations is there:

ls -ltr /etc/hadoop
total 8
drwxr-xr-x 2 root root 4096 Mar 28 16:07 conf.backup
drwxr-xr-x 3 root root 4096 Mar 28 16:07 2.3.4.0-3485
lrwxrwxrwx 1 root root   35 Mar 28 16:07 conf -> /usr/hdp/current/hadoop-client/conf
ls -ltr /usr/hdp/current/
total 0
lrwxrwxrwx 1 root root 27 Mar 29 11:27 atlas-server -> /usr/hdp/2.3.4.0-3485/atlas
lrwxrwxrwx 1 root root 30 Mar 29 11:27 accumulo-tracer -> /usr/hdp/2.3.4.0-3485/accumulo
lrwxrwxrwx 1 root root 30 Mar 29 11:27 accumulo-tablet -> /usr/hdp/2.3.4.0-3485/accumulo
lrwxrwxrwx 1 root root 30 Mar 29 11:27 accumulo-monitor -> /usr/hdp/2.3.4.0-3485/accumulo
lrwxrwxrwx 1 root root 30 Mar 29 11:27 accumulo-master -> /usr/hdp/2.3.4.0-3485/accumulo
lrwxrwxrwx 1 root root 30 Mar 29 11:27 accumulo-gc -> /usr/hdp/2.3.4.0-3485/accumulo
lrwxrwxrwx 1 root root 30 Mar 29 11:27 accumulo-client -> /usr/hdp/2.3.4.0-3485/accumulo
lrwxrwxrwx 1 root root 31 Mar 29 11:27 zookeeper-server -> /usr/hdp/2.3.4.0-3485/zookeeper
lrwxrwxrwx 1 root root 31 Mar 29 11:27 zookeeper-client -> /usr/hdp/2.3.4.0-3485/zookeeper
lrwxrwxrwx 1 root root 25 Mar 29 11:27 tez-client -> /usr/hdp/2.3.4.0-3485/tez
lrwxrwxrwx 1 root root 27 Mar 29 11:27 storm-supervisor -> /usr/hdp/2.3.4.0-3485/storm
lrwxrwxrwx 1 root root 41 Mar 29 11:27 storm-slider-client -> /usr/hdp/2.3.4.0-3485/storm-slider-client
lrwxrwxrwx 1 root root 27 Mar 29 11:27 storm-nimbus -> /usr/hdp/2.3.4.0-3485/storm
lrwxrwxrwx 1 root root 27 Mar 29 11:27 storm-client -> /usr/hdp/2.3.4.0-3485/storm
lrwxrwxrwx 1 root root 27 Mar 29 11:27 sqoop-server -> /usr/hdp/2.3.4.0-3485/sqoop
lrwxrwxrwx 1 root root 27 Mar 29 11:27 sqoop-client -> /usr/hdp/2.3.4.0-3485/sqoop
lrwxrwxrwx 1 root root 27 Mar 29 11:27 spark-thriftserver -> /usr/hdp/2.3.4.0-3485/spark
lrwxrwxrwx 1 root root 27 Mar 29 11:27 spark-historyserver -> /usr/hdp/2.3.4.0-3485/spark
lrwxrwxrwx 1 root root 27 Mar 29 11:27 spark-client -> /usr/hdp/2.3.4.0-3485/spark
lrwxrwxrwx 1 root root 28 Mar 29 11:27 slider-client -> /usr/hdp/2.3.4.0-3485/slider
lrwxrwxrwx 1 root root 37 Mar 29 11:27 ranger-usersync -> /usr/hdp/2.3.4.0-3485/ranger-usersync
lrwxrwxrwx 1 root root 32 Mar 29 11:27 ranger-kms -> /usr/hdp/2.3.4.0-3485/ranger-kms
lrwxrwxrwx 1 root root 34 Mar 29 11:27 ranger-admin -> /usr/hdp/2.3.4.0-3485/ranger-admin
lrwxrwxrwx 1 root root 25 Mar 29 11:27 pig-client -> /usr/hdp/2.3.4.0-3485/pig
lrwxrwxrwx 1 root root 29 Mar 29 11:27 phoenix-server -> /usr/hdp/2.3.4.0-3485/phoenix
lrwxrwxrwx 1 root root 29 Mar 29 11:27 phoenix-client -> /usr/hdp/2.3.4.0-3485/phoenix
lrwxrwxrwx 1 root root 27 Mar 29 11:27 oozie-server -> /usr/hdp/2.3.4.0-3485/oozie
lrwxrwxrwx 1 root root 27 Mar 29 11:27 oozie-client -> /usr/hdp/2.3.4.0-3485/oozie
lrwxrwxrwx 1 root root 28 Mar 29 11:27 mahout-client -> /usr/hdp/2.3.4.0-3485/mahout
lrwxrwxrwx 1 root root 26 Mar 29 11:27 knox-server -> /usr/hdp/2.3.4.0-3485/knox
lrwxrwxrwx 1 root root 27 Mar 29 11:27 kafka-broker -> /usr/hdp/2.3.4.0-3485/kafka
lrwxrwxrwx 1 root root 35 Mar 29 11:27 hive-webhcat -> /usr/hdp/2.3.4.0-3485/hive-hcatalog
lrwxrwxrwx 1 root root 26 Mar 29 11:27 hive-server2 -> /usr/hdp/2.3.4.0-3485/hive
lrwxrwxrwx 1 root root 26 Mar 29 11:27 hive-metastore -> /usr/hdp/2.3.4.0-3485/hive
lrwxrwxrwx 1 root root 26 Mar 29 11:27 hive-client -> /usr/hdp/2.3.4.0-3485/hive
lrwxrwxrwx 1 root root 27 Mar 29 11:27 hbase-regionserver -> /usr/hdp/2.3.4.0-3485/hbase
lrwxrwxrwx 1 root root 27 Mar 29 11:27 hbase-master -> /usr/hdp/2.3.4.0-3485/hbase
lrwxrwxrwx 1 root root 27 Mar 29 11:27 hbase-client -> /usr/hdp/2.3.4.0-3485/hbase
lrwxrwxrwx 1 root root 33 Mar 29 11:27 hadoop-yarn-timelineserver -> /usr/hdp/2.3.4.0-3485/hadoop-yarn
lrwxrwxrwx 1 root root 33 Mar 29 11:27 hadoop-yarn-resourcemanager -> /usr/hdp/2.3.4.0-3485/hadoop-yarn
lrwxrwxrwx 1 root root 33 Mar 29 11:27 hadoop-yarn-nodemanager -> /usr/hdp/2.3.4.0-3485/hadoop-yarn
lrwxrwxrwx 1 root root 33 Mar 29 11:27 hadoop-yarn-client -> /usr/hdp/2.3.4.0-3485/hadoop-yarn
lrwxrwxrwx 1 root root 38 Mar 29 11:27 hadoop-mapreduce-historyserver -> /usr/hdp/2.3.4.0-3485/hadoop-mapreduce
lrwxrwxrwx 1 root root 38 Mar 29 11:27 hadoop-mapreduce-client -> /usr/hdp/2.3.4.0-3485/hadoop-mapreduce
lrwxrwxrwx 1 root root 35 Mar 29 11:27 hadoop-httpfs -> /usr/hdp/2.3.4.0-3485/hadoop-httpfs
lrwxrwxrwx 1 root root 33 Mar 29 11:27 hadoop-hdfs-secondarynamenode -> /usr/hdp/2.3.4.0-3485/hadoop-hdfs
lrwxrwxrwx 1 root root 33 Mar 29 11:27 hadoop-hdfs-portmap -> /usr/hdp/2.3.4.0-3485/hadoop-hdfs
lrwxrwxrwx 1 root root 33 Mar 29 11:27 hadoop-hdfs-nfs3 -> /usr/hdp/2.3.4.0-3485/hadoop-hdfs
lrwxrwxrwx 1 root root 33 Mar 29 11:27 hadoop-hdfs-namenode -> /usr/hdp/2.3.4.0-3485/hadoop-hdfs
lrwxrwxrwx 1 root root 33 Mar 29 11:27 hadoop-hdfs-journalnode -> /usr/hdp/2.3.4.0-3485/hadoop-hdfs
lrwxrwxrwx 1 root root 33 Mar 29 11:27 hadoop-hdfs-datanode -> /usr/hdp/2.3.4.0-3485/hadoop-hdfs
lrwxrwxrwx 1 root root 33 Mar 29 11:27 hadoop-hdfs-client -> /usr/hdp/2.3.4.0-3485/hadoop-hdfs
lrwxrwxrwx 1 root root 28 Mar 29 11:27 hadoop-client -> /usr/hdp/2.3.4.0-3485/hadoop
lrwxrwxrwx 1 root root 27 Mar 29 11:27 flume-server -> /usr/hdp/2.3.4.0-3485/flume
lrwxrwxrwx 1 root root 28 Mar 29 11:27 falcon-server -> /usr/hdp/2.3.4.0-3485/falcon
lrwxrwxrwx 1 root root 28 Mar 29 11:27 falcon-client -> /usr/hdp/2.3.4.0-3485/falcon
hdp-select
accumulo-client - None
accumulo-gc - None
accumulo-master - None
accumulo-monitor - None
accumulo-tablet - None
accumulo-tracer - None
atlas-server - None
falcon-client - None
falcon-server - None
flume-server - 2.3.4.0-3485
hadoop-client - 2.3.4.0-3485
hadoop-hdfs-datanode - 2.3.4.0-3485
hadoop-hdfs-journalnode - 2.3.4.0-3485
hadoop-hdfs-namenode - 2.3.4.0-3485
hadoop-hdfs-nfs3 - 2.3.4.0-3485
hadoop-hdfs-portmap - 2.3.4.0-3485
hadoop-hdfs-secondarynamenode - 2.3.4.0-3485
hadoop-httpfs - None
hadoop-mapreduce-historyserver - 2.3.4.0-3485
hadoop-yarn-nodemanager - 2.3.4.0-3485
hadoop-yarn-resourcemanager - 2.3.4.0-3485
hadoop-yarn-timelineserver - 2.3.4.0-3485
hbase-client - None
hbase-master - None
hbase-regionserver - None
hive-metastore - None
hive-server2 - None
hive-webhcat - None
kafka-broker - None
knox-server - None
mahout-client - None
oozie-client - None
oozie-server - None
phoenix-client - None
phoenix-server - None
ranger-admin - None
ranger-kms - None
ranger-usersync - None
slider-client - None
spark-client - None
spark-historyserver - None
spark-thriftserver - None
sqoop-client - None
sqoop-server - None
storm-client - None
storm-nimbus - None
storm-slider-client - None
storm-supervisor - None
zookeeper-client - 2.3.4.0-3485
zookeeper-server - 2.3.4.0-3485

@darkz yu

Did you upgrade your HDP from 2.0.6 to HDP2.3.4?

Why your logs are shwoing "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/"

Expert Contributor

@Divakar Annapureddy

No,I am not upgrade HDP,just test upgrade ambari,when I met the error:

https://community.hortonworks.com/questions/24208/help-ambari-agent-registe-fail-with-netutilpy77-er...

and the problem is resolved now.

Then I install hdp,then met the error as I show in this topic

Explorer

do you resolved this problem ? i also has this problem

Expert Contributor
@gor don

I have google it and find out some people have the same problem ,but no body resloved it yet.

Just wait some to tell us and I will have some try.

Expert Contributor
@Divakar Annapuredd

@Sagar Shimpi

How to see the real command line ambari-agent execute? So I can debug the python yo find out what problem it is??

Expert Contributor

@Sagar Shimpi @Divakar Annapureddy @gor don

I modifythe code of

/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/params.py

and find line 141:

ambari_db_rca_url = config['hostLevelParams']['ambari_db_rca_url'][0]

I add

import pprint
print "xxxxxxxxxxxxxxx"
pprint.pprint(config['hostLevelParams'])
print "xxxxxxxxxxxxxxx"

to debug the dictionary content:

{'agentCacheDir': '/var/lib/ambari-agent/cache',
 'ambari_db_rca_driver': 'com.mysql.jdbc.Driver',
 'ambari_db_rca_password': '123456',
 'ambari_db_rca_url': 'jdbc:mysql://127.0.0.1:3306/ambari',
 'ambari_db_rca_username': 'ambari',
...

So I think it call wrong,so I change

ambari_db_rca_url = config['hostLevelParams']['ambari_db_rca_url'][0]

to

ambari_db_rca_url = config['hostLevelParams']['ambari_db_rca_url']

then start the service is OK.

so I think is a bug of ambari-agent ??

Expert Contributor

I have try to use an new virtual machine install the new ambari-agent version 2.2.1,but when I start server on that virtual machine,the ,og still told me :

  File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/params.py", line 141, in <module>
    ambari_db_rca_password = config['hostLevelParams']['ambari_db_rca_password'][0]
TypeError: 'int' object is unsubscriptable

So I think there must be something wrong about the ambari-agent code.

May be the basic ambari-agent python lib change lead the problem??

PS: I use the local ambari and hdp repo.

Guru

So to confirm: this is a clean install of Ambari 2.2.1.0, not an upgrade from a previous Ambari?

What OS are you using?

cc @mahadev

Explorer

My OS is centos6.5 and I not an upgrade Ambari

Guru

Would you mind posting the content of your /etc/ambari-server/conf/ambari.properties file? Also provide the output of "ambari-server --version" and "ambari-server --hash". Thx

Expert Contributor

# ambari-agent --version

2.2.1.0

# ambari-agent --hash

6a8abfa65789b87da764549c27ca0f1440b91297

@darkz yu

Are you able to start the concerned service manually on the given node? If so, please go ahead and file a technical support case / bug. Upgrade to 2.2.1.1 if possible.

Expert Contributor

This is the contents of my local yum repo file content :

/etc/yum.repos.d/hdp.repo 
[Ambari-2.1.0]
name=Ambari-2.1.0
baseurl=http://repo.test.yunwei/repo/ambari/centos6/
gpgcheck=0
enabled=1
#VERSION_NUMBER=2.2.1.0-161
[Updates-ambari-2.2.1.0]
name=ambari-2.2.1.0 - Updates
baseurl=http://repo.test.yunwei/repo/AMBARI-2.2.1.0/centos6/2.2.1.0-161/
gpgcheck=1
gpgkey=http://repo.test.yunwei/repo/AMBARI-2.2.1.0/centos6/2.2.1.0-161/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
[HDP-2.1]
name=HDP-2.1
baseurl=http://repo.test.yunwei/repo/HDP/centos6/2.x/GA/2.1.1.0/
path=/
enabled=1
gpgcheck=0
[HDP-UTILS-1.1.0.19]
name=HDP-UTILS-1.1.0.19
baseurl=http://repo.test.yunwei/repo/HDP-UTILS-1.1.0.17/repos/centos6/
path=/
enabled=1
#VERSION_NUMBER=2.3.4.0-3485
[HDP-2.3.4.0]
name=HDP Version - HDP-2.3.4.0
baseurl=http://repo.test.yunwei/repo/HDP/centos6/2.x/updates/2.3.4.0
#gpgcheck=1
#gpgkey=http://repo.test.yunwei/repo/HDP/centos6/2.x/updates/2.3.4.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
#priority=1
[HDP-UTILS-1.1.0.20]
name=HDP Utils Version - HDP-UTILS-1.1.0.20
baseurl=http://repo.test.yunwei/repo/HDP-UTILS-1.1.0.20/repos/centos6
#gpgcheck=1
gpgkey=http://repo.test.yunwei/repo/HDP-UTILS-1.1.0.20/repos/centos6/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
#priority=1

Expert Contributor

After I change the ambari-agent side python code of file:

"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/params.py"

as the contents I post above,I can start service on the host which the agent is installed through the ambari-server web gui.

So I think it must be a bug,but I check the same file of the agent at version 2.1,the code is the same and the dumped data structure in the memory is the same between two versions.So I don't know what differents lead the problem.

Explorer

/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/params.py

How to modify params.py file ? you can start service

Expert Contributor

I post it above

Expert Contributor

I reslove the problem ,because the password of the account which ambari to access mysql is a pure number, the code read the type to int but not a sting . change your password and every thing is ok.

Expert Contributor

@jeff

I once think it was the upgrade cause this problem,so I use clean virtual machine to try thi s,but problem is still the same。

My OS is centos 6.6 64 bit in virtual box,and I use my local ambari repo ,repo name is ambari 2.2.1 upgrade。When I try to install,I use yum install,an show that 2.2.1 is installed,is this the problem?