I have a 4 node cluster with HDP 2.3.4.0 V with ambari-agent 2.2.1.0.V. I was trying to flume twitter data in to the cluster. I have name node on machine1, flume on machine4. I have added the configuration script in ambari UI in Flume agent, but when i restart the flume service, it is throwing this error stderr: /var/lib/ambari-agent/data/errors-1102.txt
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/FLUME/1.4.0.2.0/package/scripts/flume_handler.py", line 124, in <module>
FlumeHandler().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute
method(env)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 524, in restart
self.start(env, upgrade_type=upgrade_type)
File "/var/lib/ambari-agent/cache/common-services/FLUME/1.4.0.2.0/package/scripts/flume_handler.py", line 55, in start
flume(action='start')
File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
return fn(*args, **kwargs)
File "/var/lib/ambari-agent/cache/common-services/FLUME/1.4.0.2.0/package/scripts/flume.py", line 190, in flume
try_sleep=10)
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 158, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 121, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 238, in action_run
tries=self.resource.tries, try_sleep=self.resource.try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
result = function(command, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
tries=tries, try_sleep=try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of 'ambari-sudo.sh -H -E pgrep -o -u flume -f '^/usr/jdk64/jdk1.7.0_67.*TwitterAgent.*' | ambari-sudo.sh -H -E tee /var/run/flume/TwitterAgent.pid && test ${PIPESTATUS[0]} -eq 0' returned 1.
and stdout: /var/lib/ambari-agent/data/output-1102.txt
2016-06-09 08:26:03,527 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.3.4.0-3485
2016-06-09 08:26:03,527 - Checking if need to create versioned conf dir /etc/hadoop/2.3.4.0-3485/0
2016-06-09 08:26:03,527 - call['conf-select create-conf-dir --package hadoop --stack-version 2.3.4.0-3485 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1}
2016-06-09 08:26:03,563 - call returned (1, '/etc/hadoop/2.3.4.0-3485/0 exist already', '')
2016-06-09 08:26:03,563 - checked_call['conf-select set-conf-dir --package hadoop --stack-version 2.3.4.0-3485 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False}
2016-06-09 08:26:03,596 - checked_call returned (0, '/usr/hdp/2.3.4.0-3485/hadoop/conf -> /etc/hadoop/2.3.4.0-3485/0')
2016-06-09 08:26:03,596 - Ensuring that hadoop has the correct symlink structure
2016-06-09 08:26:03,596 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2016-06-09 08:26:03,738 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.3.4.0-3485
2016-06-09 08:26:03,738 - Checking if need to create versioned conf dir /etc/hadoop/2.3.4.0-3485/0
2016-06-09 08:26:03,738 - call['conf-select create-conf-dir --package hadoop --stack-version 2.3.4.0-3485 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1}
2016-06-09 08:26:03,774 - call returned (1, '/etc/hadoop/2.3.4.0-3485/0 exist already', '')
2016-06-09 08:26:03,774 - checked_call['conf-select set-conf-dir --package hadoop --stack-version 2.3.4.0-3485 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False}
2016-06-09 08:26:03,806 - checked_call returned (0, '/usr/hdp/2.3.4.0-3485/hadoop/conf -> /etc/hadoop/2.3.4.0-3485/0')
2016-06-09 08:26:03,806 - Ensuring that hadoop has the correct symlink structure
2016-06-09 08:26:03,806 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2016-06-09 08:26:03,808 - Group['hadoop'] {}
2016-06-09 08:26:03,809 - Group['users'] {}
2016-06-09 08:26:03,809 - Group['nifi'] {}
2016-06-09 08:26:03,810 - Group['spark'] {}
2016-06-09 08:26:03,810 - User['hive'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2016-06-09 08:26:03,811 - User['oozie'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users']}
2016-06-09 08:26:03,812 - User['nifi'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2016-06-09 08:26:03,812 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users']}
2016-06-09 08:26:03,813 - User['flume'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2016-06-09 08:26:03,814 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2016-06-09 08:26:03,815 - User['spark'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2016-06-09 08:26:03,815 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2016-06-09 08:26:03,816 - User['tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users']}
2016-06-09 08:26:03,817 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2016-06-09 08:26:03,818 - User['mahout'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2016-06-09 08:26:03,818 - User['kafka'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2016-06-09 08:26:03,819 - User['sqoop'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2016-06-09 08:26:03,820 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2016-06-09 08:26:03,821 - User['hcat'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2016-06-09 08:26:03,821 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2016-06-09 08:26:03,822 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2016-06-09 08:26:03,824 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2016-06-09 08:26:03,832 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] due to not_if
2016-06-09 08:26:03,833 - Group['hdfs'] {}
2016-06-09 08:26:03,833 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'hdfs']}
2016-06-09 08:26:03,835 - Directory['/etc/hadoop'] {'mode': 0755}
2016-06-09 08:26:03,855 - File['/usr/hdp/current/hadoop-client/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2016-06-09 08:26:03,856 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 0777}
2016-06-09 08:26:03,871 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
2016-06-09 08:26:03,884 - Skipping Execute[('setenforce', '0')] due to not_if
2016-06-09 08:26:03,885 - Directory['/var/log/hadoop'] {'owner': 'root', 'mode': 0775, 'group': 'hadoop', 'recursive': True, 'cd_access': 'a'}
2016-06-09 08:26:03,891 - Directory['/var/run/hadoop'] {'owner': 'root', 'group': 'root', 'recursive': True, 'cd_access': 'a'}
2016-06-09 08:26:03,891 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'recursive': True, 'cd_access': 'a'}
2016-06-09 08:26:03,898 - File['/usr/hdp/current/hadoop-client/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'hdfs'}
2016-06-09 08:26:03,900 - File['/usr/hdp/current/hadoop-client/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'hdfs'}
2016-06-09 08:26:03,901 - File['/usr/hdp/current/hadoop-client/conf/log4j.properties'] {'content': ..., 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2016-06-09 08:26:03,912 - File['/usr/hdp/current/hadoop-client/conf/hadoop-metrics2.properties'] {'content': Template('hadoop-metrics2.properties.j2'), 'owner': 'hdfs'}
2016-06-09 08:26:03,913 - File['/usr/hdp/current/hadoop-client/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755}
2016-06-09 08:26:03,914 - File['/usr/hdp/current/hadoop-client/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'}
2016-06-09 08:26:03,919 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop'}
2016-06-09 08:26:03,926 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755}
2016-06-09 08:26:04,127 - File['/var/run/flume/ambari-state.txt'] {'content': 'INSTALLED'}
2016-06-09 08:26:04,128 - Writing File['/var/run/flume/ambari-state.txt'] because contents don't match
2016-06-09 08:26:04,129 - Pid file /var/run/flume/TwitterAgent.pid does not exist or does not contain a process id number
2016-06-09 08:26:04,129 - Pid file /var/run/flume/TwitterAgent.pid does not exist or does not contain a process id number
2016-06-09 08:26:04,130 - File['/var/run/flume/TwitterAgent.pid'] {'action': ['delete']}
2016-06-09 08:26:04,130 - Deleting File['/var/run/flume/TwitterAgent.pid']
2016-06-09 08:26:04,131 - File['/usr/hdp/current/flume-server/conf/TwitterAgent/ambari-meta.json'] {'action': ['delete']}
2016-06-09 08:26:04,132 - Deleting File['/usr/hdp/current/flume-server/conf/TwitterAgent/ambari-meta.json']
2016-06-09 08:26:04,132 - Directory['/var/run/flume'] {}
2016-06-09 08:26:04,132 - Directory['/usr/hdp/current/flume-server/conf'] {'owner': 'flume', 'recursive': True}
2016-06-09 08:26:04,132 - Directory['/var/log/flume'] {'owner': 'flume', 'mode': 0755, 'cd_access': 'a'}
2016-06-09 08:26:04,133 - Directory['/usr/hdp/current/flume-server/conf/TwitterAgent'] {'owner': 'flume'}
2016-06-09 08:26:04,133 - PropertiesFile['/usr/hdp/current/flume-server/conf/TwitterAgent/flume.conf'] {'owner': 'flume', 'mode': 0644, 'properties': ...}
2016-06-09 08:26:04,142 - Generating properties file: /usr/hdp/current/flume-server/conf/TwitterAgent/flume.conf
2016-06-09 08:26:04,142 - File['/usr/hdp/current/flume-server/conf/TwitterAgent/flume.conf'] {'owner': 'flume', 'content': InlineTemplate(...), 'group': None, 'mode': 0644}
2016-06-09 08:26:04,156 - Writing File['/usr/hdp/current/flume-server/conf/TwitterAgent/flume.conf'] because contents don't match
2016-06-09 08:26:04,160 - File['/usr/hdp/current/flume-server/conf/TwitterAgent/log4j.properties'] {'owner': 'flume', 'content': Template('log4j.properties.j2'), 'mode': 0644}
2016-06-09 08:26:04,161 - File['/usr/hdp/current/flume-server/conf/TwitterAgent/ambari-meta.json'] {'owner': 'flume', 'content': '{"channels_count": 1, "sinks_count": 1, "sources_count": 1}', 'mode': 0644}
2016-06-09 08:26:04,161 - Writing File['/usr/hdp/current/flume-server/conf/TwitterAgent/ambari-meta.json'] because it doesn't exist
2016-06-09 08:26:04,161 - Changing owner for /usr/hdp/current/flume-server/conf/TwitterAgent/ambari-meta.json from 0 to flume
2016-06-09 08:26:04,163 - File['/usr/hdp/current/flume-server/conf/TwitterAgent/flume-env.sh'] {'content': InlineTemplate(...), 'owner': 'flume'}
2016-06-09 08:26:04,165 - File['/usr/hdp/current/flume-server/conf/TwitterAgent/flume-metrics2.properties'] {'content': Template('flume-metrics2.properties.j2'), 'owner': 'flume'}
2016-06-09 08:26:04,166 - File['/var/run/flume/ambari-state.txt'] {'content': 'STARTED'}
2016-06-09 08:26:04,166 - Writing File['/var/run/flume/ambari-state.txt'] because contents don't match
2016-06-09 08:26:04,167 - Pid file /var/run/flume/TwitterAgent.pid is empty or does not exist
2016-06-09 08:26:04,167 - Execute['ambari-sudo.sh su flume -l -s /bin/bash -c 'export PATH='"'"'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/var/lib/ambari-agent'"'"' JAVA_HOME=/usr/jdk64/jdk1.7.0_67 ; /usr/hdp/current/flume-server/bin/flume-ng agent --name TwitterAgent --conf /usr/hdp/current/flume-server/conf/TwitterAgent --conf-file /usr/hdp/current/flume-server/conf/TwitterAgent/flume.conf -Dflume.monitoring.type=org.apache.hadoop.metrics2.sink.flume.FlumeTimelineMetricsSink -Dflume.monitoring.node=rkdatavm1.ddns.net:6188 > /var/log/flume/TwitterAgent.out 2>&1' &'] {'environment': {'JAVA_HOME': '/usr/jdk64/jdk1.7.0_67'}, 'wait_for_finish': False}
2016-06-09 08:26:04,174 - Execute['ambari-sudo.sh -H -E pgrep -o -u flume -f '^/usr/jdk64/jdk1.7.0_67.*TwitterAgent.*' | ambari-sudo.sh -H -E tee /var/run/flume/TwitterAgent.pid && test ${PIPESTATUS[0]} -eq 0'] {'logoutput': True, 'tries': 20, 'try_sleep': 10}
2016-06-09 08:26:04,185 - Retrying after 10 seconds. Reason: Execution of 'ambari-sudo.sh -H -E pgrep -o -u flume -f '^/usr/jdk64/jdk1.7.0_67.*TwitterAgent.*' | ambari-sudo.sh -H -E tee /var/run/flume/TwitterAgent.pid && test ${PIPESTATUS[0]} -eq 0' returned 1.
2016-06-09 08:26:14,207 - Retrying after 10 seconds. Reason: Execution of 'ambari-sudo.sh -H -E pgrep -o -u flume -f '^/usr/jdk64/jdk1.7.0_67.*TwitterAgent.*' | ambari-sudo.sh -H -E tee /var/run/flume/TwitterAgent.pid && test ${PIPESTATUS[0]} -eq 0' returned 1.
2016-06-09 08:26:24,225 - Retrying after 10 seconds. Reason: Execution of 'ambari-sudo.sh -H -E pgrep -o -u flume -f '^/usr/jdk64/jdk1.7.0_67.*TwitterAgent.*' | ambari-sudo.sh -H -E tee /var/run/flume/TwitterAgent.pid && test ${PIPESTATUS[0]} -eq 0' returned 1.
2016-06-09 08:26:34,248 - Retrying after 10 seconds. Reason: Execution of 'ambari-sudo.sh -H -E pgrep -o -u flume -f '^/usr/jdk64/jdk1.7.0_67.*TwitterAgent.*' | ambari-sudo.sh -H -E tee /var/run/flume/TwitterAgent.pid && test ${PIPESTATUS[0]} -eq 0' returned 1.
2016-06-09 08:26:44,266 - Retrying after 10 seconds. Reason: Execution of 'ambari-sudo.sh -H -E pgrep -o -u flume -f '^/usr/jdk64/jdk1.7.0_67.*TwitterAgent.*' | ambari-sudo.sh -H -E tee /var/run/flume/TwitterAgent.pid && test ${PIPESTATUS[0]} -eq 0' returned 1.
2016-06-09 08:26:54,288 - Retrying after 10 seconds. Reason: Execution of 'ambari-sudo.sh -H -E pgrep -o -u flume -f '^/usr/jdk64/jdk1.7.0_67.*TwitterAgent.*' | ambari-sudo.sh -H -E tee /var/run/flume/TwitterAgent.pid && test ${PIPESTATUS[0]} -eq 0' returned 1.
2016-06-09 08:27:04,310 - Retrying after 10 seconds. Reason: Execution of 'ambari-sudo.sh -H -E pgrep -o -u flume -f '^/usr/jdk64/jdk1.7.0_67.*TwitterAgent.*' | ambari-sudo.sh -H -E tee /var/run/flume/TwitterAgent.pid && test ${PIPESTATUS[0]} -eq 0' returned 1.
2016-06-09 08:27:14,326 - Retrying after 10 seconds. Reason: Execution of 'ambari-sudo.sh -H -E pgrep -o -u flume -f '^/usr/jdk64/jdk1.7.0_67.*TwitterAgent.*' | ambari-sudo.sh -H -E tee /var/run/flume/TwitterAgent.pid && test ${PIPESTATUS[0]} -eq 0' returned 1.
2016-06-09 08:27:24,347 - Retrying after 10 seconds. Reason: Execution of 'ambari-sudo.sh -H -E pgrep -o -u flume -f '^/usr/jdk64/jdk1.7.0_67.*TwitterAgent.*' | ambari-sudo.sh -H -E tee /var/run/flume/TwitterAgent.pid && test ${PIPESTATUS[0]} -eq 0' returned 1.
Please anyone help me sort out this problem.
... View more
