Member since
02-16-2016
20
Posts
10
Kudos Received
0
Solutions
04-09-2018
10:30 AM
Emil, there is a very simple workaround. On each node edit /usr/lib/python2.6/site-packages/ambari_agent/main.py and comment out the line: fix_subprocess_popen() I have been running without that with no issues
... View more
11-09-2017
09:59 AM
If that's how it is I wonder what is the criteria to publish or not. I would rather if all the official releases were also published, like it happens with all the other artifacts. So if I'm using 2.5.1.0.159 in prod, I would like to link to exactly those build libraries, not a vanilla 2.5.1 built by myself and published on my own repository so my build server can reach it. Thanks again for your assistance.
... View more
11-09-2017
08:59 AM
I appreciate your help but the question is not fully answered. 2.5.1, 2.5.2 and 2.6.0 have been released but are not there. Is this a mistake or they won't be published anymore? Hopefully somebody from Hortonworks can answer.
... View more
11-08-2017
03:40 PM
Wasn't aware of that repo. For some reason the latest they have is Ambari 2.5.0 but it's better that nothing. Thanks
... View more
11-08-2017
02:35 PM
I have been using this repository when building my own Ambari views: http://repo.hortonworks.com/content/repositories/releases/org/apache/ambari/ambari-views/ But now I'm trying to upgrade to Ambari 2.5 or later and I see they have been not published there. Does anybody know where I can find the Hortonworks Ambari artifacts published as a repository (not as tars)?
... View more
Labels:
- Labels:
-
Apache Ambari
11-08-2017
02:30 PM
In Ambari you cannot pick and choose, either the whole cluster uses kerberos or not and that trigger a number of configuration and keytab creation done automatically. The only non hacky solution I see is to have a separate ambari cluster for kafka but it's not ideal since each node can only belong to one cluster
... View more
07-26-2017
07:32 AM
Since the server started and finished startup, there was only one message printed every 5 minutes: 25 Jul 2017 14:12:22,785 INFO [pool-18-thread-1] MetricsServiceImpl:64 - Checking for metrics sink initialization
However, since I restarted now is gone and so far the heartbeat is fine. Maybe is a coincidence.
I'm thinking, I don't know when the heartbeat was lost since it's only noticeable if you go to the hosts or the specific host screen, it doesn't show in the main screen because the services are fine
... View more
07-25-2017
02:47 PM
I have never seen this before upgrading to 2.5.1 A cluster of 6 nodes, after 2 weeks running, one of the hosts is listed as having lost the heartbeat.
The agent is reporting metrics and all the components are running fine without alerts. It's only that the node actions are disabled. However looking at the agents log, the heartbeat seems to be running normally and continuously, e.g: INFO 2017-07-25 01:55:31,110 Controller.py:304 - Heartbeat (response id = 621881) with server is running...
INFO 2017-07-25 01:55:31,110 Controller.py:311 - Building heartbeat message
INFO 2017-07-25 01:55:31,112 Heartbeat.py:90 - Adding host info/state to heartbeat message.
INFO 2017-07-25 01:55:31,163 logger.py:75 - Testing the JVM's JCE policy to see it if supports an unlimited key length.
INFO 2017-07-25 01:55:31,163 logger.py:75 - Testing the JVM's JCE policy to see it if supports an unlimited key length.
INFO 2017-07-25 01:55:31,289 Hardware.py:176 - Some mount points were ignored: /, /dev, /dev/shm, /sys/fs/cgroup, /run, /boot, /var/log, /hadoop, /hadoop/druid, /hadoop/yarn/local, /run/user/1017, /run/user/1006, /run/user/1002, /run/user/1003
INFO 2017-07-25 01:55:31,291 Controller.py:320 - Sending Heartbeat (id = 621881)
INFO 2017-07-25 01:55:31,335 Controller.py:332 - Heartbeat response received (id = 621882)
INFO 2017-07-25 01:55:31,336 Controller.py:341 - Heartbeat interval is 1 seconds
INFO 2017-07-25 01:55:31,336 Controller.py:377 - Updating configurations from heartbeat
INFO 2017-07-25 01:55:31,336 Controller.py:386 - Adding cancel/execution commands
INFO 2017-07-25 01:55:31,336 Controller.py:471 - Waiting 0.9 for next heartbeat
INFO 2017-07-25 01:55:32,236 Controller.py:478 - Wait for next heartbeat over Both server and agent are clean installations and all the ambari packages are on the same version: 2.5.1.0-159 Restarting the server solved it without having to restart the agent. Has anybody seen this behavior? So far has only happened once but I have only recently started using 2.5.1
... View more
Labels:
- Labels:
-
Apache Ambari
07-18-2017
07:59 AM
Now in 2.6, I understand Hive LLAP is production ready but it still only allows a single HSI which becomes a SPOF
There was a patch contributed but it was never accepted: https://issues.apache.org/jira/browse/AMBARI-18917
Do you have any information about why something as important yet apparently trivial hasn't been addressed?
... View more
07-10-2017
08:43 AM
I'm evaluating the recently released 2.5.1 to consider an upgrade. However, I found that in a period ranging from a few minutes to a few hours, the ambari agents one by one lose the heartbeat until eventually all are in a zombie state.
The process are running find and there is no sign of errors on the logs, it just stops in the middle of its routine checks. Doing a thread dump, I got that consistently the issue is that a fork call doesn't return for some reason and the rest of the thread are waiting for the lock that fork call holds (looking that the code that synchronization is something new in 2.5).
The only solution is to restart ambari-agent, the script after a way needs to kill -9 the process since it doesn't respond to the stop signal.
I'm running on VMs which might be more error prone to racing issues. I'm using a couple of homemade components but they have been working fine for a year with Ambari 2.4, I don't see how they could affect the fork call not returning. Following a sample thread dump of the deadlock. *** STACKTRACE - START ***
# ThreadID: 139794667529984
File: "/usr/lib64/python2.7/threading.py", line 784, in __bootstrap
self.__bootstrap_inner()
File: "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
self.run()
File: "/usr/lib64/python2.7/threading.py", line 764, in run
self.__target(*self.__args, **self.__kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/threadpool.py", line 95, in _run_jobs
func(*args, **kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/scheduler.py", line 512, in _run_job
retval = job.func(*job.args, **job.kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/AlertSchedulerHandler.py", line 155, in <lambda>
return lambda: alert_def.collect()
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/base_alert.py", line 112, in collect
res = self._collect()
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/metric_alert.py", line 97, in _collect
jmx_property_values, http_code = self._load_jmx(alert_uri.is_ssl_enabled, host, port, self.metric_info)
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/metric_alert.py", line 212, in _load_jmx
connection_timeout=self.curl_connection_timeout, kinit_timer_ms = self.kinit_timeout)
File: "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/curl_krb_request.py", line 94, in curl_krb_request
import uuid
# ThreadID: 139794650744576
File: "/usr/lib64/python2.7/threading.py", line 784, in __bootstrap
self.__bootstrap_inner()
File: "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
self.run()
File: "/usr/lib64/python2.7/threading.py", line 764, in run
self.__target(*self.__args, **self.__kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/threadpool.py", line 95, in _run_jobs
func(*args, **kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/scheduler.py", line 512, in _run_job
retval = job.func(*job.args, **job.kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/AlertSchedulerHandler.py", line 155, in <lambda>
return lambda: alert_def.collect()
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/base_alert.py", line 112, in collect
res = self._collect()
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/metric_alert.py", line 97, in _collect
jmx_property_values, http_code = self._load_jmx(alert_uri.is_ssl_enabled, host, port, self.metric_info)
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/metric_alert.py", line 212, in _load_jmx
connection_timeout=self.curl_connection_timeout, kinit_timer_ms = self.kinit_timeout)
File: "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/curl_krb_request.py", line 94, in curl_krb_request
import uuid
# ThreadID: 139795212793600
File: "/usr/lib64/python2.7/threading.py", line 784, in __bootstrap
self.__bootstrap_inner()
File: "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
self.run()
File: "/usr/lib/python2.6/site-packages/ambari_agent/Controller.py", line 497, in run
self.registerAndHeartbeat()
File: "/usr/lib/python2.6/site-packages/ambari_agent/Controller.py", line 525, in registerAndHeartbeat
self.heartbeatWithServer()
File: "/usr/lib/python2.6/site-packages/ambari_agent/Controller.py", line 313, in heartbeatWithServer
data = json.dumps(self.heartbeat.build(self.responseId, send_state, self.hasMappedComponents))
File: "/usr/lib/python2.6/site-packages/ambari_agent/Heartbeat.py", line 46, in build
queueResult = self.actionQueue.result()
File: "/usr/lib/python2.6/site-packages/ambari_agent/ActionQueue.py", line 571, in result
return self.commandStatuses.generate_report()
File: "/usr/lib/python2.6/site-packages/ambari_agent/CommandStatusDict.py", line 88, in generate_report
from ActionQueue import ActionQueue
# ThreadID: 139795162437376
File: "/usr/lib64/python2.7/threading.py", line 784, in __bootstrap
self.__bootstrap_inner()
File: "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
self.run()
File: "/usr/lib64/python2.7/threading.py", line 764, in run
self.__target(*self.__args, **self.__kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/threadpool.py", line 95, in _run_jobs
func(*args, **kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/scheduler.py", line 512, in _run_job
retval = job.func(*job.args, **job.kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/AlertSchedulerHandler.py", line 155, in <lambda>
return lambda: alert_def.collect()
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/base_alert.py", line 112, in collect
res = self._collect()
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/script_alert.py", line 90, in _collect
cmd_module = self._load_source()
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/script_alert.py", line 172, in _load_source
return imp.load_source(self._get_alert_meta_value_safely('name'), self.path_to_script)
File: "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/alerts/alert_ambari_metrics_monitor.py", line 21, in <module>
import os
# ThreadID: 139795435849472
File: "/usr/lib64/python2.7/threading.py", line 784, in __bootstrap
self.__bootstrap_inner()
File: "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
self.run()
File: "/usr/lib/python2.6/site-packages/ambari_agent/PingPortListener.py", line 67, in run
conn, addr = self.socket.accept()
File: "/usr/lib64/python2.7/socket.py", line 202, in accept
sock, addr = self._sock.accept()
# ThreadID: 139795179222784
File: "/usr/lib64/python2.7/threading.py", line 784, in __bootstrap
self.__bootstrap_inner()
File: "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
self.run()
File: "/usr/lib64/python2.7/threading.py", line 764, in run
self.__target(*self.__args, **self.__kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/threadpool.py", line 95, in _run_jobs
func(*args, **kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/scheduler.py", line 512, in _run_job
retval = job.func(*job.args, **job.kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/AlertSchedulerHandler.py", line 155, in <lambda>
return lambda: alert_def.collect()
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/base_alert.py", line 112, in collect
res = self._collect()
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/script_alert.py", line 115, in _collect
result = cmd_module.execute(configurations, self.parameters, self.host_name)
File: "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/alerts/alert_checkpoint_time.py", line 189, in execute
kinit_timer_ms = kinit_timer_ms)
File: "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/curl_krb_request.py", line 122, in curl_krb_request
kinit_lock.acquire()
File: "/usr/lib64/python2.7/threading.py", line 173, in acquire
rc = self.__block.acquire(blocking)
# ThreadID: 139794625566464
File: "/usr/lib64/python2.7/threading.py", line 784, in __bootstrap
self.__bootstrap_inner()
File: "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
self.run()
File: "/usr/lib64/python2.7/threading.py", line 764, in run
self.__target(*self.__args, **self.__kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/threadpool.py", line 95, in _run_jobs
func(*args, **kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/scheduler.py", line 512, in _run_job
retval = job.func(*job.args, **job.kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/AlertSchedulerHandler.py", line 155, in <lambda>
return lambda: alert_def.collect()
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/base_alert.py", line 112, in collect
res = self._collect()
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/web_alert.py", line 102, in _collect
web_response = self._make_web_request(url)
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/web_alert.py", line 201, in _make_web_request
connection_timeout=self.curl_connection_timeout, kinit_timer_ms = self.kinit_timeout)
File: "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/curl_krb_request.py", line 94, in curl_krb_request
import uuid
# ThreadID: 139795423160064
File: "/usr/lib64/python2.7/threading.py", line 784, in __bootstrap
self.__bootstrap_inner()
File: "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
self.run()
File: "/usr/lib64/python2.7/threading.py", line 764, in run
self.__target(*self.__args, **self.__kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/scheduler.py", line 590, in _main_loop
self._wakeup.wait(wait_seconds)
File: "/usr/lib64/python2.7/threading.py", line 621, in wait
self.__cond.wait(timeout, balancing)
File: "/usr/lib64/python2.7/threading.py", line 361, in wait
_sleep(delay)
# ThreadID: 139795170830080
File: "/usr/lib64/python2.7/threading.py", line 784, in __bootstrap
self.__bootstrap_inner()
File: "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
self.run()
File: "/usr/lib64/python2.7/threading.py", line 764, in run
self.__target(*self.__args, **self.__kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/threadpool.py", line 95, in _run_jobs
func(*args, **kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/scheduler.py", line 512, in _run_job
retval = job.func(*job.args, **job.kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/AlertSchedulerHandler.py", line 155, in <lambda>
return lambda: alert_def.collect()
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/base_alert.py", line 112, in collect
res = self._collect()
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/metric_alert.py", line 97, in _collect
jmx_property_values, http_code = self._load_jmx(alert_uri.is_ssl_enabled, host, port, self.metric_info)
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/metric_alert.py", line 212, in _load_jmx
connection_timeout=self.curl_connection_timeout, kinit_timer_ms = self.kinit_timeout)
File: "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/curl_krb_request.py", line 122, in curl_krb_request
kinit_lock.acquire()
File: "/usr/lib64/python2.7/threading.py", line 173, in acquire
rc = self.__block.acquire(blocking)
# ThreadID: 139794105480960
File: "/usr/lib64/python2.7/threading.py", line 784, in __bootstrap
self.__bootstrap_inner()
File: "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
self.run()
File: "/usr/lib64/python2.7/threading.py", line 764, in run
self.__target(*self.__args, **self.__kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/threadpool.py", line 95, in _run_jobs
func(*args, **kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/scheduler.py", line 512, in _run_job
retval = job.func(*job.args, **job.kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/AlertSchedulerHandler.py", line 155, in <lambda>
return lambda: alert_def.collect()
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/base_alert.py", line 112, in collect
res = self._collect()
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/web_alert.py", line 102, in _collect
web_response = self._make_web_request(url)
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/web_alert.py", line 201, in _make_web_request
connection_timeout=self.curl_connection_timeout, kinit_timer_ms = self.kinit_timeout)
File: "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/curl_krb_request.py", line 94, in curl_krb_request
import uuid
# ThreadID: 139795187615488
File: "/usr/lib64/python2.7/threading.py", line 784, in __bootstrap
self.__bootstrap_inner()
File: "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
self.run()
File: "/usr/lib64/python2.7/threading.py", line 764, in run
self.__target(*self.__args, **self.__kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/threadpool.py", line 95, in _run_jobs
func(*args, **kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/scheduler.py", line 512, in _run_job
retval = job.func(*job.args, **job.kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/AlertSchedulerHandler.py", line 155, in <lambda>
return lambda: alert_def.collect()
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/base_alert.py", line 112, in collect
res = self._collect()
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/web_alert.py", line 102, in _collect
web_response = self._make_web_request(url)
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/web_alert.py", line 201, in _make_web_request
connection_timeout=self.curl_connection_timeout, kinit_timer_ms = self.kinit_timeout)
File: "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/curl_krb_request.py", line 94, in curl_krb_request
import uuid
# ThreadID: 139793585395456
File: "/usr/lib64/python2.7/threading.py", line 784, in __bootstrap
self.__bootstrap_inner()
File: "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
self.run()
File: "/usr/lib64/python2.7/threading.py", line 764, in run
self.__target(*self.__args, **self.__kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/threadpool.py", line 95, in _run_jobs
func(*args, **kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/scheduler.py", line 512, in _run_job
retval = job.func(*job.args, **job.kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/AlertSchedulerHandler.py", line 155, in <lambda>
return lambda: alert_def.collect()
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/base_alert.py", line 112, in collect
res = self._collect()
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/web_alert.py", line 102, in _collect
web_response = self._make_web_request(url)
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/web_alert.py", line 201, in _make_web_request
connection_timeout=self.curl_connection_timeout, kinit_timer_ms = self.kinit_timeout)
File: "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/curl_krb_request.py", line 122, in curl_krb_request
kinit_lock.acquire()
File: "/usr/lib64/python2.7/threading.py", line 173, in acquire
rc = self.__block.acquire(blocking)
# ThreadID: 139793593788160
File: "/usr/lib64/python2.7/threading.py", line 784, in __bootstrap
self.__bootstrap_inner()
File: "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
self.run()
File: "/usr/lib64/python2.7/threading.py", line 764, in run
self.__target(*self.__args, **self.__kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/threadpool.py", line 95, in _run_jobs
func(*args, **kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/scheduler.py", line 512, in _run_job
retval = job.func(*job.args, **job.kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/AlertSchedulerHandler.py", line 155, in <lambda>
return lambda: alert_def.collect()
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/base_alert.py", line 112, in collect
res = self._collect()
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/metric_alert.py", line 97, in _collect
jmx_property_values, http_code = self._load_jmx(alert_uri.is_ssl_enabled, host, port, self.metric_info)
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/metric_alert.py", line 216, in _load_jmx
url_opener = urllib2.build_opener(RefreshHeaderProcessor())
File: "/usr/lib64/python2.7/urllib2.py", line 490, in build_opener
import types
# ThreadID: 139794642351872
File: "/usr/lib64/python2.7/threading.py", line 784, in __bootstrap
self.__bootstrap_inner()
File: "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
self.run()
File: "/usr/lib64/python2.7/threading.py", line 764, in run
self.__target(*self.__args, **self.__kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/threadpool.py", line 95, in _run_jobs
func(*args, **kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/scheduler.py", line 512, in _run_job
retval = job.func(*job.args, **job.kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/AlertSchedulerHandler.py", line 155, in <lambda>
return lambda: alert_def.collect()
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/base_alert.py", line 112, in collect
res = self._collect()
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/script_alert.py", line 115, in _collect
result = cmd_module.execute(configurations, self.parameters, self.host_name)
File: "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/alerts/alert_nodemanagers_summary.py", line 138, in execute
kinit_timer_ms = kinit_timer_ms)
File: "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/curl_krb_request.py", line 142, in curl_krb_request
is_kinit_required = (shell.call(klist_command, user=user)[0] != 0)
File: "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 72, in inner
result = function(command, **kwargs)
File: "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 115, in call
tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
File: "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 150, in _call_wrapper
result = _call(command, **kwargs_copy)
File: "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 223, in _call
preexec_fn=preexec_fn)
File: "/usr/lib/python2.6/site-packages/ambari_agent/main.py", line 66, in sp_locked_init
sp_original_init(self, *a, **kw)
File: "/usr/lib64/python2.7/subprocess.py", line 711, in __init__
errread, errwrite)
File: "/usr/lib64/python2.7/subprocess.py", line 1224, in _execute_child
self.pid = os.fork()
# ThreadID: 139794633959168
File: "/usr/lib64/python2.7/threading.py", line 784, in __bootstrap
self.__bootstrap_inner()
File: "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
self.run()
File: "/usr/lib64/python2.7/threading.py", line 764, in run
self.__target(*self.__args, **self.__kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/threadpool.py", line 95, in _run_jobs
func(*args, **kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/scheduler.py", line 512, in _run_job
retval = job.func(*job.args, **job.kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/AlertSchedulerHandler.py", line 155, in <lambda>
return lambda: alert_def.collect()
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/base_alert.py", line 112, in collect
res = self._collect()
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/script_alert.py", line 115, in _collect
result = cmd_module.execute(configurations, self.parameters, self.host_name)
File: "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/alerts/alert_nodemanager_health.py", line 166, in execute
connection_timeout=curl_connection_timeout, kinit_timer_ms = kinit_timer_ms)
File: "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/curl_krb_request.py", line 209, in curl_krb_request
_, curl_stdout, curl_stderr = get_user_call_output(curl_command, user=user, env=kerberos_env)
File: "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/get_user_call_output.py", line 50, in get_user_call_output
code, _ = shell.call(shell.as_user(command_string, user), quiet=quiet, **call_kwargs)
File: "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 61, in inner
Logger.info(log_msg)
File: "/usr/lib/python2.6/site-packages/resource_management/core/logger.py", line 75, in info
Logger.logger.info(Logger.filter_text(text))
File: "/usr/lib/python2.6/site-packages/resource_management/core/logger.py", line 102, in filter_text
from resource_management.core.shell import PLACEHOLDERS_TO_STR
# ThreadID: 139795196008192
File: "/usr/lib64/python2.7/threading.py", line 784, in __bootstrap
self.__bootstrap_inner()
File: "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
self.run()
File: "/usr/lib64/python2.7/threading.py", line 764, in run
self.__target(*self.__args, **self.__kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/threadpool.py", line 95, in _run_jobs
func(*args, **kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/scheduler.py", line 512, in _run_job
retval = job.func(*job.args, **job.kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/AlertSchedulerHandler.py", line 155, in <lambda>
return lambda: alert_def.collect()
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/base_alert.py", line 112, in collect
res = self._collect()
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/script_alert.py", line 115, in _collect
result = cmd_module.execute(configurations, self.parameters, self.host_name)
File: "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/alerts/alert_webhcat_server.py", line 160, in execute
kinit_timer_ms = kinit_timer_ms)
File: "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/curl_krb_request.py", line 122, in curl_krb_request
kinit_lock.acquire()
File: "/usr/lib64/python2.7/threading.py", line 173, in acquire
rc = self.__block.acquire(blocking)
# ThreadID: 139794675922688
File: "/usr/lib64/python2.7/threading.py", line 784, in __bootstrap
self.__bootstrap_inner()
File: "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
self.run()
File: "/usr/lib64/python2.7/threading.py", line 764, in run
self.__target(*self.__args, **self.__kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/threadpool.py", line 95, in _run_jobs
func(*args, **kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/scheduler.py", line 512, in _run_job
retval = job.func(*job.args, **job.kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/AlertSchedulerHandler.py", line 155, in <lambda>
return lambda: alert_def.collect()
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/base_alert.py", line 112, in collect
res = self._collect()
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/metric_alert.py", line 97, in _collect
jmx_property_values, http_code = self._load_jmx(alert_uri.is_ssl_enabled, host, port, self.metric_info)
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/metric_alert.py", line 212, in _load_jmx
connection_timeout=self.curl_connection_timeout, kinit_timer_ms = self.kinit_timeout)
File: "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/curl_krb_request.py", line 122, in curl_krb_request
kinit_lock.acquire()
File: "/usr/lib64/python2.7/threading.py", line 173, in acquire
rc = self.__block.acquire(blocking)
# ThreadID: 139795680462656
File: "/usr/lib/python2.6/site-packages/ambari_agent/main.py", line 472, in <module>
main(heartbeat_stop_callback)
File: "/usr/lib/python2.6/site-packages/ambari_agent/main.py", line 451, in main
run_threads(server_hostname, heartbeat_stop_callback)
File: "/usr/lib/python2.6/site-packages/ambari_agent/main.py", line 335, in run_threads
time.sleep(0.1)
File: "/usr/lib/python2.6/site-packages/ambari_agent/RemoteDebugUtils.py", line 35, in print_threads_stack_traces
for filename, lineno, name, line in traceback.extract_stack(stack):
# ThreadID: 139794097088256
File: "/usr/lib64/python2.7/threading.py", line 784, in __bootstrap
self.__bootstrap_inner()
File: "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
self.run()
File: "/usr/lib64/python2.7/threading.py", line 764, in run
self.__target(*self.__args, **self.__kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/threadpool.py", line 95, in _run_jobs
func(*args, **kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/scheduler.py", line 512, in _run_job
retval = job.func(*job.args, **job.kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/AlertSchedulerHandler.py", line 155, in <lambda>
return lambda: alert_def.collect()
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/base_alert.py", line 112, in collect
res = self._collect()
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/web_alert.py", line 102, in _collect
web_response = self._make_web_request(url)
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/web_alert.py", line 201, in _make_web_request
connection_timeout=self.curl_connection_timeout, kinit_timer_ms = self.kinit_timeout)
File: "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/curl_krb_request.py", line 122, in curl_krb_request
kinit_lock.acquire()
File: "/usr/lib64/python2.7/threading.py", line 173, in acquire
rc = self.__block.acquire(blocking)
# ThreadID: 139794113873664
File: "/usr/lib64/python2.7/threading.py", line 784, in __bootstrap
self.__bootstrap_inner()
File: "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
self.run()
File: "/usr/lib64/python2.7/threading.py", line 764, in run
self.__target(*self.__args, **self.__kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/threadpool.py", line 95, in _run_jobs
func(*args, **kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/scheduler.py", line 512, in _run_job
retval = job.func(*job.args, **job.kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/AlertSchedulerHandler.py", line 155, in <lambda>
return lambda: alert_def.collect()
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/base_alert.py", line 112, in collect
res = self._collect()
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/script_alert.py", line 115, in _collect
result = cmd_module.execute(configurations, self.parameters, self.host_name)
File: "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/alerts/alert_upgrade_finalized.py", line 132, in execute
"HDFS Upgrade Finalized State", smokeuser, kinit_timer_ms = kinit_timer_ms
File: "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/curl_krb_request.py", line 122, in curl_krb_request
kinit_lock.acquire()
File: "/usr/lib64/python2.7/threading.py", line 173, in acquire
rc = self.__block.acquire(blocking)
# ThreadID: 139795444242176
File: "/usr/lib64/python2.7/threading.py", line 784, in __bootstrap
self.__bootstrap_inner()
File: "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
self.run()
File: "/usr/lib/python2.6/site-packages/ambari_agent/DataCleaner.py", line 123, in run
time.sleep(self.cleanup_interval)
# ThreadID: 139794139051776
File: "/usr/lib64/python2.7/threading.py", line 784, in __bootstrap
self.__bootstrap_inner()
File: "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
self.run()
File: "/usr/lib64/python2.7/threading.py", line 764, in run
self.__target(*self.__args, **self.__kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/threadpool.py", line 95, in _run_jobs
func(*args, **kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/scheduler.py", line 512, in _run_job
retval = job.func(*job.args, **job.kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/AlertSchedulerHandler.py", line 155, in <lambda>
return lambda: alert_def.collect()
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/base_alert.py", line 112, in collect
res = self._collect()
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/web_alert.py", line 102, in _collect
web_response = self._make_web_request(url)
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/web_alert.py", line 201, in _make_web_request
connection_timeout=self.curl_connection_timeout, kinit_timer_ms = self.kinit_timeout)
File: "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/curl_krb_request.py", line 122, in curl_krb_request
kinit_lock.acquire()
File: "/usr/lib64/python2.7/threading.py", line 173, in acquire
rc = self.__block.acquire(blocking)
# ThreadID: 139794659137280
File: "/usr/lib64/python2.7/threading.py", line 784, in __bootstrap
self.__bootstrap_inner()
File: "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
self.run()
File: "/usr/lib64/python2.7/threading.py", line 764, in run
self.__target(*self.__args, **self.__kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/threadpool.py", line 95, in _run_jobs
func(*args, **kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/scheduler.py", line 512, in _run_job
retval = job.func(*job.args, **job.kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/AlertSchedulerHandler.py", line 155, in <lambda>
return lambda: alert_def.collect()
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/base_alert.py", line 112, in collect
res = self._collect()
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/web_alert.py", line 102, in _collect
web_response = self._make_web_request(url)
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/web_alert.py", line 201, in _make_web_request
connection_timeout=self.curl_connection_timeout, kinit_timer_ms = self.kinit_timeout)
File: "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/curl_krb_request.py", line 122, in curl_krb_request
kinit_lock.acquire()
File: "/usr/lib64/python2.7/threading.py", line 173, in acquire
rc = self.__block.acquire(blocking)
# ThreadID: 139795204400896
File: "/usr/lib64/python2.7/threading.py", line 784, in __bootstrap
self.__bootstrap_inner()
File: "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
self.run()
File: "/usr/lib/python2.6/site-packages/ambari_agent/ActionQueue.py", line 149, in run
self.controller.get_status_commands_executor().process_results() # process status commands
File: "/usr/lib/python2.6/site-packages/ambari_agent/StatusCommandsExecutor.py", line 76, in process_results
self.actionQueue.process_status_command_result(self.actionQueue.execute_status_command_and_security_status(command))
File: "/usr/lib/python2.6/site-packages/ambari_agent/ActionQueue.py", line 500, in execute_status_command_and_security_status
component_status_result = self.customServiceOrchestrator.requestComponentStatus(command)
File: "/usr/lib/python2.6/site-packages/ambari_agent/CustomServiceOrchestrator.py", line 471, in requestComponentStatus
override_output_files=override_output_files)
File: "/usr/lib/python2.6/site-packages/ambari_agent/CustomServiceOrchestrator.py", line 412, in runCommand
handle = handle, log_info_on_failure=log_info_on_failure)
File: "/usr/lib/python2.6/site-packages/ambari_agent/PythonReflectiveExecutor.py", line 59, in run_file
imp.load_source('__main__', script)
File: "/var/lib/ambari-agent/cache/common-services/REST_API/1.1.0-SNAPSHOT/package/scripts/play.py", line 192, in <module>
PlayServer().execute()
File: "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 329, in execute
method(env)
File: "/var/lib/ambari-agent/cache/common-services/REST_API/1.1.0-SNAPSHOT/package/scripts/play.py", line 41, in status
from env_params import pid_file
File: "/var/lib/ambari-agent/cache/common-services/REST_API/1.1.0-SNAPSHOT/package/scripts/env_params.py", line 4, in <module>
from install_params import deploy_dir
File: "/var/lib/ambari-agent/cache/common-services/REST_API/1.1.0-SNAPSHOT/package/scripts/install_params.py", line 6, in <module>
code, hdp_version = call("hdp-select versions | tail -1")
File: "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 72, in inner
result = function(command, **kwargs)
File: "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 115, in call
tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
File: "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 150, in _call_wrapper
result = _call(command, **kwargs_copy)
File: "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 223, in _call
preexec_fn=preexec_fn)
File: "/usr/lib/python2.6/site-packages/ambari_agent/main.py", line 65, in sp_locked_init
with lock:
File: "/usr/lib64/python2.7/threading.py", line 173, in acquire
rc = self.__block.acquire(blocking)
# ThreadID: 139794130659072
File: "/usr/lib64/python2.7/threading.py", line 784, in __bootstrap
self.__bootstrap_inner()
File: "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
self.run()
File: "/usr/lib64/python2.7/threading.py", line 764, in run
self.__target(*self.__args, **self.__kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/threadpool.py", line 95, in _run_jobs
func(*args, **kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/scheduler.py", line 512, in _run_job
retval = job.func(*job.args, **job.kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/AlertSchedulerHandler.py", line 155, in <lambda>
return lambda: alert_def.collect()
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/base_alert.py", line 112, in collect
res = self._collect()
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/metric_alert.py", line 97, in _collect
jmx_property_values, http_code = self._load_jmx(alert_uri.is_ssl_enabled, host, port, self.metric_info)
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/metric_alert.py", line 212, in _load_jmx
connection_timeout=self.curl_connection_timeout, kinit_timer_ms = self.kinit_timeout)
File: "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/curl_krb_request.py", line 94, in curl_krb_request
import uuid
# ThreadID: 139794122266368
File: "/usr/lib64/python2.7/threading.py", line 784, in __bootstrap
self.__bootstrap_inner()
File: "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
self.run()
File: "/usr/lib64/python2.7/threading.py", line 764, in run
self.__target(*self.__args, **self.__kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/threadpool.py", line 95, in _run_jobs
func(*args, **kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/scheduler.py", line 512, in _run_job
retval = job.func(*job.args, **job.kwargs)
File: "/usr/lib/python2.6/site-packages/ambari_agent/AlertSchedulerHandler.py", line 155, in <lambda>
return lambda: alert_def.collect()
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/base_alert.py", line 112, in collect
res = self._collect()
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/web_alert.py", line 102, in _collect
web_response = self._make_web_request(url)
File: "/usr/lib/python2.6/site-packages/ambari_agent/alerts/web_alert.py", line 201, in _make_web_request
connection_timeout=self.curl_connection_timeout, kinit_timer_ms = self.kinit_timeout)
File: "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/curl_krb_request.py", line 122, in curl_krb_request
kinit_lock.acquire()
File: "/usr/lib64/python2.7/threading.py", line 173, in acquire
rc = self.__block.acquire(blocking)
*** STACKTRACE - END ***
... View more
Labels:
- Labels:
-
Apache Ambari
04-03-2017
12:56 PM
Thanks for your response.
I did kinit and can -cat the file correctly if I don't use webhdfs (so all the permissions and policies are correct for that user)
I read about the hdfs blacklist in another issue in the forum, changed it restarted everything but still I get the "No valid credentials provided error".
Are you able to use File View in Ambari to preview/download encrypted files?
... View more
03-31-2017
11:56 AM
I'm using HDP 2.5 and I'm experiencing the issue HADOOP-12787 even though it was merged in 2.4.2 To reproduce it first validate you can correctly read a file in an encryption zone: hdfs dfs -cat /user/me/somefile.txt now try going through webhdfs: hdfs dfs -cat webhdfs://hadoop1:50070/user/me/somefile.txt I get:
org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) I noticed this because the FileView preview wasn't working with encrypted files
... View more
Labels:
- Labels:
-
Apache Hadoop
03-08-2016
11:06 AM
Thanks. That's really disappointing.
Ambari knows the packages installed by the stack so it should be easy to implement that feature.
Even better if it moved away from packages altogether, as Cloudera did Somebody mentioned the Ambari cleanup script in an idea proposed. Will check it out but I believe that script will remove everything not just old versions.
... View more
03-08-2016
09:11 AM
3 Kudos
How are you supposed to remove an old version of HDP once you have successfully upgraded to a new version? The old version is still listed as installed and the "Deregister" button is disabled because "it is installed". It would be easy to delete the /usr/hdp/[old version] folder but all the packages would still be considered installed by the OS. On the other hand trying to remove the old packages manually on each node is cumbersome and risky. Leaving behing GBs of data and lots of packages will easily buildup as you upgrade to new versions over the years.
... View more
Labels:
- Labels:
-
Apache Ambari
