SYMPTOM: After upgrading ambari from 2.1.1 to 2.2.2.2 tried restarting oozie service which failed with error - " su: cannot set user id: Resource temporarily unavailable" ERROR: Below are the error logs- Execution, [[0000002-160227115902137-oozie-oozi-C@4]::CoordActionInputCheck:: Ignoring action. Coordinator job is not in RUNNING/RUNNINGWITHERROR/PAUSED/PAUSEDWITHERROR state, but state=SUSPENDED], Error Code: E1100 2016-07-02 13:04:42,457 WARN CoordActionInputCheckXCommand:523 - SERVER[hdmlup000a.machine.group] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-160227115902137-oozie-oozi-C] ACTION[0000002-160227115902137-oozie-oozi-C@5] E1100: Command precondition does not hold before execution, [[0000002-160227115902137-oozie-oozi-C@5]::CoordActionInputCheck:: Ignoring action. Coordinator job is not in RUNNING/RUNNINGWITHERROR/PAUSED/PAUSEDWITHERROR state, but state=SUSPENDED], Error Code: E1100 2016-07-02 13:04:42,459 WARN CoordActionInputCheckXCommand:523 - SERVER[hdmlup000a.machine.group] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-160227115902137-oozie-oozi-C] ACTION[0000002-160227115902137-oozie-oozi-C@6] E1100: Command precondition does not hold before execution, [[0000002-160227115902137-oozie-oozi-C@6]::CoordActionInputCheck:: Ignoring action. Coordinator job is not in RUNNING/RUNNINGWITHERROR/PAUSED/PAUSEDWITHERROR state, but state=SUSPENDED], Error Code: E1100 2016-07-02 13:04:42,460 WARN CoordActionReadyXCommand:523 - SERVER[hdmlup000a.machine.group] USER[falcon] GROUP[-] TOKEN[] APP[FALCON_PROCESS_DEFAULT_Push03to04run03] JOB[0000002-160227115902137-oozie-oozi-C] ACTION[] E1100: Command precondition does not hold before execution, [[0000002-160227115902137-oozie-oozi-C]::CoordActionReady:: Ignoring job. Coordinator job is not in RUNNING state, but state=SUSPENDED], Error Code: E1100 2016-07-02 13:04:53,076 INFO PauseTransitService:520 - SERVER[hdmlup000a.machine.group] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Acquired lock for [org.apache.oozie.service.PauseTransitService] 2016-07-02 13:04:53,086 INFO PauseTransitService:520 - SERVER[hdmlup000a.machine.group] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Released lock for [org.apache.oozie.service.PauseTransitService] ROOT CAUSE: The issue is probably due to nproc settings. You need to modify the nproc settings for particular service user. RESOLUTION: Below were steps performed for resolution 1.Check output of "ps -u oozie -L | wc -l" Nproc limit for oozie was set to 16000 in ambari oozie config. 2. Modified the nproc limit from 16000 to 32000 using ambari->services->oozie->configs 3. Restarted oozie.The oozie process was down from ambari UI but was showing running using ps command. 4.The issue was with the process was in stale state and was showing running from X no of days. 5.We tried restarting oozie server but still the process was not getting restarted as checked from cli. 6.Killed the oozie server process from cli also tried clearing agent cache using below command - mv /var/lib/ambari-agent/data/structured-out-status.json /var/lib/ambari-agent/data/structured-out-status.json.bak 7. Restarted ambari agent process. 8. Restarted oozie server process which worked well and now oozie process is showing right status in ps command output. ... View more

SYMPTOM: HDFS service is not able to start and throwing below error in logs - python error " File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/ranger_functions.py", line 124, in create_ranger_repository" ERROR: Ambari operation logs shows below message - stderr: /var/lib/ambari-agent/data/errors-22280.txt Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py", line 433, in <module> NameNode().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute method(env) File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 524, in restart self.start(env, upgrade_type=upgrade_type) File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py", line 102, in start namenode(action="start", hdfs_binary=hdfs_binary, upgrade_type=upgrade_type, env=env) File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk return fn(*args, **kwargs) File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py", line 60, in namenode setup_ranger_hdfs(upgrade_type=upgrade_type) File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/setup_ranger_hdfs.py", line 61, in setup_ranger_hdfs hdp_version_override = hdp_version, skip_if_rangeradmin_down= not params.retryAble) File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/setup_ranger_plugin_xml.py", line 78, in setup_ranger_plugin policy_user) File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/ranger_functions.py", line 124, in create_ranger_repository repo = self.get_repository_by_name_urllib2(repo_name, component, 'true', ambari_username_password_for_ranger) File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/decorator.py", line 82, in wrapper return function(*args, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/ranger_functions.py", line 77, in get_repository_by_name_urllib2 response = json.loads(result.read()) File "/usr/lib/python2.6/site-packages/ambari_simplejson/__init__.py", line 307, in loads return _default_decoder.decode(s) File "/usr/lib/python2.6/site-packages/ambari_simplejson/decoder.py", line 335, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python2.6/site-packages/ambari_simplejson/decoder.py", line 353, in raw_decode raise ValueError("No JSON object could be decoded") ValueError: No JSON object could be decoded ROOT CAUSE: This is due to issue with "amb_ranger_admin" user password and hence ranger service are not able to communicate with Ranger admin. RESOLUTION: Below were steps performed for resolution - 1. Tried disabling the HDFS plugin for ranger and restarting HDFS worked well. 2. Removed the HDFS repository policy cache files from both namenodes. 3. Enabled HDFS plugin and restarted standby namenode which got failed again with same error. 4. Checked in Ranger UI in Audit -> Access->login tab was displaying wrong credentials for ambari admin user 5. We Reset the password for amb_ranger_admin from Ranger UI also modified same value in Ambari -> Services->Ranger->Configs 6. Restarted ranger ... View more

SYMPTOM: Ambari smoke test fails for hbase service. Below is the current scenario - Ranger is installed in the cluster HBase policy has been enabled ambari-qa user has the privileges correctly defined in the HBase policy ERROR: ERROR: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient permissions (user=ambari-qa, scope=default, params=[namespace=default,table=default:ambarismoketest,family=family],action=CREATE) 2015-10-27 09:52:03,342 ERROR [main] client.AsyncProcess: Failed to get region location org.apache.hadoop.hbase.TableNotFoundException: Table 'ambarismoketest' was not found, got: XXXXX01. ROOT CAUSE: If the Ranger co-processor is not correctly defined in the HBase configuration, the smoke test from Ambari would fail. Any table creation as non-hbase user could also fail. RESOLUTION: Verify the Ranger configuration for HBase. Ensure that the following properties are set correctly and that the co-processors include Ranger classes hbase.coprocessor.master.classes hbase.coprocessor.region.classes hbase.coprocessor.regionserver.classes All of the above should include org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor hbase.security.authorization should be enabled i.e. set to true. ... View more

SYMPTOM: Ranger plugin is enabled for HIVE. Restarting HIVE service its not able to start and stucking on below error ERROR: 2015-10-15 13:02:51,683 - u"File['/var/lib/ambari-agent/data/tmp/ojdbc6.jar']" {'content': DownloadSource('http://sjcservicenode04-prod.xxxinternal.com:8080/resources//oracle-jdbc-driver.jar')} 2015-10-15 13:02:51,796 - Not downloading the file from http://sjcservicenode04-prod.xxxinternal.com:8080/resources//oracle-jdbc-driver.jar, because /var/lib/ambari-agent/data/tmp/oracle-jdbc-driver.jar already exists 2015-10-15 13:02:51,996 - call['hdp-select status hadoop-client'] {'timeout': 20} ROOT CAUSE: Ranger Hive policies http url calls were taking forever to return results Ranger makes a lot of calls to urllib2.urlopen(request) that don't have a timeout in Ambari 2.0 Opened Ambari BUG to put time=5 in the ranger_function.py file. https://hortonworks.jira.com/browse/BUG-46275   RESOLUTION: 1) Edit /usr/lib/python2.6/site-packages/resource-management/libraries/functions/ranger_functions.py and copy to all host to be safe (only Hive nodes) all urllib2.urlopen(request) do urllib2.urlopen(request, timeout=5) 2) Delete duplicate x_group_users - MYSQL https://hortonworks.jira.com/browse/BUG-43119 ... View more

Scenario: Lets say you have 2 ranger admin instances configured in your cluster. Now if you want to enable Ranger HA, you need to delete one of the ranger admin instance[since Ranger HA admin wizard will create 2nd instance of ranger for you] In such case you need to remove one instance of Ranger admin which is already installed. The following steps will guide you how to remove Ranger Admin instance using Ambari API Backup Ambari Server database [https://ambari.apache.org/current/installing-hadoop-using-ambari/content/ambari-chap11-1.html] Stop the Ranger service using Ambari. In case the Ranger Admin fails to stop, try stopping the Ranger Service as follows using Ambari API: curl -u admin:admin -H 'X-Requested-By: ambari' -X PUT -d '{"RequestInfo":{"context":"Stop Service"},"Body":{"ServiceInfo":{"state":"INSTALLED"}}}' \ http://xxx.hostname:8080/api/v1/clusters/TEST/services/RANGER curl -u admin:admin -H 'X-Requested-By: ambari' -X PUT -d '{"RequestInfo": {"context" :"Stop Service"}, "Body": {"ServiceComponentInfo": {"state": "INSTALLED"}}}' \ http://xxx.hostname:8080/api/v1/clusters/TEST/hosts/xxx.hostname/host_components/RANGER_ADMIN Remove Ranger Admin using API: curl -u admin:admin -H "X-Requested-By: ambari" -X DELETE http://xxx.hostname:8080/api/v1/clusters/TEST/hosts/xxx.hostname/host_components/RANGER_ADMIN ... View more

SYMPTOM: User was not able to browse ambari UI after ambari server restart. Ambari version : 2.1.2 Below was the error seen in logs ERROR: 06 Jul 2016 09:40:26,505 ERROR [Stack Version Loading Thread] LatestRepoCallable:93 - Could not load the URI for stack HDP-2.1 from http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json (connect timed out) 06 Jul 2016 09:40:26,506 INFO [Stack Version Loading Thread] LatestRepoCallable:74 - Loading latest URL info for stack HDP-2.2 from http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json 06 Jul 2016 09:40:28,508 ERROR [Stack VersionLoading Thread] LatestRepoCallable:93 - Could not load the URI for stack HDP-2.2 from http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json (connect timed out) 06 Jul 2016 09:40:28,509 INFO [Stack Version Loading Thread] LatestRepoCallable:74 - Loading latest URL info for stack HDP-2.3 from http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json 06 Jul 2016 09:40:30,511 ERROR [Stack Version Loading Thread] LatestRepoCallable:93 - Could not load the URI for stack HDP-2.3 from http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json (connect timed out) 06 Jul 2016 09:40:30,511 INFO [Stack Version Loading Thread] LatestRepoCallable:74 - Loading latest URL info for stack HDP-2.0 from http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json 06 Jul 2016 09:40:32,514 ERROR [Stack Version Loading Thread] LatestRepoCallable:93 - Could not load the URI for stack HDP-2.0 from http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json (connect timed out) 06 Jul 2016 09:40:32,514 INFO [Stack VersionL oading Thread] LatestRepoCallable:74 - Loading latest URL info for stack HDP-2.3.GlusterFS from http://s3.amazonaws.com/dev.hortonworks.com/HDP/hdp_urlinfo.json 06 Jul 2016 09:40:34,519 ERROR [Stack Version Loading Thread] LatestRepoCallable:93 - Could not load the URI for stack HDP-2.3.GlusterFS from http://s3.amazonaws.com/dev.hortonworks.com/HDP/hdp_urlinfo.json ROOT CAUSE: This is a BUG in Ambari 2.1.2 version and below are the jira - https://hortonworks.jira.com/browse/BUG-46081 RESOLUTION: Upgrading Ambari from 2.1.2 to 2.1.2.1 resolved the issue. ... View more

SYMPTOM: RM is down due to below error. Earlier we were suspicion the ulimit could be culprit though we have increased it to 128K. But still no luck. ERROR: 2016-07-25 12:19:47,125 WARN security.DelegationTokenRenewer (DelegationTokenRenewer.java:handleDTRenewerAppSubmitEvent(873)) - Unable to add the application to the delegation token renewer. java.lang.OutOfMemoryError: unable to create new native thread. Below was few steps followed - 1. Checked the error and saw that previously the same issue and increasing ulimit resolved the issue. 2. Checked the ulimit and lsof output - $ulimit -n 131072 $lsof |grep yarn |wc 1726 15553 242741 3. Checked the heap size for yarn process which was set to 8Gb and looks good. Below error was displayed in RM out.log file Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f89641cf000, 12288, 0) failed; error='Cannot allocate memory' (errno=12) # # There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (malloc) failed to allocate 12288 bytes for commtting reserved memory. # An error report file with more information is saved as: # /tmp/hs_err_pid56149.log Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed. Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f89642d0000, 12288, 0) failed; error='Cannot allocate memory' (errno=12) Below was log in "/tmp/hs_err_pid56149.log" this looks a problem with memory allocation for threads at OS level === Stack: [0x00007f89641cf000,0x00007f89642d0000], sp=0x00007f89642ce900, free space=1022k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [libjvm.so+0x99eb8a] VMError::report_and_die()+0x2ea V [libjvm.so+0x49721b] report_vm_out_of_memory(char const*, int, unsigned long, char const*)+0x9b V [libjvm.so+0x81d9ae] os::Linux::commit_memory_impl(char*, unsigned long, bool)+0xfe V [libjvm.so+0x81da6c] os::pd_commit_memory(char*, unsigned long, bool)+0xc V [libjvm.so+0x8157fa] os::commit_memory(char*, unsigned long, bool)+0x2a V [libjvm.so+0x81bf5d] os::pd_create_stack_guard_pages(char*, unsigned long)+0x6d V [libjvm.so+0x95249e] JavaThread::create_stack_guard_pages()+0x5e V [libjvm.so+0x958de4] JavaThread::run()+0x34 V [libjvm.so+0x81f988] java_start(Thread*)+0x108 === stack suggest memory allocation (malloc) failed at OS level.check you have enough physical memory available at host. ROOT CAUSE: Collected the jstack logs for process and found that - the 'Truststore reloader thread' count is increasing which is the same issue what i earlier mentioned - https://issues.apache.org/jira/browse/YARN-5309. $grep 'Truststore reloader thread' threadDump|wc -l    14873 $ grep 'Truststore reloader thread' threadDump1|wc -l    14999 $grep 'Truststore reloader thread' threadDump2|wc -l    15063 $grep 'Truststore reloader thread' threadDump3|wc -l    15149 $grep 'Truststore reloader thread' threadDump4|wc -l    15230 $grep 'Truststore reloader thread' threadDump5|wc -l    15347 RESOLUTION: This is confirmed as BUG and patch has been provided to resolve the issue https://issues.apache.org/jira/browse/YARN-5309 https://hortonworks.jira.com/browse/BUG-63499 ... View more