Support Questions
Find answers, ask questions, and share your expertise

Services are Not starting post the integration with Kerberos HDP version 3.1.4 and Ambari Version 2.7.4

Services are Not starting post the integration with Kerberos HDP version 3.1.4 and Ambari Version 2.7.4

Explorer
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 314, in _call
    raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: 
Execution of 'ambari-sudo.sh -H -E /usr/hdp/3.1.4.0-315/hadoop/bin/hdfs --config /usr/hdp/3.1.4.0-315/hadoop/conf --daemon start datanode' returned 1.
ERROR: Cannot set priority of datanode process 45359 stdout: 2021-04-22 03:25:38,875 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.4.0-315 -> 3.1.4.0-315 2021-04-22 03:25:38,931 - Using hadoop conf dir: /usr/hdp/3.1.4.0-315/hadoop/conf 2021-04-22 03:25:39,273 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.4.0-315 -> 3.1.4.0-315 2021-04-22 03:25:39,289 - Using hadoop conf dir: /usr/hdp/3.1.4.0-315/hadoop/conf 2021-04-22 03:25:39,292 - Group['hdfs'] {} 2021-04-22 03:25:39,294 - Group['hadoop'] {} 2021-04-22 03:25:39,295 - Group['users'] {} 2021-04-22 03:25:39,296 - User['yarn-ats'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None} 2021-04-22 03:25:39,297 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}

 

1 REPLY 1

Re: Services are Not starting post the integration with Kerberos HDP version 3.1.4 and Ambari Version 2.7.4

Cloudera Employee

@PrernaU I hope you haven't changed any config in HDFS. Please compare pre & Post upgrade configs using ambari UI.

Considering the exception below, I have seen this issue once due to memory issue on a single DataNode.

 

ERROR: Cannot set priority of datanode process 45359

 

i.e. The available memory was 12GB while the DN heap was set to 16GB & due to this the DN JVM was failing to start. 

 

/proc/meminfo:
MemTotal:       131407744 kB
MemFree:         2180792 kB
MemAvailable:   12004080 kB


You can try checking which process is using RAM by running below command and try to reduce the RAM utilization and start the DN process

#ps aux --sort -rss

 

There could be something else but probably some host level crunch is causing the JVM to not get started properly. Please check DN.out and DN.log file once for more details.