About Amithsha

Amithsha · ‎02-15-2022

Can you run fsck with -blocks option to get the datanode address. hadoop fsck /user/oozie/tmp/test2/workflow.xml -files -blocks Login to the datanode and check/grep for that particular blockId/filename on the datanode log. Also grep for blockId/filename on the namenode log.

Amithsha · ‎02-12-2022

I NFO: Exception in thread "main" java.lang.IllegalArgumentException: Required AM memory Above error is for AM and not for executors, hence you need to set the AM memory as spark.yarn.am.memory=2g

Amithsha · ‎02-10-2022

Hi, Error is ldapsearch: command not found Make sure ldapsearch command installed on your node.

Amithsha · ‎01-27-2022

Yes the config changes are getting reflected on the services as expected. Can you post the complete error ?

Amithsha · ‎01-27-2022

Ideally in this case, increasing the yarn.scheduler.maximum-allocation-mb should solve. But from your comments i can understand that the changes are not reflecting on the yarn service. To confirm the same you can check via this http://active_rm_hostname:8088/conf Under this url search for yarn.scheduler.maximum-allocation-mb and check the value. Make sure the client configs are deployed via ambari. Check the status of YARN service in ambari.

Amithsha · ‎01-27-2022

Can you try passing this spark config on your spark shell /spark submit spark.yarn.am.memory=1g Make sure to mark the answer as the accepted solution. If it resolves your issue !

Amithsha · ‎01-26-2022

Can you check the userlimit of the queue and the max AM resource percentage ? RM UI -> Scheduler -> Expand your queue( take screenshot and attach to this case)

Amithsha · ‎01-26-2022

AM Memory is from this property yarn.app.mapreduce.am.resource.mb You can set the am memory and tunning the value of the above property. Make sure to mark the answer as the accepted solution. If it resolves your issue!

Amithsha · ‎01-18-2022

I suspect that your datanodes report is slow.Because after restart of namenode you are trigger the datanode restart so it will take time to come up with reports during that interval you can except for missing blocks this will be an intermediate issue. So that you can wait for few more min's and check the namenode ui. Else during the time of issue copy the logs and share it. Make sure to mark the answer as the accepted solution. If it resolves your issue !

Amithsha · ‎01-17-2022

Particular datanode has been excluded from write operation. Why excluded ? Need to check the namenode log and datanode log. You can share the logs to debug further. Also check then namenode UI and Datanodes link for errors.

Amithsha · ‎01-17-2022

Since its moved from INFO level you can set the NM log level to DEBUG mode to get the container metrics on nm log. But that will be applied globally hence you can try to enable only on ContainersMonitorImpl. Append to nodemanager log4j log4j.logger.org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl.audit=DEBUG Make sure to mark the answer as the accepted solution. If it resolves your issue !

Amithsha · ‎01-16-2022

2022-01-13 09:53:19,043 ERROR main:com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean: Unable to access schema version in database. It says unable to access the schema version from the database. Can you access the scm databsase and check the version scm=> select * from SCHEMA_VERSION; version | old_version ---------+------------- |

Amithsha · ‎01-16-2022

When I restarted the cluster earlier,there are also many missingBlocks before the DataNode restarts completely. This can cause the missing blocks.ex: Namenode got restarted but still the datanode restart is inprogress. So the heartbeat from datanode might be missed. To confirm that you can check the namenode UI post restart and during the time of missing blocks.

Amithsha · ‎01-16-2022

Missing blocks can happen because of 2 reasons, 1. Data corruption - Either the disk is corrupted or the VM is down. This is a permanently failure means the data is lost. 2. Delay is Datanode report - This is because of a delay in datanode report and this is a temporary failure and in some time if datanode reports back the report then the cluster will be back to normal. How to find and solve the issue. 1. Make sure no data volume failures or VM failures. 2. Ensure that the namenode received heartbeats from all datanodes. Namenode UI -> Datanodes -> last contact If all datanodes found in the list and not in dead list or not found then no issues with receiving block report. Apart from this to debug further we need Namenode and datanode logs.

Amithsha · ‎01-16-2022

Currently on HDP3.1.5.0-152 logging of container memory usage is not possible. Because it got removed from the source code of HDP3. Make sure to mark the answer as the accepted solution. If it resolves your issue !

Amithsha · ‎01-16-2022

Hadoop hosts with different OS version is not recommended because higher OS version might come up with higher JAVA version hence that can cause inconsistencies. Due to this either you can see either performance improvement or performance degradation on both java and OS level. Because all hadoop processes are isolated in JVM env. But still we can run a cluster with different OS versions but make sure the OS is supported. https://docs.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_os_requirements.html#os_requirements Make sure to mark the answer as the accepted solution. If it resolves your issue !

Amithsha · ‎01-15-2022

HI, YARN Graceful decommission will wait for jobs to complete. You can pass the timeout value so that YARN will start decommission after x seconds. If no jobs running within x secs then automatically YARN will start decommission without waiting for timeout to happen. CM -> Clusters -> yarn -> Configuration -> In search bar ( yarn.resourcemanager.nodemanager-graceful-decommission-timeout-secs) Set the value and save the configuration and do restart to deploy configs. To decommission a specific host/more hosts CM -> Clusters -> yarn -> Instances (Select the hosts that you want to decommission) Click -> Actions for selected hosts -> Decommission In case you want to decommission all the roles of a host then follow this doc https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/cm_mc_host_maint.html#decomm_host Make sure to mark the answer as the accepted solution. If it resolves you issue !

Online	Offline
Last Visited	‎04-29-2022 01:31 AM

Date Registered	‎04-20-2021 03:45 AM
Last Visited	‎04-29-2022 01:31 AM
Total Messages Posted	17
Total Kudos Received	3

Re: NodeManger doesn’t log the Memory usage mess...

Re: How to gracefully stop YARN role on a datanode...

Re: Why hdfs dfs -setrep -w 2 / keeps waiting?

Re: Yarn: Required AM memory(5273+527) is above th...

Re: Kerberos issue

Re: Yarn: Required AM memory(5273+527) is above th...

Re: Yarn: Required AM memory(5273+527) is above th...

Re: Yarn: Required AM memory(5273+527) is above th...

Re: About the scheduler-capacity prop to launch ma...

Re: Yarn: Required AM memory(5273+527) is above th...

Re: HDFS blocks missing at every restart of CM clu...

Re: There are 1 datanode(s) running and 1 node(s) ...

Re: NodeManger doesn’t log the Memory usage mess...

Re: Cloudera server fails after migrating from pos...

Re: HDFS blocks missing at every restart of CM clu...

Re: HDFS blocks missing at every restart of CM clu...

Re: NodeManger doesn’t log the Memory usage mess...

Re: Different OS version in Hadoop Cluster

Re: How to gracefully stop YARN role on a datanode...