INFO: Exception in thread "main" java.lang.IllegalArgumentException: Required AM memory Above error is for AM and not for executors, hence you need to set the AM memory as spark.yarn.am.memory=2g ... View more

Yes the config changes are getting reflected on the services as expected. Can you post the complete error ?  ... View more

Ideally in this case, increasing the yarn.scheduler.maximum-allocation-mb  should solve. But from your comments i can understand that the changes are not reflecting on the yarn service. To confirm the same you can check via this  http://active_rm_hostname:8088/conf Under this url search for yarn.scheduler.maximum-allocation-mb and check the value.  Make sure the client configs are deployed via ambari. Check the status of YARN service in ambari. ... View more

Can you try passing this spark config on your spark shell /spark submit  spark.yarn.am.memory=1g Make sure to mark the answer as the accepted solution. If it resolves your issue !  ... View more

AM Memory is from this property  yarn.app.mapreduce.am.resource.mb  You can set the am memory and tunning the value of the above property.   Make sure to mark the answer as the accepted solution. If it resolves your issue! ... View more

I suspect that your datanodes report is slow.Because after restart of namenode you are trigger the datanode restart so it will take time to come up with reports during that interval you can except for missing blocks this will be an intermediate issue. So that you can wait for few more min's and check the namenode ui. Else during the time of issue copy the logs and share it.   Make sure to mark the answer as the accepted solution. If it resolves your issue ! ... View more

Particular datanode has been excluded from write operation. Why excluded ? Need to check the namenode log and datanode log. You can share the logs to debug further. Also check then namenode UI and Datanodes link for errors. ... View more

2022-01-13 09:53:19,043 ERROR main:com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean: Unable to access schema version in database. It says unable to access the schema version from the database. Can you access the scm databsase and check the version  scm=> select * from SCHEMA_VERSION; version | old_version ---------+-------------    |       ... View more

When I restarted the cluster earlier,there are also many missingBlocks before the DataNode restarts completely. This can cause the missing blocks.ex: Namenode got restarted but still the datanode restart is inprogress. So the heartbeat from datanode might be missed. To confirm that you can check the namenode UI post restart and during the time of missing blocks. ... View more