You are a SAVIOUR !! I made a script to assign all regions with ""not deployed on any region server"" and now it works fine!! Awesome thanks a lot mate! ... View more

@loridigia ,   The best way to avoid this problem is to configure and use Knox Gateway to access those services. Knox will handle the authentication and will ensure you can use those services without requiring any specific browser configuration.   Cheers, André   ... View more

Someone know the solution?   ... View more

INFO: Exception in thread "main" java.lang.IllegalArgumentException: Required AM memory Above error is for AM and not for executors, hence you need to set the AM memory as spark.yarn.am.memory=2g ... View more

Hi @loridigia    If cluster/application is not enabled dynamic allocation and if you set --conf spark.executor.instances=1 then it will launch only 1 executor. Apart from executor, you will see AM/driver in the Executor tab Spark UI. ... View more

Hello @loridigia  I don't think there is a direct way to achieve this. But we have a workaround to do that. We can start the Spark jobs with Dynamic Allocation enabled. And we can set the Minimum executors to "0", initial executors to "1" and the idle timeout to "5s". With these configurations, the Spark job will start with 1 executor and after 5 seconds that container will be killed as it will be idle for more than 5 seconds. Now, we will have a Spark application only with the Driver / ApplicationMaster container running.   CONFIGS: --conf spark.dynamicAllocation.enabled=true --conf spark.shuffle.service.enabled=true --conf spark.dynamicAllocation.executorIdleTimeout=5s --conf spark.dynamicAllocation.initialExecutors=1 --conf spark.dynamicAllocation.maxExecutors=1 --conf spark.dynamicAllocation.minExecutors=1   NOTE: We can add these configs to the spark-defaults.conf so that the changes will be applied to all the Running jobs. Please be careful with other / actual Spark job configurations.   Make sure to mark the answer as the accepted solution. If it resolves your issue !  ... View more

@Lo  Good Day!! Even though you submit spark-submit via JDBC the request goes through the HMS. HMS detects the type of client for interacting with it, for example, Hive or Spark, and acts accordingly. If you see any timeout requests in the logs you can adjust the below property. hive.metastore.client.socket.timeout=<to a higher value>  Check and let us know If this helped. ... View more

Hi @loridigia, Based on the current error you provided "org.apache.hadoop.hbase.NotServingRegionException:  table XXX is not online on worker04" maybe some regions are not deployed on any RegionServers yet. please check this result to see is there any inconsistencies on this table: 1. sudo -u hbase hbase hbck -details > /tmp/hbck.txt 2. If you see inconsistencies please grep ERROR from hbck.txt you will see which region has problem. 3. Then you need to check if this region's directory is complete in this result: hdfs dfs -ls -R /hbase 4. Then need to check in hbase shell : scan 'hbase:meta', if this region's info are updated in hbase:meta table. 5. Based on type of the issue we need to use hbck2 jar to fix the inconsistencies. https://github.com/apache/hbase-operator-tools/tree/master/hbase-hbck2 These are general steps to deal with this kind of problem, there could be more complex issues behind it. We suggest you to file a case with Cloudera support.    Thanks, Will ... View more

I fixed just editing the fields directly in the Ambari Postgres DB and then Rebooting. ... View more