I think, what you need first is this https://www.cloudera.com/documentation/enterprise/5-10-x/topics/cm_mc_decomm_host.html If decommission is done successfully (all blocks are available across the remaining Datanodes according to the replication factor) then you can delete the nodes. ... View more

I think what you need is the following   hbase_config= { 'hbase_service_config_safety_valve': '<property><name>regionserver.global.memstore.upperLimit</name><value>0.15</value></property>' } hbase.update_config(hbase_config) I have tested it and it works. There is no need to set "m" with get_config and then re-apply the whole configuration back. You only have to update the specific safety_valve and not all hbase config.   The only catch is that you have to apply all hbase_service_config_safety_valve parameters. The same will happen with your approach because you are not updating only the "regionserver.global.memstore.upperLimit" but you are applying a value to safety valve. Of course you can write additional code, to parse the existing config of safety valve (the xml part) and add or update the regionserver.global.memstore.upperLimit but again change it to json format as per my example. ... View more

My mistake. Cloudera works with json not xml. Anyhow, to upload complete cluster config you can do it by following https://gist.github.com/gwenshap/7044525 but this is for serious cases. On the other hand, you can push only HBase configuration with the previous links I send you. You just have to prepare config in json format. ... View more

From command line I don't know if there is a way to upload xml for a specific service. You can do it for the cluster. But you should check https://cloudera.github.io/cm_api/apidocs/v19/path__clusters_-clusterName-_services_-serviceName-_config.html and https://cloudera.github.io/cm_api/apidocs/v19/path__clusters_-clusterName-_services_-serviceName-_roles_-roleName-_config.html You can also modify parameters by using PUT instead of GET ... View more

You can try using the API https://www.cloudera.com/documentation/enterprise/5-14-x/topics/cm_intro_api.html https://cloudera.github.io/cm_api/apidocs/v19/index.html ... View more

If it doesn't work, then probably the e-mail in action will not work. Try to send a mail from console: mail user@mail.address -s -s test_subject << EOF This is a test e-mail EOF There are various cases to send an e-mail. If you need an e-mail if an error is encountered, or action takes too much time, then you have to enable SLAs on this action and define the recipient. If you need an e-mail that the workflow executed successfully, then add a mail action just before the end of the workflow. If any previous action fails, the e-mail action will not executed. Unless you have modified the kill in one of your actions of course, and lead it to this e-mail action. You have multiple options to cover multiple scenarios. ... View more

You mean how the user can submit the job from HUE? If you save the file in HDFS as "workflow.xml", go to File browser using HUE. You will notice that if you select the check button of this file, you a "Submit" action button will appear. So the user can just hit it. ... View more

As you said, when it's executed, we don't know on which yarn node the command will be executed. So, with this storage bay mounted on each node, it doesn't matter on which node it's executed (I think :p)  Correct   Regarding the rest. First of all you don't have to be a scala developer to schedule a script in oozie 🙂 Your command should be "./runConsolidator.sh". Important is that your script has execute permissions and you define it in "Files". How it works: this shell action will be a YARN job, so YARN will create a temp folder e.g. "/yar/nm/some_id/another_id". All files defined in "Files" of this action, will automatically downloaded into this directory. This directory will be your working directory, so you should run your command with "./" in front, since by default, "./" is not defined in PATH.   NOTE: If your script is using jar files etc. then you should define all of them in "Files", so they will copied to the working directory. I suggest to proceed with this approach. Setting the xml will mess things and you need some experience to do it and avoid mistakes. Once you create a working job from HUE, you can export the xml and start playing. ... View more

Hi, sorry if the reply is not very clear. It is written in hurry. I will try to expand it later.   First of all HUE provides a very good interface to write oozie jobs and hardly you will need to write the job xml your own. You have a shell script with spark-submit. Spark submit will execute something (a jar or python file). When you define a "shell action", all files (used as parameters) should exist on the local filesystem (not in HDFS). If for example in a shel action you try "cat /etc/hosts" you will get the /etc/hosts file of  the YARN node, where this shell action will be executed. If you have a file in HDFS (e.g. my_hosts) and you define it as "File" in shell action then oozie will download this file automatically into the working directory. Working directory is a random directory in YARN node's filesystem, which leaves while this yarn job is being executed. So, if you use the command "cat ./my_hosts", then wou will get the contents of the "my_hosts" which is downloaded to the working directory.     In general is not very good idea to work with files on Slave Nodes, because you don't know on which yarn node this command will be executed each time. Unless you are sure tha you control it and you have deployed all required files to all nodes. Of course we are not discussing about temporary files that you may create during the execution, but for files with like metadata or configuration or files with results that you want to use after.   IMHO it is always better to have these files in HDFS and send the results back to HDFS, so they will be easily accessible by other actions. ... View more

Just to understand, you don't want the spark job to use these files from HDFS but from your local system. If that is the case, then you can create a shell action as you have mentioned. First of all, put all required files on HDFS. Then define these files in the shell action. Oozie will automatically download these files to the working directory of the node that job will be executed. You don't have to manually distribute anything in advance to the nodes, oozie will take care of it, you just have to define these files in the job. ... View more

My CDH version is 5.14 but there is no spark2 parcel for 5.14, there are parcels for 5.13 and 5.12. Is this the problem I faced with? This is not an issue. Spark is built on CDH5.13 but it works fine with CDH5.14. Check compatibility notes: https://www.cloudera.com/documentation/spark2/latest/topics/spark2_requirements.html#cdh_versions   According to the screenshot, the procedure failed while it was distributing configuration to namenode. Can you check the "stdout" and "stderr" output? You can copy it here so we can take a look. ... View more