As an alternative, you can change scratchdir like so https://cwiki.apache.org/confluence/display/Hive/AdminManual+Configuration Using the set command in the CLI or Beeline for setting session level values for the configuration variable for all statements subsequent to the set command. For example, the following command sets the scratch directory (which is used by Hive to store temporary output and plans) to /tmp/mydir for all subsequent statements: set hive.exec.scratchdir=/tmp/mydir; Using the --hiveconf option of the hive command (in the CLI) or beeline command for the entire session. For example: bin/hive --hiveconf hive.exec.scratchdir=/tmp/mydir In hive-site.xml. This is used for setting values for the entire Hive configuration (see hive-site.xml and hive-default.xml.template below). For example: <property> <name>hive.exec.scratchdir</name> <value>/tmp/mydir</value> <description>Scratch space for Hive jobs</description> </property> ... View more

HDFS has a mechanism called quotas, it is possible that your admin team set storage quotas on the individual user directories, you can set larger quota on your directory and avoid the situation # requires superuser privileges # set space quota of 1kb on a directory, can be k, m, g, etc. sudo -u hdfs hdfs dfsadmin -setSpaceQuota 1k /quotasdir # add a file sudo -u hdfs hdfs dfs -touchz /quotasdir/1 # notice file is 0 bytes sudo -u hdfs hdfs dfs -ls /quotasdir/ # for demo purposes, we need to upload a large file, larger than 1kb into directory, watch the prompt sudo -u hdfs hdfs dfs -chown -R root:hdfs /quotasdir hdfs dfs -put /root/install.log /quotasdir/ 15/11/25 15:10:47 WARN hdfs.DFSClient: DataStreamer Exception org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota of /quotasdir is exceeded: quota = 1024 B = 1 KB but diskspace consumed = 402653184 B = 384 MB at org.apache.hadoop.hdfs.server.namenode.DirectoryWithQuotaFeature.verifyStoragespaceQuota(DirectoryWithQuotaFeature.java:211) at org.apache.hadoop.hdfs.server.namenode.DirectoryWithQuotaFeature.verifyQuota(DirectoryWithQuotaFeature.java:239) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyQuota(FSDirectory.java:907) # remove space quota sudo -u hdfs hdfs dfsadmin -clrSpaceQuota /quotasdir ... View more

@Sam Pat first of all thanks for checking out my article, I see you have company reference in your error message, please edit your comment and remove it. Secondly, can you run your python script w/out Oozie? I have a feeling you're trying to execute a Python 2 script with Python3 as default interpreter. You should add the interpreter line to your script and try again. Take a look at my scripts I have a version for Python 2 #! /usr/bin/env python and Python 3 #! /usr/bin/env /usr/local/bin/python3.3 If your cluster has Python 3 installed, make sure it's across the whole cluster and has the same path. If it's Python2 then also make sure every node is configured correctly with the location of the interpreter. ... View more

@Amit Panda to do this on continuous basis you either need to setup an Oozie job that will run a script to determine old data and move it to new location. Alternatively, you can use Apache Nifi by watching a directory for old data and move it to new location. There's nothing out of the box that will do that for you. ... View more

Please check whether failures only occur on the exact same node? Also can you drill down into the yarn job logs and see what error you get? ... View more

@zaenal rifai the solution to your problem is to wait for Ambari 2.5 and use Workflow manager view. I just tested a workflow with 26 actions and it displayed well, except it's hard to see with so many actions. I attached screenshots to my response. The other option is of course to file a jira and either submit a patch or wait for community to work on the jira. I say use Workflow Manager when it comes out. The collapsed view is from the original design of the flow, the image titled many and many2 is result of running the workflow and representing the flow graph, which is what you're trying to do. With WFM, you actually can visually display your flow graph before and after the execution of the flow. <?xml version="1.0" encoding="UTF-8" standalone="no"?> <workflow-app xmlns="uri:oozie:workflow:0.5" name="Workflow3"> <start to="email_1"/> <action name="email_1"> <email xmlns="uri:oozie:email-action:0.2"> <to>address@hortonworks.com</to> <subject>1</subject> <body>1</body> </email> <ok to="email_2"/> <error to="kill"/> </action> <action name="email_2"> <email xmlns="uri:oozie:email-action:0.2"> <to>address@hortonworks.com</to> <subject>1</subject> <body>1</body> </email> <ok to="email_3"/> <error to="kill"/> </action> <action name="email_3"> <email xmlns="uri:oozie:email-action:0.2"> <to>address@hortonworks.com</to> <subject>1</subject> <body>1</body> </email> <ok to="email_4"/> <error to="kill"/> </action> <action name="email_4"> <email xmlns="uri:oozie:email-action:0.2"> <to>address@hortonworks.com</to> <subject>1</subject> <body>1</body> </email> <ok to="email_5"/> <error to="kill"/> </action> <action name="email_5"> <email xmlns="uri:oozie:email-action:0.2"> <to>address@hortonworks.com</to> <subject>1</subject> <body>1</body> </email> <ok to="email_6"/> <error to="kill"/> </action> <action name="email_6"> <email xmlns="uri:oozie:email-action:0.2"> <to>address@hortonworks.com</to> <subject>1</subject> <body>1</body> </email> <ok to="email_7"/> <error to="kill"/> </action> <action name="email_7"> <email xmlns="uri:oozie:email-action:0.2"> <to>address@hortonworks.com</to> <subject>1</subject> <body>1</body> </email> <ok to="email_8"/> <error to="kill"/> </action> <action name="email_8"> <email xmlns="uri:oozie:email-action:0.2"> <to>address@hortonworks.com</to> <subject>1</subject> <body>1</body> </email> <ok to="email_9"/> <error to="kill"/> </action> <action name="email_9"> <email xmlns="uri:oozie:email-action:0.2"> <to>address@hortonworks.com</to> <subject>1</subject> <body>1</body> </email> <ok to="email_10"/> <error to="kill"/> </action> <action name="email_10"> <email xmlns="uri:oozie:email-action:0.2"> <to>address@hortonworks.com</to> <subject>1</subject> <body>1</body> </email> <ok to="email_11"/> <error to="kill"/> </action> <action name="email_11"> <email xmlns="uri:oozie:email-action:0.2"> <to>address@hortonworks.com</to> <subject>1</subject> <body>1</body> </email> <ok to="email_12"/> <error to="kill"/> </action> <action name="email_12"> <email xmlns="uri:oozie:email-action:0.2"> <to>address@hortonworks.com</to> <subject>1</subject> <body>1</body> </email> <ok to="email_13"/> <error to="kill"/> </action> <action name="email_13"> <email xmlns="uri:oozie:email-action:0.2"> <to>address@hortonworks.com</to> <subject>1</subject> <body>1</body> </email> <ok to="email_14"/> <error to="kill"/> </action> <action name="email_14"> <email xmlns="uri:oozie:email-action:0.2"> <to>address@hortonworks.com</to> <subject>1</subject> <body>1</body> </email> <ok to="email_15"/> <error to="kill"/> </action> <action name="email_15"> <email xmlns="uri:oozie:email-action:0.2"> <to>address@hortonworks.com</to> <subject>1</subject> <body>1</body> </email> <ok to="email_16"/> <error to="kill"/> </action> <action name="email_16"> <email xmlns="uri:oozie:email-action:0.2"> <to>address@hortonworks.com</to> <subject>1</subject> <body>1</body> </email> <ok to="email_17"/> <error to="kill"/> </action> <action name="email_17"> <email xmlns="uri:oozie:email-action:0.2"> <to>address@hortonworks.com</to> <subject>1</subject> <body>1</body> </email> <ok to="email_18"/> <error to="kill"/> </action> <action name="email_18"> <email xmlns="uri:oozie:email-action:0.2"> <to>address@hortonworks.com</to> <subject>1</subject> <body>1</body> </email> <ok to="email_19"/> <error to="kill"/> </action> <action name="email_19"> <email xmlns="uri:oozie:email-action:0.2"> <to>address@hortonworks.com</to> <subject>1</subject> <body>1</body> </email> <ok to="email_20"/> <error to="kill"/> </action> <action name="email_20"> <email xmlns="uri:oozie:email-action:0.2"> <to>address@hortonworks.com</to> <subject>1</subject> <body>1</body> </email> <ok to="email_21"/> <error to="kill"/> </action> <action name="email_21"> <email xmlns="uri:oozie:email-action:0.2"> <to>address@hortonworks.com</to> <subject>1</subject> <body>1</body> </email> <ok to="email_22"/> <error to="kill"/> </action> <action name="email_22"> <email xmlns="uri:oozie:email-action:0.2"> <to>address@hortonworks.com</to> <subject>1</subject> <body>1</body> </email> <ok to="email_23"/> <error to="kill"/> </action> <action name="email_23"> <email xmlns="uri:oozie:email-action:0.2"> <to>address@hortonworks.com</to> <subject>1</subject> <body>1</body> </email> <ok to="email_24"/> <error to="kill"/> </action> <action name="email_24"> <email xmlns="uri:oozie:email-action:0.2"> <to>address@hortonworks.com</to> <subject>1</subject> <body>1</body> </email> <ok to="email_25"/> <error to="kill"/> </action> <action name="email_25"> <email xmlns="uri:oozie:email-action:0.2"> <to>address@hortonworks.com</to> <subject>1</subject> <body>1</body> </email> <ok to="email_26"/> <error to="kill"/> </action> <action name="email_26"> <email xmlns="uri:oozie:email-action:0.2"> <to>address@hortonworks.com</to> <subject>1</subject> <body>1</body> </email> <ok to="end"/> <error to="kill"/> </action> <kill name="kill"> <message>${wf:errorMessage(wf:lastErrorNode())}</message> </kill> <end name="end"/> </workflow-app> ... View more

to change an actual date of file you need to rewrite it. That was not the original question as far as I understand. Please open a new question with exact requirements. ... View more

if you use the log search utility, it automatically parses logs for you by severity level, if you intend to do it manually, you can search for an ERROR code. ... View more

@P D you have mismatch in version of Ambari and Grafana, can't conclusively say it is related to your issue but certainly not optimal, if your ambari version is 2.4.2, ambari-metrics-grafana must match the version. On my machine it looks like so ambari-metrics-monitor-2.5.0.1-51.x86_64 ambari-infra-solr-client-2.5.0.1-51.noarch ambari-metrics-hadoop-sink-2.5.0.1-51.x86_64 ambari-metrics-grafana-2.5.0.1-51.x86_64 ambari-agent-2.5.0.1-51.x86_64 ... View more