@Sunile Manjee you can leverage WebHCat for this as one idea, https://cwiki.apache.org/confluence/display/Hive/WebHCat+UsingWebHCat#WebHCatUsingWebHCat-ErrorCodesandResponses # this will execute a hive query and save result to hdfs file in your home directory called output curl -s -d execute="select+*+from+sample_08;" \ -d statusdir="output" \ 'http://localhost:50111/templeton/v1/hive?user.name=root' # if you ls on the directory, it will have two files, stderr and stdout hdfs dfs -ls output # if the job succeeded, you can cat the stdout file and view the results hdfs dfs -cat output/stdout when you invoke the job, you will get a response with job id, then you can also check whether output directory exists and there's no error log with webhdfs API, in that case job succeedd. curl -i "http://sandbox.hortonworks.com:50070/webhdfs/v1/user/root/output/?op=LISTSTATUS" another idea is to leverage Oozie to wire the jobs together, once job completes, you can use SLA monitoring features of Oozie to check whether job completed or send an email (SLA not needed for this) whichever way you go, you can have Nifi watch these events either from JMS topic in ActiveMQ if you intend to use SLA or email alert. https://community.hortonworks.com/articles/83787/apache-ambari-workflow-manager-view-for-apache-ooz-1.html probably even better idea is to query ATS via REST API https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/TimelineServer.html I think this is probably the most sane approach, you can query ATS for finished job and get status. So once you know the job ID, there are ways to get it, one of them is via my first example, then in the second processor you can query ATS for completion state. ... View more

@Amit Panda here's a slightly modified script from stack overflow thread #!/bin/bash usage="Usage: dir_diff.sh [directory] [days]" if [[ $# -ne 2 ]] then echo $usage exit 1 fi now=$(date +%s) hadoop fs -ls -R $1 | grep "^d" | while read f; do dir_date=`echo $f | awk '{print $6}'` difference=$(( ( $now - $(date -d "$dir_date" +%s) ) / (24 * 60 * 60 ) )) if [ $difference -gt $2 ]; then echo $f; fi done I don't have files older than 10 days on my HDFS so I execute with 1 day argument like so: sudo sh dir_diff.sh /tmp 1 drwx------ - ambari-qa hdfs 0 2017-03-11 15:41 /tmp/ambari-qa drwx------ - ambari-qa hdfs 0 2017-03-11 15:41 /tmp/ambari-qa/staging drwxr-xr-x - hdfs hdfs 0 2017-03-11 15:39 /tmp/entity-file-history drwxr-xr-x - yarn hadoop 0 2017-03-11 15:39 /tmp/entity-file-history/active drwx------ - hive hdfs 0 2017-03-11 15:42 /tmp/hive/hive/17c0213c-358a-4c89-b803-800762144a21 drwx------ - hive hdfs 0 2017-03-11 15:42 /tmp/hive/hive/17c0213c-358a-4c89-b803-800762144a21/_tmp_space.db drwx------ - hive hdfs 0 2017-03-11 15:42 /tmp/hive/hive/96049638-4aee-42cc-95f6-0652b3a66cae drwx------ - hive hdfs 0 2017-03-11 15:42 /tmp/hive/hive/96049638-4aee-42cc-95f6-0652b3a66cae/_tmp_space.db drwx------ - hive hdfs 0 2017-03-11 15:41 /tmp/hive/hive/e4fe18d1-5cb4-4088-93ff-cf4aac410301 drwx------ - hive hdfs 0 2017-03-11 15:41 /tmp/hive/hive/e4fe18d1-5cb4-4088-93ff-cf4aac410301/_tmp_space.db drwxr-xr-x - ambari-qa hdfs 0 2017-03-11 15:41 /tmp/tezsmokeinput On my 2.5 Sandbox, it returns this sh dir_diff.sh /tmp 10 drwxr-xr-x - hdfs hdfs 0 2016-10-25 07:48 /tmp/entity-file-history drwxr-xr-x - yarn hadoop 0 2016-10-25 07:48 /tmp/entity-file-history/active drwxrwxrwx - guest hdfs 0 2017-01-12 18:42 /tmp/freewheel drwxrwxrwx - guest hdfs 0 2017-01-12 18:46 /tmp/freewheel/hdfs drwx-wx-wx - ambari-qa hdfs 0 2016-10-25 07:51 /tmp/hive drwx------ - ambari-qa hdfs 0 2016-10-25 08:09 /tmp/hive/ambari-qa drwx------ - hive hdfs 0 2017-01-23 20:51 /tmp/hive/hive/_tez_session_dir drwx------ - hive hdfs 0 2017-01-16 16:03 /tmp/hive/hive/ff5fb9ba-01db-45d3-b924-e1bd6ee5203b drwx------ - hive hdfs 0 2017-01-16 16:03 /tmp/hive/hive/ff5fb9ba-01db-45d3-b924-e1bd6ee5203b/_tmp_space.db Once you get a list of those files, you can issue hdfs dfs -mv file newdir We're adding some new Grafana dashboards in the next release of Ambari that can tell with granularity who are hdfs users and what files they're creating. There's also an activity explorer dashboard you can check out in latest Ambari + Smartsense for some other HDFS file statistics, especially when you're looking for small files. ... View more

it is best to keep them separate, here are two threads for you to review with findings from the field. https://community.hortonworks.com/questions/10868/best-practices-for-storm-deployment-on-a-hadoop-cl.html https://community.hortonworks.com/articles/550/unofficial-storm-and-kafka-best-practices-guide.html ... View more