@vamsi valiveti I have to side with @Santhosh B Gowda typically scheduled jobs are handled by coordinator workflow. Implement condition logic in your Oozie workflow and if there's new data, run the actions, otherwise proceed to end action. ... View more

HDFS spout assumes that files visible in the monitored directory are not actively being updated. Only after a file is completely written should it be made visible to the spout. Following are two approaches for ensuring this: Write the file to another directory. When the write operation is finished, move the file to the monitored directory. Create the file in the monitored directory with an '.ignore' suffix; HDFS spout ignores files with an '.ignore' suffix. When the write operation is finished, rename the file to omit the suffix. When the spout is actively consuming a file, it renames the file with an .inprogress suffix. After consuming all contents in the file, the file is moved to a configurable done directory and the .inprogress suffix is dropped. @Jasper http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_storm-component-guide/content/storm-ingest-hdfs.html ... View more

@Anant Rathi I have some verified answers in this thread from engineering and also another answer from @Chris Nauroth there's a reference blog http://gbif.blogspot.com/2015/05/dont-fill-your-hdfs-disks-upgrading-to.html we don't have field agreement to one or the other policy p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Calibri} span.s1 {font-kerning: none} AvailableSpaceVolumeChoosingPolicy is not something that we have ever formally tested or certified. It was developed at Cloudera. We do not certify it under our support. ... View more

@Saumil Mayani please confirm the steps to add user/group described in this thread https://community.hortonworks.com/questions/50073/adding-new-ambari-user-with-assigned-group-with-ap.html work. ... View more

It's a good practice to accept answer if it satisfies your needs. ... View more

@Saumil Mayani that's a great suggestion, it was on my follow up list as when you provision a new user and/or new view, permissions need to be added. Will follow up with that suggestion soon. ... View more

You can learn that by following this guide https://community.hortonworks.com/content/kbentry/9148/troubleshooting-an-oozie-flow.html ... View more

That also works, I'm going to look at the effort to contribute an enhancement for FS action to add this functionality. Seems only a few FS actions were implemented where so many are available with hdfs shell. ... View more

@justlearning same version of Python needs to be installed on every node that will run oozie containers (nodemanager). Same goes for any Python libraries you're importing into your script. I usually create the following tree admin@u1201:~/oozie/apps/python$ tree . |-- job.properties |-- scripts | `-- script.py `-- workflow.xml 1 directory, 3 files so what you want is a workflow directory on hdfs with at least workflow.xml and optionally another directory within it with a Python script. job.properties file needs to be on your local filesystem. Then you would execute the oozie wf the following way: oozie job -oozie http://u1203.ambari.apache.org:11000/oozie -config oozie/apps/python/job.properties -run ... View more