question Re: I am working with Falcon, while creating the entity for cluster i got the error for the location of staging directory does not exists. here is my xml file in Archives of Support Questions (Read Only)

I am working with Falcon, while creating the entity for cluster i got the error for the location of staging directory does not exists. here is my xml file

khushikalra_060 — Thu, 28 Jan 2016 02:11:33 GMT

<?xml version="1.0" encoding="UTF-8" standalone="yes"?><clustername="primaryCluster"description="this is primary cluster"colo="primaryColo"xmlns="uri:falcon:cluster:0.1"><tags>primaryKey=primaryValue</tags><interfaces><interfacetype="readonly"endpoint="hftp://sandbox.hortonworks.com:50070"version="2.2.0"/><interfacetype="write"endpoint="hdfs://sandbox.hortonworks.com:8020"version="2.2.0"/><interfacetype="execute"endpoint="sandbox.hortonworks.com:8050"version="2.2.0"/><interfacetype="workflow"endpoint="http://sandbox.hortonworks.com:11000/oozie/"version="4.0.0"/><interfacetype="messaging"endpoint="tcp://sandbox.hortonworks.com:61616?daemon=true"version="5.1.6"/></interfaces><locations><locationname="staging"path="/apps/falcon/primaryCluster/staging"/><locationname="temp"path="/tmp"/><locationname="working"path="/apps/falcon/primaryCluster/working"/></locations><ACLowner="ambari-qa"group="users"permission="0x755"/><properties><propertyname="test"value="value1"/></properties></cluster>

Re: I am working with Falcon, while creating the entity for cluster i got the error for the location of staging directory does not exists. here is my xml file

sramesh — Thu, 28 Jan 2016 03:11:43 GMT

Before creating the cluster entities, we need to create the directories on HDFS representing the cluster that we are going to define, namely primaryCluster in your case.

su - falcon
hadoop fs -mkdir /apps/falcon/primaryCluster

Further create directories called staging and working

hadoop fs -mkdir /apps/falcon/primaryCluster/staging

hadoop fs -mkdir /apps/falcon/primaryCluster/working

Finally you need to set the proper permissions on the staging/working directories:

hadoop fs -chmod 777 /apps/falcon/primaryCluster/staging
hadoop fs -chmod 755 /apps/falcon/primaryCluster/working
hadoop fs -chown -R falcon /apps/falcon/*

You can refer to http://hortonworks.com/hadoop-tutorial/processing-data-pipeline-with-apache-falcon/ for more details.

Re: I am working with Falcon, while creating the entity for cluster i got the error for the location of staging directory does not exists. here is my xml file

khushikalra_060 — Thu, 28 Jan 2016 04:08:19 GMT

i have done that. but its still showing the same error.

Re: I am working with Falcon, while creating the entity for cluster i got the error for the location of staging directory does not exists. here is my xml file

khushikalra_060 — Thu, 28 Jan 2016 05:03:13 GMT

i have done that before the cluster entity.. but its still showing the same error

Re: I am working with Falcon, while creating the entity for cluster i got the error for the location of staging directory does not exists. here is my xml file

sramesh — Thu, 28 Jan 2016 06:23:09 GMT

@khushi kalra: Hard to figure out what's going wrong without looking at logs. Can you tail & provide falcon.application.log when the error occurs? Also can you paste the output for

hadoop fs -ls -R /apps/falcon? Thanks!

Re: I am working with Falcon, while creating the entity for cluster i got the error for the location of staging directory does not exists. here is my xml file

khushikalra_060 — Fri, 29 Jan 2016 00:09:08 GMT

thank you its working now.

But i want to ask about the validity start date and end date of the feeds and process?

Re: I am working with Falcon, while creating the entity for cluster i got the error for the location of staging directory does not exists. here is my xml file

sramesh — Fri, 29 Jan 2016 02:36:51 GMT

khushi kalra:

Validity of a feed on cluster specifies duration for which this feed is valid on this cluster.

Process validity defines how long the workflow should run. It has 3 components - start time, end time and timezone. Start time and end time are timestamps defined in yyyy-MM-dd'T'HH:mm'Z' format and should always be in UTC. Timezone is used to compute the next instances starting from start time. The workflow will start at start time and end before end time specified on a given cluster.

Please refer this doc for more details. Thanks!

Re: I am working with Falcon, while creating the entity for cluster i got the error for the location of staging directory does not exists. here is my xml file

dconnolly — Wed, 30 Mar 2016 23:19:12 GMT

Just out of curiosity, how did you get this working, I am working through the same problem on sandbox 2.4, log attachedfalcon-app-log.txt

[falcon@sandbox logs]$ hadoop fs -ls -R /apps/falcon drwxrwxrwx - falcon hdfs 0 2016-03-30 14:56 /apps/falcon/backupCluster drwxrwxrwx - falcon hdfs 0 2016-03-30 14:54 /apps/falcon/backupCluster/staging drwxr-xr-x - falcon hdfs 0 2016-03-30 14:56 /apps/falcon/backupCluster/working drwxrwxrwx - falcon hdfs 0 2016-03-30 14:55 /apps/falcon/primaryCluster drwxrwxrwx - falcon hdfs 0 2016-03-30 14:53 /apps/falcon/primaryCluster/staging drwxr-xr-x - falcon hdfs 0 2016-03-30 14:55 /apps/falcon/primaryCluster/working