question Re: Could not submit falcon feed schedule with different user in Support Questions

Could not submit falcon feed schedule with different user

venkata_gangava — Sat, 12 Dec 2015 05:36:37 GMT

Hi,

We are using falcon-0.6.

I am trying to submit Falcon feed schedule as user1 but it throws below error

Permission denied: user=user1, access=WRITE, inode="/apps/falcon/backupCluster/staging/falcon/workflows/feed":falcon:hdfs:drwxr-xr-x

I gave 777 permissions to staging directory but I still see the same error.

I am able to submit the schedule as falcon user but could not as any other user.

Please help me with this

Thanks,

Venkat

Re: Could not submit falcon feed schedule with different user

bvellanki — Sat, 12 Dec 2015 05:47:06 GMT

Hi this is a known bug https://issues.apache.org/jira/browse/FALCON-1647. The workaround until we release next version of Falcon is to give 777 permissions to all directories under /apps/falcon/backupCluster/staging/falcon.

Re: Could not submit falcon feed schedule with different user

yzheng — Sat, 12 Dec 2015 05:52:50 GMT

This is a known bug in Falcon and was recently fixed in FALCON-1647 (https://issues.apache.org/jira/browse/FALCON-1647). To fixed your issue, please gave 777 permission to the following two directories: /apps/falcon/backupCluster/staging/falcon/workflows/feed and /apps/falcon/backupCluster/staging/falcon/workflows/process. Otherwise, it only allows the first user to submit entities. Thanks.

Re: Could not submit falcon feed schedule with different user

venkata_gangava — Mon, 14 Dec 2015 23:42:37 GMT

Thank you for your quick response.

I am seeing that error even after giving 777 permission to all directories within staging directory for both primaryCluster and backupCluster.

Re: Could not submit falcon feed schedule with different user

bvellanki — Wed, 16 Dec 2015 03:41:03 GMT

Hi Venkata, Can you please go to the falcon.application.log file, and paste the full exception you see when you attempt to submit the feed? This way, we can debug better.

Re: Could not submit falcon feed schedule with different user

venkata_gangava — Thu, 17 Dec 2015 02:17:46 GMT

T//entities/schedule/feed/hdfspdrrep] ~ Unable to schedule workflow (AbstractSchedulableEntityManager:66) org.apache.falcon.FalconException: Error preparing base staging dirs: /apps/falcon/backupCluster/staging/falcon/workflows/feed/hdfspdrrep at org.apache.falcon.workflow.engine.OozieWorkflowEngine.prepareEntityBuildPath(OozieWorkflowEngine.java:185) at org.apache.falcon.workflow.engine.OozieWorkflowEngine.schedule(OozieWorkflowEngine.java:153) at org.apache.falcon.resource.AbstractSchedulableEntityManager.scheduleInternal(AbstractSchedulableEntityManager.java:76) at org.apache.falcon.resource.AbstractSchedulableEntityManager.schedule(AbstractSchedulableEntityManager.java:63) at org.apache.falcon.resource.SchedulableEntityManager.schedule(SchedulableEntityManager.java:116) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

Caused by: org.apache.hadoop.security.AccessControlException: Permission denied: user=hdpexeusr, access=WRITE, inode="/apps/falcon/backupCluster/staging/falcon/workflows/feed":falcon:hdfs:drwxr-xr-x at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:271) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:257) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:238) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:179) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6795) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6777) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:6729) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:4495) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:4465) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:4438) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:830) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:614)

.......................

... 61 more Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=hdpexeusr, access=WRITE, inode="/apps/falcon/backupCluster/staging/falcon/workflows/feed":falcon:hdfs:drwxr-xr-x at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:271) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:257) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:238) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:179) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6795) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6777) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:6729) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:4495) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:4465) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:4438) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:830) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:614)

......................

2015-12-11 14:22:12,668 ERROR - [1088325169@qtp-1225320031-7009:717f3588-92ea-412b-a666-698ac323b961 hdpexeusr:POST//entities/schedule/feed/hdfspdrrep] ~ Action failed: Bad Request Error: Error preparing base staging dirs: /apps/falcon/backupCluster/staging/falcon/workflows/feed/hdfspdrrep (FalconWebException:68) 2015-12-11 14:22:12,669 INFO - [1088325169@qtp-1225320031-7009:717f3588-92ea-412b-a666-698ac323b961 hdpexeusr:POST//entities/schedule/feed/hdfspdrrep] ~ {Action:schedule, Dimensions:{entityType=feed, colo=*, entityName=hdfspdrrep}, Status: FAILED, Time-taken:232141743 ns} (METRIC:38)

Re: Could not submit falcon feed schedule with different user

bvellanki — Thu, 17 Dec 2015 02:22:52 GMT

The error says

... 61 more Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=hdpexeusr, access=WRITE, inode="/apps/falcon/backupCluster/staging/falcon/workflows/feed":falcon:hdfs:drwxr-xr-x

It is not sufficient to give perms to just the feed or process dir under /apps/falcon/backupCluster/staging/falcon/workflows , you must have permission 777 for ALL directories under /apps/falcon/backupCluster/staging.

Re: Could not submit falcon feed schedule with different user

venkata_gangava — Fri, 18 Dec 2015 23:40:32 GMT

Hey Balu,

Thanks for quick responses on the issue. I really appreciate it.

I gave recursive 777 permissions to the entire directory.

drwxrwxrwx - falcon hdfs 0 2015-12-10 15:42 /apps/falcon/backupCluster/staging

I still see the below error

Re: Could not submit falcon feed schedule with different user

bvellanki — Fri, 18 Dec 2015 23:45:09 GMT

Hi, the error clearly says you did not give 777 permissions. Can you please try this command?

hadoop fs -chmod -R 777 /apps/falcon/backupCluster/staging

Re: Could not submit falcon feed schedule with different user

venkata_gangava — Fri, 18 Dec 2015 23:58:06 GMT

I think the problems is, though I gave 777 over /apps/falcon/backupCluster/staging, when I tried creating directory under /apps/falcon/backupCluster/staging/falcon/workflows/feed as falcon user, it is created with 755 permissions as below

[falcon@hostname ~]$ hdfs dfs -mkdir /apps/falcon/backupCluster/staging/falcon/workflows/feed/test1

hdfs dfs -ls /apps/falcon/backupCluster/staging/falcon/workflows/feed

drwxr-xr-x - falcon hdfs 0 2015-12-18 09:55 /apps/falcon/backupCluster/staging/falcon/workflows/feed/test1

The directory is not created with 777 permission (umask 022).

Do you think it could be the reason?

Re: Could not submit falcon feed schedule with different user

bvellanki — Sat, 19 Dec 2015 00:39:33 GMT

Yes. That is correct.

Re: Could not submit falcon feed schedule with different user

mrakki3110 — Thu, 24 Dec 2015 16:56:04 GMT

My falcon.application.log file shows the following exception :

Caused by: org.apache.falcon.FalconException: E0501 : E0501: Could not perform authorization operation, Unauthorized connection for super-user: oozie from IP 52.8.144.45
	at org.apache.falcon.workflow.engine.OozieWorkflowEngine.dryRunInternal(OozieWorkflowEngine.java:234)
	at org.apache.falcon.workflow.engine.OozieWorkflowEngine.schedule(OozieWorkflowEngine.java:172)
	at org.apache.falcon.resource.AbstractSchedulableEntityManager.scheduleInternal(AbstractSchedulableEntityManager.java:94)
What & How to configure oozie for falcon ??