oozie submit spark action but get a error: Not implemented by the TFS FileSystem implementation

[#1]

I install the ambari and the hdp 2.3.4

Resently, I wanted to use the oozie as a job to execute the spark action, so I tested the spark examples but I found a problem.

--jobproperties

--lib

----spark-assembly.jar

----test-spark.jar

--workflow.xml

When I submit the oozie job, the action was killed.

the logs say:

2016-02-17 17:55:47,252 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1455672954475_0016_000002
2016-02-17 17:55:47,682 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2016-02-17 17:55:47,708 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
2016-02-17 17:55:47,966 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (appAttemptId { application_id { id: 16 cluster_timestamp: 1455672954475 } attemptId: 2 } keyId: 1590943555)
2016-02-17 17:55:47,968 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: RM_DELEGATION_TOKEN, Service: 192.168.1.200:8032,192.168.1.202:8032, Ident: (owner=root, renewer=oozie mr token, realUser=oozie, issueDate=1455702938114, maxDate=1456307738114, sequenceNumber=134, masterKeyId=46)
2016-02-17 17:55:48,068 INFO [main] org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED; cause: java.lang.UnsupportedOperationException: Not implemented by the TFS FileSystem implementation
java.lang.UnsupportedOperationException: Not implemented by the TFS FileSystem implementation
	at org.apache.hadoop.fs.FileSystem.getScheme(FileSystem.java:216)
	at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2564)
	at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2574)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.getFileSystem(MRAppMaster.java:497)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:281)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1499)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1496)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1429)

so, Have anyone know how to solve this problem?

Thanks.

[#2]

@allen huang

oozie Spark action is not supported at this time.

[#3]

@allen huang Spark action is not supported, feel free to search HCC for workarounds as they do exist but it's a waste of time as we will add support to oozie jobs in future releases and it will most likely break compatibility with your workarounds. https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_spark-guide/content/ch_introduction-spar...

[#4 · Accepted solution]

I found a method to submit a spark job, but it is not oozie.

I use pentaho kettle as a workflow engine and it can run well.

But anyway , I should thinks for your help.

[#5]

@allen huang you could also try Oozie shell action calling Spark but I'm glad you got a workaround

[#6]

I included spark-assembly-1.4.1.2.3.2.0-2950-hadoop2.7.1.2.3.2.0-2950.jar file under lib folder ,

still i am gettingthe following exception

016-03-09 16:44:31,942 INFO [main] org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED; cause: java.lang.UnsupportedOperationException: Not implemented by the TFS FileSystem implementation
java.lang.UnsupportedOperationException: Not implemented by the TFS FileSystem implementation
	at org.apache.hadoop.fs.FileSystem.getScheme(FileSystem.java:217)
	at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2624)
	at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2634)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2651)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:170)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.getFileSystem(MRAppMaster.java:503)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:287)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1518)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1515)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1448)

[#7]

Have you any tachyon.jar somewhere? Because that's caused by an old version getting on the classpath

see: https://issues.apache.org/jira/browse/SPARK-8385?focusedCommentId=14643652&page=com.atlassian.jira.p...