oozie submit spark action but get a error: Not implemented by the TFS FileSystem implementation

[#1]

I install the ambari and the hdp 2.3.4

Resently, I wanted to use the oozie as a job to execute the spark action, so I tested the spark examples but I found a problem.

--jobproperties

--lib

----spark-assembly.jar

----test-spark.jar

--workflow.xml

When I submit the oozie job, the action was killed.

the logs say:

2016-02-17 17:55:47,252 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1455672954475_0016_000002
2016-02-17 17:55:47,682 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2016-02-17 17:55:47,708 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
2016-02-17 17:55:47,966 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (appAttemptId { application_id { id: 16 cluster_timestamp: 1455672954475 } attemptId: 2 } keyId: 1590943555)
2016-02-17 17:55:47,968 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: RM_DELEGATION_TOKEN, Service: 192.168.1.200:8032,192.168.1.202:8032, Ident: (owner=root, renewer=oozie mr token, realUser=oozie, issueDate=1455702938114, maxDate=1456307738114, sequenceNumber=134, masterKeyId=46)
2016-02-17 17:55:48,068 INFO [main] org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED; cause: java.lang.UnsupportedOperationException: Not implemented by the TFS FileSystem implementation
java.lang.UnsupportedOperationException: Not implemented by the TFS FileSystem implementation
	at org.apache.hadoop.fs.FileSystem.getScheme(FileSystem.java:216)
	at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2564)
	at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2574)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.getFileSystem(MRAppMaster.java:497)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:281)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1499)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1496)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1429)

so, Have anyone know how to solve this problem?

Thanks.

[#2]

@allen huang

Not implemented by the TFS FileSystem implementation

Tachyon File system is not supported.

It has to do with Tachyon jars

[#3]

More details https://issues.apache.org/jira/browse/SPARK-8385

At this point in time Hortonworks does not support Oozie Spark action. We have it listed here as not supported at this time:

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_HDP_RelNotes/content/community_features.h...

[#4]

Thanks for your help.

but as I know, The TFS is need by spark-assembly.jar and it has bundle to spark-assembly.jar. So I don't need to add a new TFS jar.

Have anyone use oozie submit the spark action success? Please tell me how to do this.

Thanks.

[#5]

@allen huang Please see my response. "At this point in time Hortonworks does not support Oozie Spark action. We have it listed here as not supported at this time:

[#6]

Oh , Thanks for your response.

but, I need a method to make a job to execute a spark program. Could you help me?

[#7]

@allen huang

https://oozie.apache.org/docs/4.2.0/DG_SparkActionExtension.html

" If you have Tachyon package included at a lower version that would caused this. Another reason is if you have Tachyon packages in oozie shared lib."

[#8]

Yes, I see a reason to solve this problem in this

https://issues.apache.org/jira/browse/SPARK-8385

org.tachyonproject:tachyon:0.5.0:jar => org.tachyonproject:tachyon:0.6.4:jar

the effect is spark version 1.4.0

-----------------------------------------------------------------------------------------------------

but I don't know how to check if I have Tachyon package with a lower version.

so I download spark 1.5.2 from spark offical website and change the spark-assembly.jar , but the problem is also exist.

and I think you said the oozie shared lib is the "/usr/hdp/2.3.4.0-3485/oozie/libext" , but this folder is empty.

so what can I do ,next.

think for you help.

[#9]

@allen huang I would recommend to wait and check on the official release. I have this

[root@sandbox libext]# ls

ext-2.2.zip

falcon-oozie-el-extension-0.6.1.2.3.2.0-2950.jar

[root@sandbox libext]# pwd

/usr/hdp/2.3.2.0-2950/oozie/libext

[root@sandbox libext]#

You can copy jars manullay under the directory and try it

[#10]

you can search for a file across the entire filesystem.

This won't find libraries which have copied the contents of the JAR in (spark-assembly), but it will find self-contained tachyon releases

find . -name tachyon\*.jar -print