question oozie submit spark action but get a error: Not implemented by the TFS FileSystem implementation in Support Questions

oozie submit spark action but get a error: Not implemented by the TFS FileSystem implementation

fengyeyouni — Thu, 18 Feb 2016 09:55:06 GMT

I install the ambari and the hdp 2.3.4

Resently, I wanted to use the oozie as a job to execute the spark action, so I tested the spark examples but I found a problem.

--jobproperties

--lib

----spark-assembly.jar

----test-spark.jar

--workflow.xml

When I submit the oozie job, the action was killed.

the logs say:

2016-02-17 17:55:47,252 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1455672954475_0016_000002
2016-02-17 17:55:47,682 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2016-02-17 17:55:47,708 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
2016-02-17 17:55:47,966 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (appAttemptId { application_id { id: 16 cluster_timestamp: 1455672954475 } attemptId: 2 } keyId: 1590943555)
2016-02-17 17:55:47,968 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: RM_DELEGATION_TOKEN, Service: 192.168.1.200:8032,192.168.1.202:8032, Ident: (owner=root, renewer=oozie mr token, realUser=oozie, issueDate=1455702938114, maxDate=1456307738114, sequenceNumber=134, masterKeyId=46)
2016-02-17 17:55:48,068 INFO [main] org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED; cause: java.lang.UnsupportedOperationException: Not implemented by the TFS FileSystem implementation
java.lang.UnsupportedOperationException: Not implemented by the TFS FileSystem implementation
	at org.apache.hadoop.fs.FileSystem.getScheme(FileSystem.java:216)
	at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2564)
	at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2574)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.getFileSystem(MRAppMaster.java:497)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:281)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1499)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1496)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1429)

so, Have anyone know how to solve this problem?

Thanks.

Re: oozie submit spark action but get a error: Not implemented by the TFS FileSystem implementation

nsabharwal — Thu, 18 Feb 2016 09:58:33 GMT

@allen huang

Not implemented by the TFS FileSystem implementation

Tachyon File system is not supported.

It has to do with Tachyon jars

Re: oozie submit spark action but get a error: Not implemented by the TFS FileSystem implementation

nsabharwal — Thu, 18 Feb 2016 10:10:17 GMT

More details https://issues.apache.org/jira/browse/SPARK-8385

At this point in time Hortonworks does not support Oozie Spark action. We have it listed here as not supported at this time:

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_HDP_RelNotes/content/community_features.html

Re: oozie submit spark action but get a error: Not implemented by the TFS FileSystem implementation

fengyeyouni — Thu, 18 Feb 2016 10:15:18 GMT

Thanks for your help.

but as I know, The TFS is need by spark-assembly.jar and it has bundle to spark-assembly.jar. So I don't need to add a new TFS jar.

Have anyone use oozie submit the spark action success? Please tell me how to do this.

Thanks.

Re: oozie submit spark action but get a error: Not implemented by the TFS FileSystem implementation

aervits — Thu, 18 Feb 2016 10:21:44 GMT

@allen huang

oozie Spark action is not supported at this time.

Re: oozie submit spark action but get a error: Not implemented by the TFS FileSystem implementation

nsabharwal — Thu, 18 Feb 2016 10:28:21 GMT

@allen huang Please see my response. "At this point in time Hortonworks does not support Oozie Spark action. We have it listed here as not supported at this time:

Re: oozie submit spark action but get a error: Not implemented by the TFS FileSystem implementation

fengyeyouni — Thu, 18 Feb 2016 10:38:22 GMT

Oh , Thanks for your response.

but, I need a method to make a job to execute a spark program. Could you help me?

Re: oozie submit spark action but get a error: Not implemented by the TFS FileSystem implementation

nsabharwal — Thu, 18 Feb 2016 10:45:57 GMT

@allen huang

https://oozie.apache.org/docs/4.2.0/DG_SparkActionExtension.html

" If you have Tachyon package included at a lower version that would caused this. Another reason is if you have Tachyon packages in oozie shared lib."

Re: oozie submit spark action but get a error: Not implemented by the TFS FileSystem implementation

fengyeyouni — Thu, 18 Feb 2016 11:24:21 GMT

Yes, I see a reason to solve this problem in this

https://issues.apache.org/jira/browse/SPARK-8385

org.tachyonproject:tachyon:0.5.0:jar => org.tachyonproject:tachyon:0.6.4:jar

the effect is spark version 1.4.0

-----------------------------------------------------------------------------------------------------

but I don't know how to check if I have Tachyon package with a lower version.

so I download spark 1.5.2 from spark offical website and change the spark-assembly.jar , but the problem is also exist.

and I think you said the oozie shared lib is the "/usr/hdp/2.3.4.0-3485/oozie/libext" , but this folder is empty.

so what can I do ,next.

think for you help.

Re: oozie submit spark action but get a error: Not implemented by the TFS FileSystem implementation

aervits — Thu, 18 Feb 2016 21:12:45 GMT

@allen huang Spark action is not supported, feel free to search HCC for workarounds as they do exist but it's a waste of time as we will add support to oozie jobs in future releases and it will most likely break compatibility with your workarounds. https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_spark-guide/content/ch_introduction-spark.html

Re: oozie submit spark action but get a error: Not implemented by the TFS FileSystem implementation

nsabharwal — Thu, 18 Feb 2016 21:16:14 GMT

@allen huang I would recommend to wait and check on the official release. I have this

[root@sandbox libext]# ls

ext-2.2.zip

falcon-oozie-el-extension-0.6.1.2.3.2.0-2950.jar

[root@sandbox libext]# pwd

/usr/hdp/2.3.2.0-2950/oozie/libext

[root@sandbox libext]#

You can copy jars manullay under the directory and try it

Re: oozie submit spark action but get a error: Not implemented by the TFS FileSystem implementation

fengyeyouni — Fri, 19 Feb 2016 09:56:30 GMT

I found a method to submit a spark job, but it is not oozie.

I use pentaho kettle as a workflow engine and it can run well.

But anyway , I should thinks for your help.

Re: oozie submit spark action but get a error: Not implemented by the TFS FileSystem implementation

aervits — Fri, 19 Feb 2016 09:58:07 GMT

@allen huang you could also try Oozie shell action calling Spark but I'm glad you got a workaround

Re: oozie submit spark action but get a error: Not implemented by the TFS FileSystem implementation

achittela — Thu, 10 Mar 2016 06:03:50 GMT

I included spark-assembly-1.4.1.2.3.2.0-2950-hadoop2.7.1.2.3.2.0-2950.jar file under lib folder ,

still i am gettingthe following exception

016-03-09 16:44:31,942 INFO [main] org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED; cause: java.lang.UnsupportedOperationException: Not implemented by the TFS FileSystem implementation
java.lang.UnsupportedOperationException: Not implemented by the TFS FileSystem implementation
	at org.apache.hadoop.fs.FileSystem.getScheme(FileSystem.java:217)
	at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2624)
	at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2634)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2651)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:170)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.getFileSystem(MRAppMaster.java:503)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:287)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1518)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1515)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1448)

Re: oozie submit spark action but get a error: Not implemented by the TFS FileSystem implementation

stevel — Fri, 18 Mar 2016 18:55:05 GMT

Have you any tachyon.jar somewhere? Because that's caused by an old version getting on the classpath

see: https://issues.apache.org/jira/browse/SPARK-8385?focusedCommentId=14643652&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14643652

Re: oozie submit spark action but get a error: Not implemented by the TFS FileSystem implementation

stevel — Sat, 19 Mar 2016 19:53:30 GMT

you can search for a file across the entire filesystem.

This won't find libraries which have copied the contents of the JAR in (spark-assembly), but it will find self-contained tachyon releases

find . -name tachyon\*.jar -print