Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Trouble with classes while using s3a filesystem in oozie coordinator input dataset

Highlighted

Trouble with classes while using s3a filesystem in oozie coordinator input dataset

New Contributor

For our hadoop clusters ( versions 2.6.0 and 2.7.1) we are using s3 buckets as the source of a number of input files. We have been successful at creating an oozie workflow with a distcp action, putting the data from the bucket to where we want it on the cluster. The filesystem we chose to do this is s3a (we expect large files).

Now we want to schedule this workflow, using a location on the bucket as the uri for the input dataset. Below is an example of the dataset.

		<dataset name="s3bucket" initial-instance="${coorStartTime}" timezone="Europe/Amsterdam" frequency="15">
 			<uri-template>s3a://our-s3-bucket/${YEAR}${MONTH}${DAY}</uri-template>
 			<done-flag></done-flag>	
		</dataset>

Since this is a coordinator dataset and no oozie workflow-action has been started yet, the hadoop-aws<version>.jar in the share lib is not picked up. As a result we get an error saying that the class org.apache.hadoop.fs.s3a.S3AFileSystem is missing.

I tried adding the hadoop-aws.jar to <path-to-oozie>/libext and then do

./oozie-setup.sh prepare-war

That gave us an error when starting the oozie service back up. (the same on both clusters)

(directly placing the jar in /libserver got me the same error)

Note: without the added hadoop-aws.jar this particular part of the oozie.log shows all the share lib jars it can find, instead of the error.

2016-01-27 12:33:29,948 ERROR ShareLibService:540 - SERVER[host] USER[-] GROUP[-] org.apache.oozie.service.ServiceException: E0104: Could not fully initialize service [org.apache.oozie.service.ShareLibService], Not able to cache sharelib. An Admin needs to install the sharelib with oozie-setup.sh and issue the 'oozie admin' CLI command to update the sharelib
org.apache.oozie.service.ServiceException: E0104: Could not fully initialize service [org.apache.oozie.service.ShareLibService], Not able to cache sharelib. An Admin needs to install the sharelib with oozie-setup.sh and issue the 'oozie admin' CLI command to update the sharelib
	at org.apache.oozie.service.ShareLibService.init(ShareLibService.java:123)
	at org.apache.oozie.service.Services.setServiceInternal(Services.java:383)
	at org.apache.oozie.service.Services.setService(Services.java:369)
	at org.apache.oozie.service.Services.loadServices(Services.java:302)
	at org.apache.oozie.service.Services.init(Services.java:210)
	at org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:45)
	at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4210)
	at org.apache.catalina.core.StandardContext.start(StandardContext.java:4709)
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:583)
	at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:675)
	at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:601)
	at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:502)
	at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317)
	at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
	at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
	at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065)
	at org.apache.catalina.core.StandardHost.start(StandardHost.java:822)
	at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057)
	at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
	at org.apache.catalina.core.StandardService.start(StandardService.java:525)
	at org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
	at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
	at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
Caused by: java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: Provider org.apache.hadoop.fs.s3a.S3AFileSystem could not be instantiated
	at java.util.ServiceLoader.fail(ServiceLoader.java:224)
	at java.util.ServiceLoader.access$100(ServiceLoader.java:181)
	at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:377)
	at java.util.ServiceLoader$1.next(ServiceLoader.java:445)
	at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2586)
	at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2597)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2614)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2653)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2635)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
	at org.apache.oozie.service.ShareLibService.init(ShareLibService.java:112)
	... 29 more
Caused by: java.lang.NoClassDefFoundError: com/amazonaws/services/s3/AmazonS3
	at java.lang.Class.getDeclaredConstructors0(Native Method)
	at java.lang.Class.privateGetDeclaredConstructors(Class.java:2532)
	at java.lang.Class.getConstructor0(Class.java:2842)
	at java.lang.Class.newInstance(Class.java:345)
	at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:373)
	... 39 more
Caused by: java.lang.ClassNotFoundException: com.amazonaws.services.s3.AmazonS3
	at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1680)
	at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
	... 44 more

Other classes seems to be missing.But... The error talks about the share lib, but I changed nothing there and it works fine with the workflow My question is: why did my adding a library to oozie /libext upset the share lib? And above all how can I get the coordinator to make use of the s3a filesystem class while trying to reach the bucket?

Edited after comment by Artem Ervits

The workflow I am talking about is a regular workflow-app, using a distcp action to move .txt files from s3 to a folder on the cluster.

<workflow-app xmlns="uri:oozie:workflow:0.5" name="100-load-full">

	<global>
		<job-tracker>${jobTracker}</job-tracker>
		<name-node>${nameNode}</name-node>
		<job-xml>${jobXML}</job-xml>
		<configuration>
			<property>
				<name>mapred.job.queue.name</name>
				<value>${queueName}</value>
			</property>
		</configuration>
	</global>

	<start to="start-distcp"/>

	<action name="start-distcp">
         <distcp xmlns="uri:oozie:distcp-action:0.2">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <prepare>
            	<mkdir path="${nameNode}${dropzoneMgm}/${loadDate}"/>
            </prepare>
	    <arg>-Dmapred.job.queue.name=${queueName}</arg>
            <arg>s3a://my-favourite-s3-bucket/*</arg>
            <arg>${nameNode}${dropzoneMgm}/${loadDate}</arg>
         </distcp> 
       
        <ok to="create-success"/>
        <error to="fail"/>
    </action>
	<action name="fail">
        <email xmlns="uri:oozie:email-action:0.1">
            <to>${mailgroep}</to>   
            <subject>050-workflow-mgm-bvn.xml</subject>
            <body>De workflow is met fout(en) beeindigd bij tabel ${tableName}</body>
        </email>
        <ok to="fail-end"/>
        <error to="fail-end"/>
	</action>

	<action name="create-success">
		<fs>
			<touchz path="${nameNode}${dropzoneMgm}/${loadDate}/_SUCCESS"/>
		</fs>
		<ok to="end"/>
		<error to="fail"/>
	</action>

	<kill name="fail-end">
		<message>job failed</message>
	</kill>
	<end name="end"/>
	
</workflow-app>

Starting this workflow by using a coordinator which polls for a dataset works when this uri is on the cluster itself (hdfs uri).

For example using the dataset:

<dataset name="s3bucket" initial-instance="${coorStartTime}" timezone="Europe/Amsterdam" frequency="15">
 		<uri-template>${nameNode}/dropzone</uri-template>
 		<done-flag>_TEST</done-flag>
</dataset>

This coordinator - workflow combination does what we expect it to. It copies everything in s3a://my-favourite-s3-bucket/ to ${nameNode}${dropzoneMgm}/${loadDate} if _TEST exists in ${nameNode}/dropzone

When I change the filesystem of the uri to a path on the bucket, then it goes wrong.

By the way we did set the property oozie.service.HadoopAccessorService.supported.filesystems to hdfs,s3a in oozie-site (because the standard was just hdfs)

3 REPLIES 3
Highlighted

Re: Trouble with classes while using s3a filesystem in oozie coordinator input dataset

Mentor

@Esther Schreuders if I understand your problem, you are modifying coordinator wf. I usually leave coordinator wf alone and make sure regular workflow works. Once it works I place the whole wf directory within coordinator wf directory, modify schedule and launch. Try a sample coordinator before complicating your wf with S3 and confirm that works. Them try my suggested approach.

Highlighted

Re: Trouble with classes while using s3a filesystem in oozie coordinator input dataset

New Contributor

Thanks for the reply, I already tested whether the workflow and coordinator worked without the s3 dataset. I added some information about those to the question.

Highlighted

Re: Trouble with classes while using s3a filesystem in oozie coordinator input dataset

Mentor

Have you looked at Falcon, it has S3 support and works on top of Oozie Link @bsaini @Esther Schreuders

Don't have an account?
Coming from Hortonworks? Activate your account here