Member since
09-06-2016
9
Posts
4
Kudos Received
0
Solutions
09-16-2016
03:48 PM
I just noticed that when a path does not exist for a given hour falcon/oozie just get stuck!.. rather than check for the next hour? My misunderstanding I guess. Have got it working now.
... View more
09-15-2016
11:49 AM
Hi Sowmya, Is there another debug information I can provide to help solve the cause of the problem? Kind Regards, Liam
... View more
09-13-2016
07:45 AM
Thanks Saumitra
... View more
09-12-2016
10:14 AM
Hi Sowmya, Attached file contains the feed definition, falcon and oozie logs. I submitted and scheduled the feed around the 14:40 timestamp file.tar.gz Thanks for your help Liam
... View more
09-08-2016
07:24 PM
Hi Venkat, The property is set to *=/etc/hadoop/conf. This is just a simple single node cluster (HDP 2.3 sandbox). The s3a properties have been added to both core and hdfs site files, but still the same problem I'm afraid.
... View more
09-08-2016
12:23 PM
Full exception in oozie log is as follows: org.apache.oozie.action.hadoop.JavaMainException: com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:59) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47) at org.apache.oozie.action.hadoop.JavaMain.main(JavaMain.java:35) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:236) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:117) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3521) at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1031) at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:994) at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:297) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371) at org.apache.falcon.hadoop.HadoopClientFactory$1.run(HadoopClientFactory.java:200) at org.apache.falcon.hadoop.HadoopClientFactory$1.run(HadoopClientFactory.java:198) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.falcon.hadoop.HadoopClientFactory.createFileSystem(HadoopClientFactory.java:198) at org.apache.falcon.hadoop.HadoopClientFactory.createProxiedFileSystem(HadoopClientFactory.java:153) at org.apache.falcon.hadoop.HadoopClientFactory.createProxiedFileSystem(HadoopClientFactory.java:145) at org.apache.falcon.entity.FileSystemStorage.fileSystemEvictor(FileSystemStorage.java:317) at org.apache.falcon.entity.FileSystemStorage.evict(FileSystemStorage.java:300) at org.apache.falcon.retention.FeedEvictor.run(FeedEvictor.java:76) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.falcon.retention.FeedEvictor.main(FeedEvictor.java:52) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:56) ... 15 more I have defined fs.s3a.acccess.key, fs.s3a.secret.key, fs.s3a.endpoint in hdfs-stite.xml. I can use hdfs dfs -ls s3a://<my-buckt> from the command line, and it works. I've also set the path in the feed example to be s3a://<my-bucket>... But this exception would seem to day oozie can't see the AWS access/secret key from some reason? Regards, Liam
... View more
09-08-2016
11:35 AM
Thanks for that Sowyma, This is definitely trying to do something! But I now see an exception in oozie logs which says 160908110420441-oozie-oozi-W] ACTION[0000034-160908110420441-oozie-oozi-W@eviction] Launcher exception: com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain org.apache.oozie.action.hadoop.JavaMainException: com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:59) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47) ..
... View more
09-06-2016
10:29 AM
2 Kudos
I have not seen any example of using s3 in Falcon except for mirroring. Is it possible to use an S3-bucket as location path for a feed?
... View more
Labels:
- Labels:
-
Apache Falcon
09-06-2016
10:26 AM
2 Kudos
The scenario I have is a HDFS cluster, and a separate Object Store. The Object store provides an HDFS Compatible File System jar that I can use from the native HDFS Cluster to read/write to the object store. However, referencing the Object store requires using a distinct URL i.e. For the HDFS Cluster I use hdfs dfs -ls hdfs://<namenode>:8020/... But for the Object store I have to use a custom url hdfs dfs -ls vipers://<namenode>:8020/... If I define a falcon mirroring job on the HDFS Cluster, but using paths on the HDFS Object store, then I get URL Exceptions thrown when the job is submitted. This is because the path URI is appended to the Cluster URI.... What I think is happening is as follows: hdfs://<namenode>:8020/ can only be used to address the HDFS Cluster Files and blocks. And, while I can make the cluster access a third party HDFS compatible Object store, there is no way to make a cluster name node 'proxy' for the Object store itself. Is that right?
... View more
Labels:
- Labels:
-
Apache Hadoop