Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.
Nukala
Nukala
Contributor
My Accepted Solutions
Show All

Re: Facing problem in S3 metadata and lineage extr...

by Contributor Nukala in Support Questions
‎01-06-2020 01:07 AM
‎01-06-2020 01:07 AM
Hi Iwang,   The cloudera manager and CDH version is 5.13   ... View more

Re: Facing problem in S3 metadata and lineage extr...

by Contributor Nukala in Support Questions
‎10-07-2019 12:54 AM
‎10-07-2019 12:54 AM
Hi,   Below are the 2 screen shots. If you see hivesampledirectory is not showing in cloudera Navigator, but it is present in S3. And other directory like SampleDirectory, Devfolder and all are present in S3, but it is showing in deleted status in ClouderaNavigator.   I checked log files (stdout, std err), don't see any error. But in Role log file I can see some error that specified bucket doesn't exist. I am giving the log information after the screen shot.  CloudEra navigator image S3 Image     Log file :   2019-10-07 09:15:12,324 INFO com.cloudera.nav.utils.JsonModelLoader [main]: Added audit event model for HUE 2019-10-07 09:15:12,333 INFO com.cloudera.nav.utils.JsonModelLoader [main]: Added audit event model for SOLR 2019-10-07 09:15:12,337 INFO com.cloudera.nav.utils.JsonModelLoader [main]: Added audit event model for SENTRY 2019-10-07 09:15:12,343 INFO com.cloudera.nav.utils.JsonModelLoader [main]: Added audit event model for NAVMS 2019-10-07 09:15:12,901 INFO com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource [main]: Initializing c3p0 pool... com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3, acquireRetryAttempts -> 30, acquireRetryDelay -> 1000, autoCommitOnClose -> false, automaticTestTable -> null, breakAfterAcquireFailure -> false, checkoutTimeout -> 0, connectionCustomizerClassName -> null, connectionTesterClassName -> com.mchange.v2.c3p0.impl.DefaultConnectionTester, dataSourceName -> 2t7fqea519ss8gsjrxr8i|45099dd3, debugUnreturnedConnectionStackTraces -> false, description -> null, driverClass -> com.mysql.jdbc.Driver, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> false, identityToken -> 2t7fqea519ss8gsjrxr8i|45099dd3, idleConnectionTestPeriod -> 300, initialPoolSize -> 2, jdbcUrl -> jdbc:mysql://quickstart.cloudera/navms?useUnicode=true&characterEncoding=UTF-8, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 0, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 50, maxStatements -> 0, maxStatementsPerConnection -> 0, minPoolSize -> 5, numHelperThreads -> 3, numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery -> null, properties -> {user=******, password=BUNDLE-REDACTED propertyCycle -> 0, testConnectionOnCheckin -> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout -> 0, usesTraditionalReflectiveProxies -> false ] 2019-10-07 09:15:16,831 INFO com.cloudera.nav.analytics.dataservices.api.service.DataService [main]: Caching is enabled for SOLR based metrics for usage in dashboard. 2019-10-07 09:15:19,627 WARN org.springframework.web.context.support.XmlWebApplicationContext [main]: Exception encountered during context initialization - cancelling refresh attempt org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'CDXExtractorService' defined in URL [jar:file:/var/lib/cloudera-scm-navigator/temp/jetty-0.0.0.0-7187-nav-core-webapp-2.12.0.war-_-any-/webapp/WEB-INF/lib/nav-service-2.12.0.jar!/com/cloudera/nav/extract/CDXExtractorService.class]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.cloudera.nav.extract.CDXExtractorService]: Constructor threw exception; nested exception is com.amazonaws.services.s3.model.AmazonS3Exception: The specified bucket does not exist (Service: Amazon S3; Status Code: 404; Error Code: NoSuchBucket; Request ID: 864FF96F219DEC19), S3 Extended Request ID: 7uWwv/0AOsBSSP0z+hHez1a82xhjdNvAv/g+ZKr5PxQ6Nq9q1VI6f+alFAf7pEYytaKU5mQa1tc= at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:278) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1115) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1018) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:504) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:475) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:302) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:298) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:706) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:762) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:482) at org.apache.cxf.transport.servlet.CXFServlet.createSpringContext(CXFServlet.java:160) at org.apache.cxf.transport.servlet.CXFServlet.loadBus(CXFServlet.java:74) at org.apache.cxf.transport.servlet.CXFNonSpringServlet.init(CXFNonSpringServlet.java:71) at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:532) at org.eclipse.jetty.servlet.ServletHolder.doStart(ServletHolder.java:344) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64) at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:791) at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:265) at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1242) at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:717) at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:494) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64) at org.eclipse.jetty.server.handler.HandlerCollection.doStart(HandlerCollection.java:229) at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:172) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64) at org.eclipse.jetty.server.handler.HandlerWrapper.doStart(HandlerWrapper.java:95) at org.eclipse.jetty.server.Server.doStart(Server.java:282) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64) at com.cloudera.nav.server.NavServer.run(NavServer.java:422) at com.cloudera.nav.server.NavServer.main(NavServer.java:117) Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.cloudera.nav.extract.CDXExtractorService]: Constructor threw exception; nested exception is com.amazonaws.services.s3.model.AmazonS3Exception: The specified bucket does not exist (Service: Amazon S3; Status Code: 404; Error Code: NoSuchBucket; Request ID: 864FF96F219DEC19), S3 Extended Request ID: 7uWwv/0AOsBSSP0z+hHez1a82xhjdNvAv/g+ZKr5PxQ6Nq9q1VI6f+alFAf7pEYytaKU5mQa1tc= at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:164) at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:125) at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:270) ... 31 more Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: The specified bucket does not exist (Service: Amazon S3; Status Code: 404; Error Code: NoSuchBucket; Request ID: 864FF96F219DEC19), S3 Extended Request ID: 7uWwv/0AOsBSSP0z+hHez1a82xhjdNvAv/g+ZKr5PxQ6Nq9q1VI6f+alFAf7pEYytaKU5mQa1tc= at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1545) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1183) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:964) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:676) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:650) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:633) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:601) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:583) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:447) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4137) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4079) at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:819) at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:797) at com.cloudera.cdx.client.impl.bulk.download.S3Downloader.getAllKeysAtPrefix(S3Downloader.java:215) at com.cloudera.cdx.client.impl.bulk.download.S3Downloader.recoverUncommitted(S3Downloader.java:241) at com.cloudera.cdx.client.impl.bulk.download.S3Downloader.<init>(S3Downloader.java:75) at com.cloudera.cdx.client.impl.bulk.CdxBulkImporter.<init>(CdxBulkImporter.java:61) at com.cloudera.cdx.client.CdxImporterFactory.newImporter(CdxImporterFactory.java:37) at com.cloudera.nav.extract.AbstractCDXImporterExtractorFactory.createImporter(AbstractCDXImporterExtractorFactory.java:46) at com.cloudera.nav.extract.AbstractCDXImporterExtractorFactory.<init>(AbstractCDXImporterExtractorFactory.java:36) at com.cloudera.nav.extract.AbstractCDXExtractorFactory.<init>(AbstractCDXExtractorFactory.java:49) at com.cloudera.nav.extract.CDXSourceExtractor.<init>(CDXSourceExtractor.java:45) at com.cloudera.nav.extract.CDXExtractorService.<init>(CDXExtractorService.java:88) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:148) ... 33 more 2019-10-07 09:15:19,794 WARN / [main]: unavailable org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'CDXExtractorService' defined in URL [jar:file:/var/lib/cloudera-scm-navigator/temp/jetty-0.0.0.0-7187-nav-core-webapp-2.12.0.war-_-any-/webapp/WEB-INF/lib/nav-service-2.12.0.jar!/com/cloudera/nav/extract/CDXExtractorService.class]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.cloudera.nav.extract.CDXExtractorService]: Constructor threw exception; nested exception is com.amazonaws.services.s3.model.AmazonS3Exception: The specified bucket does not exist (Service: Amazon S3; Status Code: 404; Error Code: NoSuchBucket; Request ID: 864FF96F219DEC19), S3 Extended Request ID: 7uWwv/0AOsBSSP0z+hHez1a82xhjdNvAv/g+ZKr5PxQ6Nq9q1VI6f+alFAf7pEYytaKU5mQa1tc= at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:278) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1115) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1018) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:504) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:475) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:302) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:298) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:706) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:762) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:482) at org.apache.cxf.transport.servlet.CXFServlet.createSpringContext(CXFServlet.java:160) at org.apache.cxf.transport.servlet.CXFServlet.loadBus(CXFServlet.java:74) at org.apache.cxf.transport.servlet.CXFNonSpringServlet.init(CXFNonSpringServlet.java:71) ... View more

Facing problem in S3 metadata and lineage extracti...

by Contributor Nukala in Support Questions
‎09-23-2019 01:46 AM
‎09-23-2019 01:46 AM
Hi,   Below are my questions.   1. Observed that in Cloudera Navigator, once the metadata is collected from Amazon s3 I see that the data is updating its status to deleted (deleted property to true). But actually in S3 the file/directory is not deleted. What is the reason for this? 2.Created an external table in hive specifying its location to AmazonS3 path using Hue. Observed that the S3 data is not reflecting automatically. Some time it reflects after restarting and some time it is not. Is there any specific configurations for this to reflect S3 items immediately in Cloudera Navigator? 3. I followed the steps to extract S3 metadata and lineage information from Amazon s3 to cloudera navigator, but in lineage tab folder hierarchy is showing. I follwed below url. https://www.cloudera.com/documentation/enterprise/5-14-x/topics/navigator_s3_metadata_extraction.html Do I need to create any cloudera altus account for lineage extraction? 4. Can we able to see any lineage for the below query(expecting that there is lineage between hive table and s3 directory), Is my expectation is correct? CREATE EXTERNAL TABLE sampletable (key STRING, value INT) LOCATION 's3n://mybucketName/directoryName'; Also, I have created a backup job between HDFS : S3 and Hive : S3, but was not able to see the lineage.   Thanks, ... View more

Cloudera Navigator and S3 source type

by Contributor Nukala in Support Questions
‎07-23-2019 10:21 AM
‎07-23-2019 10:21 AM
Hello,   I am trying to add AWS credentials as External Account and getting the S3 metadata information in Cloudera Navigator. For this when i use the personal account I was able to get the S3 entities metadata information, but when i do with my corp AWS account I was not able to get the S3 entities metadata information. Could you please help in providing the policies/permissions from AWS side to get access to S3 from Cloudera Navigator.     Thanks & Regards, Rajesh ... View more

Re: Not able to find Cloudera Navigator in clouder...

by Contributor Nukala in Support Questions
‎07-23-2019 03:51 AM
‎07-23-2019 03:51 AM
Thanks. It worked. ... View more

Not able to find Cloudera Navigator in cloudera-qu...

by Contributor Nukala in Support Questions
‎07-15-2019 11:49 AM
‎07-15-2019 11:49 AM
I am trying to install Cloudera using VMware and docker image and after installtion I am not able to find the Cloudera Navigator and metadataserver running. Could you please quickly help on this. Version used: cloudera-quickstart-vm-5.13.0-0-beta-docker cloudera-quickstart-vm-5.13.0-0-vmware   Thanks in advance.   Regards, Rajesh ... View more

Re: S3 Data discovery in Cloudera navigator

by Contributor Nukala in Support Questions
‎05-15-2019 01:36 PM
‎05-15-2019 01:36 PM
Hi Seth,   Thanks for the response. I have generated the bundle data file and not sure that it can contain any credentials. Let me confirm from my side. Can you please confirm that it will get credentials information too?     Thanks & Regards, Rajesh ... View more

Re: S3 Data discovery in Cloudera navigator

by Contributor Nukala in Support Questions
‎05-13-2019 02:11 AM
‎05-13-2019 02:11 AM
Hello, I have tried to do custom configuration and done setup for SQS and SNS and I see that the messages are part of the Queue but not received in Clouderanavigator. Could you please help on this. SQSMessages   Thanks & Regards, Rajesh ... View more

Re: S3 Data discovery in Cloudera navigator

by Contributor Nukala in Support Questions
‎05-10-2019 06:43 PM
‎05-10-2019 06:43 PM
Hello, Can you please help on this issue. Thanks & Regards, Rajesh ... View more

Re: S3 Data discovery in Cloudera navigator

by Contributor Nukala in Support Questions
‎05-08-2019 11:20 PM
‎05-08-2019 11:20 PM
Hello @seth ,   Can you please help on this issue?   Thanks & Regards, Rajesh ... View more

Re: S3 Data discovery in Cloudera navigator

by Contributor Nukala in Support Questions
‎05-07-2019 03:10 AM
‎05-07-2019 03:10 AM
Hi Seth,   Thank you for the link. I followed the document and created default configuration and also provided Amazon S3 and SQS access to the user I used. Still I was not able to see any metadata or source type for S3 in Cloudera navigator. Could you please help on this? Also, please let me know how I can get the live support to resolve this issue quickly.   Thanks & Regards, Rajesh ... View more

Re: S3 Data discovery in Cloudera navigator

by Contributor Nukala in Support Questions
‎05-03-2019 12:44 AM
‎05-03-2019 12:44 AM
Hi,   I am trying to collect S3 objects metadata in ClouderaNavigator and did the following configuration after which I was not able to see any sourceType for S3 in Cloudera Navigator. Could you please help on this Cloudera Version: CDH 6.1.0, Parcels Added S3 account as an External Account in Cloudera Manager and enabled metadata   After that I see the S3 Extraction was enabled in Cloudera Navigator Administration Advanced tab After that I was not able to get any metadata or sourcetype added for S3 in Cloudera Navigator. Did i miss something here?   Thanks & Regards, Rajesh ... View more

Re: S3 connectivity using impala shell

by Contributor Nukala in Support Questions
‎12-13-2018 03:35 AM
‎12-13-2018 03:35 AM
Hello,   Now I was able to run the same query by specifying like this. Not sure does the one which is failed requires any more configuration to be set.   alter table mostly_on_hdfs add partition (year=2015) location 's3a://AWS_ACCESS_KEY_ID:AWS_SECRET_ACCESS_KEY@myclouderaraj/root';   My intension of running these queries to get metadata and lineage information of S3 items in Cloudera Navigator. But I see nothing related to S3 in Cloudera Navigator.     Thanks, Rajesh   ... View more

S3 Data discovery in Cloudera navigator

by Contributor Nukala in Support Questions
‎12-12-2018 06:50 PM
‎12-12-2018 06:50 PM
Hello,   I am trying to create an environment to have metadata and lineage information in Cloudera Navigator. For this I am trying to add the connection of S3 and thinking of running the backup sync from HDFS to S3. Does this collects the metadata lineage information in Cloudera Navigator.   Is there any other way to collect metadata in Cloudera Navigator.   Thanks & Regards, Rajesh ... View more

S3 connectivity using impala shell

by Contributor Nukala in Support Questions
‎12-12-2018 06:41 PM
‎12-12-2018 06:41 PM
Hello,   I tried to do the following operations in Cloudera machine for impala using impala-shell https://www.cloudera.com/documentation/enterprise/5-7-x/topics/impala_s3.html   In this Cloudera machine I verified the S3 connectivity using the following command and was able to access hdfs dfs -Dfs.s3a.access.key=myaccesskey -Dfs.s3a.secret.key=mysecretkey -ls s3a://myclouderaraj/root   Updated [root@quickstart ~]# cd /etc/hadoop/conf/hdfs-site.xml file for access and secret key. Restarted all the services, also tried restarting cloudera-scm-server. Then ran the following from impala-shell [localhost:21000] > create database db_on_hdfs; [localhost:21000] > use db_on_hdfs; [localhost:21000] > create table mostly_on_hdfs (x int) partitioned by (year int); [localhost:21000] > alter table mostly_on_hdfs add partition (year=2013); [localhost:21000] > alter table mostly_on_hdfs add partition (year=2014); [localhost:21000] > alter table mostly_on_hdfs add partition (year=2015) location 's3a://impala-demo/dir1/dir2/dir3/t1'; [quickstart.cloudera:21000] > alter table mostly_on_hdfs add partition (year=2015) LOCATION 's3a://myclouderaraj/root'; Query: alter table mostly_on_hdfs add partition (year=2015) LOCATION 's3a://myclouderaraj/root' ERROR: AnalysisException: null CAUSED BY: AmazonClientException: Unable to load AWS credentials from any provider in the chain [quickstart.cloudera:21000] >   Can you please help.   Thanks & Regards, Rajesh ... View more

Re: Metadata and lineage collection for S3

by Contributor Nukala in Support Questions
‎12-12-2018 06:35 PM
‎12-12-2018 06:35 PM
Yes, it was successful and i was not able to attach the complete logs.   Yes, I see that there is some change in the request headers/user-agent. Not sure why that was. I have not done any specific changes to different versions of Cloudera and trying to connect the same S3 account.   18/12/11 19:12:54 DEBUG conn.DefaultClientConnection: Sending request: GET /?prefix=root%2F&delimiter=%2F&max-keys=1 HTTP/1.1 18/12/11 19:12:54 DEBUG http.wire: >> "GET /?prefix=root%2F&delimiter=%2F&max-keys=1 HTTP/1.1[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Host: myclouderaraj.s3.amazonaws.com[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:xhHdbA514F67IgOMPolX00Rv1h4=[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Date: Wed, 12 Dec 2018 03:12:54 GMT[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Content-Type: application/x-www-form-urlencoded; charset=utf-8[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Connection: Keep-Alive[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "[\r][\n]" 18/12/11 19:12:54 DEBUG http.headers: >> GET /?prefix=root%2F&delimiter=%2F&max-keys=1 HTTP/1.1 18/12/11 19:12:54 DEBUG http.headers: >> Host: myclouderaraj.s3.amazonaws.com 18/12/11 19:12:54 DEBUG http.headers: >> Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:xhHdbA514F67IgOMPolX00Rv1h4= 18/12/11 19:12:54 DEBUG http.headers: >> User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192 18/12/11 19:12:54 DEBUG http.headers: >> Date: Wed, 12 Dec 2018 03:12:54 GMT 18/12/11 19:12:54 DEBUG http.headers: >> Content-Type: application/x-www-form-urlencoded; charset=utf-8 18/12/11 19:12:54 DEBUG http.headers: >> Connection: Keep-Alive 18/12/11 19:12:54 DEBUG http.wire: << "HTTP/1.1 200 OK[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "x-amz-id-2: n67wc46WhWXcw4ygKZ0I2NEYJXqmyEG2sDyv8+hKbZaGCflTI6YB4gvIpuqQgHb+LCSgCQ33MlU=[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "x-amz-request-id: BE60D884EB5E4D60[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Date: Wed, 12 Dec 2018 03:12:55 GMT[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "x-amz-bucket-region: us-east-1[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Content-Type: application/xml[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Transfer-Encoding: chunked[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Server: AmazonS3[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:54 DEBUG conn.DefaultClientConnection: Receiving response: HTTP/1.1 200 OK 18/12/11 19:12:54 DEBUG http.headers: << HTTP/1.1 200 OK 18/12/11 19:12:54 DEBUG http.headers: << x-amz-id-2: n67wc46WhWXcw4ygKZ0I2NEYJXqmyEG2sDyv8+hKbZaGCflTI6YB4gvIpuqQgHb+LCSgCQ33MlU= 18/12/11 19:12:54 DEBUG http.headers: << x-amz-request-id: BE60D884EB5E4D60 18/12/11 19:12:54 DEBUG http.headers: << Date: Wed, 12 Dec 2018 03:12:55 GMT 18/12/11 19:12:54 DEBUG http.headers: << x-amz-bucket-region: us-east-1 18/12/11 19:12:54 DEBUG http.headers: << Content-Type: application/xml 18/12/11 19:12:54 DEBUG http.headers: << Transfer-Encoding: chunked 18/12/11 19:12:54 DEBUG http.headers: << Server: AmazonS3 18/12/11 19:12:54 DEBUG client.SdkHttpClient: Connection can be kept alive indefinitely 18/12/11 19:12:54 DEBUG amazonaws.requestId: x-amzn-RequestId: not available 18/12/11 19:12:54 TRACE amazonaws.request: Beginning to parse service response XML 18/12/11 19:12:54 DEBUG transform.XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.cloudera.com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler 18/12/11 19:12:54 DEBUG http.wire: << "286[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "<?xml version="1.0" encoding="UTF-8"?>[\n]" 18/12/11 19:12:54 DEBUG http.wire: << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>myclouderaraj</Name><Prefix>root/</Prefix><Marker></Marker><NextMarker>root/bulkInsert.csv</NextMarker><MaxKeys>1</MaxKeys><Delimiter>/</Delimiter><IsTruncated>true</IsTruncated><Contents><Key>root/bulkInsert.csv</Key><LastModified>2018-11-30T10:30:01.000Z</LastModified><ETag>&quot;963e1e40ac0247f23ebf138755ceabfb&quot;</ETag><Size>224</Size><Owner><ID>fdbe33e179958960b9d1fab4b547c1b8ccf7a9ed1b17b3ecc20f39d0866957ab</ID><DisplayName>rajesh210</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents></ListBucketResult>" 18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "0[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com] can be kept alive indefinitely 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection released: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:54 DEBUG transform.XmlResponsesSaxParser: Parsing XML response document with handler: class com.cloudera.com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler 18/12/11 19:12:54 DEBUG transform.XmlResponsesSaxParser: Examining listing for bucket: myclouderaraj 18/12/11 19:12:54 TRACE amazonaws.request: Done parsing service response XML 18/12/11 19:12:54 DEBUG amazonaws.request: Received successful response: 200, AWS Request ID: BE60D884EB5E4D60 18/12/11 19:12:54 DEBUG amazonaws.requestId: AWS Request ID: BE60D884EB5E4D60 18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: Found path as directory (with /): 0/1 18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: Summary: root/bulkInsert.csv 224 18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: listStatus: doing listObjects for directory root/ 18/12/11 19:12:54 DEBUG amazonaws.request: Sending Request: GET https://myclouderaraj.s3.amazonaws.com / Parameters: ({"prefix":["root/"],"delimiter":["/"],"max-keys":["5000"]}Headers: (User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192, Content-Type: application/x-www-form-urlencoded; charset=utf-8, ) 18/12/11 19:12:54 DEBUG internal.S3Signer: Calculated string to sign: "GET application/x-www-form-urlencoded; charset=utf-8 Wed, 12 Dec 2018 03:12:54 GMT /myclouderaraj/" 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection request: [route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection leased: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 0; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:54 DEBUG client.SdkHttpClient: Stale connection check 18/12/11 19:12:54 DEBUG protocol.RequestAddCookies: CookieSpec selected: best-match 18/12/11 19:12:54 DEBUG protocol.RequestAuthCache: Auth cache not set in the context 18/12/11 19:12:54 DEBUG protocol.RequestProxyAuthentication: Proxy auth state: UNCHALLENGED 18/12/11 19:12:54 DEBUG client.SdkHttpClient: Attempt 1 to execute request 18/12/11 19:12:54 DEBUG conn.DefaultClientConnection: Sending request: GET /?prefix=root%2F&delimiter=%2F&max-keys=5000 HTTP/1.1 18/12/11 19:12:54 DEBUG http.wire: >> "GET /?prefix=root%2F&delimiter=%2F&max-keys=5000 HTTP/1.1[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Host: myclouderaraj.s3.amazonaws.com[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:xhHdbA514F67IgOMPolX00Rv1h4=[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Date: Wed, 12 Dec 2018 03:12:54 GMT[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Content-Type: application/x-www-form-urlencoded; charset=utf-8[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Connection: Keep-Alive[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "[\r][\n]" 18/12/11 19:12:54 DEBUG http.headers: >> GET /?prefix=root%2F&delimiter=%2F&max-keys=5000 HTTP/1.1 18/12/11 19:12:54 DEBUG http.headers: >> Host: myclouderaraj.s3.amazonaws.com 18/12/11 19:12:54 DEBUG http.headers: >> Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:xhHdbA514F67IgOMPolX00Rv1h4= 18/12/11 19:12:54 DEBUG http.headers: >> User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192 18/12/11 19:12:54 DEBUG http.headers: >> Date: Wed, 12 Dec 2018 03:12:54 GMT 18/12/11 19:12:54 DEBUG http.headers: >> Content-Type: application/x-www-form-urlencoded; charset=utf-8 18/12/11 19:12:54 DEBUG http.headers: >> Connection: Keep-Alive 18/12/11 19:12:54 DEBUG http.wire: << "HTTP/1.1 200 OK[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "x-amz-id-2: qt25vFz8qCf82h/fM/Pl2kHdk8eWPN12AmczWd25XULPEZbxmvgkoWmnMkDdaK9O13AEcF7V5Fs=[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "x-amz-request-id: 70E4AB92D29F2154[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Date: Wed, 12 Dec 2018 03:12:55 GMT[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "x-amz-bucket-region: us-east-1[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Content-Type: application/xml[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Transfer-Encoding: chunked[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Server: AmazonS3[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:54 DEBUG conn.DefaultClientConnection: Receiving response: HTTP/1.1 200 OK 18/12/11 19:12:54 DEBUG http.headers: << HTTP/1.1 200 OK 18/12/11 19:12:54 DEBUG http.headers: << x-amz-id-2: qt25vFz8qCf82h/fM/Pl2kHdk8eWPN12AmczWd25XULPEZbxmvgkoWmnMkDdaK9O13AEcF7V5Fs= 18/12/11 19:12:54 DEBUG http.headers: << x-amz-request-id: 70E4AB92D29F2154 18/12/11 19:12:54 DEBUG http.headers: << Date: Wed, 12 Dec 2018 03:12:55 GMT 18/12/11 19:12:54 DEBUG http.headers: << x-amz-bucket-region: us-east-1 18/12/11 19:12:54 DEBUG http.headers: << Content-Type: application/xml 18/12/11 19:12:54 DEBUG http.headers: << Transfer-Encoding: chunked 18/12/11 19:12:54 DEBUG http.headers: << Server: AmazonS3 18/12/11 19:12:54 DEBUG client.SdkHttpClient: Connection can be kept alive indefinitely 18/12/11 19:12:54 DEBUG amazonaws.requestId: x-amzn-RequestId: not available 18/12/11 19:12:54 TRACE amazonaws.request: Beginning to parse service response XML 18/12/11 19:12:54 DEBUG transform.XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.cloudera.com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler 18/12/11 19:12:54 DEBUG http.wire: << "64b[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "<?xml version="1.0" encoding="UTF-8"?>[\n]" 18/12/11 19:12:54 DEBUG http.wire: << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>myclouderaraj</Name><Prefix>root/</Prefix><Marker></Marker><MaxKeys>5000</MaxKeys><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Contents><Key>root/bulkInsert.csv</Key><LastModified>2018-11-30T10:30:01.000Z</LastModified><ETag>&quot;963e1e40ac0247f23ebf138755ceabfb&quot;</ETag><Size>224</Size><Owner><ID>fdbe33e179958960b9d1fab4b547c1b8ccf7a9ed1b17b3ecc20f39d0866957ab</ID><DisplayName>rajesh210</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>root/customers</Key><LastModified>2018-11-30T05:57:52.000Z</LastModified><ETag>&quot;8811238ea9271f2441f6e9a96b1393a9&quot;</ETag><Size>15812</Size><Owner><ID>fdbe33e179958960b9d1fab4b547c1b8ccf7a9ed1b17b3ecc20f39d0866957ab</ID><DisplayName>rajesh210</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>root/table_test.csv</Key><LastModified>2018-11-30T10:30:26.000Z</LastModified><ETag>&quot;d41d8cd98f00b204e9800998ecf8427e&quot;</ETag><Size>0</Size><Owner><ID>fdbe33e179958960b9d1fab4b547c1b8ccf7a9ed1b17b3ecc20f39d0866957ab</ID><DisplayName>rajesh210</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>root/temp.json</Key><LastModified>2018-11-29T18:32:55.000Z</LastModified><ETag>&quot;bdd3d8948e659d61ab0438132780e116&quot;</ETag><Size>2121</Size><Owner><ID>fdbe33e179958960b9d1fab4b547c1b8ccf7a9ed1b17b3ecc20f39d0866957ab</ID><DisplayName>rajesh210</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents></ListBucketResult>" 18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "0[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com] can be kept alive indefinitely 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection released: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:54 DEBUG transform.XmlResponsesSaxParser: Parsing XML response document with handler: class com.cloudera.com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler 18/12/11 19:12:54 DEBUG transform.XmlResponsesSaxParser: Examining listing for bucket: myclouderaraj 18/12/11 19:12:54 TRACE amazonaws.request: Done parsing service response XML 18/12/11 19:12:54 DEBUG amazonaws.request: Received successful response: 200, AWS Request ID: 70E4AB92D29F2154 18/12/11 19:12:54 DEBUG amazonaws.requestId: AWS Request ID: 70E4AB92D29F2154 18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: Adding: fi: s3a://myclouderaraj/root/bulkInsert.csv 18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: Adding: fi: s3a://myclouderaraj/root/customers 18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: Adding: fi: s3a://myclouderaraj/root/table_test.csv 18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: Adding: fi: s3a://myclouderaraj/root/temp.json Found 4 items -rw-rw-rw- 1 root 224 2018-11-30 02:30 s3a://myclouderaraj/root/bulkInsert.csv -rw-rw-rw- 1 root 15812 2018-11-29 21:57 s3a://myclouderaraj/root/customers -rw-rw-rw- 1 root 0 2018-11-30 02:30 s3a://myclouderaraj/root/table_test.csv -rw-rw-rw- 1 root 2121 2018-11-29 10:32 s3a://myclouderaraj/root/temp.json 18/12/11 19:12:54 TRACE core.TracerPool: TracerPool(Global): removing tracer Tracer(FsShell/10.33.6.146) 18/12/11 19:12:55 DEBUG conn.PoolingClientConnectionManager: Connection manager is shutting down 18/12/11 19:12:55 DEBUG internal.SdkSSLSocket: shutting down output of myclouderaraj.s3.amazonaws.com/52.216.107.100:443 18/12/11 19:12:55 DEBUG internal.SdkSSLSocket: closing myclouderaraj.s3.amazonaws.com/52.216.107.100:443 18/12/11 19:12:55 DEBUG conn.DefaultClientConnection: Connection 0.0.0.0:51025<->52.216.107.100:443 closed 18/12/11 19:12:55 DEBUG conn.DefaultClientConnection: Connection 0.0.0.0:51025<->52.216.107.100:443 closed 18/12/11 19:12:55 DEBUG conn.PoolingClientConnectionManager: Connection manager shut down [root@quickstart ~]#     Thanks & Regards, Rajesh ... View more

Re: Metadata and lineage collection for S3

by Contributor Nukala in Support Questions
‎12-12-2018 05:09 AM
‎12-12-2018 05:09 AM
Seth,   This is the success log   [root@quickstart ~]# hdfs dfs -Dfs.s3a.access.key=myaccesskey -Dfs.s3a.secret.key=mysecretkey -ls s3a://myclouderaraj/root 18/12/11 19:09:45 DEBUG util.Shell: setsid exited with exit code 0 18/12/11 19:10:10 DEBUG conf.Configuration: parsing URL jar:file:/usr/jars/hadoop-common-2.6.0-cdh5.7.0.jar!/core-default.xml 18/12/11 19:10:10 DEBUG conf.Configuration: parsing input stream sun.net.www.protocol.jar.JarURLConnection$JarURLInputStream@ea1a8d5 18/12/11 19:11:30 DEBUG conf.Configuration: parsing URL file:/etc/hadoop/conf.cloudera.yarn/core-site.xml 18/12/11 19:11:31 DEBUG conf.Configuration: parsing input stream java.io.BufferedInputStream@fcd6521 18/12/11 19:12:34 DEBUG core.Tracer: sampler.classes = ; loaded no samplers 18/12/11 19:12:35 TRACE core.TracerId: ProcessID(fmt=%{tname}/%{ip}): computed process ID of "FsShell/10.33.6.146" 18/12/11 19:12:35 TRACE core.TracerPool: TracerPool(Global): adding tracer Tracer(FsShell/10.33.6.146) 18/12/11 19:12:35 DEBUG core.Tracer: span.receiver.classes = ; loaded no span receivers 18/12/11 19:12:35 TRACE core.Tracer: Created Tracer(FsShell/10.33.6.146) for FsShell 18/12/11 19:12:37 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of successful kerberos logins and latency (milliseconds)]) 18/12/11 19:12:37 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of failed kerberos logins and latency (milliseconds)]) 18/12/11 19:12:37 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[GetGroups]) 18/12/11 19:12:37 DEBUG impl.MetricsSystemImpl: UgiMetrics, User and group related metrics Java config name: null Native config name: /etc/krb5.conf Loaded from native config 18/12/11 19:12:38 DEBUG security.Groups: Creating new Groups object 18/12/11 19:12:38 DEBUG security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000; warningDeltaMs=5000 [UnixLoginModule]: succeeded importing info: uid = 0 gid = 0 supp gid = 0 18/12/11 19:12:38 DEBUG security.UserGroupInformation: hadoop login [UnixLoginModule]: added UnixPrincipal, UnixNumericUserPrincipal, UnixNumericGroupPrincipal(s), to Subject 18/12/11 19:12:38 DEBUG security.UserGroupInformation: hadoop login commit 18/12/11 19:12:38 DEBUG security.UserGroupInformation: using local user:UnixPrincipal: root 18/12/11 19:12:38 DEBUG security.UserGroupInformation: Using user: "UnixPrincipal: root" with name root 18/12/11 19:12:38 DEBUG security.UserGroupInformation: User entry: "root" 18/12/11 19:12:38 DEBUG security.UserGroupInformation: UGI loginUser:root (auth:SIMPLE) 18/12/11 19:12:38 DEBUG core.Tracer: sampler.classes = ; loaded no samplers 18/12/11 19:12:38 TRACE core.TracerId: ProcessID(fmt=%{tname}/%{ip}): computed process ID of "FSClient/10.33.6.146" 18/12/11 19:12:38 TRACE core.TracerPool: TracerPool(Global): adding tracer Tracer(FSClient/10.33.6.146) 18/12/11 19:12:38 DEBUG core.Tracer: span.receiver.classes = ; loaded no span receivers 18/12/11 19:12:38 TRACE core.Tracer: Created Tracer(FSClient/10.33.6.146) for FSClient 18/12/11 19:12:40 DEBUG azure.NativeAzureFileSystem: finalize() called. 18/12/11 19:12:40 DEBUG azure.NativeAzureFileSystem: finalize() called. 18/12/11 19:12:41 DEBUG amazonaws.AmazonWebServiceClient: Internal logging succesfully configured to commons logger: true 18/12/11 19:12:44 DEBUG metrics.AwsSdkMetrics: Admin mbean registered under com.cloudera.com.amazonaws.management:type=AwsSdkMetrics 18/12/11 19:12:50 DEBUG auth.AWSCredentialsProviderChain: Loading credentials from BasicAWSCredentialsProvider 18/12/11 19:12:50 DEBUG amazonaws.request: Sending Request: HEAD https://myclouderaraj.s3.amazonaws.com / Headers: (User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192, Content-Type: application/x-www-form-urlencoded; charset=utf-8, ) 18/12/11 19:12:51 DEBUG internal.S3Signer: Calculated string to sign: "HEAD application/x-www-form-urlencoded; charset=utf-8 Wed, 12 Dec 2018 03:12:50 GMT /myclouderaraj/" 18/12/11 19:12:51 DEBUG conn.PoolingClientConnectionManager: Connection request: [route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 0; route allocated: 0 of 15; total allocated: 0 of 15] 18/12/11 19:12:51 DEBUG conn.PoolingClientConnectionManager: Connection leased: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 0; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:51 DEBUG ssl.SdkTLSSocketFactory: socket.getSupportedProtocols(): [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2], socket.getEnabledProtocols(): [TLSv1, TLSv1.1, TLSv1.2] 18/12/11 19:12:51 DEBUG ssl.SdkTLSSocketFactory: TLS protocol enabled for SSL handshake: [TLSv1.2, TLSv1.1, TLSv1] 18/12/11 19:12:51 DEBUG conn.DefaultClientConnectionOperator: Connecting to myclouderaraj.s3.amazonaws.com:443 18/12/11 19:12:51 DEBUG ssl.SdkTLSSocketFactory: connecting to myclouderaraj.s3.amazonaws.com/52.216.107.100:443 18/12/11 19:12:53 DEBUG internal.SdkSSLSocket: created: myclouderaraj.s3.amazonaws.com/52.216.107.100:443 18/12/11 19:12:53 DEBUG protocol.RequestAddCookies: CookieSpec selected: best-match 18/12/11 19:12:53 DEBUG protocol.RequestAuthCache: Auth cache not set in the context 18/12/11 19:12:53 DEBUG protocol.RequestProxyAuthentication: Proxy auth state: UNCHALLENGED 18/12/11 19:12:53 DEBUG client.SdkHttpClient: Attempt 1 to execute request 18/12/11 19:12:53 DEBUG conn.DefaultClientConnection: Sending request: HEAD / HTTP/1.1 18/12/11 19:12:53 DEBUG http.wire: >> "HEAD / HTTP/1.1[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Host: myclouderaraj.s3.amazonaws.com[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:XU3AFfGphmQy0rEhRrWjajyeFbA=[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Date: Wed, 12 Dec 2018 03:12:50 GMT[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Content-Type: application/x-www-form-urlencoded; charset=utf-8[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Connection: Keep-Alive[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "[\r][\n]" 18/12/11 19:12:53 DEBUG http.headers: >> HEAD / HTTP/1.1 18/12/11 19:12:53 DEBUG http.headers: >> Host: myclouderaraj.s3.amazonaws.com 18/12/11 19:12:53 DEBUG http.headers: >> Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:XU3AFfGphmQy0rEhRrWjajyeFbA= 18/12/11 19:12:53 DEBUG http.headers: >> User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192 18/12/11 19:12:53 DEBUG http.headers: >> Date: Wed, 12 Dec 2018 03:12:50 GMT 18/12/11 19:12:53 DEBUG http.headers: >> Content-Type: application/x-www-form-urlencoded; charset=utf-8 18/12/11 19:12:53 DEBUG http.headers: >> Connection: Keep-Alive 18/12/11 19:12:53 DEBUG http.wire: << "HTTP/1.1 200 OK[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "x-amz-id-2: sjybfZSJELaf9eW/QJHWV/FcObyfD5ElzagJj8SqLiPwVogmvp1lih+csHvi73bNoHKuS/OLqSI=[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "x-amz-request-id: 03AD0786BC412E7C[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "Date: Wed, 12 Dec 2018 03:12:54 GMT[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "x-amz-bucket-region: us-east-1[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "Content-Type: application/xml[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "Transfer-Encoding: chunked[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "Server: AmazonS3[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:53 DEBUG conn.DefaultClientConnection: Receiving response: HTTP/1.1 200 OK 18/12/11 19:12:53 DEBUG http.headers: << HTTP/1.1 200 OK 18/12/11 19:12:53 DEBUG http.headers: << x-amz-id-2: sjybfZSJELaf9eW/QJHWV/FcObyfD5ElzagJj8SqLiPwVogmvp1lih+csHvi73bNoHKuS/OLqSI= 18/12/11 19:12:53 DEBUG http.headers: << x-amz-request-id: 03AD0786BC412E7C 18/12/11 19:12:53 DEBUG http.headers: << Date: Wed, 12 Dec 2018 03:12:54 GMT 18/12/11 19:12:53 DEBUG http.headers: << x-amz-bucket-region: us-east-1 18/12/11 19:12:53 DEBUG http.headers: << Content-Type: application/xml 18/12/11 19:12:53 DEBUG http.headers: << Transfer-Encoding: chunked 18/12/11 19:12:53 DEBUG http.headers: << Server: AmazonS3 18/12/11 19:12:53 DEBUG client.SdkHttpClient: Connection can be kept alive indefinitely 18/12/11 19:12:53 DEBUG conn.PoolingClientConnectionManager: Connection [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com] can be kept alive indefinitely 18/12/11 19:12:53 DEBUG conn.PoolingClientConnectionManager: Connection released: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:53 DEBUG amazonaws.requestId: x-amzn-RequestId: not available 18/12/11 19:12:53 DEBUG amazonaws.request: Received successful response: 200, AWS Request ID: 03AD0786BC412E7C 18/12/11 19:12:53 DEBUG amazonaws.requestId: AWS Request ID: 03AD0786BC412E7C 18/12/11 19:12:53 DEBUG s3a.S3AFileSystem: Getting path status for s3a://myclouderaraj/root (root) 18/12/11 19:12:53 DEBUG amazonaws.request: Sending Request: HEAD https://myclouderaraj.s3.amazonaws.com /root Headers: (User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192, Content-Type: application/x-www-form-urlencoded; charset=utf-8, ) 18/12/11 19:12:53 DEBUG internal.S3Signer: Calculated string to sign: "HEAD application/x-www-form-urlencoded; charset=utf-8 Wed, 12 Dec 2018 03:12:53 GMT /myclouderaraj/root" 18/12/11 19:12:53 DEBUG conn.PoolingClientConnectionManager: Connection request: [route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:53 DEBUG conn.PoolingClientConnectionManager: Connection leased: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 0; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:53 DEBUG client.SdkHttpClient: Stale connection check 18/12/11 19:12:53 DEBUG protocol.RequestAddCookies: CookieSpec selected: best-match 18/12/11 19:12:53 DEBUG protocol.RequestAuthCache: Auth cache not set in the context 18/12/11 19:12:53 DEBUG protocol.RequestProxyAuthentication: Proxy auth state: UNCHALLENGED 18/12/11 19:12:53 DEBUG client.SdkHttpClient: Attempt 1 to execute request 18/12/11 19:12:53 DEBUG conn.DefaultClientConnection: Sending request: HEAD /root HTTP/1.1 18/12/11 19:12:53 DEBUG http.wire: >> "HEAD /root HTTP/1.1[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Host: myclouderaraj.s3.amazonaws.com[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:SLVFDzdTaXiUPcomkZCGPdiXP+I=[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Date: Wed, 12 Dec 2018 03:12:53 GMT[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Content-Type: application/x-www-form-urlencoded; charset=utf-8[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Connection: Keep-Alive[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "[\r][\n]" 18/12/11 19:12:53 DEBUG http.headers: >> HEAD /root HTTP/1.1 18/12/11 19:12:53 DEBUG http.headers: >> Host: myclouderaraj.s3.amazonaws.com 18/12/11 19:12:53 DEBUG http.headers: >> Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:SLVFDzdTaXiUPcomkZCGPdiXP+I= 18/12/11 19:12:53 DEBUG http.headers: >> User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192 18/12/11 19:12:53 DEBUG http.headers: >> Date: Wed, 12 Dec 2018 03:12:53 GMT 18/12/11 19:12:53 DEBUG http.headers: >> Content-Type: application/x-www-form-urlencoded; charset=utf-8 18/12/11 19:12:53 DEBUG http.headers: >> Connection: Keep-Alive 18/12/11 19:12:53 DEBUG http.wire: << "HTTP/1.1 404 Not Found[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "x-amz-request-id: 562A66F35BB7E4A1[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "x-amz-id-2: FC+MjpOY8O1AM4iHk4leLkFYxsqUqsOGka5QBXub98ve+SirWRY2PXH8778eWyv9UUFbFwVn0LQ=[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "Content-Type: application/xml[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "Transfer-Encoding: chunked[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "Date: Wed, 12 Dec 2018 03:12:53 GMT[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "Server: AmazonS3[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:53 DEBUG conn.DefaultClientConnection: Receiving response: HTTP/1.1 404 Not Found 18/12/11 19:12:53 DEBUG http.headers: << HTTP/1.1 404 Not Found 18/12/11 19:12:53 DEBUG http.headers: << x-amz-request-id: 562A66F35BB7E4A1 18/12/11 19:12:53 DEBUG http.headers: << x-amz-id-2: FC+MjpOY8O1AM4iHk4leLkFYxsqUqsOGka5QBXub98ve+SirWRY2PXH8778eWyv9UUFbFwVn0LQ= 18/12/11 19:12:53 DEBUG http.headers: << Content-Type: application/xml 18/12/11 19:12:53 DEBUG http.headers: << Transfer-Encoding: chunked 18/12/11 19:12:53 DEBUG http.headers: << Date: Wed, 12 Dec 2018 03:12:53 GMT 18/12/11 19:12:53 DEBUG http.headers: << Server: AmazonS3 18/12/11 19:12:53 DEBUG client.SdkHttpClient: Connection can be kept alive indefinitely 18/12/11 19:12:53 DEBUG conn.PoolingClientConnectionManager: Connection [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com] can be kept alive indefinitely 18/12/11 19:12:53 DEBUG conn.PoolingClientConnectionManager: Connection released: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:53 DEBUG amazonaws.requestId: x-amzn-RequestId: not available 18/12/11 19:12:53 DEBUG amazonaws.request: Received error response: com.cloudera.com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: null; Status Code: 404; Error Code: 404 Not Found; Request ID: 562A66F35BB7E4A1), S3 Extended Request ID: FC+MjpOY8O1AM4iHk4leLkFYxsqUqsOGka5QBXub98ve+SirWRY2PXH8778eWyv9UUFbFwVn0LQ= 18/12/11 19:12:53 DEBUG amazonaws.request: Sending Request: HEAD https://myclouderaraj.s3.amazonaws.com /root/ Headers: (User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192, Content-Type: application/x-www-form-urlencoded; charset=utf-8, ) 18/12/11 19:12:53 DEBUG internal.S3Signer: Calculated string to sign: "HEAD application/x-www-form-urlencoded; charset=utf-8 Wed, 12 Dec 2018 03:12:53 GMT /myclouderaraj/root/" 18/12/11 19:12:53 DEBUG conn.PoolingClientConnectionManager: Connection request: [route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:53 DEBUG conn.PoolingClientConnectionManager: Connection leased: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 0; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:53 DEBUG client.SdkHttpClient: Stale connection check 18/12/11 19:12:53 DEBUG protocol.RequestAddCookies: CookieSpec selected: best-match 18/12/11 19:12:53 DEBUG protocol.RequestAuthCache: Auth cache not set in the context 18/12/11 19:12:53 DEBUG protocol.RequestProxyAuthentication: Proxy auth state: UNCHALLENGED 18/12/11 19:12:53 DEBUG client.SdkHttpClient: Attempt 1 to execute request 18/12/11 19:12:53 DEBUG conn.DefaultClientConnection: Sending request: HEAD /root/ HTTP/1.1 18/12/11 19:12:53 DEBUG http.wire: >> "HEAD /root/ HTTP/1.1[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Host: myclouderaraj.s3.amazonaws.com[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:nWk9yWYYe8oEEGtfxRoqQmfgSK0=[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Date: Wed, 12 Dec 2018 03:12:53 GMT[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Content-Type: application/x-www-form-urlencoded; charset=utf-8[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Connection: Keep-Alive[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "[\r][\n]" 18/12/11 19:12:53 DEBUG http.headers: >> HEAD /root/ HTTP/1.1 18/12/11 19:12:53 DEBUG http.headers: >> Host: myclouderaraj.s3.amazonaws.com 18/12/11 19:12:53 DEBUG http.headers: >> Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:nWk9yWYYe8oEEGtfxRoqQmfgSK0= 18/12/11 19:12:53 DEBUG http.headers: >> User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192 18/12/11 19:12:53 DEBUG http.headers: >> Date: Wed, 12 Dec 2018 03:12:53 GMT 18/12/11 19:12:53 DEBUG http.headers: >> Content-Type: application/x-www-form-urlencoded; charset=utf-8 18/12/11 19:12:53 DEBUG http.headers: >> Connection: Keep-Alive 18/12/11 19:12:53 DEBUG http.wire: << "HTTP/1.1 404 Not Found[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "x-amz-request-id: C5C8989A67B509AC[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "x-amz-id-2: 19gaP2XtwSoU9Bl1Y+jO3A8rPAr+v00XbdZQDD08v6u3XJKiB3uRfLSF+E4vvivzeiwCK74Vy+I=[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "Content-Type: application/xml[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "Transfer-Encoding: chunked[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "Date: Wed, 12 Dec 2018 03:12:53 GMT[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "Server: AmazonS3[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:53 DEBUG conn.DefaultClientConnection: Receiving response: HTTP/1.1 404 Not Found 18/12/11 19:12:53 DEBUG http.headers: << HTTP/1.1 404 Not Found 18/12/11 19:12:53 DEBUG http.headers: << x-amz-request-id: C5C8989A67B509AC 18/12/11 19:12:53 DEBUG http.headers: << x-amz-id-2: 19gaP2XtwSoU9Bl1Y+jO3A8rPAr+v00XbdZQDD08v6u3XJKiB3uRfLSF+E4vvivzeiwCK74Vy+I= 18/12/11 19:12:53 DEBUG http.headers: << Content-Type: application/xml 18/12/11 19:12:53 DEBUG http.headers: << Transfer-Encoding: chunked 18/12/11 19:12:53 DEBUG http.headers: << Date: Wed, 12 Dec 2018 03:12:53 GMT 18/12/11 19:12:53 DEBUG http.headers: << Server: AmazonS3 18/12/11 19:12:53 DEBUG client.SdkHttpClient: Connection can be kept alive indefinitely 18/12/11 19:12:53 DEBUG conn.PoolingClientConnectionManager: Connection [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com] can be kept alive indefinitely 18/12/11 19:12:53 DEBUG conn.PoolingClientConnectionManager: Connection released: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:53 DEBUG amazonaws.requestId: x-amzn-RequestId: not available 18/12/11 19:12:53 DEBUG amazonaws.request: Received error response: com.cloudera.com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: null; Status Code: 404; Error Code: 404 Not Found; Request ID: C5C8989A67B509AC), S3 Extended Request ID: 19gaP2XtwSoU9Bl1Y+jO3A8rPAr+v00XbdZQDD08v6u3XJKiB3uRfLSF+E4vvivzeiwCK74Vy+I= 18/12/11 19:12:53 DEBUG amazonaws.request: Sending Request: GET https://myclouderaraj.s3.amazonaws.com / Parameters: ({"prefix":["root/"],"delimiter":["/"],"max-keys":["1"]}Headers: (User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192, Content-Type: application/x-www-form-urlencoded; charset=utf-8, ) 18/12/11 19:12:53 DEBUG internal.S3Signer: Calculated string to sign: "GET application/x-www-form-urlencoded; charset=utf-8 Wed, 12 Dec 2018 03:12:53 GMT /myclouderaraj/" 18/12/11 19:12:53 DEBUG conn.PoolingClientConnectionManager: Connection request: [route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:53 DEBUG conn.PoolingClientConnectionManager: Connection leased: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 0; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:53 DEBUG client.SdkHttpClient: Stale connection check 18/12/11 19:12:53 DEBUG protocol.RequestAddCookies: CookieSpec selected: best-match 18/12/11 19:12:53 DEBUG protocol.RequestAuthCache: Auth cache not set in the context 18/12/11 19:12:53 DEBUG protocol.RequestProxyAuthentication: Proxy auth state: UNCHALLENGED 18/12/11 19:12:53 DEBUG client.SdkHttpClient: Attempt 1 to execute request 18/12/11 19:12:53 DEBUG conn.DefaultClientConnection: Sending request: GET /?prefix=root%2F&delimiter=%2F&max-keys=1 HTTP/1.1 18/12/11 19:12:53 DEBUG http.wire: >> "GET /?prefix=root%2F&delimiter=%2F&max-keys=1 HTTP/1.1[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Host: myclouderaraj.s3.amazonaws.com[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:VEhMOYpTlsvVdG9Q0F7ehtOuxyg=[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Date: Wed, 12 Dec 2018 03:12:53 GMT[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Content-Type: application/x-www-form-urlencoded; charset=utf-8[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Connection: Keep-Alive[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "[\r][\n]" 18/12/11 19:12:53 DEBUG http.headers: >> GET /?prefix=root%2F&delimiter=%2F&max-keys=1 HTTP/1.1 18/12/11 19:12:53 DEBUG http.headers: >> Host: myclouderaraj.s3.amazonaws.com 18/12/11 19:12:53 DEBUG http.headers: >> Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:VEhMOYpTlsvVdG9Q0F7ehtOuxyg= 18/12/11 19:12:53 DEBUG http.headers: >> User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192 18/12/11 19:12:53 DEBUG http.headers: >> Date: Wed, 12 Dec 2018 03:12:53 GMT 18/12/11 19:12:53 DEBUG http.headers: >> Content-Type: application/x-www-form-urlencoded; charset=utf-8 18/12/11 19:12:53 DEBUG http.headers: >> Connection: Keep-Alive 18/12/11 19:12:54 DEBUG http.wire: << "HTTP/1.1 200 OK[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "x-amz-id-2: fcNfT1n+Dhn7lPqVGiKhrTq36arxsAAG1QNTFjpiJ+O8vSqBUumwNFw7Nmv6onHdi4Y0t0wmYGE=[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "x-amz-request-id: 741C65B73A0E800F[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Date: Wed, 12 Dec 2018 03:12:54 GMT[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "x-amz-bucket-region: us-east-1[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Content-Type: application/xml[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Transfer-Encoding: chunked[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Server: AmazonS3[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:54 DEBUG conn.DefaultClientConnection: Receiving response: HTTP/1.1 200 OK 18/12/11 19:12:54 DEBUG http.headers: << HTTP/1.1 200 OK 18/12/11 19:12:54 DEBUG http.headers: << x-amz-id-2: fcNfT1n+Dhn7lPqVGiKhrTq36arxsAAG1QNTFjpiJ+O8vSqBUumwNFw7Nmv6onHdi4Y0t0wmYGE= 18/12/11 19:12:54 DEBUG http.headers: << x-amz-request-id: 741C65B73A0E800F 18/12/11 19:12:54 DEBUG http.headers: << Date: Wed, 12 Dec 2018 03:12:54 GMT 18/12/11 19:12:54 DEBUG http.headers: << x-amz-bucket-region: us-east-1 18/12/11 19:12:54 DEBUG http.headers: << Content-Type: application/xml 18/12/11 19:12:54 DEBUG http.headers: << Transfer-Encoding: chunked 18/12/11 19:12:54 DEBUG http.headers: << Server: AmazonS3 18/12/11 19:12:54 DEBUG client.SdkHttpClient: Connection can be kept alive indefinitely 18/12/11 19:12:54 DEBUG amazonaws.requestId: x-amzn-RequestId: not available 18/12/11 19:12:54 TRACE amazonaws.request: Beginning to parse service response XML 18/12/11 19:12:54 DEBUG transform.XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.cloudera.com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler 18/12/11 19:12:54 DEBUG http.wire: << "286[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "<?xml version="1.0" encoding="UTF-8"?>[\n]" 18/12/11 19:12:54 DEBUG http.wire: << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>myclouderaraj</Name><Prefix>root/</Prefix><Marker></Marker><NextMarker>root/bulkInsert.csv</NextMarker><MaxKeys>1</MaxKeys><Delimiter>/</Delimiter><IsTruncated>true</IsTruncated><Contents><Key>root/bulkInsert.csv</Key><LastModified>2018-11-30T10:30:01.000Z</LastModified><ETag>&quot;963e1e40ac0247f23ebf138755ceabfb&quot;</ETag><Size>224</Size><Owner><ID>fdbe33e179958960b9d1fab4b547c1b8ccf7a9ed1b17b3ecc20f39d0866957ab</ID><DisplayName>rajesh210</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents></ListBucketResult>" 18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "0[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com] can be kept alive indefinitely 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection released: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:54 DEBUG transform.XmlResponsesSaxParser: Parsing XML response document with handler: class com.cloudera.com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler 18/12/11 19:12:54 DEBUG transform.XmlResponsesSaxParser: Examining listing for bucket: myclouderaraj 18/12/11 19:12:54 TRACE amazonaws.request: Done parsing service response XML 18/12/11 19:12:54 DEBUG amazonaws.request: Received successful response: 200, AWS Request ID: 741C65B73A0E800F 18/12/11 19:12:54 DEBUG amazonaws.requestId: AWS Request ID: 741C65B73A0E800F 18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: Found path as directory (with /): 0/1 18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: Summary: root/bulkInsert.csv 224 18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: List status for path: s3a://myclouderaraj/root 18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: Getting path status for s3a://myclouderaraj/root (root) 18/12/11 19:12:54 DEBUG amazonaws.request: Sending Request: HEAD https://myclouderaraj.s3.amazonaws.com /root Headers: (User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192, Content-Type: application/x-www-form-urlencoded; charset=utf-8, ) 18/12/11 19:12:54 DEBUG internal.S3Signer: Calculated string to sign: "HEAD application/x-www-form-urlencoded; charset=utf-8 Wed, 12 Dec 2018 03:12:54 GMT /myclouderaraj/root" 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection request: [route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection leased: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 0; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:54 DEBUG client.SdkHttpClient: Stale connection check 18/12/11 19:12:54 DEBUG protocol.RequestAddCookies: CookieSpec selected: best-match 18/12/11 19:12:54 DEBUG protocol.RequestAuthCache: Auth cache not set in the context 18/12/11 19:12:54 DEBUG protocol.RequestProxyAuthentication: Proxy auth state: UNCHALLENGED 18/12/11 19:12:54 DEBUG client.SdkHttpClient: Attempt 1 to execute request 18/12/11 19:12:54 DEBUG conn.DefaultClientConnection: Sending request: HEAD /root HTTP/1.1 18/12/11 19:12:54 DEBUG http.wire: >> "HEAD /root HTTP/1.1[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Host: myclouderaraj.s3.amazonaws.com[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:/G0a8NrUHFBGmNvT/YV5nkFRbKU=[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Date: Wed, 12 Dec 2018 03:12:54 GMT[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Content-Type: application/x-www-form-urlencoded; charset=utf-8[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Connection: Keep-Alive[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "[\r][\n]" 18/12/11 19:12:54 DEBUG http.headers: >> HEAD /root HTTP/1.1 18/12/11 19:12:54 DEBUG http.headers: >> Host: myclouderaraj.s3.amazonaws.com 18/12/11 19:12:54 DEBUG http.headers: >> Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:/G0a8NrUHFBGmNvT/YV5nkFRbKU= 18/12/11 19:12:54 DEBUG http.headers: >> User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192 18/12/11 19:12:54 DEBUG http.headers: >> Date: Wed, 12 Dec 2018 03:12:54 GMT 18/12/11 19:12:54 DEBUG http.headers: >> Content-Type: application/x-www-form-urlencoded; charset=utf-8 18/12/11 19:12:54 DEBUG http.headers: >> Connection: Keep-Alive 18/12/11 19:12:54 DEBUG http.wire: << "HTTP/1.1 404 Not Found[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "x-amz-request-id: 48EB44080F918683[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "x-amz-id-2: MLddIb2VuN1THhnjZI4diJfaB37g0zGNOzIFrEIfgOCMYeJBJEb2xV16+VQsv/2Z2kGhoQyhLhM=[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Content-Type: application/xml[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Transfer-Encoding: chunked[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Date: Wed, 12 Dec 2018 03:12:54 GMT[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Server: AmazonS3[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:54 DEBUG conn.DefaultClientConnection: Receiving response: HTTP/1.1 404 Not Found 18/12/11 19:12:54 DEBUG http.headers: << HTTP/1.1 404 Not Found 18/12/11 19:12:54 DEBUG http.headers: << x-amz-request-id: 48EB44080F918683 18/12/11 19:12:54 DEBUG http.headers: << x-amz-id-2: MLddIb2VuN1THhnjZI4diJfaB37g0zGNOzIFrEIfgOCMYeJBJEb2xV16+VQsv/2Z2kGhoQyhLhM= 18/12/11 19:12:54 DEBUG http.headers: << Content-Type: application/xml 18/12/11 19:12:54 DEBUG http.headers: << Transfer-Encoding: chunked 18/12/11 19:12:54 DEBUG http.headers: << Date: Wed, 12 Dec 2018 03:12:54 GMT 18/12/11 19:12:54 DEBUG http.headers: << Server: AmazonS3 18/12/11 19:12:54 DEBUG client.SdkHttpClient: Connection can be kept alive indefinitely 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com] can be kept alive indefinitely 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection released: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:54 DEBUG amazonaws.requestId: x-amzn-RequestId: not available 18/12/11 19:12:54 DEBUG amazonaws.request: Received error response: com.cloudera.com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: null; Status Code: 404; Error Code: 404 Not Found; Request ID: 48EB44080F918683), S3 Extended Request ID: MLddIb2VuN1THhnjZI4diJfaB37g0zGNOzIFrEIfgOCMYeJBJEb2xV16+VQsv/2Z2kGhoQyhLhM= 18/12/11 19:12:54 DEBUG amazonaws.request: Sending Request: HEAD https://myclouderaraj.s3.amazonaws.com /root/ Headers: (User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192, Content-Type: application/x-www-form-urlencoded; charset=utf-8, ) 18/12/11 19:12:54 DEBUG internal.S3Signer: Calculated string to sign: "HEAD application/x-www-form-urlencoded; charset=utf-8 Wed, 12 Dec 2018 03:12:54 GMT /myclouderaraj/root/" 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection request: [route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection leased: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 0; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:54 DEBUG client.SdkHttpClient: Stale connection check 18/12/11 19:12:54 DEBUG protocol.RequestAddCookies: CookieSpec selected: best-match 18/12/11 19:12:54 DEBUG protocol.RequestAuthCache: Auth cache not set in the context 18/12/11 19:12:54 DEBUG protocol.RequestProxyAuthentication: Proxy auth state: UNCHALLENGED 18/12/11 19:12:54 DEBUG client.SdkHttpClient: Attempt 1 to execute request 18/12/11 19:12:54 DEBUG conn.DefaultClientConnection: Sending request: HEAD /root/ HTTP/1.1 18/12/11 19:12:54 DEBUG http.wire: >> "HEAD /root/ HTTP/1.1[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Host: myclouderaraj.s3.amazonaws.com[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:bAUfpGUWpSAqbkk2vygScGZO7Qw=[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Date: Wed, 12 Dec 2018 03:12:54 GMT[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Content-Type: application/x-www-form-urlencoded; charset=utf-8[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Connection: Keep-Alive[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "[\r][\n]" 18/12/11 19:12:54 DEBUG http.headers: >> HEAD /root/ HTTP/1.1 18/12/11 19:12:54 DEBUG http.headers: >> Host: myclouderaraj.s3.amazonaws.com 18/12/11 19:12:54 DEBUG http.headers: >> Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:bAUfpGUWpSAqbkk2vygScGZO7Qw= 18/12/11 19:12:54 DEBUG http.headers: >> User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192 18/12/11 19:12:54 DEBUG http.headers: >> Date: Wed, 12 Dec 2018 03:12:54 GMT 18/12/11 19:12:54 DEBUG http.headers: >> Content-Type: application/x-www-form-urlencoded; charset=utf-8 18/12/11 19:12:54 DEBUG http.headers: >> Connection: Keep-Alive 18/12/11 19:12:54 DEBUG http.wire: << "HTTP/1.1 404 Not Found[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "x-amz-request-id: D79FFCC520EEA4D7[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "x-amz-id-2: TZC+frfDHCmwLqED0QD8Cg/lABBzKNkVwRIto+wmg3IVMm6+9KGRE87O3cr/jpQ8LFM3YvfaUuI=[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Content-Type: application/xml[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Transfer-Encoding: chunked[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Date: Wed, 12 Dec 2018 03:12:54 GMT[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Server: AmazonS3[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:54 DEBUG conn.DefaultClientConnection: Receiving response: HTTP/1.1 404 Not Found 18/12/11 19:12:54 DEBUG http.headers: << HTTP/1.1 404 Not Found 18/12/11 19:12:54 DEBUG http.headers: << x-amz-request-id: D79FFCC520EEA4D7 18/12/11 19:12:54 DEBUG http.headers: << x-amz-id-2: TZC+frfDHCmwLqED0QD8Cg/lABBzKNkVwRIto+wmg3IVMm6+9KGRE87O3cr/jpQ8LFM3YvfaUuI= 18/12/11 19:12:54 DEBUG http.headers: << Content-Type: application/xml 18/12/11 19:12:54 DEBUG http.headers: << Transfer-Encoding: chunked 18/12/11 19:12:54 DEBUG http.headers: << Date: Wed, 12 Dec 2018 03:12:54 GMT 18/12/11 19:12:54 DEBUG http.headers: << Server: AmazonS3 18/12/11 19:12:54 DEBUG client.SdkHttpClient: Connection can be kept alive indefinitely 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com] can be kept alive indefinitely 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection released: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:54 DEBUG amazonaws.requestId: x-amzn-RequestId: not available 18/12/11 19:12:54 DEBUG amazonaws.request: Received error response: com.cloudera.com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: null; Status Code: 404; Error Code: 404 Not Found; Request ID: D79FFCC520EEA4D7), S3 Extended Request ID: TZC+frfDHCmwLqED0QD8Cg/lABBzKNkVwRIto+wmg3IVMm6+9KGRE87O3cr/jpQ8LFM3YvfaUuI= 18/12/11 19:12:54 DEBUG amazonaws.request: Sending Request: GET https://myclouderaraj.s3.amazonaws.com / Parameters: ({"prefix":["root/"],"delimiter":["/"],"max-keys":["1"]}Headers: (User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192, Content-Type: application/x-www-form-urlencoded; charset=utf-8, ) 18/12/11 19:12:54 DEBUG internal.S3Signer: Calculated string to sign: "GET application/x-www-form-urlencoded; charset=utf-8 Wed, 12 Dec 2018 03:12:54 GMT /myclouderaraj/" 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection request: [route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection leased: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 0; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:54 DEBUG client.SdkHttpClient: Stale connection check 18/12/11 19:12:54 DEBUG protocol.RequestAddCookies: CookieSpec selected: best-match 18/12/11 19:12:54 DEBUG protocol.RequestAuthCache: Auth cache not set in the context 18/12/11 19:12:54 DEBUG protocol.RequestProxyAuthentication: Proxy auth state: UNCHALLENGED 18/12/11 19:12:54 DEBUG client.SdkHttpClient: Attempt 1 to execute request 18/12/11 19:12:54 DEBUG conn.DefaultClientConnection: Sending request: GET /?prefix=root%2F&delimiter=%2F&max-keys=1 HTTP/1.1 18/12/11 19:12:54 DEBUG http.wire: >> "GET /?prefix=root%2F&delimiter=%2F&max-keys=1 HTTP/1.1[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Host: myclouderaraj.s3.amazonaws.com[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:xhHdbA514F67IgOMPolX00Rv1h4=[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Date: Wed, 12 Dec 2018 03:12:54 GMT[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Content-Type: application/x-www-form-urlencoded; charset=utf-8[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Connection: Keep-Alive[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "[\r][\n]" 18/12/11 19:12:54 DEBUG http.headers: >> GET /?prefix=root%2F&delimiter=%2F&max-keys=1 HTTP/1.1 18/12/11 19:12:54 DEBUG http.headers: >> Host: myclouderaraj.s3.amazonaws.com 18/12/11 19:12:54 DEBUG http.headers: >> Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:xhHdbA514F67IgOMPolX00Rv1h4= 18/12/11 19:12:54 DEBUG http.headers: >> User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192 18/12/11 19:12:54 DEBUG http.headers: >> Date: Wed, 12 Dec 2018 03:12:54 GMT 18/12/11 19:12:54 DEBUG http.headers: >> Content-Type: application/x-www-form-urlencoded; charset=utf-8 18/12/11 19:12:54 DEBUG http.headers: >> Connection: Keep-Alive 18/12/11 19:12:54 DEBUG http.wire: << "HTTP/1.1 200 OK[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "x-amz-id-2: n67wc46WhWXcw4ygKZ0I2NEYJXqmyEG2sDyv8+hKbZaGCflTI6YB4gvIpuqQgHb+LCSgCQ33MlU=[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "x-amz-request-id: BE60D884EB5E4D60[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Date: Wed, 12 Dec 2018 03:12:55 GMT[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "x-amz-bucket-region: us-east-1[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Content-Type: application/xml[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Transfer-Encoding: chunked[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Server: AmazonS3[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:54 DEBUG conn.DefaultClientConnection: Receiving response: HTTP/1.1 200 OK 18/12/11 19:12:54 DEBUG http.headers: << HTTP/1.1 200 OK 18/12/11 19:12:54 DEBUG http.headers: << x-amz-id-2: n67wc46WhWXcw4ygKZ0I2NEYJXqmyEG2sDyv8+hKbZaGCflTI6YB4gvIpuqQgHb+LCSgCQ33MlU= 18/12/11 19:12:54 DEBUG http.headers: << x-amz-request-id: BE60D884EB5E4D60 18/12/11 19:12:54 DEBUG http.headers: << Date: Wed, 12 Dec 2018 03:12:55 GMT 18/12/11 19:12:54 DEBUG http.headers: << x-amz-bucket-region: us-east-1 18/12/11 19:12:54 DEBUG http.headers: << Content-Type: application/xml 18/12/11 19:12:54 DEBUG http.headers: << Transfer-Encoding: chunked 18/12/11 19:12:54 DEBUG http.headers: << Server: AmazonS3 18/12/11 19:12:54 DEBUG client.SdkHttpClient: Connection can be kept alive indefinitely 18/12/11 19:12:54 DEBUG amazonaws.requestId: x-amzn-RequestId: not available 18/12/11 19:12:54 TRACE amazonaws.request: Beginning to parse service response XML 18/12/11 19:12:54 DEBUG transform.XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.cloudera.com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler 18/12/11 19:12:54 DEBUG http.wire: << "286[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "<?xml version="1.0" encoding="UTF-8"?>[\n]" 18/12/11 19:12:54 DEBUG http.wire: << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>myclouderaraj</Name><Prefix>root/</Prefix><Marker></Marker><NextMarker>root/bulkInsert.csv</NextMarker><MaxKeys>1</MaxKeys><Delimiter>/</Delimiter><IsTruncated>true</IsTruncated><Contents><Key>root/bulkInsert.csv</Key><LastModified>2018-11-30T10:30:01.000Z</LastModified><ETag>&quot;963e1e40ac0247f23ebf138755ceabfb&quot;</ETag><Size>224</Size><Owner><ID>fdbe33e179958960b9d1fab4b547c1b8ccf7a9ed1b17b3ecc20f39d0866957ab</ID><DisplayName>rajesh210</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents></ListBucketResult>" 18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "0[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com] can be kept alive indefinitely 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection released: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:54 DEBUG transform.XmlResponsesSaxParser: Parsing XML response document with handler: class com.cloudera.com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler 18/12/11 19:12:54 DEBUG transform.XmlResponsesSaxParser: Examining listing for bucket: myclouderaraj 18/12/11 19:12:54 TRACE amazonaws.request: Done parsing service response XML 18/12/11 19:12:54 DEBUG amazonaws.request: Received successful response: 200, AWS Request ID: BE60D884EB5E4D60 18/12/11 19:12:54 DEBUG amazonaws.requestId: AWS Request ID: BE60D884EB5E4D60 18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: Found path as directory (with /): 0/1 18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: Summary: root/bulkInsert.csv 224 18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: listStatus: doing listObjects for directory root/ 18/12/11 19:12:54 DEBUG amazonaws.request: Sending Request: GET https://myclouderaraj.s3.amazonaws.com / Parameters: ({"prefix":["root/"],"delimiter":["/"],"max-keys":["5000"]}Headers: (User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192, Content-Type: application/x-www-form-urlencoded; charset=utf-8, ) 18/12/11 19:12:54 DEBUG internal.S3Signer: Calculated string to sign: "GET application/x-www-form-urlencoded; charset=utf-8 Wed, 12 Dec 2018 03:12:54 GMT /myclouderaraj/" 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection request: [route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection leased: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 0; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:54 DEBUG client.SdkHttpClient: Stale connection check 18/12/11 19:12:54 DEBUG protocol.RequestAddCookies: CookieSpec selected: best-match 18/12/11 19:12:54 DEBUG protocol.RequestAuthCache: Auth cache not set in the context 18/12/11 19:12:54 DEBUG protocol.RequestProxyAuthentication: Proxy auth state: UNCHALLENGED 18/12/11 19:12:54 DEBUG client.SdkHttpClient: Attempt 1 to execute request 18/12/11 19:12:54 DEBUG conn.DefaultClientConnection: Sending request: GET /?prefix=root%2F&delimiter=%2F&max-keys=5000 HTTP/1.1 18/12/11 19:12:54 DEBUG http.wire: >> "GET /?prefix=root%2F&delimiter=%2F&max-keys=5000 HTTP/1.1[\r][\n]" ... View more

Re: Metadata and lineage collection for S3

by Contributor Nukala in Support Questions
‎12-11-2018 09:24 AM
‎12-11-2018 09:24 AM
Hi Seth,   I did as you mentioned and the logs are as memtioned below   [root@quickstart ~]# hdfs dfs -Dfs.s3a.access.key=myaccesskey -Dfs.s3a.secret.key=mysecretkey -ls s3a://myclouderaraj/root 18/12/11 17:45:55 DEBUG util.Shell: setsid exited with exit code 0 18/12/11 17:45:55 DEBUG conf.Configuration: parsing URL jar:file:/usr/lib/hadoop/hadoop-common-2.6.0-cdh5.13.0.jar!/core-default.xml 18/12/11 17:45:55 DEBUG conf.Configuration: parsing input stream sun.net.www.protocol.jar.JarURLConnection$JarURLInputStream@4232c52b 18/12/11 17:45:55 DEBUG conf.Configuration: parsing URL file:/etc/hadoop/conf.cloudera.yarn/core-site.xml 18/12/11 17:45:55 DEBUG conf.Configuration: parsing input stream java.io.BufferedInputStream@4671e53b 18/12/11 17:45:56 DEBUG core.Tracer: sampler.classes = ; loaded no samplers 18/12/11 17:45:56 TRACE core.TracerId: ProcessID(fmt=%{tname}/%{ip}): computed process ID of "FsShell/10.33.6.155" 18/12/11 17:45:56 TRACE core.TracerPool: TracerPool(Global): adding tracer Tracer(FsShell/10.33.6.155) 18/12/11 17:45:56 DEBUG core.Tracer: span.receiver.classes = ; loaded no span receivers 18/12/11 17:45:56 TRACE core.Tracer: Created Tracer(FsShell/10.33.6.155) for FsShell 18/12/11 17:45:56 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of successful kerberos logins and latency (milliseconds)]) 18/12/11 17:45:56 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of failed kerberos logins and latency (milliseconds)]) 18/12/11 17:45:56 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[GetGroups]) 18/12/11 17:45:56 DEBUG lib.MutableMetricsFactory: field private org.apache.hadoop.metrics2.lib.MutableGaugeLong org.apache.hadoop.security.UserGroupInformation$UgiMetrics.renewalFailuresTotal with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Renewal failures since startup]) 18/12/11 17:45:56 DEBUG lib.MutableMetricsFactory: field private org.apache.hadoop.metrics2.lib.MutableGaugeInt org.apache.hadoop.security.UserGroupInformation$UgiMetrics.renewalFailures with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Renewal failures since last successful login]) 18/12/11 17:45:56 DEBUG impl.MetricsSystemImpl: UgiMetrics, User and group related metrics 18/12/11 17:45:56 DEBUG security.SecurityUtil: Setting hadoop.security.token.service.use_ip to true Java config name: null Native config name: /etc/krb5.conf Loaded from native config 18/12/11 17:45:56 DEBUG security.Groups: Creating new Groups object 18/12/11 17:45:57 DEBUG security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000; warningDeltaMs=5000 [UnixLoginModule]: succeeded importing info: uid = 0 gid = 0 supp gid = 0 18/12/11 17:45:57 DEBUG security.UserGroupInformation: hadoop login [UnixLoginModule]: added UnixPrincipal, UnixNumericUserPrincipal, UnixNumericGroupPrincipal(s), to Subject 18/12/11 17:45:57 DEBUG security.UserGroupInformation: hadoop login commit 18/12/11 17:45:57 DEBUG security.UserGroupInformation: using local user:UnixPrincipal: root 18/12/11 17:45:57 DEBUG security.UserGroupInformation: Using user: "UnixPrincipal: root" with name root 18/12/11 17:45:57 DEBUG security.UserGroupInformation: User entry: "root" 18/12/11 17:45:57 DEBUG security.UserGroupInformation: UGI loginUser:root (auth:SIMPLE) 18/12/11 17:45:57 DEBUG core.Tracer: sampler.classes = ; loaded no samplers 18/12/11 17:45:57 TRACE core.TracerId: ProcessID(fmt=%{tname}/%{ip}): computed process ID of "FSClient/10.33.6.155" 18/12/11 17:45:57 TRACE core.TracerPool: TracerPool(Global): adding tracer Tracer(FSClient/10.33.6.155) 18/12/11 17:45:57 DEBUG core.Tracer: span.receiver.classes = ; loaded no span receivers 18/12/11 17:45:57 TRACE core.Tracer: Created Tracer(FSClient/10.33.6.155) for FSClient 18/12/11 17:45:57 DEBUG conf.Configuration: Reloading 3 existing configurations 18/12/11 17:45:58 DEBUG s3a.S3AFileSystem: Propagating entries under fs.s3a.bucket.myclouderaraj. 18/12/11 17:45:58 DEBUG s3a.S3AFileSystem: For URI s3a://myclouderaraj//root, using credentials AWSCredentialProviderList: BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider org.apache.hadoop.fs.s3a.SharedInstanceProfileCredentialsProvider@3f191845 18/12/11 17:45:59 DEBUG azure.NativeAzureFileSystem: finalize() called. 18/12/11 17:45:59 DEBUG s3a.S3AFileSystem: Value of fs.s3a.connection.maximum is 100 18/12/11 17:45:59 DEBUG s3a.S3AFileSystem: Value of fs.s3a.attempts.maximum is 20 18/12/11 17:45:59 DEBUG s3a.S3AFileSystem: Value of fs.s3a.connection.establish.timeout is 5000 18/12/11 17:45:59 DEBUG azure.NativeAzureFileSystem: finalize() called. 18/12/11 17:45:59 DEBUG s3a.S3AFileSystem: Value of fs.s3a.connection.timeout is 200000 18/12/11 17:45:59 DEBUG s3a.S3AFileSystem: Value of fs.s3a.socket.send.buffer is 8192 18/12/11 17:45:59 DEBUG s3a.S3AFileSystem: Value of fs.s3a.socket.recv.buffer is 8192 18/12/11 17:45:59 DEBUG s3a.S3AFileSystem: Using User-Agent: Hadoop 2.6.0-cdh5.13.0 18/12/11 17:45:59 DEBUG amazonaws.AmazonWebServiceClient: Internal logging successfully configured to commons logger: true 18/12/11 17:45:59 DEBUG metrics.AwsSdkMetrics: Admin mbean registered under com.amazonaws.management:type=AwsSdkMetrics 18/12/11 17:46:01 DEBUG s3a.S3AFileSystem: Value of fs.s3a.paging.maximum is 5000 18/12/11 17:46:01 DEBUG s3a.S3AFileSystem: Value of fs.s3a.block.size is 33554432 18/12/11 17:46:01 DEBUG s3a.S3AFileSystem: Value of fs.s3a.readahead.range is 65536 18/12/11 17:46:01 DEBUG s3a.S3AFileSystem: Value of fs.s3a.max.total.tasks is 5 18/12/11 17:46:01 DEBUG s3a.S3AFileSystem: Value of fs.s3a.threads.keepalivetime is 60 18/12/11 17:46:01 DEBUG s3.AmazonS3Client: Bucket region cache doesn't have an entry for myclouderaraj. Trying to get bucket region from Amazon S3. 18/12/11 17:46:01 DEBUG s3a.AWSCredentialProviderList: Using credentials from BasicAWSCredentialsProvider 18/12/11 17:46:01 DEBUG amazonaws.request: Sending Request: HEAD https://myclouderaraj.s3.amazonaws.com / Headers: (User-Agent: Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161, amz-sdk-invocation-id: 9c0b91f3-3175-fc6a-3669-b075a6a89b9a, Content-Type: application/octet-stream, ) 18/12/11 17:46:01 DEBUG auth.AWS4Signer: AWS4 Canonical Request: '"HEAD / amz-sdk-invocation-id:9c0b91f3-3175-fc6a-3669-b075a6a89b9a amz-sdk-retry:0/0/500 content-type:application/octet-stream host:myclouderaraj.s3.amazonaws.com user-agent:Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161 x-amz-content-sha256:UNSIGNED-PAYLOAD x-amz-date:20181211T164601Z amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date UNSIGNED-PAYLOAD" 18/12/11 17:46:01 DEBUG auth.AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256 20181211T164601Z 20181211/us-east-1/s3/aws4_request 767662b5323a076b055a71d9bf21e1b437c5791f56d0efcafab5311d6e510cfd" 18/12/11 17:46:01 DEBUG auth.AWS4Signer: Generating a new signing key as the signing key not available in the cache for the date 1544486400000 18/12/11 17:46:02 DEBUG protocol.RequestAddCookies: CookieSpec selected: default 18/12/11 17:46:02 DEBUG protocol.RequestAuthCache: Auth cache not set in the context 18/12/11 17:46:02 DEBUG conn.PoolingHttpClientConnectionManager: Connection request: [route: {s}->https://myclouderaraj.s3.amazonaws.com:443][total kept alive: 0; route allocated: 0 of 100; total allocated: 0 of 100] 18/12/11 17:46:02 DEBUG conn.PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com:443][total kept alive: 0; route allocated: 1 of 100; total allocated: 1 of 100] 18/12/11 17:46:02 DEBUG execchain.MainClientExec: Opening connection {s}->https://myclouderaraj.s3.amazonaws.com:443 18/12/11 17:46:02 DEBUG conn.DefaultHttpClientConnectionOperator: Connecting to myclouderaraj.s3.amazonaws.com/52.217.1.140:443 18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory: connecting to myclouderaraj.s3.amazonaws.com/52.217.1.140:443 18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory: Connecting socket to myclouderaraj.s3.amazonaws.com/52.217.1.140:443 with timeout 5000 18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory: Enabled protocols: [TLSv1, TLSv1.1, TLSv1.2] 18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory: Enabled cipher suites:[TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV] 18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory: socket.getSupportedProtocols(): [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2], socket.getEnabledProtocols(): [TLSv1, TLSv1.1, TLSv1.2] 18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory: TLS protocol enabled for SSL handshake: [TLSv1.2, TLSv1.1, TLSv1] 18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory: Starting handshake 18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory: Secure session established 18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory: negotiated protocol: TLSv1.2 18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory: negotiated cipher suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA 18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory: peer principal: CN=*.s3.amazonaws.com, O=Amazon.com Inc., L=Seattle, ST=Washington, C=US 18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory: peer alternative names: [*.s3.amazonaws.com, s3.amazonaws.com] 18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory: issuer principal: CN=DigiCert Baltimore CA-2 G2, OU=www.digicert.com, O=DigiCert Inc, C=US 18/12/11 17:46:03 DEBUG internal.SdkSSLSocket: created: myclouderaraj.s3.amazonaws.com/52.217.1.140:443 18/12/11 17:46:03 DEBUG conn.DefaultHttpClientConnectionOperator: Connection established 10.33.6.155:38019<->52.217.1.140:443 18/12/11 17:46:03 DEBUG conn.DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 18/12/11 17:46:03 DEBUG execchain.MainClientExec: Executing request HEAD / HTTP/1.1 18/12/11 17:46:03 DEBUG execchain.MainClientExec: Proxy auth state: UNCHALLENGED 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> HEAD / HTTP/1.1 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Host: myclouderaraj.s3.amazonaws.com 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> x-amz-content-sha256: UNSIGNED-PAYLOAD 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=AKIAIYFZQ7LF7F5MJL4Q/20181211/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=bc733f2ab05759751c4eeaf11095f286eceba2647ee8b014211cab2e03f967db 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> X-Amz-Date: 20181211T164601Z 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> User-Agent: Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> amz-sdk-invocation-id: 9c0b91f3-3175-fc6a-3669-b075a6a89b9a 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Content-Type: application/octet-stream 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Connection: Keep-Alive 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "HEAD / HTTP/1.1[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Host: myclouderaraj.s3.amazonaws.com[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "x-amz-content-sha256: UNSIGNED-PAYLOAD[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=AKIAIYFZQ7LF7F5MJL4Q/20181211/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=bc733f2ab05759751c4eeaf11095f286eceba2647ee8b014211cab2e03f967db[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "X-Amz-Date: 20181211T164601Z[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "User-Agent: Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "amz-sdk-invocation-id: 9c0b91f3-3175-fc6a-3669-b075a6a89b9a[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "HTTP/1.1 403 Forbidden[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "x-amz-bucket-region: us-east-1[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "x-amz-request-id: 93DABB98D9C20FB2[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "x-amz-id-2: GImfViIrJZLJvNPS0MGpxZSfsmJqibcEtOLcBK86sdCo79CpRHxlKQhANkddSqVMPIIBtIGEZho=[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "Transfer-Encoding: chunked[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "Date: Tue, 11 Dec 2018 17:08:34 GMT[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "Server: AmazonS3[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "[\r][\n]" 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << HTTP/1.1 403 Forbidden 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << x-amz-bucket-region: us-east-1 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << x-amz-request-id: 93DABB98D9C20FB2 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << x-amz-id-2: GImfViIrJZLJvNPS0MGpxZSfsmJqibcEtOLcBK86sdCo79CpRHxlKQhANkddSqVMPIIBtIGEZho= 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << Content-Type: application/xml 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << Transfer-Encoding: chunked 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << Date: Tue, 11 Dec 2018 17:08:34 GMT 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << Server: AmazonS3 18/12/11 17:46:03 DEBUG execchain.MainClientExec: Connection can be kept alive for 60000 MILLISECONDS 18/12/11 17:46:03 DEBUG conn.PoolingHttpClientConnectionManager: Connection [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com:443] can be kept alive for 60.0 seconds 18/12/11 17:46:03 DEBUG conn.PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com:443][total kept alive: 1; route allocated: 1 of 100; total allocated: 1 of 100] 18/12/11 17:46:03 DEBUG amazonaws.request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 93DABB98D9C20FB2), S3 Extended Request ID: GImfViIrJZLJvNPS0MGpxZSfsmJqibcEtOLcBK86sdCo79CpRHxlKQhANkddSqVMPIIBtIGEZho= 18/12/11 17:46:03 DEBUG s3.AmazonS3Client: Region for myclouderaraj is us-east-1 18/12/11 17:46:03 DEBUG amazonaws.request: Sending Request: HEAD https://myclouderaraj.s3.amazonaws.com / Headers: (User-Agent: Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161, amz-sdk-invocation-id: fa6a15f3-60e0-0dd4-85de-ce71e5864550, Content-Type: application/octet-stream, ) 18/12/11 17:46:03 DEBUG auth.AWS4Signer: AWS4 Canonical Request: '"HEAD / amz-sdk-invocation-id:fa6a15f3-60e0-0dd4-85de-ce71e5864550 amz-sdk-retry:0/0/500 content-type:application/octet-stream host:myclouderaraj.s3.amazonaws.com user-agent:Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161 x-amz-content-sha256:UNSIGNED-PAYLOAD x-amz-date:20181211T164603Z amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date UNSIGNED-PAYLOAD" 18/12/11 17:46:03 DEBUG auth.AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256 20181211T164603Z 20181211/us-east-1/s3/aws4_request 7af1674e22b74abaf21c562903716022a599fb7ae7c2cd46b493450627c84858" 18/12/11 17:46:03 DEBUG protocol.RequestAddCookies: CookieSpec selected: default 18/12/11 17:46:03 DEBUG protocol.RequestAuthCache: Auth cache not set in the context 18/12/11 17:46:03 DEBUG conn.PoolingHttpClientConnectionManager: Connection request: [route: {s}->https://myclouderaraj.s3.amazonaws.com:443][total kept alive: 1; route allocated: 1 of 100; total allocated: 1 of 100] 18/12/11 17:46:03 DEBUG conn.PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com:443][total kept alive: 0; route allocated: 1 of 100; total allocated: 1 of 100] 18/12/11 17:46:03 DEBUG conn.DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 18/12/11 17:46:03 DEBUG execchain.MainClientExec: Executing request HEAD / HTTP/1.1 18/12/11 17:46:03 DEBUG execchain.MainClientExec: Proxy auth state: UNCHALLENGED 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> HEAD / HTTP/1.1 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Host: myclouderaraj.s3.amazonaws.com 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> x-amz-content-sha256: UNSIGNED-PAYLOAD 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=AKIAIYFZQ7LF7F5MJL4Q/20181211/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=6e8a6bbf0e44be59a0894e30c4645049de3f3c8e2c774077a081885702236aaa 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> X-Amz-Date: 20181211T164603Z 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> User-Agent: Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> amz-sdk-invocation-id: fa6a15f3-60e0-0dd4-85de-ce71e5864550 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Content-Type: application/octet-stream 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Connection: Keep-Alive 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "HEAD / HTTP/1.1[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Host: myclouderaraj.s3.amazonaws.com[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "x-amz-content-sha256: UNSIGNED-PAYLOAD[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=AKIAIYFZQ7LF7F5MJL4Q/20181211/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=6e8a6bbf0e44be59a0894e30c4645049de3f3c8e2c774077a081885702236aaa[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "X-Amz-Date: 20181211T164603Z[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "User-Agent: Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "amz-sdk-invocation-id: fa6a15f3-60e0-0dd4-85de-ce71e5864550[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "HTTP/1.1 403 Forbidden[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "x-amz-bucket-region: us-east-1[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "x-amz-request-id: 0D4E622D9A858BD3[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "x-amz-id-2: dwOjYsPPO7Hn7MlUZPAZ6VVOVIWg6gGbQ0i9JY0YRfLnOV+GEhagAsXJuB1S0dt4bgo4VwZzpYk=[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "Transfer-Encoding: chunked[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "Date: Tue, 11 Dec 2018 17:08:35 GMT[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "Server: AmazonS3[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "[\r][\n]" 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << HTTP/1.1 403 Forbidden 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << x-amz-bucket-region: us-east-1 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << x-amz-request-id: 0D4E622D9A858BD3 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << x-amz-id-2: dwOjYsPPO7Hn7MlUZPAZ6VVOVIWg6gGbQ0i9JY0YRfLnOV+GEhagAsXJuB1S0dt4bgo4VwZzpYk= 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << Content-Type: application/xml 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << Transfer-Encoding: chunked 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << Date: Tue, 11 Dec 2018 17:08:35 GMT 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << Server: AmazonS3 18/12/11 17:46:03 DEBUG execchain.MainClientExec: Connection can be kept alive for 60000 MILLISECONDS 18/12/11 17:46:03 DEBUG conn.PoolingHttpClientConnectionManager: Connection [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com:443] can be kept alive for 60.0 seconds 18/12/11 17:46:03 DEBUG conn.PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com:443][total kept alive: 1; route allocated: 1 of 100; total allocated: 1 of 100] 18/12/11 17:46:03 DEBUG amazonaws.request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 0D4E622D9A858BD3), S3 Extended Request ID: dwOjYsPPO7Hn7MlUZPAZ6VVOVIWg6gGbQ0i9JY0YRfLnOV+GEhagAsXJuB1S0dt4bgo4VwZzpYk= 18/12/11 17:46:03 DEBUG s3a.S3AFileSystem: Value of fs.s3a.multipart.purge.age is 86400 18/12/11 17:46:03 INFO Configuration.deprecation: fs.s3a.server-side-encryption-key is deprecated. Instead, use fs.s3a.server-side-encryption.key 18/12/11 17:46:03 DEBUG s3a.S3AFileSystem: Data is unencrypted 18/12/11 17:46:03 DEBUG s3a.S3AFileSystem: Using SSE-C with null key 18/12/11 17:46:03 DEBUG s3a.S3AFileSystem: Using S3AOutputStream 18/12/11 17:46:03 DEBUG s3guard.S3Guard: Using NullMetadataStore metadata store for s3a filesystem 18/12/11 17:46:03 DEBUG s3a.S3AFileSystem: op_glob_status += 1 -> 1 18/12/11 17:46:03 DEBUG s3a.S3AFileSystem: op_get_file_status += 1 -> 1 18/12/11 17:46:03 DEBUG s3a.S3AFileSystem: Getting path status for s3a://myclouderaraj/root (root) 18/12/11 17:46:03 DEBUG s3a.S3AFileSystem: object_metadata_requests += 1 -> 1 18/12/11 17:46:03 DEBUG s3.AmazonS3Client: Region for myclouderaraj is us-east-1 18/12/11 17:46:03 DEBUG amazonaws.request: Sending Request: HEAD https://myclouderaraj.s3.amazonaws.com /root Headers: (User-Agent: Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161, amz-sdk-invocation-id: 1777bc11-f8c0-c825-58e6-113912bcd9cd, Content-Type: application/octet-stream, ) 18/12/11 17:46:03 DEBUG auth.AWS4Signer: AWS4 Canonical Request: '"HEAD /root amz-sdk-invocation-id:1777bc11-f8c0-c825-58e6-113912bcd9cd amz-sdk-retry:0/0/500 content-type:application/octet-stream host:myclouderaraj.s3.amazonaws.com user-agent:Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161 x-amz-content-sha256:UNSIGNED-PAYLOAD x-amz-date:20181211T164603Z amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date UNSIGNED-PAYLOAD" 18/12/11 17:46:03 DEBUG auth.AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256 20181211T164603Z 20181211/us-east-1/s3/aws4_request ed1bf2a929e630f833c3eb1f6639dbcb84723d570e76ed41cc99ddbc8b33d3a1" 18/12/11 17:46:03 DEBUG protocol.RequestAddCookies: CookieSpec selected: default 18/12/11 17:46:03 DEBUG protocol.RequestAuthCache: Auth cache not set in the context 18/12/11 17:46:03 DEBUG conn.PoolingHttpClientConnectionManager: Connection request: [route: {s}->https://myclouderaraj.s3.amazonaws.com:443][total kept alive: 1; route allocated: 1 of 100; total allocated: 1 of 100] 18/12/11 17:46:03 DEBUG conn.PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com:443][total kept alive: 0; route allocated: 1 of 100; total allocated: 1 of 100] 18/12/11 17:46:03 DEBUG conn.DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 18/12/11 17:46:03 DEBUG execchain.MainClientExec: Executing request HEAD /root HTTP/1.1 18/12/11 17:46:03 DEBUG execchain.MainClientExec: Proxy auth state: UNCHALLENGED 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> HEAD /root HTTP/1.1 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Host: myclouderaraj.s3.amazonaws.com 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> x-amz-content-sha256: UNSIGNED-PAYLOAD 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=AKIAIYFZQ7LF7F5MJL4Q/20181211/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=0e06c4f75551b83e1e122abdbf7092ae2c4413f38419d89572b77ecdc4da1d5d 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> X-Amz-Date: 20181211T164603Z 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> User-Agent: Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> amz-sdk-invocation-id: 1777bc11-f8c0-c825-58e6-113912bcd9cd 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Content-Type: application/octet-stream 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Connection: Keep-Alive 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "HEAD /root HTTP/1.1[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Host: myclouderaraj.s3.amazonaws.com[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "x-amz-content-sha256: UNSIGNED-PAYLOAD[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=AKIAIYFZQ7LF7F5MJL4Q/20181211/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=0e06c4f75551b83e1e122abdbf7092ae2c4413f38419d89572b77ecdc4da1d5d[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "X-Amz-Date: 20181211T164603Z[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "User-Agent: Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "amz-sdk-invocation-id: 1777bc11-f8c0-c825-58e6-113912bcd9cd[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "HTTP/1.1 403 Forbidden[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "x-amz-request-id: 1F1ED2B55D3DBA53[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "x-amz-id-2: oSqyv7R/B05MjjUWQflFaLoOv5Mb2ycprL+PcmqjussFP+11AiJxpKwj+MlDGrfULeumJRHkTjY=[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "Transfer-Encoding: chunked[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "Date: Tue, 11 Dec 2018 17:08:35 GMT[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "Server: AmazonS3[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "[\r][\n]" 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << HTTP/1.1 403 Forbidden 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << x-amz-request-id: 1F1ED2B55D3DBA53 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << x-amz-id-2: oSqyv7R/B05MjjUWQflFaLoOv5Mb2ycprL+PcmqjussFP+11AiJxpKwj+MlDGrfULeumJRHkTjY= 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << Content-Type: application/xml 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << Transfer-Encoding: chunked 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << Date: Tue, 11 Dec 2018 17:08:35 GMT 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << Server: AmazonS3 18/12/11 17:46:03 DEBUG execchain.MainClientExec: Connection can be kept alive for 60000 MILLISECONDS 18/12/11 17:46:03 DEBUG conn.PoolingHttpClientConnectionManager: Connection [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com:443] can be kept alive for 60.0 seconds 18/12/11 17:46:03 DEBUG conn.PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com:443][total kept alive: 1; route allocated: 1 of 100; total allocated: 1 of 100] 18/12/11 17:46:03 DEBUG amazonaws.request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 1F1ED2B55D3DBA53), S3 Extended Request ID: oSqyv7R/B05MjjUWQflFaLoOv5Mb2ycprL+PcmqjussFP+11AiJxpKwj+MlDGrfULeumJRHkTjY= ls: s3a://myclouderaraj/root: getFileStatus on s3a://myclouderaraj/root: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 1F1ED2B55D3DBA53), S3 Extended Request ID: oSqyv7R/B05MjjUWQflFaLoOv5Mb2ycprL+PcmqjussFP+11AiJxpKwj+MlDGrfULeumJRHkTjY= 18/12/11 17:46:03 TRACE core.TracerPool: TracerPool(Global): removing tracer Tracer(FsShell/10.33.6.155) 18/12/11 17:46:03 DEBUG http.IdleConnectionReaper: Reaper thread: java.lang.InterruptedException: sleep interrupted at java.lang.Thread.sleep(Native Method) at com.amazonaws.http.IdleConnectionReaper.run(IdleConnectionReaper.java:181) 18/12/11 17:46:03 DEBUG conn.PoolingHttpClientConnectionManager: Connection manager is shutting down 18/12/11 17:46:03 DEBUG http.IdleConnectionReaper: Shutting down reaper thread. 18/12/11 17:46:03 DEBUG conn.DefaultManagedHttpClientConnection: http-outgoing-0: Close connection 18/12/11 17:46:03 DEBUG internal.SdkSSLSocket: shutting down output of myclouderaraj.s3.amazonaws.com/52.217.1.140:443 18/12/11 17:46:03 DEBUG internal.SdkSSLSocket: closing myclouderaraj.s3.amazonaws.com/52.217.1.140:443 18/12/11 17:46:03 DEBUG conn.PoolingHttpClientConnectionManager: Connection manager shut down [root@quickstart ~]# I used the same command in both the machine where one is able to connect and the other is giving this issue.   Next try:   Also, I tried to do the following operations in the working machine for impala using impala-shell https://www.cloudera.com/documentation/enterprise/5-7-x/topics/impala_s3.html Updated [root@quickstart ~]# cd /etc/hadoop/conf/hdfs-site.xml file for access and secret key. Restarted all the services, also tried restarting cloudera-scm-server. Then ran the following from impala-shell [localhost:21000] > create database db_on_hdfs; [localhost:21000] > use db_on_hdfs; [localhost:21000] > create table mostly_on_hdfs (x int) partitioned by (year int); [localhost:21000] > alter table mostly_on_hdfs add partition (year=2013); [localhost:21000] > alter table mostly_on_hdfs add partition (year=2014); [localhost:21000] > alter table mostly_on_hdfs add partition (year=2015) location 's3a://impala-demo/dir1/dir2/dir3/t1'; [quickstart.cloudera:21000] > alter table mostly_on_hdfs add partition (year=2015) LOCATION 's3a://myclouderaraj/root'; Query: alter table mostly_on_hdfs add partition (year=2015) LOCATION 's3a://myclouderaraj/root' ERROR: AnalysisException: null CAUSED BY: AmazonClientException: Unable to load AWS credentials from any provider in the chain [quickstart.cloudera:21000] >   Can you please help.   Thansk & Regards, Rajesh   ... View more

Re: Metadata and lineage collection for S3

by Contributor Nukala in Support Questions
‎12-03-2018 09:40 AM
‎12-03-2018 09:40 AM
Hi Seth,   Tried the following which are not working   [root@quickstart ~]# hdfs dfs -Dfs.s3a.access.key=myaccesskey -Dfs.s3a.secret.key=mysecretkey -DFS.s3a -ls s3a://myclouderaraj/root 18/12/03 18:01:27 INFO Configuration.deprecation: fs.s3a.server-side-encryption-key is deprecated. Instead, use fs.s3a.server-side-encryption.key ls: s3a://myclouderaraj/root: getFileStatus on s3a://myclouderaraj/root: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 78B0773829C23DB5), S3 Extended Request ID: gOFP7ib+VT7lkUo3A/AMjXGKrqNebIPNHSbHRXGsZU9dzkxtw8dceLCNfkkFkypbfPNFN6Pqe1M= [root@quickstart ~]#   [root@quickstart ~]# hdfs dfs -Dfs.s3a.access.key= myaccesskey   -Dfs.s3a.secret.key= mysecretkey   -ls s3a://myclouderaraj/root 18/12/03 18:07:11 INFO Configuration.deprecation: fs.s3a.server-side-encryption-key is deprecated. Instead, use fs.s3a.server-side-encryption.key ls: s3a://myclouderaraj/root: getFileStatus on s3a://myclouderaraj/root: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 4BC4A44FB6133284), S3 Extended Request ID: dD8H3FqvwSrWxqfIWmNbjS+xzC5v8t0yUtQOGwJztzpKtMZ2sZ0k73RbUDxsv3pZzsC5u9Hg4as= [root@quickstart ~]#     [cloudera@quickstart ~]$ hdfs dfs -Dfs.s3a.access.key= myaccesskey   -Dfs.s3a.secret.key= mysecretkey   -DFS.s3a -ls s3a://myclouderaraj/root 18/12/03 18:04:56 INFO Configuration.deprecation: fs.s3a.server-side-encryption-key is deprecated. Instead, use fs.s3a.server-side-encryption.key ls: s3a://myclouderaraj/root: getFileStatus on s3a://myclouderaraj/root: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 9CCAB47BBC71B470), S3 Extended Request ID: Z/N0DveFbFio6AmyEk5EkIn4BfgNnRutSIdQ26xNYy9PDyliY87AadWM8ONEDggd1hRN+XVu1yw= [cloudera@quickstart ~]$ [cloudera@quickstart ~]$ [cloudera@quickstart ~]$ hdfs dfs -Dfs.s3a.access.key= myaccesskey   -Dfs.s3a.secret.key= mysecretkey   -ls s3a://myclouderaraj/root 18/12/03 18:05:12 INFO Configuration.deprecation: fs.s3a.server-side-encryption-key is deprecated. Instead, use fs.s3a.server-side-encryption.key ls: s3a://myclouderaraj/root: getFileStatus on s3a://myclouderaraj/root: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 75349677808F4357), S3 Extended Request ID: ObgEtqWMjCZQ4R8uMJf/R+x0msuPE5IQiMLnSFFTTJSH2NvZf4AtPUM8m5uncrxLeLqDfe8jEIM= [cloudera@quickstart ~]$   Thanks & Regards, Rajesh ... View more

Re: Metadata and lineage collection for S3

by Contributor Nukala in Support Questions
‎12-03-2018 03:41 AM
‎12-03-2018 03:41 AM
I see I can copy in CDH 5.7, but the error i mentioned is seen in CDH 5.13. To get work in CDH 5.13, do we need to update any thing. Can you please help on this issue.   Thanks & Regards, Rajesh ... View more

Re: Metadata and lineage collection for S3

by Contributor Nukala in Support Questions
‎11-29-2018 11:07 AM
‎11-29-2018 11:07 AM
In Hortonworks I was able run the following command with same credentials   [root@sandbox ~]# hdfs dfs -Dfs.s3a.access.key=myaccesskey -Dfs.s3a.secret.key=mysecretkey -ls s3a://myclouderaraj/root Found 1 items -rw-rw-rw- 1 root 2121 2018-11-29 18:32 s3a://myclouderaraj/root/temp.json   Is there anything specific to Cloudera? ... View more

Re: Metadata and lineage collection for S3

by Contributor Nukala in Support Questions
‎11-29-2018 10:49 AM
‎11-29-2018 10:49 AM
Seth,   From AWS CLI i tried the following and it was success   C:\Users\rajesh.nukala\Desktop>aws s3 cp temp.json s3://myclouderaraj/root/ upload: .\temp.json to s3://myclouderaraj/root/temp.json     Also,I tried the one which was provided by you but seeing the error as mentioned below   [root@quickstart ~]# hdfs dfs -ls -Dfs.s3a.access.key=<myaccesskey> -Dfs.s3a.secret.key=<mysecretkey> s3a://myclouderaraj/root -ls: Illegal option -Dfs.s3a.access.key= <myaccesskey> Usage: hadoop fs [generic options] -ls [-C] [-d] [-h] [-q] [-R] [-t] [-S] [-r] [-u] [<path> ...] [root@quickstart ~]#     [root@quickstart ~]# hdfs dfs -Dfs.s3a.access.key=<myaccesskey> -Dfs.s3a.secret.key=<mysecretkey> -ls s3a://myclouderaraj/root 18/11/29 19:19:31 INFO Configuration.deprecation: fs.s3a.server-side-encryption-key is deprecated. Instead, use fs.s3a.server-side-encryption.key ls: s3a://myclouderaraj/root: getFileStatus on s3a://myclouderaraj/root: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 062979117C7D7717), S3 Extended Request ID: HKTksk3mPhuVDGDxTfZjE6ElvqYGwO8+a7ryv5IQ14mBF721gfNGI6Xluvv/m0csI05KH0mLO7g= [root@quickstart ~]# ... View more

Re: Metadata and lineage collection for S3

by Contributor Nukala in Support Questions
‎11-29-2018 07:56 AM
‎11-29-2018 07:56 AM
Hello Seth,   I tried through CLI and now I am getting the below error  I used the below command hadoop distcp -Dfs.s3a.access.key=<myaccesskey> -Dfs.s3a.secret.key=<mysecretkey> /user/cloudera/hdfs_error.xml s3a://myclouderaraj 18/11/29 17:14:32 INFO tools.OptionsParser: parseChunkSize: blocksperchunk false 18/11/29 17:14:36 INFO Configuration.deprecation: fs.s3a.server-side-encryption-key is deprecated. Instead, use fs.s3a.server-side-encryption.key 18/11/29 17:14:36 ERROR tools.DistCp: Invalid arguments: java.lang.IllegalArgumentException: path must be absolute at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88) at org.apache.hadoop.fs.s3a.s3guard.PathMetadata.<init>(PathMetadata.java:63) at org.apache.hadoop.fs.s3a.s3guard.PathMetadata.<init>(PathMetadata.java:55) at org.apache.hadoop.fs.s3a.s3guard.PathMetadata.<init>(PathMetadata.java:51) at org.apache.hadoop.fs.s3a.s3guard.S3Guard.putAndReturn(S3Guard.java:138) at org.apache.hadoop.fs.s3a.   Also i tried this command and found the following issue   Not sure why this is looking for s3a://myclouderaraj/user/root Copy to S3 error Thanks & Regards, Rajesh ... View more

Metadata and lineage collection for S3

by Contributor Nukala in Support Questions
‎11-26-2018 06:47 PM
‎11-26-2018 06:47 PM
Hello,    I am trying to collect metadata and lineage information in Cloudera Navigator for S3. For this I have done the following configuration and while doing HDFS replication I am getting the following error   Error Message for HDFS replication ExternalAccounts added in Cloudera Manager Configuration to Access key credentials Configruarion to IAM based Configuration for S3 connector service CDH version used: 5.13.0   Could you please help in creading metadata and lineage for S3 in Cloudera Navigator. ... View more

Control Flows for sources other than Hive\Impala

by Contributor Nukala in Support Questions
‎10-22-2018 04:40 AM
‎10-22-2018 04:40 AM
Hello, I can see control flow in lineage diagrams for Hive and Impala operations. Are there any other sources which can provide the same?  Is it possible to see control flows for jobs executed by Spark or Yarn? ... View more

Re: Using REST API, how to search an operation wit...

by Contributor Nukala in Support Questions
‎10-22-2018 03:43 AM
‎10-22-2018 03:43 AM
Hello Perry,  Yes, exactly. I believe replacing everything suspicious is not a proper way of dealing with such a stuff because query can contain asterisk itself... ... View more

Re: How to setup mapreduce not to use yarn

by Contributor Nukala in Support Questions
‎10-22-2018 12:01 AM
‎10-22-2018 12:01 AM
Thanks. ... View more

Re: Cloudera Navigator Cluster Instance and Group

by Contributor Nukala in Support Questions
‎10-21-2018 02:17 AM
‎10-21-2018 02:17 AM
Thank you Elias.   Here is my understanding on Cluster, Cluster Group and Cluster Instances. Please correct me if I am wrong and also please clarify my doubts   1. Cluster Group and Cluster Instances are mostly related to AWS. 2. When you add  AWS Cluster as an external account to Cloudera, It will have a Cluster Group and Cluster Instance 3. Cluster Group will have multiple Cluster Instance which are same as different Clusters(AWS AMI's) added as external accounts to Cloudera from same AWS account 4. Suppose, I have a Cloudera Cluster then I have one Cluster Group and Cluster Instance. - When I add AWS as an external account to the Cluster then I will have two Cluster Group and two Cluster Instances where the AWS Cluster Group will have the AWS Cluster Instance. - Similarly if I add another AWS Cluster from the same account, then it will add another Cluster Instance to the same Cluster Group. So, now we will have two Cluster Groups and three Cluster Instances.   How we can identify different source types from different Clusters using Cloudera Navigator REST API's?   I see the following in Cloudera Navigator UI. Could you please how we can relate this with REST API response. For Operation entities - in metadata it is having Cluster Group     http://10.33.6.155:7187/api/v9/entities?query=(sourceType:mapreduce)AND(originalName:wordcount)   [ { "originalName" : "wordcount", "originalDescription" : null, "sourceId" : "4692666", "firstClassParentId" : null, "parentPath" : null, "deleteTime" : null, "extractorRunId" : "4692666##525", "customProperties" : null, "name" : "wordcount", "description" : null, "tags" : null, "properties" : null, "technicalProperties" : null, "wfIds" : null, "inputFormat" : null, "outputFormat" : null, "outputKey" : "org.apache.hadoop.io.Text", "outputValue" : "org.apache.hadoop.io.IntWritable", "mapper" : "org.myorg.WordCount$Map", "reducer" : "org.myorg.WordCount$Reduce", "sourceType" : "MAPREDUCE", "type" : "OPERATION", "metaClassName" : "mrjobspec", "deleted" : null, "packageName" : "nav", "userEntity" : false, "identity" : "5431085", "internalType" : "mrjobspec" } ]   For Operation Execution - in metadata it is having Cluster Instance   http://10.33.6.155:7187/api/v9/entities?query=(sourceType:mapreduce)AND(originalName:job_201810202026_0003)   [ { "originalName" : "job_201810202026_0003", "originalDescription" : null, "sourceId" : "4692666", "firstClassParentId" : null, "parentPath" : null, "deleteTime" : null, "extractorRunId" : "4692666##525", "customProperties" : null, "name" : null, "description" : null, "tags" : null, "properties" : null, "technicalProperties" : null, "started" : "2018-10-20T18:47:20.530Z", "ended" : "2018-10-20T18:47:50.778Z", "principal" : "cloudera", "inputs" : [ "hdfs://quickstart.cloudera:8020/user/cloudera/wordcount/input" ], "outputs" : [ "hdfs://quickstart.cloudera:8020/user/cloudera/wordcount/output13" ], "wfInstId" : null, "errorCodes" : null, "jobID" : "job_201810202026_0003", "sourceType" : "MAPREDUCE", "inputRecursive" : false, "type" : "OPERATION_EXECUTION", "metaClassName" : "mrjobinstance", "deleted" : null, "packageName" : "nav", "userEntity" : false, "identity" : "5435211", "internalType" : "mrjobinstance" } ]   ... View more

Re: Cloudera Pavigator PARTITION output format cha...

by Contributor Nukala in Support Questions
‎10-20-2018 11:31 AM
‎10-20-2018 11:31 AM
Could you please let me know how to submit a case? ... View more

Re: How to setup mapreduce not to use yarn

by Contributor Nukala in Support Questions
‎10-18-2018 10:36 AM
‎10-18-2018 10:36 AM
Yes, but I still want to have some data to be created with sourceType:Mapreduce. So, I have to execute by mapreduce job not to use Yarn. Please let me know the configuration to achive this. ... View more
Contact Me
Online
Offline
Last Visited
‎01-08-2020 02:20 AM
Public Statistics
Date Registered ‎09-05-2018 10:48 PM
Last Visited ‎01-08-2020 02:20 AM
Total Messages Posted 49
Total Kudos Received 2