Member since
09-05-2018
48
Posts
2
Kudos Received
1
Solution
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 112 | 07-23-2019 03:51 AM |
10-07-2019
12:54 AM
Hi, Below are the 2 screen shots. If you see hivesampledirectory is not showing in cloudera Navigator, but it is present in S3. And other directory like SampleDirectory, Devfolder and all are present in S3, but it is showing in deleted status in ClouderaNavigator. I checked log files (stdout, std err), don't see any error. But in Role log file I can see some error that specified bucket doesn't exist. I am giving the log information after the screen shot. CloudEra navigator image S3 Image Log file : 2019-10-07 09:15:12,324 INFO com.cloudera.nav.utils.JsonModelLoader [main]: Added audit event model for HUE 2019-10-07 09:15:12,333 INFO com.cloudera.nav.utils.JsonModelLoader [main]: Added audit event model for SOLR 2019-10-07 09:15:12,337 INFO com.cloudera.nav.utils.JsonModelLoader [main]: Added audit event model for SENTRY 2019-10-07 09:15:12,343 INFO com.cloudera.nav.utils.JsonModelLoader [main]: Added audit event model for NAVMS 2019-10-07 09:15:12,901 INFO com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource [main]: Initializing c3p0 pool... com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3, acquireRetryAttempts -> 30, acquireRetryDelay -> 1000, autoCommitOnClose -> false, automaticTestTable -> null, breakAfterAcquireFailure -> false, checkoutTimeout -> 0, connectionCustomizerClassName -> null, connectionTesterClassName -> com.mchange.v2.c3p0.impl.DefaultConnectionTester, dataSourceName -> 2t7fqea519ss8gsjrxr8i|45099dd3, debugUnreturnedConnectionStackTraces -> false, description -> null, driverClass -> com.mysql.jdbc.Driver, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> false, identityToken -> 2t7fqea519ss8gsjrxr8i|45099dd3, idleConnectionTestPeriod -> 300, initialPoolSize -> 2, jdbcUrl -> jdbc:mysql://quickstart.cloudera/navms?useUnicode=true&characterEncoding=UTF-8, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 0, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 50, maxStatements -> 0, maxStatementsPerConnection -> 0, minPoolSize -> 5, numHelperThreads -> 3, numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery -> null, properties -> {user=******, password=BUNDLE-REDACTED propertyCycle -> 0, testConnectionOnCheckin -> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout -> 0, usesTraditionalReflectiveProxies -> false ] 2019-10-07 09:15:16,831 INFO com.cloudera.nav.analytics.dataservices.api.service.DataService [main]: Caching is enabled for SOLR based metrics for usage in dashboard. 2019-10-07 09:15:19,627 WARN org.springframework.web.context.support.XmlWebApplicationContext [main]: Exception encountered during context initialization - cancelling refresh attempt org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'CDXExtractorService' defined in URL [jar:file:/var/lib/cloudera-scm-navigator/temp/jetty-0.0.0.0-7187-nav-core-webapp-2.12.0.war-_-any-/webapp/WEB-INF/lib/nav-service-2.12.0.jar!/com/cloudera/nav/extract/CDXExtractorService.class]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.cloudera.nav.extract.CDXExtractorService]: Constructor threw exception; nested exception is com.amazonaws.services.s3.model.AmazonS3Exception: The specified bucket does not exist (Service: Amazon S3; Status Code: 404; Error Code: NoSuchBucket; Request ID: 864FF96F219DEC19), S3 Extended Request ID: 7uWwv/0AOsBSSP0z+hHez1a82xhjdNvAv/g+ZKr5PxQ6Nq9q1VI6f+alFAf7pEYytaKU5mQa1tc= at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:278) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1115) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1018) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:504) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:475) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:302) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:298) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:706) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:762) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:482) at org.apache.cxf.transport.servlet.CXFServlet.createSpringContext(CXFServlet.java:160) at org.apache.cxf.transport.servlet.CXFServlet.loadBus(CXFServlet.java:74) at org.apache.cxf.transport.servlet.CXFNonSpringServlet.init(CXFNonSpringServlet.java:71) at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:532) at org.eclipse.jetty.servlet.ServletHolder.doStart(ServletHolder.java:344) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64) at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:791) at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:265) at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1242) at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:717) at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:494) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64) at org.eclipse.jetty.server.handler.HandlerCollection.doStart(HandlerCollection.java:229) at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:172) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64) at org.eclipse.jetty.server.handler.HandlerWrapper.doStart(HandlerWrapper.java:95) at org.eclipse.jetty.server.Server.doStart(Server.java:282) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64) at com.cloudera.nav.server.NavServer.run(NavServer.java:422) at com.cloudera.nav.server.NavServer.main(NavServer.java:117) Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.cloudera.nav.extract.CDXExtractorService]: Constructor threw exception; nested exception is com.amazonaws.services.s3.model.AmazonS3Exception: The specified bucket does not exist (Service: Amazon S3; Status Code: 404; Error Code: NoSuchBucket; Request ID: 864FF96F219DEC19), S3 Extended Request ID: 7uWwv/0AOsBSSP0z+hHez1a82xhjdNvAv/g+ZKr5PxQ6Nq9q1VI6f+alFAf7pEYytaKU5mQa1tc= at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:164) at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:125) at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:270) ... 31 more Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: The specified bucket does not exist (Service: Amazon S3; Status Code: 404; Error Code: NoSuchBucket; Request ID: 864FF96F219DEC19), S3 Extended Request ID: 7uWwv/0AOsBSSP0z+hHez1a82xhjdNvAv/g+ZKr5PxQ6Nq9q1VI6f+alFAf7pEYytaKU5mQa1tc= at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1545) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1183) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:964) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:676) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:650) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:633) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:601) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:583) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:447) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4137) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4079) at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:819) at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:797) at com.cloudera.cdx.client.impl.bulk.download.S3Downloader.getAllKeysAtPrefix(S3Downloader.java:215) at com.cloudera.cdx.client.impl.bulk.download.S3Downloader.recoverUncommitted(S3Downloader.java:241) at com.cloudera.cdx.client.impl.bulk.download.S3Downloader.<init>(S3Downloader.java:75) at com.cloudera.cdx.client.impl.bulk.CdxBulkImporter.<init>(CdxBulkImporter.java:61) at com.cloudera.cdx.client.CdxImporterFactory.newImporter(CdxImporterFactory.java:37) at com.cloudera.nav.extract.AbstractCDXImporterExtractorFactory.createImporter(AbstractCDXImporterExtractorFactory.java:46) at com.cloudera.nav.extract.AbstractCDXImporterExtractorFactory.<init>(AbstractCDXImporterExtractorFactory.java:36) at com.cloudera.nav.extract.AbstractCDXExtractorFactory.<init>(AbstractCDXExtractorFactory.java:49) at com.cloudera.nav.extract.CDXSourceExtractor.<init>(CDXSourceExtractor.java:45) at com.cloudera.nav.extract.CDXExtractorService.<init>(CDXExtractorService.java:88) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:148) ... 33 more 2019-10-07 09:15:19,794 WARN / [main]: unavailable org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'CDXExtractorService' defined in URL [jar:file:/var/lib/cloudera-scm-navigator/temp/jetty-0.0.0.0-7187-nav-core-webapp-2.12.0.war-_-any-/webapp/WEB-INF/lib/nav-service-2.12.0.jar!/com/cloudera/nav/extract/CDXExtractorService.class]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.cloudera.nav.extract.CDXExtractorService]: Constructor threw exception; nested exception is com.amazonaws.services.s3.model.AmazonS3Exception: The specified bucket does not exist (Service: Amazon S3; Status Code: 404; Error Code: NoSuchBucket; Request ID: 864FF96F219DEC19), S3 Extended Request ID: 7uWwv/0AOsBSSP0z+hHez1a82xhjdNvAv/g+ZKr5PxQ6Nq9q1VI6f+alFAf7pEYytaKU5mQa1tc= at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:278) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1115) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1018) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:504) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:475) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:302) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:298) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:706) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:762) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:482) at org.apache.cxf.transport.servlet.CXFServlet.createSpringContext(CXFServlet.java:160) at org.apache.cxf.transport.servlet.CXFServlet.loadBus(CXFServlet.java:74) at org.apache.cxf.transport.servlet.CXFNonSpringServlet.init(CXFNonSpringServlet.java:71)
... View more
09-23-2019
01:46 AM
Hi,
Below are my questions.
1. Observed that in Cloudera Navigator, once the metadata is collected from Amazon s3 I see that the data is updating its status to deleted (deleted property to true). But actually in S3 the file/directory is not deleted. What is the reason for this?
2.Created an external table in hive specifying its location to AmazonS3 path using Hue. Observed that the S3 data is not reflecting automatically. Some time it reflects after restarting and some time it is not. Is there any specific configurations for this to reflect S3 items immediately in Cloudera Navigator?
3. I followed the steps to extract S3 metadata and lineage information from Amazon s3 to cloudera navigator, but in lineage tab folder hierarchy is showing. I follwed below url.
https://www.cloudera.com/documentation/enterprise/5-14-x/topics/navigator_s3_metadata_extraction.html Do I need to create any cloudera altus account for lineage extraction?
4. Can we able to see any lineage for the below query(expecting that there is lineage between hive table and s3 directory), Is my expectation is correct?
CREATE EXTERNAL TABLE sampletable (key STRING, value INT) LOCATION 's3n://mybucketName/directoryName';
Also, I have created a backup job between HDFS : S3 and Hive : S3, but was not able to see the lineage.
Thanks,
... View more
07-23-2019
10:21 AM
Hello, I am trying to add AWS credentials as External Account and getting the S3 metadata information in Cloudera Navigator. For this when i use the personal account I was able to get the S3 entities metadata information, but when i do with my corp AWS account I was not able to get the S3 entities metadata information. Could you please help in providing the policies/permissions from AWS side to get access to S3 from Cloudera Navigator. Thanks & Regards, Rajesh
... View more
Labels:
07-23-2019
03:51 AM
Thanks. It worked.
... View more
07-15-2019
11:49 AM
I am trying to install Cloudera using VMware and docker image and after installtion I am not able to find the Cloudera Navigator and metadataserver running. Could you please quickly help on this. Version used: cloudera-quickstart-vm-5.13.0-0-beta-docker cloudera-quickstart-vm-5.13.0-0-vmware Thanks in advance. Regards, Rajesh
... View more
05-15-2019
01:36 PM
Hi Seth, Thanks for the response. I have generated the bundle data file and not sure that it can contain any credentials. Let me confirm from my side. Can you please confirm that it will get credentials information too? Thanks & Regards, Rajesh
... View more
05-13-2019
02:11 AM
Hello, I have tried to do custom configuration and done setup for SQS and SNS and I see that the messages are part of the Queue but not received in Clouderanavigator. Could you please help on this. SQSMessages Thanks & Regards, Rajesh
... View more
05-10-2019
06:43 PM
Hello, Can you please help on this issue. Thanks & Regards, Rajesh
... View more
05-08-2019
11:20 PM
Hello @seth ,
Can you please help on this issue?
Thanks & Regards,
Rajesh
... View more
05-07-2019
03:10 AM
Hi Seth, Thank you for the link. I followed the document and created default configuration and also provided Amazon S3 and SQS access to the user I used. Still I was not able to see any metadata or source type for S3 in Cloudera navigator. Could you please help on this? Also, please let me know how I can get the live support to resolve this issue quickly. Thanks & Regards, Rajesh
... View more
05-03-2019
12:44 AM
Hi, I am trying to collect S3 objects metadata in ClouderaNavigator and did the following configuration after which I was not able to see any sourceType for S3 in Cloudera Navigator. Could you please help on this Cloudera Version: CDH 6.1.0, Parcels Added S3 account as an External Account in Cloudera Manager and enabled metadata After that I see the S3 Extraction was enabled in Cloudera Navigator Administration Advanced tab After that I was not able to get any metadata or sourcetype added for S3 in Cloudera Navigator. Did i miss something here? Thanks & Regards, Rajesh
... View more
12-13-2018
03:35 AM
Hello, Now I was able to run the same query by specifying like this. Not sure does the one which is failed requires any more configuration to be set. alter table mostly_on_hdfs add partition (year=2015) location 's3a://AWS_ACCESS_KEY_ID:AWS_SECRET_ACCESS_KEY@myclouderaraj/root'; My intension of running these queries to get metadata and lineage information of S3 items in Cloudera Navigator. But I see nothing related to S3 in Cloudera Navigator. Thanks, Rajesh
... View more
12-12-2018
06:50 PM
Hello, I am trying to create an environment to have metadata and lineage information in Cloudera Navigator. For this I am trying to add the connection of S3 and thinking of running the backup sync from HDFS to S3. Does this collects the metadata lineage information in Cloudera Navigator. Is there any other way to collect metadata in Cloudera Navigator. Thanks & Regards, Rajesh
... View more
Labels:
12-12-2018
06:41 PM
Hello,
I tried to do the following operations in Cloudera machine for impala using impala-shell
https://www.cloudera.com/documentation/enterprise/5-7-x/topics/impala_s3.html
In this Cloudera machine I verified the S3 connectivity using the following command and was able to access
hdfs dfs -Dfs.s3a.access.key=myaccesskey -Dfs.s3a.secret.key=mysecretkey -ls s3a://myclouderaraj/root
Updated [root@quickstart ~]# cd /etc/hadoop/conf/hdfs-site.xml file for access and secret key.
Restarted all the services, also tried restarting cloudera-scm-server.
Then ran the following from impala-shell
[localhost:21000] > create database db_on_hdfs;
[localhost:21000] > use db_on_hdfs;
[localhost:21000] > create table mostly_on_hdfs (x int) partitioned by (year int);
[localhost:21000] > alter table mostly_on_hdfs add partition (year=2013);
[localhost:21000] > alter table mostly_on_hdfs add partition (year=2014);
[localhost:21000] > alter table mostly_on_hdfs add partition (year=2015) location 's3a://impala-demo/dir1/dir2/dir3/t1';
[quickstart.cloudera:21000] > alter table mostly_on_hdfs add partition (year=2015) LOCATION 's3a://myclouderaraj/root'; Query: alter table mostly_on_hdfs add partition (year=2015) LOCATION 's3a://myclouderaraj/root' ERROR: AnalysisException: null CAUSED BY: AmazonClientException: Unable to load AWS credentials from any provider in the chain
[quickstart.cloudera:21000] >
Can you please help.
Thanks & Regards,
Rajesh
... View more
Labels:
12-12-2018
06:35 PM
Yes, it was successful and i was not able to attach the complete logs. Yes, I see that there is some change in the request headers/user-agent. Not sure why that was. I have not done any specific changes to different versions of Cloudera and trying to connect the same S3 account. 18/12/11 19:12:54 DEBUG conn.DefaultClientConnection: Sending request: GET /?prefix=root%2F&delimiter=%2F&max-keys=1 HTTP/1.1 18/12/11 19:12:54 DEBUG http.wire: >> "GET /?prefix=root%2F&delimiter=%2F&max-keys=1 HTTP/1.1[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Host: myclouderaraj.s3.amazonaws.com[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:xhHdbA514F67IgOMPolX00Rv1h4=[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Date: Wed, 12 Dec 2018 03:12:54 GMT[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Content-Type: application/x-www-form-urlencoded; charset=utf-8[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Connection: Keep-Alive[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "[\r][\n]" 18/12/11 19:12:54 DEBUG http.headers: >> GET /?prefix=root%2F&delimiter=%2F&max-keys=1 HTTP/1.1 18/12/11 19:12:54 DEBUG http.headers: >> Host: myclouderaraj.s3.amazonaws.com 18/12/11 19:12:54 DEBUG http.headers: >> Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:xhHdbA514F67IgOMPolX00Rv1h4= 18/12/11 19:12:54 DEBUG http.headers: >> User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192 18/12/11 19:12:54 DEBUG http.headers: >> Date: Wed, 12 Dec 2018 03:12:54 GMT 18/12/11 19:12:54 DEBUG http.headers: >> Content-Type: application/x-www-form-urlencoded; charset=utf-8 18/12/11 19:12:54 DEBUG http.headers: >> Connection: Keep-Alive 18/12/11 19:12:54 DEBUG http.wire: << "HTTP/1.1 200 OK[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "x-amz-id-2: n67wc46WhWXcw4ygKZ0I2NEYJXqmyEG2sDyv8+hKbZaGCflTI6YB4gvIpuqQgHb+LCSgCQ33MlU=[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "x-amz-request-id: BE60D884EB5E4D60[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Date: Wed, 12 Dec 2018 03:12:55 GMT[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "x-amz-bucket-region: us-east-1[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Content-Type: application/xml[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Transfer-Encoding: chunked[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Server: AmazonS3[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:54 DEBUG conn.DefaultClientConnection: Receiving response: HTTP/1.1 200 OK 18/12/11 19:12:54 DEBUG http.headers: << HTTP/1.1 200 OK 18/12/11 19:12:54 DEBUG http.headers: << x-amz-id-2: n67wc46WhWXcw4ygKZ0I2NEYJXqmyEG2sDyv8+hKbZaGCflTI6YB4gvIpuqQgHb+LCSgCQ33MlU= 18/12/11 19:12:54 DEBUG http.headers: << x-amz-request-id: BE60D884EB5E4D60 18/12/11 19:12:54 DEBUG http.headers: << Date: Wed, 12 Dec 2018 03:12:55 GMT 18/12/11 19:12:54 DEBUG http.headers: << x-amz-bucket-region: us-east-1 18/12/11 19:12:54 DEBUG http.headers: << Content-Type: application/xml 18/12/11 19:12:54 DEBUG http.headers: << Transfer-Encoding: chunked 18/12/11 19:12:54 DEBUG http.headers: << Server: AmazonS3 18/12/11 19:12:54 DEBUG client.SdkHttpClient: Connection can be kept alive indefinitely 18/12/11 19:12:54 DEBUG amazonaws.requestId: x-amzn-RequestId: not available 18/12/11 19:12:54 TRACE amazonaws.request: Beginning to parse service response XML 18/12/11 19:12:54 DEBUG transform.XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.cloudera.com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler 18/12/11 19:12:54 DEBUG http.wire: << "286[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "<?xml version="1.0" encoding="UTF-8"?>[\n]" 18/12/11 19:12:54 DEBUG http.wire: << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>myclouderaraj</Name><Prefix>root/</Prefix><Marker></Marker><NextMarker>root/bulkInsert.csv</NextMarker><MaxKeys>1</MaxKeys><Delimiter>/</Delimiter><IsTruncated>true</IsTruncated><Contents><Key>root/bulkInsert.csv</Key><LastModified>2018-11-30T10:30:01.000Z</LastModified><ETag>"963e1e40ac0247f23ebf138755ceabfb"</ETag><Size>224</Size><Owner><ID>fdbe33e179958960b9d1fab4b547c1b8ccf7a9ed1b17b3ecc20f39d0866957ab</ID><DisplayName>rajesh210</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents></ListBucketResult>" 18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "0[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com] can be kept alive indefinitely 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection released: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:54 DEBUG transform.XmlResponsesSaxParser: Parsing XML response document with handler: class com.cloudera.com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler 18/12/11 19:12:54 DEBUG transform.XmlResponsesSaxParser: Examining listing for bucket: myclouderaraj 18/12/11 19:12:54 TRACE amazonaws.request: Done parsing service response XML 18/12/11 19:12:54 DEBUG amazonaws.request: Received successful response: 200, AWS Request ID: BE60D884EB5E4D60 18/12/11 19:12:54 DEBUG amazonaws.requestId: AWS Request ID: BE60D884EB5E4D60 18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: Found path as directory (with /): 0/1 18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: Summary: root/bulkInsert.csv 224 18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: listStatus: doing listObjects for directory root/ 18/12/11 19:12:54 DEBUG amazonaws.request: Sending Request: GET https://myclouderaraj.s3.amazonaws.com / Parameters: ({"prefix":["root/"],"delimiter":["/"],"max-keys":["5000"]}Headers: (User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192, Content-Type: application/x-www-form-urlencoded; charset=utf-8, ) 18/12/11 19:12:54 DEBUG internal.S3Signer: Calculated string to sign: "GET application/x-www-form-urlencoded; charset=utf-8 Wed, 12 Dec 2018 03:12:54 GMT /myclouderaraj/" 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection request: [route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection leased: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 0; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:54 DEBUG client.SdkHttpClient: Stale connection check 18/12/11 19:12:54 DEBUG protocol.RequestAddCookies: CookieSpec selected: best-match 18/12/11 19:12:54 DEBUG protocol.RequestAuthCache: Auth cache not set in the context 18/12/11 19:12:54 DEBUG protocol.RequestProxyAuthentication: Proxy auth state: UNCHALLENGED 18/12/11 19:12:54 DEBUG client.SdkHttpClient: Attempt 1 to execute request 18/12/11 19:12:54 DEBUG conn.DefaultClientConnection: Sending request: GET /?prefix=root%2F&delimiter=%2F&max-keys=5000 HTTP/1.1 18/12/11 19:12:54 DEBUG http.wire: >> "GET /?prefix=root%2F&delimiter=%2F&max-keys=5000 HTTP/1.1[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Host: myclouderaraj.s3.amazonaws.com[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:xhHdbA514F67IgOMPolX00Rv1h4=[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Date: Wed, 12 Dec 2018 03:12:54 GMT[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Content-Type: application/x-www-form-urlencoded; charset=utf-8[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Connection: Keep-Alive[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "[\r][\n]" 18/12/11 19:12:54 DEBUG http.headers: >> GET /?prefix=root%2F&delimiter=%2F&max-keys=5000 HTTP/1.1 18/12/11 19:12:54 DEBUG http.headers: >> Host: myclouderaraj.s3.amazonaws.com 18/12/11 19:12:54 DEBUG http.headers: >> Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:xhHdbA514F67IgOMPolX00Rv1h4= 18/12/11 19:12:54 DEBUG http.headers: >> User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192 18/12/11 19:12:54 DEBUG http.headers: >> Date: Wed, 12 Dec 2018 03:12:54 GMT 18/12/11 19:12:54 DEBUG http.headers: >> Content-Type: application/x-www-form-urlencoded; charset=utf-8 18/12/11 19:12:54 DEBUG http.headers: >> Connection: Keep-Alive 18/12/11 19:12:54 DEBUG http.wire: << "HTTP/1.1 200 OK[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "x-amz-id-2: qt25vFz8qCf82h/fM/Pl2kHdk8eWPN12AmczWd25XULPEZbxmvgkoWmnMkDdaK9O13AEcF7V5Fs=[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "x-amz-request-id: 70E4AB92D29F2154[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Date: Wed, 12 Dec 2018 03:12:55 GMT[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "x-amz-bucket-region: us-east-1[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Content-Type: application/xml[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Transfer-Encoding: chunked[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Server: AmazonS3[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:54 DEBUG conn.DefaultClientConnection: Receiving response: HTTP/1.1 200 OK 18/12/11 19:12:54 DEBUG http.headers: << HTTP/1.1 200 OK 18/12/11 19:12:54 DEBUG http.headers: << x-amz-id-2: qt25vFz8qCf82h/fM/Pl2kHdk8eWPN12AmczWd25XULPEZbxmvgkoWmnMkDdaK9O13AEcF7V5Fs= 18/12/11 19:12:54 DEBUG http.headers: << x-amz-request-id: 70E4AB92D29F2154 18/12/11 19:12:54 DEBUG http.headers: << Date: Wed, 12 Dec 2018 03:12:55 GMT 18/12/11 19:12:54 DEBUG http.headers: << x-amz-bucket-region: us-east-1 18/12/11 19:12:54 DEBUG http.headers: << Content-Type: application/xml 18/12/11 19:12:54 DEBUG http.headers: << Transfer-Encoding: chunked 18/12/11 19:12:54 DEBUG http.headers: << Server: AmazonS3 18/12/11 19:12:54 DEBUG client.SdkHttpClient: Connection can be kept alive indefinitely 18/12/11 19:12:54 DEBUG amazonaws.requestId: x-amzn-RequestId: not available 18/12/11 19:12:54 TRACE amazonaws.request: Beginning to parse service response XML 18/12/11 19:12:54 DEBUG transform.XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.cloudera.com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler 18/12/11 19:12:54 DEBUG http.wire: << "64b[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "<?xml version="1.0" encoding="UTF-8"?>[\n]" 18/12/11 19:12:54 DEBUG http.wire: << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>myclouderaraj</Name><Prefix>root/</Prefix><Marker></Marker><MaxKeys>5000</MaxKeys><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Contents><Key>root/bulkInsert.csv</Key><LastModified>2018-11-30T10:30:01.000Z</LastModified><ETag>"963e1e40ac0247f23ebf138755ceabfb"</ETag><Size>224</Size><Owner><ID>fdbe33e179958960b9d1fab4b547c1b8ccf7a9ed1b17b3ecc20f39d0866957ab</ID><DisplayName>rajesh210</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>root/customers</Key><LastModified>2018-11-30T05:57:52.000Z</LastModified><ETag>"8811238ea9271f2441f6e9a96b1393a9"</ETag><Size>15812</Size><Owner><ID>fdbe33e179958960b9d1fab4b547c1b8ccf7a9ed1b17b3ecc20f39d0866957ab</ID><DisplayName>rajesh210</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>root/table_test.csv</Key><LastModified>2018-11-30T10:30:26.000Z</LastModified><ETag>"d41d8cd98f00b204e9800998ecf8427e"</ETag><Size>0</Size><Owner><ID>fdbe33e179958960b9d1fab4b547c1b8ccf7a9ed1b17b3ecc20f39d0866957ab</ID><DisplayName>rajesh210</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>root/temp.json</Key><LastModified>2018-11-29T18:32:55.000Z</LastModified><ETag>"bdd3d8948e659d61ab0438132780e116"</ETag><Size>2121</Size><Owner><ID>fdbe33e179958960b9d1fab4b547c1b8ccf7a9ed1b17b3ecc20f39d0866957ab</ID><DisplayName>rajesh210</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents></ListBucketResult>" 18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "0[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com] can be kept alive indefinitely 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection released: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:54 DEBUG transform.XmlResponsesSaxParser: Parsing XML response document with handler: class com.cloudera.com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler 18/12/11 19:12:54 DEBUG transform.XmlResponsesSaxParser: Examining listing for bucket: myclouderaraj 18/12/11 19:12:54 TRACE amazonaws.request: Done parsing service response XML 18/12/11 19:12:54 DEBUG amazonaws.request: Received successful response: 200, AWS Request ID: 70E4AB92D29F2154 18/12/11 19:12:54 DEBUG amazonaws.requestId: AWS Request ID: 70E4AB92D29F2154 18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: Adding: fi: s3a://myclouderaraj/root/bulkInsert.csv 18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: Adding: fi: s3a://myclouderaraj/root/customers 18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: Adding: fi: s3a://myclouderaraj/root/table_test.csv 18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: Adding: fi: s3a://myclouderaraj/root/temp.json Found 4 items -rw-rw-rw- 1 root 224 2018-11-30 02:30 s3a://myclouderaraj/root/bulkInsert.csv -rw-rw-rw- 1 root 15812 2018-11-29 21:57 s3a://myclouderaraj/root/customers -rw-rw-rw- 1 root 0 2018-11-30 02:30 s3a://myclouderaraj/root/table_test.csv -rw-rw-rw- 1 root 2121 2018-11-29 10:32 s3a://myclouderaraj/root/temp.json 18/12/11 19:12:54 TRACE core.TracerPool: TracerPool(Global): removing tracer Tracer(FsShell/10.33.6.146) 18/12/11 19:12:55 DEBUG conn.PoolingClientConnectionManager: Connection manager is shutting down 18/12/11 19:12:55 DEBUG internal.SdkSSLSocket: shutting down output of myclouderaraj.s3.amazonaws.com/52.216.107.100:443 18/12/11 19:12:55 DEBUG internal.SdkSSLSocket: closing myclouderaraj.s3.amazonaws.com/52.216.107.100:443 18/12/11 19:12:55 DEBUG conn.DefaultClientConnection: Connection 0.0.0.0:51025<->52.216.107.100:443 closed 18/12/11 19:12:55 DEBUG conn.DefaultClientConnection: Connection 0.0.0.0:51025<->52.216.107.100:443 closed 18/12/11 19:12:55 DEBUG conn.PoolingClientConnectionManager: Connection manager shut down [root@quickstart ~]# Thanks & Regards, Rajesh
... View more
12-12-2018
05:09 AM
Seth, This is the success log [root@quickstart ~]# hdfs dfs -Dfs.s3a.access.key=myaccesskey -Dfs.s3a.secret.key=mysecretkey -ls s3a://myclouderaraj/root 18/12/11 19:09:45 DEBUG util.Shell: setsid exited with exit code 0 18/12/11 19:10:10 DEBUG conf.Configuration: parsing URL jar:file:/usr/jars/hadoop-common-2.6.0-cdh5.7.0.jar!/core-default.xml 18/12/11 19:10:10 DEBUG conf.Configuration: parsing input stream sun.net.www.protocol.jar.JarURLConnection$JarURLInputStream@ea1a8d5 18/12/11 19:11:30 DEBUG conf.Configuration: parsing URL file:/etc/hadoop/conf.cloudera.yarn/core-site.xml 18/12/11 19:11:31 DEBUG conf.Configuration: parsing input stream java.io.BufferedInputStream@fcd6521 18/12/11 19:12:34 DEBUG core.Tracer: sampler.classes = ; loaded no samplers 18/12/11 19:12:35 TRACE core.TracerId: ProcessID(fmt=%{tname}/%{ip}): computed process ID of "FsShell/10.33.6.146" 18/12/11 19:12:35 TRACE core.TracerPool: TracerPool(Global): adding tracer Tracer(FsShell/10.33.6.146) 18/12/11 19:12:35 DEBUG core.Tracer: span.receiver.classes = ; loaded no span receivers 18/12/11 19:12:35 TRACE core.Tracer: Created Tracer(FsShell/10.33.6.146) for FsShell 18/12/11 19:12:37 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of successful kerberos logins and latency (milliseconds)]) 18/12/11 19:12:37 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of failed kerberos logins and latency (milliseconds)]) 18/12/11 19:12:37 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[GetGroups]) 18/12/11 19:12:37 DEBUG impl.MetricsSystemImpl: UgiMetrics, User and group related metrics Java config name: null Native config name: /etc/krb5.conf Loaded from native config 18/12/11 19:12:38 DEBUG security.Groups: Creating new Groups object 18/12/11 19:12:38 DEBUG security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000; warningDeltaMs=5000 [UnixLoginModule]: succeeded importing info: uid = 0 gid = 0 supp gid = 0 18/12/11 19:12:38 DEBUG security.UserGroupInformation: hadoop login [UnixLoginModule]: added UnixPrincipal, UnixNumericUserPrincipal, UnixNumericGroupPrincipal(s), to Subject 18/12/11 19:12:38 DEBUG security.UserGroupInformation: hadoop login commit 18/12/11 19:12:38 DEBUG security.UserGroupInformation: using local user:UnixPrincipal: root 18/12/11 19:12:38 DEBUG security.UserGroupInformation: Using user: "UnixPrincipal: root" with name root 18/12/11 19:12:38 DEBUG security.UserGroupInformation: User entry: "root" 18/12/11 19:12:38 DEBUG security.UserGroupInformation: UGI loginUser:root (auth:SIMPLE) 18/12/11 19:12:38 DEBUG core.Tracer: sampler.classes = ; loaded no samplers 18/12/11 19:12:38 TRACE core.TracerId: ProcessID(fmt=%{tname}/%{ip}): computed process ID of "FSClient/10.33.6.146" 18/12/11 19:12:38 TRACE core.TracerPool: TracerPool(Global): adding tracer Tracer(FSClient/10.33.6.146) 18/12/11 19:12:38 DEBUG core.Tracer: span.receiver.classes = ; loaded no span receivers 18/12/11 19:12:38 TRACE core.Tracer: Created Tracer(FSClient/10.33.6.146) for FSClient 18/12/11 19:12:40 DEBUG azure.NativeAzureFileSystem: finalize() called. 18/12/11 19:12:40 DEBUG azure.NativeAzureFileSystem: finalize() called. 18/12/11 19:12:41 DEBUG amazonaws.AmazonWebServiceClient: Internal logging succesfully configured to commons logger: true 18/12/11 19:12:44 DEBUG metrics.AwsSdkMetrics: Admin mbean registered under com.cloudera.com.amazonaws.management:type=AwsSdkMetrics 18/12/11 19:12:50 DEBUG auth.AWSCredentialsProviderChain: Loading credentials from BasicAWSCredentialsProvider 18/12/11 19:12:50 DEBUG amazonaws.request: Sending Request: HEAD https://myclouderaraj.s3.amazonaws.com / Headers: (User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192, Content-Type: application/x-www-form-urlencoded; charset=utf-8, ) 18/12/11 19:12:51 DEBUG internal.S3Signer: Calculated string to sign: "HEAD application/x-www-form-urlencoded; charset=utf-8 Wed, 12 Dec 2018 03:12:50 GMT /myclouderaraj/" 18/12/11 19:12:51 DEBUG conn.PoolingClientConnectionManager: Connection request: [route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 0; route allocated: 0 of 15; total allocated: 0 of 15] 18/12/11 19:12:51 DEBUG conn.PoolingClientConnectionManager: Connection leased: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 0; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:51 DEBUG ssl.SdkTLSSocketFactory: socket.getSupportedProtocols(): [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2], socket.getEnabledProtocols(): [TLSv1, TLSv1.1, TLSv1.2] 18/12/11 19:12:51 DEBUG ssl.SdkTLSSocketFactory: TLS protocol enabled for SSL handshake: [TLSv1.2, TLSv1.1, TLSv1] 18/12/11 19:12:51 DEBUG conn.DefaultClientConnectionOperator: Connecting to myclouderaraj.s3.amazonaws.com:443 18/12/11 19:12:51 DEBUG ssl.SdkTLSSocketFactory: connecting to myclouderaraj.s3.amazonaws.com/52.216.107.100:443 18/12/11 19:12:53 DEBUG internal.SdkSSLSocket: created: myclouderaraj.s3.amazonaws.com/52.216.107.100:443 18/12/11 19:12:53 DEBUG protocol.RequestAddCookies: CookieSpec selected: best-match 18/12/11 19:12:53 DEBUG protocol.RequestAuthCache: Auth cache not set in the context 18/12/11 19:12:53 DEBUG protocol.RequestProxyAuthentication: Proxy auth state: UNCHALLENGED 18/12/11 19:12:53 DEBUG client.SdkHttpClient: Attempt 1 to execute request 18/12/11 19:12:53 DEBUG conn.DefaultClientConnection: Sending request: HEAD / HTTP/1.1 18/12/11 19:12:53 DEBUG http.wire: >> "HEAD / HTTP/1.1[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Host: myclouderaraj.s3.amazonaws.com[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:XU3AFfGphmQy0rEhRrWjajyeFbA=[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Date: Wed, 12 Dec 2018 03:12:50 GMT[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Content-Type: application/x-www-form-urlencoded; charset=utf-8[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Connection: Keep-Alive[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "[\r][\n]" 18/12/11 19:12:53 DEBUG http.headers: >> HEAD / HTTP/1.1 18/12/11 19:12:53 DEBUG http.headers: >> Host: myclouderaraj.s3.amazonaws.com 18/12/11 19:12:53 DEBUG http.headers: >> Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:XU3AFfGphmQy0rEhRrWjajyeFbA= 18/12/11 19:12:53 DEBUG http.headers: >> User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192 18/12/11 19:12:53 DEBUG http.headers: >> Date: Wed, 12 Dec 2018 03:12:50 GMT 18/12/11 19:12:53 DEBUG http.headers: >> Content-Type: application/x-www-form-urlencoded; charset=utf-8 18/12/11 19:12:53 DEBUG http.headers: >> Connection: Keep-Alive 18/12/11 19:12:53 DEBUG http.wire: << "HTTP/1.1 200 OK[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "x-amz-id-2: sjybfZSJELaf9eW/QJHWV/FcObyfD5ElzagJj8SqLiPwVogmvp1lih+csHvi73bNoHKuS/OLqSI=[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "x-amz-request-id: 03AD0786BC412E7C[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "Date: Wed, 12 Dec 2018 03:12:54 GMT[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "x-amz-bucket-region: us-east-1[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "Content-Type: application/xml[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "Transfer-Encoding: chunked[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "Server: AmazonS3[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:53 DEBUG conn.DefaultClientConnection: Receiving response: HTTP/1.1 200 OK 18/12/11 19:12:53 DEBUG http.headers: << HTTP/1.1 200 OK 18/12/11 19:12:53 DEBUG http.headers: << x-amz-id-2: sjybfZSJELaf9eW/QJHWV/FcObyfD5ElzagJj8SqLiPwVogmvp1lih+csHvi73bNoHKuS/OLqSI= 18/12/11 19:12:53 DEBUG http.headers: << x-amz-request-id: 03AD0786BC412E7C 18/12/11 19:12:53 DEBUG http.headers: << Date: Wed, 12 Dec 2018 03:12:54 GMT 18/12/11 19:12:53 DEBUG http.headers: << x-amz-bucket-region: us-east-1 18/12/11 19:12:53 DEBUG http.headers: << Content-Type: application/xml 18/12/11 19:12:53 DEBUG http.headers: << Transfer-Encoding: chunked 18/12/11 19:12:53 DEBUG http.headers: << Server: AmazonS3 18/12/11 19:12:53 DEBUG client.SdkHttpClient: Connection can be kept alive indefinitely 18/12/11 19:12:53 DEBUG conn.PoolingClientConnectionManager: Connection [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com] can be kept alive indefinitely 18/12/11 19:12:53 DEBUG conn.PoolingClientConnectionManager: Connection released: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:53 DEBUG amazonaws.requestId: x-amzn-RequestId: not available 18/12/11 19:12:53 DEBUG amazonaws.request: Received successful response: 200, AWS Request ID: 03AD0786BC412E7C 18/12/11 19:12:53 DEBUG amazonaws.requestId: AWS Request ID: 03AD0786BC412E7C 18/12/11 19:12:53 DEBUG s3a.S3AFileSystem: Getting path status for s3a://myclouderaraj/root (root) 18/12/11 19:12:53 DEBUG amazonaws.request: Sending Request: HEAD https://myclouderaraj.s3.amazonaws.com /root Headers: (User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192, Content-Type: application/x-www-form-urlencoded; charset=utf-8, ) 18/12/11 19:12:53 DEBUG internal.S3Signer: Calculated string to sign: "HEAD application/x-www-form-urlencoded; charset=utf-8 Wed, 12 Dec 2018 03:12:53 GMT /myclouderaraj/root" 18/12/11 19:12:53 DEBUG conn.PoolingClientConnectionManager: Connection request: [route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:53 DEBUG conn.PoolingClientConnectionManager: Connection leased: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 0; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:53 DEBUG client.SdkHttpClient: Stale connection check 18/12/11 19:12:53 DEBUG protocol.RequestAddCookies: CookieSpec selected: best-match 18/12/11 19:12:53 DEBUG protocol.RequestAuthCache: Auth cache not set in the context 18/12/11 19:12:53 DEBUG protocol.RequestProxyAuthentication: Proxy auth state: UNCHALLENGED 18/12/11 19:12:53 DEBUG client.SdkHttpClient: Attempt 1 to execute request 18/12/11 19:12:53 DEBUG conn.DefaultClientConnection: Sending request: HEAD /root HTTP/1.1 18/12/11 19:12:53 DEBUG http.wire: >> "HEAD /root HTTP/1.1[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Host: myclouderaraj.s3.amazonaws.com[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:SLVFDzdTaXiUPcomkZCGPdiXP+I=[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Date: Wed, 12 Dec 2018 03:12:53 GMT[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Content-Type: application/x-www-form-urlencoded; charset=utf-8[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Connection: Keep-Alive[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "[\r][\n]" 18/12/11 19:12:53 DEBUG http.headers: >> HEAD /root HTTP/1.1 18/12/11 19:12:53 DEBUG http.headers: >> Host: myclouderaraj.s3.amazonaws.com 18/12/11 19:12:53 DEBUG http.headers: >> Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:SLVFDzdTaXiUPcomkZCGPdiXP+I= 18/12/11 19:12:53 DEBUG http.headers: >> User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192 18/12/11 19:12:53 DEBUG http.headers: >> Date: Wed, 12 Dec 2018 03:12:53 GMT 18/12/11 19:12:53 DEBUG http.headers: >> Content-Type: application/x-www-form-urlencoded; charset=utf-8 18/12/11 19:12:53 DEBUG http.headers: >> Connection: Keep-Alive 18/12/11 19:12:53 DEBUG http.wire: << "HTTP/1.1 404 Not Found[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "x-amz-request-id: 562A66F35BB7E4A1[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "x-amz-id-2: FC+MjpOY8O1AM4iHk4leLkFYxsqUqsOGka5QBXub98ve+SirWRY2PXH8778eWyv9UUFbFwVn0LQ=[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "Content-Type: application/xml[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "Transfer-Encoding: chunked[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "Date: Wed, 12 Dec 2018 03:12:53 GMT[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "Server: AmazonS3[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:53 DEBUG conn.DefaultClientConnection: Receiving response: HTTP/1.1 404 Not Found 18/12/11 19:12:53 DEBUG http.headers: << HTTP/1.1 404 Not Found 18/12/11 19:12:53 DEBUG http.headers: << x-amz-request-id: 562A66F35BB7E4A1 18/12/11 19:12:53 DEBUG http.headers: << x-amz-id-2: FC+MjpOY8O1AM4iHk4leLkFYxsqUqsOGka5QBXub98ve+SirWRY2PXH8778eWyv9UUFbFwVn0LQ= 18/12/11 19:12:53 DEBUG http.headers: << Content-Type: application/xml 18/12/11 19:12:53 DEBUG http.headers: << Transfer-Encoding: chunked 18/12/11 19:12:53 DEBUG http.headers: << Date: Wed, 12 Dec 2018 03:12:53 GMT 18/12/11 19:12:53 DEBUG http.headers: << Server: AmazonS3 18/12/11 19:12:53 DEBUG client.SdkHttpClient: Connection can be kept alive indefinitely 18/12/11 19:12:53 DEBUG conn.PoolingClientConnectionManager: Connection [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com] can be kept alive indefinitely 18/12/11 19:12:53 DEBUG conn.PoolingClientConnectionManager: Connection released: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:53 DEBUG amazonaws.requestId: x-amzn-RequestId: not available 18/12/11 19:12:53 DEBUG amazonaws.request: Received error response: com.cloudera.com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: null; Status Code: 404; Error Code: 404 Not Found; Request ID: 562A66F35BB7E4A1), S3 Extended Request ID: FC+MjpOY8O1AM4iHk4leLkFYxsqUqsOGka5QBXub98ve+SirWRY2PXH8778eWyv9UUFbFwVn0LQ= 18/12/11 19:12:53 DEBUG amazonaws.request: Sending Request: HEAD https://myclouderaraj.s3.amazonaws.com /root/ Headers: (User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192, Content-Type: application/x-www-form-urlencoded; charset=utf-8, ) 18/12/11 19:12:53 DEBUG internal.S3Signer: Calculated string to sign: "HEAD application/x-www-form-urlencoded; charset=utf-8 Wed, 12 Dec 2018 03:12:53 GMT /myclouderaraj/root/" 18/12/11 19:12:53 DEBUG conn.PoolingClientConnectionManager: Connection request: [route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:53 DEBUG conn.PoolingClientConnectionManager: Connection leased: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 0; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:53 DEBUG client.SdkHttpClient: Stale connection check 18/12/11 19:12:53 DEBUG protocol.RequestAddCookies: CookieSpec selected: best-match 18/12/11 19:12:53 DEBUG protocol.RequestAuthCache: Auth cache not set in the context 18/12/11 19:12:53 DEBUG protocol.RequestProxyAuthentication: Proxy auth state: UNCHALLENGED 18/12/11 19:12:53 DEBUG client.SdkHttpClient: Attempt 1 to execute request 18/12/11 19:12:53 DEBUG conn.DefaultClientConnection: Sending request: HEAD /root/ HTTP/1.1 18/12/11 19:12:53 DEBUG http.wire: >> "HEAD /root/ HTTP/1.1[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Host: myclouderaraj.s3.amazonaws.com[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:nWk9yWYYe8oEEGtfxRoqQmfgSK0=[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Date: Wed, 12 Dec 2018 03:12:53 GMT[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Content-Type: application/x-www-form-urlencoded; charset=utf-8[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Connection: Keep-Alive[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "[\r][\n]" 18/12/11 19:12:53 DEBUG http.headers: >> HEAD /root/ HTTP/1.1 18/12/11 19:12:53 DEBUG http.headers: >> Host: myclouderaraj.s3.amazonaws.com 18/12/11 19:12:53 DEBUG http.headers: >> Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:nWk9yWYYe8oEEGtfxRoqQmfgSK0= 18/12/11 19:12:53 DEBUG http.headers: >> User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192 18/12/11 19:12:53 DEBUG http.headers: >> Date: Wed, 12 Dec 2018 03:12:53 GMT 18/12/11 19:12:53 DEBUG http.headers: >> Content-Type: application/x-www-form-urlencoded; charset=utf-8 18/12/11 19:12:53 DEBUG http.headers: >> Connection: Keep-Alive 18/12/11 19:12:53 DEBUG http.wire: << "HTTP/1.1 404 Not Found[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "x-amz-request-id: C5C8989A67B509AC[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "x-amz-id-2: 19gaP2XtwSoU9Bl1Y+jO3A8rPAr+v00XbdZQDD08v6u3XJKiB3uRfLSF+E4vvivzeiwCK74Vy+I=[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "Content-Type: application/xml[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "Transfer-Encoding: chunked[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "Date: Wed, 12 Dec 2018 03:12:53 GMT[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "Server: AmazonS3[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:53 DEBUG conn.DefaultClientConnection: Receiving response: HTTP/1.1 404 Not Found 18/12/11 19:12:53 DEBUG http.headers: << HTTP/1.1 404 Not Found 18/12/11 19:12:53 DEBUG http.headers: << x-amz-request-id: C5C8989A67B509AC 18/12/11 19:12:53 DEBUG http.headers: << x-amz-id-2: 19gaP2XtwSoU9Bl1Y+jO3A8rPAr+v00XbdZQDD08v6u3XJKiB3uRfLSF+E4vvivzeiwCK74Vy+I= 18/12/11 19:12:53 DEBUG http.headers: << Content-Type: application/xml 18/12/11 19:12:53 DEBUG http.headers: << Transfer-Encoding: chunked 18/12/11 19:12:53 DEBUG http.headers: << Date: Wed, 12 Dec 2018 03:12:53 GMT 18/12/11 19:12:53 DEBUG http.headers: << Server: AmazonS3 18/12/11 19:12:53 DEBUG client.SdkHttpClient: Connection can be kept alive indefinitely 18/12/11 19:12:53 DEBUG conn.PoolingClientConnectionManager: Connection [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com] can be kept alive indefinitely 18/12/11 19:12:53 DEBUG conn.PoolingClientConnectionManager: Connection released: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:53 DEBUG amazonaws.requestId: x-amzn-RequestId: not available 18/12/11 19:12:53 DEBUG amazonaws.request: Received error response: com.cloudera.com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: null; Status Code: 404; Error Code: 404 Not Found; Request ID: C5C8989A67B509AC), S3 Extended Request ID: 19gaP2XtwSoU9Bl1Y+jO3A8rPAr+v00XbdZQDD08v6u3XJKiB3uRfLSF+E4vvivzeiwCK74Vy+I= 18/12/11 19:12:53 DEBUG amazonaws.request: Sending Request: GET https://myclouderaraj.s3.amazonaws.com / Parameters: ({"prefix":["root/"],"delimiter":["/"],"max-keys":["1"]}Headers: (User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192, Content-Type: application/x-www-form-urlencoded; charset=utf-8, ) 18/12/11 19:12:53 DEBUG internal.S3Signer: Calculated string to sign: "GET application/x-www-form-urlencoded; charset=utf-8 Wed, 12 Dec 2018 03:12:53 GMT /myclouderaraj/" 18/12/11 19:12:53 DEBUG conn.PoolingClientConnectionManager: Connection request: [route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:53 DEBUG conn.PoolingClientConnectionManager: Connection leased: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 0; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:53 DEBUG client.SdkHttpClient: Stale connection check 18/12/11 19:12:53 DEBUG protocol.RequestAddCookies: CookieSpec selected: best-match 18/12/11 19:12:53 DEBUG protocol.RequestAuthCache: Auth cache not set in the context 18/12/11 19:12:53 DEBUG protocol.RequestProxyAuthentication: Proxy auth state: UNCHALLENGED 18/12/11 19:12:53 DEBUG client.SdkHttpClient: Attempt 1 to execute request 18/12/11 19:12:53 DEBUG conn.DefaultClientConnection: Sending request: GET /?prefix=root%2F&delimiter=%2F&max-keys=1 HTTP/1.1 18/12/11 19:12:53 DEBUG http.wire: >> "GET /?prefix=root%2F&delimiter=%2F&max-keys=1 HTTP/1.1[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Host: myclouderaraj.s3.amazonaws.com[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:VEhMOYpTlsvVdG9Q0F7ehtOuxyg=[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Date: Wed, 12 Dec 2018 03:12:53 GMT[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Content-Type: application/x-www-form-urlencoded; charset=utf-8[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Connection: Keep-Alive[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "[\r][\n]" 18/12/11 19:12:53 DEBUG http.headers: >> GET /?prefix=root%2F&delimiter=%2F&max-keys=1 HTTP/1.1 18/12/11 19:12:53 DEBUG http.headers: >> Host: myclouderaraj.s3.amazonaws.com 18/12/11 19:12:53 DEBUG http.headers: >> Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:VEhMOYpTlsvVdG9Q0F7ehtOuxyg= 18/12/11 19:12:53 DEBUG http.headers: >> User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192 18/12/11 19:12:53 DEBUG http.headers: >> Date: Wed, 12 Dec 2018 03:12:53 GMT 18/12/11 19:12:53 DEBUG http.headers: >> Content-Type: application/x-www-form-urlencoded; charset=utf-8 18/12/11 19:12:53 DEBUG http.headers: >> Connection: Keep-Alive 18/12/11 19:12:54 DEBUG http.wire: << "HTTP/1.1 200 OK[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "x-amz-id-2: fcNfT1n+Dhn7lPqVGiKhrTq36arxsAAG1QNTFjpiJ+O8vSqBUumwNFw7Nmv6onHdi4Y0t0wmYGE=[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "x-amz-request-id: 741C65B73A0E800F[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Date: Wed, 12 Dec 2018 03:12:54 GMT[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "x-amz-bucket-region: us-east-1[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Content-Type: application/xml[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Transfer-Encoding: chunked[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Server: AmazonS3[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:54 DEBUG conn.DefaultClientConnection: Receiving response: HTTP/1.1 200 OK 18/12/11 19:12:54 DEBUG http.headers: << HTTP/1.1 200 OK 18/12/11 19:12:54 DEBUG http.headers: << x-amz-id-2: fcNfT1n+Dhn7lPqVGiKhrTq36arxsAAG1QNTFjpiJ+O8vSqBUumwNFw7Nmv6onHdi4Y0t0wmYGE= 18/12/11 19:12:54 DEBUG http.headers: << x-amz-request-id: 741C65B73A0E800F 18/12/11 19:12:54 DEBUG http.headers: << Date: Wed, 12 Dec 2018 03:12:54 GMT 18/12/11 19:12:54 DEBUG http.headers: << x-amz-bucket-region: us-east-1 18/12/11 19:12:54 DEBUG http.headers: << Content-Type: application/xml 18/12/11 19:12:54 DEBUG http.headers: << Transfer-Encoding: chunked 18/12/11 19:12:54 DEBUG http.headers: << Server: AmazonS3 18/12/11 19:12:54 DEBUG client.SdkHttpClient: Connection can be kept alive indefinitely 18/12/11 19:12:54 DEBUG amazonaws.requestId: x-amzn-RequestId: not available 18/12/11 19:12:54 TRACE amazonaws.request: Beginning to parse service response XML 18/12/11 19:12:54 DEBUG transform.XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.cloudera.com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler 18/12/11 19:12:54 DEBUG http.wire: << "286[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "<?xml version="1.0" encoding="UTF-8"?>[\n]" 18/12/11 19:12:54 DEBUG http.wire: << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>myclouderaraj</Name><Prefix>root/</Prefix><Marker></Marker><NextMarker>root/bulkInsert.csv</NextMarker><MaxKeys>1</MaxKeys><Delimiter>/</Delimiter><IsTruncated>true</IsTruncated><Contents><Key>root/bulkInsert.csv</Key><LastModified>2018-11-30T10:30:01.000Z</LastModified><ETag>"963e1e40ac0247f23ebf138755ceabfb"</ETag><Size>224</Size><Owner><ID>fdbe33e179958960b9d1fab4b547c1b8ccf7a9ed1b17b3ecc20f39d0866957ab</ID><DisplayName>rajesh210</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents></ListBucketResult>" 18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "0[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com] can be kept alive indefinitely 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection released: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:54 DEBUG transform.XmlResponsesSaxParser: Parsing XML response document with handler: class com.cloudera.com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler 18/12/11 19:12:54 DEBUG transform.XmlResponsesSaxParser: Examining listing for bucket: myclouderaraj 18/12/11 19:12:54 TRACE amazonaws.request: Done parsing service response XML 18/12/11 19:12:54 DEBUG amazonaws.request: Received successful response: 200, AWS Request ID: 741C65B73A0E800F 18/12/11 19:12:54 DEBUG amazonaws.requestId: AWS Request ID: 741C65B73A0E800F 18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: Found path as directory (with /): 0/1 18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: Summary: root/bulkInsert.csv 224 18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: List status for path: s3a://myclouderaraj/root 18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: Getting path status for s3a://myclouderaraj/root (root) 18/12/11 19:12:54 DEBUG amazonaws.request: Sending Request: HEAD https://myclouderaraj.s3.amazonaws.com /root Headers: (User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192, Content-Type: application/x-www-form-urlencoded; charset=utf-8, ) 18/12/11 19:12:54 DEBUG internal.S3Signer: Calculated string to sign: "HEAD application/x-www-form-urlencoded; charset=utf-8 Wed, 12 Dec 2018 03:12:54 GMT /myclouderaraj/root" 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection request: [route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection leased: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 0; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:54 DEBUG client.SdkHttpClient: Stale connection check 18/12/11 19:12:54 DEBUG protocol.RequestAddCookies: CookieSpec selected: best-match 18/12/11 19:12:54 DEBUG protocol.RequestAuthCache: Auth cache not set in the context 18/12/11 19:12:54 DEBUG protocol.RequestProxyAuthentication: Proxy auth state: UNCHALLENGED 18/12/11 19:12:54 DEBUG client.SdkHttpClient: Attempt 1 to execute request 18/12/11 19:12:54 DEBUG conn.DefaultClientConnection: Sending request: HEAD /root HTTP/1.1 18/12/11 19:12:54 DEBUG http.wire: >> "HEAD /root HTTP/1.1[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Host: myclouderaraj.s3.amazonaws.com[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:/G0a8NrUHFBGmNvT/YV5nkFRbKU=[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Date: Wed, 12 Dec 2018 03:12:54 GMT[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Content-Type: application/x-www-form-urlencoded; charset=utf-8[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Connection: Keep-Alive[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "[\r][\n]" 18/12/11 19:12:54 DEBUG http.headers: >> HEAD /root HTTP/1.1 18/12/11 19:12:54 DEBUG http.headers: >> Host: myclouderaraj.s3.amazonaws.com 18/12/11 19:12:54 DEBUG http.headers: >> Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:/G0a8NrUHFBGmNvT/YV5nkFRbKU= 18/12/11 19:12:54 DEBUG http.headers: >> User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192 18/12/11 19:12:54 DEBUG http.headers: >> Date: Wed, 12 Dec 2018 03:12:54 GMT 18/12/11 19:12:54 DEBUG http.headers: >> Content-Type: application/x-www-form-urlencoded; charset=utf-8 18/12/11 19:12:54 DEBUG http.headers: >> Connection: Keep-Alive 18/12/11 19:12:54 DEBUG http.wire: << "HTTP/1.1 404 Not Found[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "x-amz-request-id: 48EB44080F918683[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "x-amz-id-2: MLddIb2VuN1THhnjZI4diJfaB37g0zGNOzIFrEIfgOCMYeJBJEb2xV16+VQsv/2Z2kGhoQyhLhM=[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Content-Type: application/xml[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Transfer-Encoding: chunked[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Date: Wed, 12 Dec 2018 03:12:54 GMT[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Server: AmazonS3[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:54 DEBUG conn.DefaultClientConnection: Receiving response: HTTP/1.1 404 Not Found 18/12/11 19:12:54 DEBUG http.headers: << HTTP/1.1 404 Not Found 18/12/11 19:12:54 DEBUG http.headers: << x-amz-request-id: 48EB44080F918683 18/12/11 19:12:54 DEBUG http.headers: << x-amz-id-2: MLddIb2VuN1THhnjZI4diJfaB37g0zGNOzIFrEIfgOCMYeJBJEb2xV16+VQsv/2Z2kGhoQyhLhM= 18/12/11 19:12:54 DEBUG http.headers: << Content-Type: application/xml 18/12/11 19:12:54 DEBUG http.headers: << Transfer-Encoding: chunked 18/12/11 19:12:54 DEBUG http.headers: << Date: Wed, 12 Dec 2018 03:12:54 GMT 18/12/11 19:12:54 DEBUG http.headers: << Server: AmazonS3 18/12/11 19:12:54 DEBUG client.SdkHttpClient: Connection can be kept alive indefinitely 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com] can be kept alive indefinitely 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection released: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:54 DEBUG amazonaws.requestId: x-amzn-RequestId: not available 18/12/11 19:12:54 DEBUG amazonaws.request: Received error response: com.cloudera.com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: null; Status Code: 404; Error Code: 404 Not Found; Request ID: 48EB44080F918683), S3 Extended Request ID: MLddIb2VuN1THhnjZI4diJfaB37g0zGNOzIFrEIfgOCMYeJBJEb2xV16+VQsv/2Z2kGhoQyhLhM= 18/12/11 19:12:54 DEBUG amazonaws.request: Sending Request: HEAD https://myclouderaraj.s3.amazonaws.com /root/ Headers: (User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192, Content-Type: application/x-www-form-urlencoded; charset=utf-8, ) 18/12/11 19:12:54 DEBUG internal.S3Signer: Calculated string to sign: "HEAD application/x-www-form-urlencoded; charset=utf-8 Wed, 12 Dec 2018 03:12:54 GMT /myclouderaraj/root/" 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection request: [route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection leased: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 0; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:54 DEBUG client.SdkHttpClient: Stale connection check 18/12/11 19:12:54 DEBUG protocol.RequestAddCookies: CookieSpec selected: best-match 18/12/11 19:12:54 DEBUG protocol.RequestAuthCache: Auth cache not set in the context 18/12/11 19:12:54 DEBUG protocol.RequestProxyAuthentication: Proxy auth state: UNCHALLENGED 18/12/11 19:12:54 DEBUG client.SdkHttpClient: Attempt 1 to execute request 18/12/11 19:12:54 DEBUG conn.DefaultClientConnection: Sending request: HEAD /root/ HTTP/1.1 18/12/11 19:12:54 DEBUG http.wire: >> "HEAD /root/ HTTP/1.1[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Host: myclouderaraj.s3.amazonaws.com[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:bAUfpGUWpSAqbkk2vygScGZO7Qw=[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Date: Wed, 12 Dec 2018 03:12:54 GMT[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Content-Type: application/x-www-form-urlencoded; charset=utf-8[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Connection: Keep-Alive[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "[\r][\n]" 18/12/11 19:12:54 DEBUG http.headers: >> HEAD /root/ HTTP/1.1 18/12/11 19:12:54 DEBUG http.headers: >> Host: myclouderaraj.s3.amazonaws.com 18/12/11 19:12:54 DEBUG http.headers: >> Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:bAUfpGUWpSAqbkk2vygScGZO7Qw= 18/12/11 19:12:54 DEBUG http.headers: >> User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192 18/12/11 19:12:54 DEBUG http.headers: >> Date: Wed, 12 Dec 2018 03:12:54 GMT 18/12/11 19:12:54 DEBUG http.headers: >> Content-Type: application/x-www-form-urlencoded; charset=utf-8 18/12/11 19:12:54 DEBUG http.headers: >> Connection: Keep-Alive 18/12/11 19:12:54 DEBUG http.wire: << "HTTP/1.1 404 Not Found[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "x-amz-request-id: D79FFCC520EEA4D7[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "x-amz-id-2: TZC+frfDHCmwLqED0QD8Cg/lABBzKNkVwRIto+wmg3IVMm6+9KGRE87O3cr/jpQ8LFM3YvfaUuI=[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Content-Type: application/xml[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Transfer-Encoding: chunked[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Date: Wed, 12 Dec 2018 03:12:54 GMT[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Server: AmazonS3[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:54 DEBUG conn.DefaultClientConnection: Receiving response: HTTP/1.1 404 Not Found 18/12/11 19:12:54 DEBUG http.headers: << HTTP/1.1 404 Not Found 18/12/11 19:12:54 DEBUG http.headers: << x-amz-request-id: D79FFCC520EEA4D7 18/12/11 19:12:54 DEBUG http.headers: << x-amz-id-2: TZC+frfDHCmwLqED0QD8Cg/lABBzKNkVwRIto+wmg3IVMm6+9KGRE87O3cr/jpQ8LFM3YvfaUuI= 18/12/11 19:12:54 DEBUG http.headers: << Content-Type: application/xml 18/12/11 19:12:54 DEBUG http.headers: << Transfer-Encoding: chunked 18/12/11 19:12:54 DEBUG http.headers: << Date: Wed, 12 Dec 2018 03:12:54 GMT 18/12/11 19:12:54 DEBUG http.headers: << Server: AmazonS3 18/12/11 19:12:54 DEBUG client.SdkHttpClient: Connection can be kept alive indefinitely 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com] can be kept alive indefinitely 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection released: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:54 DEBUG amazonaws.requestId: x-amzn-RequestId: not available 18/12/11 19:12:54 DEBUG amazonaws.request: Received error response: com.cloudera.com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: null; Status Code: 404; Error Code: 404 Not Found; Request ID: D79FFCC520EEA4D7), S3 Extended Request ID: TZC+frfDHCmwLqED0QD8Cg/lABBzKNkVwRIto+wmg3IVMm6+9KGRE87O3cr/jpQ8LFM3YvfaUuI= 18/12/11 19:12:54 DEBUG amazonaws.request: Sending Request: GET https://myclouderaraj.s3.amazonaws.com / Parameters: ({"prefix":["root/"],"delimiter":["/"],"max-keys":["1"]}Headers: (User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192, Content-Type: application/x-www-form-urlencoded; charset=utf-8, ) 18/12/11 19:12:54 DEBUG internal.S3Signer: Calculated string to sign: "GET application/x-www-form-urlencoded; charset=utf-8 Wed, 12 Dec 2018 03:12:54 GMT /myclouderaraj/" 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection request: [route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection leased: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 0; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:54 DEBUG client.SdkHttpClient: Stale connection check 18/12/11 19:12:54 DEBUG protocol.RequestAddCookies: CookieSpec selected: best-match 18/12/11 19:12:54 DEBUG protocol.RequestAuthCache: Auth cache not set in the context 18/12/11 19:12:54 DEBUG protocol.RequestProxyAuthentication: Proxy auth state: UNCHALLENGED 18/12/11 19:12:54 DEBUG client.SdkHttpClient: Attempt 1 to execute request 18/12/11 19:12:54 DEBUG conn.DefaultClientConnection: Sending request: GET /?prefix=root%2F&delimiter=%2F&max-keys=1 HTTP/1.1 18/12/11 19:12:54 DEBUG http.wire: >> "GET /?prefix=root%2F&delimiter=%2F&max-keys=1 HTTP/1.1[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Host: myclouderaraj.s3.amazonaws.com[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:xhHdbA514F67IgOMPolX00Rv1h4=[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Date: Wed, 12 Dec 2018 03:12:54 GMT[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Content-Type: application/x-www-form-urlencoded; charset=utf-8[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Connection: Keep-Alive[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "[\r][\n]" 18/12/11 19:12:54 DEBUG http.headers: >> GET /?prefix=root%2F&delimiter=%2F&max-keys=1 HTTP/1.1 18/12/11 19:12:54 DEBUG http.headers: >> Host: myclouderaraj.s3.amazonaws.com 18/12/11 19:12:54 DEBUG http.headers: >> Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:xhHdbA514F67IgOMPolX00Rv1h4= 18/12/11 19:12:54 DEBUG http.headers: >> User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192 18/12/11 19:12:54 DEBUG http.headers: >> Date: Wed, 12 Dec 2018 03:12:54 GMT 18/12/11 19:12:54 DEBUG http.headers: >> Content-Type: application/x-www-form-urlencoded; charset=utf-8 18/12/11 19:12:54 DEBUG http.headers: >> Connection: Keep-Alive 18/12/11 19:12:54 DEBUG http.wire: << "HTTP/1.1 200 OK[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "x-amz-id-2: n67wc46WhWXcw4ygKZ0I2NEYJXqmyEG2sDyv8+hKbZaGCflTI6YB4gvIpuqQgHb+LCSgCQ33MlU=[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "x-amz-request-id: BE60D884EB5E4D60[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Date: Wed, 12 Dec 2018 03:12:55 GMT[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "x-amz-bucket-region: us-east-1[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Content-Type: application/xml[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Transfer-Encoding: chunked[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Server: AmazonS3[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:54 DEBUG conn.DefaultClientConnection: Receiving response: HTTP/1.1 200 OK 18/12/11 19:12:54 DEBUG http.headers: << HTTP/1.1 200 OK 18/12/11 19:12:54 DEBUG http.headers: << x-amz-id-2: n67wc46WhWXcw4ygKZ0I2NEYJXqmyEG2sDyv8+hKbZaGCflTI6YB4gvIpuqQgHb+LCSgCQ33MlU= 18/12/11 19:12:54 DEBUG http.headers: << x-amz-request-id: BE60D884EB5E4D60 18/12/11 19:12:54 DEBUG http.headers: << Date: Wed, 12 Dec 2018 03:12:55 GMT 18/12/11 19:12:54 DEBUG http.headers: << x-amz-bucket-region: us-east-1 18/12/11 19:12:54 DEBUG http.headers: << Content-Type: application/xml 18/12/11 19:12:54 DEBUG http.headers: << Transfer-Encoding: chunked 18/12/11 19:12:54 DEBUG http.headers: << Server: AmazonS3 18/12/11 19:12:54 DEBUG client.SdkHttpClient: Connection can be kept alive indefinitely 18/12/11 19:12:54 DEBUG amazonaws.requestId: x-amzn-RequestId: not available 18/12/11 19:12:54 TRACE amazonaws.request: Beginning to parse service response XML 18/12/11 19:12:54 DEBUG transform.XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.cloudera.com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler 18/12/11 19:12:54 DEBUG http.wire: << "286[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "<?xml version="1.0" encoding="UTF-8"?>[\n]" 18/12/11 19:12:54 DEBUG http.wire: << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>myclouderaraj</Name><Prefix>root/</Prefix><Marker></Marker><NextMarker>root/bulkInsert.csv</NextMarker><MaxKeys>1</MaxKeys><Delimiter>/</Delimiter><IsTruncated>true</IsTruncated><Contents><Key>root/bulkInsert.csv</Key><LastModified>2018-11-30T10:30:01.000Z</LastModified><ETag>"963e1e40ac0247f23ebf138755ceabfb"</ETag><Size>224</Size><Owner><ID>fdbe33e179958960b9d1fab4b547c1b8ccf7a9ed1b17b3ecc20f39d0866957ab</ID><DisplayName>rajesh210</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents></ListBucketResult>" 18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "0[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com] can be kept alive indefinitely 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection released: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:54 DEBUG transform.XmlResponsesSaxParser: Parsing XML response document with handler: class com.cloudera.com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler 18/12/11 19:12:54 DEBUG transform.XmlResponsesSaxParser: Examining listing for bucket: myclouderaraj 18/12/11 19:12:54 TRACE amazonaws.request: Done parsing service response XML 18/12/11 19:12:54 DEBUG amazonaws.request: Received successful response: 200, AWS Request ID: BE60D884EB5E4D60 18/12/11 19:12:54 DEBUG amazonaws.requestId: AWS Request ID: BE60D884EB5E4D60 18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: Found path as directory (with /): 0/1 18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: Summary: root/bulkInsert.csv 224 18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: listStatus: doing listObjects for directory root/ 18/12/11 19:12:54 DEBUG amazonaws.request: Sending Request: GET https://myclouderaraj.s3.amazonaws.com / Parameters: ({"prefix":["root/"],"delimiter":["/"],"max-keys":["5000"]}Headers: (User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192, Content-Type: application/x-www-form-urlencoded; charset=utf-8, ) 18/12/11 19:12:54 DEBUG internal.S3Signer: Calculated string to sign: "GET application/x-www-form-urlencoded; charset=utf-8 Wed, 12 Dec 2018 03:12:54 GMT /myclouderaraj/" 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection request: [route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection leased: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 0; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:54 DEBUG client.SdkHttpClient: Stale connection check 18/12/11 19:12:54 DEBUG protocol.RequestAddCookies: CookieSpec selected: best-match 18/12/11 19:12:54 DEBUG protocol.RequestAuthCache: Auth cache not set in the context 18/12/11 19:12:54 DEBUG protocol.RequestProxyAuthentication: Proxy auth state: UNCHALLENGED 18/12/11 19:12:54 DEBUG client.SdkHttpClient: Attempt 1 to execute request 18/12/11 19:12:54 DEBUG conn.DefaultClientConnection: Sending request: GET /?prefix=root%2F&delimiter=%2F&max-keys=5000 HTTP/1.1 18/12/11 19:12:54 DEBUG http.wire: >> "GET /?prefix=root%2F&delimiter=%2F&max-keys=5000 HTTP/1.1[\r][\n]"
... View more
12-11-2018
09:24 AM
Hi Seth, I did as you mentioned and the logs are as memtioned below [root@quickstart ~]# hdfs dfs -Dfs.s3a.access.key=myaccesskey -Dfs.s3a.secret.key=mysecretkey -ls s3a://myclouderaraj/root
18/12/11 17:45:55 DEBUG util.Shell: setsid exited with exit code 0
18/12/11 17:45:55 DEBUG conf.Configuration: parsing URL jar:file:/usr/lib/hadoop/hadoop-common-2.6.0-cdh5.13.0.jar!/core-default.xml
18/12/11 17:45:55 DEBUG conf.Configuration: parsing input stream sun.net.www.protocol.jar.JarURLConnection$JarURLInputStream@4232c52b
18/12/11 17:45:55 DEBUG conf.Configuration: parsing URL file:/etc/hadoop/conf.cloudera.yarn/core-site.xml
18/12/11 17:45:55 DEBUG conf.Configuration: parsing input stream java.io.BufferedInputStream@4671e53b
18/12/11 17:45:56 DEBUG core.Tracer: sampler.classes = ; loaded no samplers
18/12/11 17:45:56 TRACE core.TracerId: ProcessID(fmt=%{tname}/%{ip}): computed process ID of "FsShell/10.33.6.155"
18/12/11 17:45:56 TRACE core.TracerPool: TracerPool(Global): adding tracer Tracer(FsShell/10.33.6.155)
18/12/11 17:45:56 DEBUG core.Tracer: span.receiver.classes = ; loaded no span receivers
18/12/11 17:45:56 TRACE core.Tracer: Created Tracer(FsShell/10.33.6.155) for FsShell
18/12/11 17:45:56 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of successful kerberos logins and latency (milliseconds)])
18/12/11 17:45:56 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of failed kerberos logins and latency (milliseconds)])
18/12/11 17:45:56 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[GetGroups])
18/12/11 17:45:56 DEBUG lib.MutableMetricsFactory: field private org.apache.hadoop.metrics2.lib.MutableGaugeLong org.apache.hadoop.security.UserGroupInformation$UgiMetrics.renewalFailuresTotal with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Renewal failures since startup])
18/12/11 17:45:56 DEBUG lib.MutableMetricsFactory: field private org.apache.hadoop.metrics2.lib.MutableGaugeInt org.apache.hadoop.security.UserGroupInformation$UgiMetrics.renewalFailures with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Renewal failures since last successful login])
18/12/11 17:45:56 DEBUG impl.MetricsSystemImpl: UgiMetrics, User and group related metrics
18/12/11 17:45:56 DEBUG security.SecurityUtil: Setting hadoop.security.token.service.use_ip to true
Java config name: null
Native config name: /etc/krb5.conf
Loaded from native config
18/12/11 17:45:56 DEBUG security.Groups: Creating new Groups object
18/12/11 17:45:57 DEBUG security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000; warningDeltaMs=5000
[UnixLoginModule]: succeeded importing info:
uid = 0
gid = 0
supp gid = 0
18/12/11 17:45:57 DEBUG security.UserGroupInformation: hadoop login
[UnixLoginModule]: added UnixPrincipal,
UnixNumericUserPrincipal,
UnixNumericGroupPrincipal(s),
to Subject
18/12/11 17:45:57 DEBUG security.UserGroupInformation: hadoop login commit
18/12/11 17:45:57 DEBUG security.UserGroupInformation: using local user:UnixPrincipal: root
18/12/11 17:45:57 DEBUG security.UserGroupInformation: Using user: "UnixPrincipal: root" with name root
18/12/11 17:45:57 DEBUG security.UserGroupInformation: User entry: "root"
18/12/11 17:45:57 DEBUG security.UserGroupInformation: UGI loginUser:root (auth:SIMPLE)
18/12/11 17:45:57 DEBUG core.Tracer: sampler.classes = ; loaded no samplers
18/12/11 17:45:57 TRACE core.TracerId: ProcessID(fmt=%{tname}/%{ip}): computed process ID of "FSClient/10.33.6.155"
18/12/11 17:45:57 TRACE core.TracerPool: TracerPool(Global): adding tracer Tracer(FSClient/10.33.6.155)
18/12/11 17:45:57 DEBUG core.Tracer: span.receiver.classes = ; loaded no span receivers
18/12/11 17:45:57 TRACE core.Tracer: Created Tracer(FSClient/10.33.6.155) for FSClient
18/12/11 17:45:57 DEBUG conf.Configuration: Reloading 3 existing configurations
18/12/11 17:45:58 DEBUG s3a.S3AFileSystem: Propagating entries under fs.s3a.bucket.myclouderaraj.
18/12/11 17:45:58 DEBUG s3a.S3AFileSystem: For URI s3a://myclouderaraj//root, using credentials AWSCredentialProviderList: BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider org.apache.hadoop.fs.s3a.SharedInstanceProfileCredentialsProvider@3f191845
18/12/11 17:45:59 DEBUG azure.NativeAzureFileSystem: finalize() called.
18/12/11 17:45:59 DEBUG s3a.S3AFileSystem: Value of fs.s3a.connection.maximum is 100
18/12/11 17:45:59 DEBUG s3a.S3AFileSystem: Value of fs.s3a.attempts.maximum is 20
18/12/11 17:45:59 DEBUG s3a.S3AFileSystem: Value of fs.s3a.connection.establish.timeout is 5000
18/12/11 17:45:59 DEBUG azure.NativeAzureFileSystem: finalize() called.
18/12/11 17:45:59 DEBUG s3a.S3AFileSystem: Value of fs.s3a.connection.timeout is 200000
18/12/11 17:45:59 DEBUG s3a.S3AFileSystem: Value of fs.s3a.socket.send.buffer is 8192
18/12/11 17:45:59 DEBUG s3a.S3AFileSystem: Value of fs.s3a.socket.recv.buffer is 8192
18/12/11 17:45:59 DEBUG s3a.S3AFileSystem: Using User-Agent: Hadoop 2.6.0-cdh5.13.0
18/12/11 17:45:59 DEBUG amazonaws.AmazonWebServiceClient: Internal logging successfully configured to commons logger: true
18/12/11 17:45:59 DEBUG metrics.AwsSdkMetrics: Admin mbean registered under com.amazonaws.management:type=AwsSdkMetrics
18/12/11 17:46:01 DEBUG s3a.S3AFileSystem: Value of fs.s3a.paging.maximum is 5000
18/12/11 17:46:01 DEBUG s3a.S3AFileSystem: Value of fs.s3a.block.size is 33554432
18/12/11 17:46:01 DEBUG s3a.S3AFileSystem: Value of fs.s3a.readahead.range is 65536
18/12/11 17:46:01 DEBUG s3a.S3AFileSystem: Value of fs.s3a.max.total.tasks is 5
18/12/11 17:46:01 DEBUG s3a.S3AFileSystem: Value of fs.s3a.threads.keepalivetime is 60
18/12/11 17:46:01 DEBUG s3.AmazonS3Client: Bucket region cache doesn't have an entry for myclouderaraj. Trying to get bucket region from Amazon S3.
18/12/11 17:46:01 DEBUG s3a.AWSCredentialProviderList: Using credentials from BasicAWSCredentialsProvider
18/12/11 17:46:01 DEBUG amazonaws.request: Sending Request: HEAD https://myclouderaraj.s3.amazonaws.com / Headers: (User-Agent: Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161, amz-sdk-invocation-id: 9c0b91f3-3175-fc6a-3669-b075a6a89b9a, Content-Type: application/octet-stream, )
18/12/11 17:46:01 DEBUG auth.AWS4Signer: AWS4 Canonical Request: '"HEAD
/
amz-sdk-invocation-id:9c0b91f3-3175-fc6a-3669-b075a6a89b9a
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:myclouderaraj.s3.amazonaws.com
user-agent:Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20181211T164601Z
amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
UNSIGNED-PAYLOAD"
18/12/11 17:46:01 DEBUG auth.AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20181211T164601Z
20181211/us-east-1/s3/aws4_request
767662b5323a076b055a71d9bf21e1b437c5791f56d0efcafab5311d6e510cfd"
18/12/11 17:46:01 DEBUG auth.AWS4Signer: Generating a new signing key as the signing key not available in the cache for the date 1544486400000
18/12/11 17:46:02 DEBUG protocol.RequestAddCookies: CookieSpec selected: default
18/12/11 17:46:02 DEBUG protocol.RequestAuthCache: Auth cache not set in the context
18/12/11 17:46:02 DEBUG conn.PoolingHttpClientConnectionManager: Connection request: [route: {s}->https://myclouderaraj.s3.amazonaws.com:443][total kept alive: 0; route allocated: 0 of 100; total allocated: 0 of 100]
18/12/11 17:46:02 DEBUG conn.PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com:443][total kept alive: 0; route allocated: 1 of 100; total allocated: 1 of 100]
18/12/11 17:46:02 DEBUG execchain.MainClientExec: Opening connection {s}->https://myclouderaraj.s3.amazonaws.com:443
18/12/11 17:46:02 DEBUG conn.DefaultHttpClientConnectionOperator: Connecting to myclouderaraj.s3.amazonaws.com/52.217.1.140:443
18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory: connecting to myclouderaraj.s3.amazonaws.com/52.217.1.140:443
18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory: Connecting socket to myclouderaraj.s3.amazonaws.com/52.217.1.140:443 with timeout 5000
18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory: Enabled protocols: [TLSv1, TLSv1.1, TLSv1.2]
18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory: Enabled cipher suites:[TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory: socket.getSupportedProtocols(): [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2], socket.getEnabledProtocols(): [TLSv1, TLSv1.1, TLSv1.2]
18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory: TLS protocol enabled for SSL handshake: [TLSv1.2, TLSv1.1, TLSv1]
18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory: Starting handshake
18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory: Secure session established
18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory: negotiated protocol: TLSv1.2
18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory: negotiated cipher suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory: peer principal: CN=*.s3.amazonaws.com, O=Amazon.com Inc., L=Seattle, ST=Washington, C=US
18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory: peer alternative names: [*.s3.amazonaws.com, s3.amazonaws.com]
18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory: issuer principal: CN=DigiCert Baltimore CA-2 G2, OU=www.digicert.com, O=DigiCert Inc, C=US
18/12/11 17:46:03 DEBUG internal.SdkSSLSocket: created: myclouderaraj.s3.amazonaws.com/52.217.1.140:443
18/12/11 17:46:03 DEBUG conn.DefaultHttpClientConnectionOperator: Connection established 10.33.6.155:38019<->52.217.1.140:443
18/12/11 17:46:03 DEBUG conn.DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
18/12/11 17:46:03 DEBUG execchain.MainClientExec: Executing request HEAD / HTTP/1.1
18/12/11 17:46:03 DEBUG execchain.MainClientExec: Proxy auth state: UNCHALLENGED
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> HEAD / HTTP/1.1
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Host: myclouderaraj.s3.amazonaws.com
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> x-amz-content-sha256: UNSIGNED-PAYLOAD
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=AKIAIYFZQ7LF7F5MJL4Q/20181211/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=bc733f2ab05759751c4eeaf11095f286eceba2647ee8b014211cab2e03f967db
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> X-Amz-Date: 20181211T164601Z
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> User-Agent: Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> amz-sdk-invocation-id: 9c0b91f3-3175-fc6a-3669-b075a6a89b9a
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Content-Type: application/octet-stream
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Connection: Keep-Alive
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "HEAD / HTTP/1.1[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Host: myclouderaraj.s3.amazonaws.com[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "x-amz-content-sha256: UNSIGNED-PAYLOAD[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=AKIAIYFZQ7LF7F5MJL4Q/20181211/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=bc733f2ab05759751c4eeaf11095f286eceba2647ee8b014211cab2e03f967db[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "X-Amz-Date: 20181211T164601Z[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "User-Agent: Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "amz-sdk-invocation-id: 9c0b91f3-3175-fc6a-3669-b075a6a89b9a[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "HTTP/1.1 403 Forbidden[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "x-amz-bucket-region: us-east-1[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "x-amz-request-id: 93DABB98D9C20FB2[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "x-amz-id-2: GImfViIrJZLJvNPS0MGpxZSfsmJqibcEtOLcBK86sdCo79CpRHxlKQhANkddSqVMPIIBtIGEZho=[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "Transfer-Encoding: chunked[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "Date: Tue, 11 Dec 2018 17:08:34 GMT[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "Server: AmazonS3[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "[\r][\n]"
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << HTTP/1.1 403 Forbidden
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << x-amz-bucket-region: us-east-1
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << x-amz-request-id: 93DABB98D9C20FB2
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << x-amz-id-2: GImfViIrJZLJvNPS0MGpxZSfsmJqibcEtOLcBK86sdCo79CpRHxlKQhANkddSqVMPIIBtIGEZho=
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << Content-Type: application/xml
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << Transfer-Encoding: chunked
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << Date: Tue, 11 Dec 2018 17:08:34 GMT
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << Server: AmazonS3
18/12/11 17:46:03 DEBUG execchain.MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
18/12/11 17:46:03 DEBUG conn.PoolingHttpClientConnectionManager: Connection [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com:443] can be kept alive for 60.0 seconds
18/12/11 17:46:03 DEBUG conn.PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com:443][total kept alive: 1; route allocated: 1 of 100; total allocated: 1 of 100]
18/12/11 17:46:03 DEBUG amazonaws.request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 93DABB98D9C20FB2), S3 Extended Request ID: GImfViIrJZLJvNPS0MGpxZSfsmJqibcEtOLcBK86sdCo79CpRHxlKQhANkddSqVMPIIBtIGEZho=
18/12/11 17:46:03 DEBUG s3.AmazonS3Client: Region for myclouderaraj is us-east-1
18/12/11 17:46:03 DEBUG amazonaws.request: Sending Request: HEAD https://myclouderaraj.s3.amazonaws.com / Headers: (User-Agent: Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161, amz-sdk-invocation-id: fa6a15f3-60e0-0dd4-85de-ce71e5864550, Content-Type: application/octet-stream, )
18/12/11 17:46:03 DEBUG auth.AWS4Signer: AWS4 Canonical Request: '"HEAD
/
amz-sdk-invocation-id:fa6a15f3-60e0-0dd4-85de-ce71e5864550
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:myclouderaraj.s3.amazonaws.com
user-agent:Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20181211T164603Z
amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
UNSIGNED-PAYLOAD"
18/12/11 17:46:03 DEBUG auth.AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20181211T164603Z
20181211/us-east-1/s3/aws4_request
7af1674e22b74abaf21c562903716022a599fb7ae7c2cd46b493450627c84858"
18/12/11 17:46:03 DEBUG protocol.RequestAddCookies: CookieSpec selected: default
18/12/11 17:46:03 DEBUG protocol.RequestAuthCache: Auth cache not set in the context
18/12/11 17:46:03 DEBUG conn.PoolingHttpClientConnectionManager: Connection request: [route: {s}->https://myclouderaraj.s3.amazonaws.com:443][total kept alive: 1; route allocated: 1 of 100; total allocated: 1 of 100]
18/12/11 17:46:03 DEBUG conn.PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com:443][total kept alive: 0; route allocated: 1 of 100; total allocated: 1 of 100]
18/12/11 17:46:03 DEBUG conn.DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
18/12/11 17:46:03 DEBUG execchain.MainClientExec: Executing request HEAD / HTTP/1.1
18/12/11 17:46:03 DEBUG execchain.MainClientExec: Proxy auth state: UNCHALLENGED
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> HEAD / HTTP/1.1
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Host: myclouderaraj.s3.amazonaws.com
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> x-amz-content-sha256: UNSIGNED-PAYLOAD
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=AKIAIYFZQ7LF7F5MJL4Q/20181211/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=6e8a6bbf0e44be59a0894e30c4645049de3f3c8e2c774077a081885702236aaa
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> X-Amz-Date: 20181211T164603Z
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> User-Agent: Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> amz-sdk-invocation-id: fa6a15f3-60e0-0dd4-85de-ce71e5864550
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Content-Type: application/octet-stream
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Connection: Keep-Alive
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "HEAD / HTTP/1.1[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Host: myclouderaraj.s3.amazonaws.com[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "x-amz-content-sha256: UNSIGNED-PAYLOAD[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=AKIAIYFZQ7LF7F5MJL4Q/20181211/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=6e8a6bbf0e44be59a0894e30c4645049de3f3c8e2c774077a081885702236aaa[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "X-Amz-Date: 20181211T164603Z[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "User-Agent: Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "amz-sdk-invocation-id: fa6a15f3-60e0-0dd4-85de-ce71e5864550[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "HTTP/1.1 403 Forbidden[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "x-amz-bucket-region: us-east-1[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "x-amz-request-id: 0D4E622D9A858BD3[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "x-amz-id-2: dwOjYsPPO7Hn7MlUZPAZ6VVOVIWg6gGbQ0i9JY0YRfLnOV+GEhagAsXJuB1S0dt4bgo4VwZzpYk=[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "Transfer-Encoding: chunked[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "Date: Tue, 11 Dec 2018 17:08:35 GMT[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "Server: AmazonS3[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "[\r][\n]"
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << HTTP/1.1 403 Forbidden
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << x-amz-bucket-region: us-east-1
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << x-amz-request-id: 0D4E622D9A858BD3
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << x-amz-id-2: dwOjYsPPO7Hn7MlUZPAZ6VVOVIWg6gGbQ0i9JY0YRfLnOV+GEhagAsXJuB1S0dt4bgo4VwZzpYk=
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << Content-Type: application/xml
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << Transfer-Encoding: chunked
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << Date: Tue, 11 Dec 2018 17:08:35 GMT
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << Server: AmazonS3
18/12/11 17:46:03 DEBUG execchain.MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
18/12/11 17:46:03 DEBUG conn.PoolingHttpClientConnectionManager: Connection [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com:443] can be kept alive for 60.0 seconds
18/12/11 17:46:03 DEBUG conn.PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com:443][total kept alive: 1; route allocated: 1 of 100; total allocated: 1 of 100]
18/12/11 17:46:03 DEBUG amazonaws.request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 0D4E622D9A858BD3), S3 Extended Request ID: dwOjYsPPO7Hn7MlUZPAZ6VVOVIWg6gGbQ0i9JY0YRfLnOV+GEhagAsXJuB1S0dt4bgo4VwZzpYk=
18/12/11 17:46:03 DEBUG s3a.S3AFileSystem: Value of fs.s3a.multipart.purge.age is 86400
18/12/11 17:46:03 INFO Configuration.deprecation: fs.s3a.server-side-encryption-key is deprecated. Instead, use fs.s3a.server-side-encryption.key
18/12/11 17:46:03 DEBUG s3a.S3AFileSystem: Data is unencrypted
18/12/11 17:46:03 DEBUG s3a.S3AFileSystem: Using SSE-C with null key
18/12/11 17:46:03 DEBUG s3a.S3AFileSystem: Using S3AOutputStream
18/12/11 17:46:03 DEBUG s3guard.S3Guard: Using NullMetadataStore metadata store for s3a filesystem
18/12/11 17:46:03 DEBUG s3a.S3AFileSystem: op_glob_status += 1 -> 1
18/12/11 17:46:03 DEBUG s3a.S3AFileSystem: op_get_file_status += 1 -> 1
18/12/11 17:46:03 DEBUG s3a.S3AFileSystem: Getting path status for s3a://myclouderaraj/root (root)
18/12/11 17:46:03 DEBUG s3a.S3AFileSystem: object_metadata_requests += 1 -> 1
18/12/11 17:46:03 DEBUG s3.AmazonS3Client: Region for myclouderaraj is us-east-1
18/12/11 17:46:03 DEBUG amazonaws.request: Sending Request: HEAD https://myclouderaraj.s3.amazonaws.com /root Headers: (User-Agent: Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161, amz-sdk-invocation-id: 1777bc11-f8c0-c825-58e6-113912bcd9cd, Content-Type: application/octet-stream, )
18/12/11 17:46:03 DEBUG auth.AWS4Signer: AWS4 Canonical Request: '"HEAD
/root
amz-sdk-invocation-id:1777bc11-f8c0-c825-58e6-113912bcd9cd
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:myclouderaraj.s3.amazonaws.com
user-agent:Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20181211T164603Z
amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
UNSIGNED-PAYLOAD"
18/12/11 17:46:03 DEBUG auth.AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20181211T164603Z
20181211/us-east-1/s3/aws4_request
ed1bf2a929e630f833c3eb1f6639dbcb84723d570e76ed41cc99ddbc8b33d3a1"
18/12/11 17:46:03 DEBUG protocol.RequestAddCookies: CookieSpec selected: default
18/12/11 17:46:03 DEBUG protocol.RequestAuthCache: Auth cache not set in the context
18/12/11 17:46:03 DEBUG conn.PoolingHttpClientConnectionManager: Connection request: [route: {s}->https://myclouderaraj.s3.amazonaws.com:443][total kept alive: 1; route allocated: 1 of 100; total allocated: 1 of 100]
18/12/11 17:46:03 DEBUG conn.PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com:443][total kept alive: 0; route allocated: 1 of 100; total allocated: 1 of 100]
18/12/11 17:46:03 DEBUG conn.DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
18/12/11 17:46:03 DEBUG execchain.MainClientExec: Executing request HEAD /root HTTP/1.1
18/12/11 17:46:03 DEBUG execchain.MainClientExec: Proxy auth state: UNCHALLENGED
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> HEAD /root HTTP/1.1
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Host: myclouderaraj.s3.amazonaws.com
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> x-amz-content-sha256: UNSIGNED-PAYLOAD
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=AKIAIYFZQ7LF7F5MJL4Q/20181211/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=0e06c4f75551b83e1e122abdbf7092ae2c4413f38419d89572b77ecdc4da1d5d
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> X-Amz-Date: 20181211T164603Z
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> User-Agent: Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> amz-sdk-invocation-id: 1777bc11-f8c0-c825-58e6-113912bcd9cd
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Content-Type: application/octet-stream
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Connection: Keep-Alive
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "HEAD /root HTTP/1.1[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Host: myclouderaraj.s3.amazonaws.com[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "x-amz-content-sha256: UNSIGNED-PAYLOAD[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=AKIAIYFZQ7LF7F5MJL4Q/20181211/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=0e06c4f75551b83e1e122abdbf7092ae2c4413f38419d89572b77ecdc4da1d5d[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "X-Amz-Date: 20181211T164603Z[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "User-Agent: Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "amz-sdk-invocation-id: 1777bc11-f8c0-c825-58e6-113912bcd9cd[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "HTTP/1.1 403 Forbidden[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "x-amz-request-id: 1F1ED2B55D3DBA53[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "x-amz-id-2: oSqyv7R/B05MjjUWQflFaLoOv5Mb2ycprL+PcmqjussFP+11AiJxpKwj+MlDGrfULeumJRHkTjY=[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "Transfer-Encoding: chunked[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "Date: Tue, 11 Dec 2018 17:08:35 GMT[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "Server: AmazonS3[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "[\r][\n]"
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << HTTP/1.1 403 Forbidden
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << x-amz-request-id: 1F1ED2B55D3DBA53
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << x-amz-id-2: oSqyv7R/B05MjjUWQflFaLoOv5Mb2ycprL+PcmqjussFP+11AiJxpKwj+MlDGrfULeumJRHkTjY=
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << Content-Type: application/xml
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << Transfer-Encoding: chunked
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << Date: Tue, 11 Dec 2018 17:08:35 GMT
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << Server: AmazonS3
18/12/11 17:46:03 DEBUG execchain.MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
18/12/11 17:46:03 DEBUG conn.PoolingHttpClientConnectionManager: Connection [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com:443] can be kept alive for 60.0 seconds
18/12/11 17:46:03 DEBUG conn.PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com:443][total kept alive: 1; route allocated: 1 of 100; total allocated: 1 of 100]
18/12/11 17:46:03 DEBUG amazonaws.request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 1F1ED2B55D3DBA53), S3 Extended Request ID: oSqyv7R/B05MjjUWQflFaLoOv5Mb2ycprL+PcmqjussFP+11AiJxpKwj+MlDGrfULeumJRHkTjY=
ls: s3a://myclouderaraj/root: getFileStatus on s3a://myclouderaraj/root: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 1F1ED2B55D3DBA53), S3 Extended Request ID: oSqyv7R/B05MjjUWQflFaLoOv5Mb2ycprL+PcmqjussFP+11AiJxpKwj+MlDGrfULeumJRHkTjY=
18/12/11 17:46:03 TRACE core.TracerPool: TracerPool(Global): removing tracer Tracer(FsShell/10.33.6.155)
18/12/11 17:46:03 DEBUG http.IdleConnectionReaper: Reaper thread:
java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)
at com.amazonaws.http.IdleConnectionReaper.run(IdleConnectionReaper.java:181)
18/12/11 17:46:03 DEBUG conn.PoolingHttpClientConnectionManager: Connection manager is shutting down
18/12/11 17:46:03 DEBUG http.IdleConnectionReaper: Shutting down reaper thread.
18/12/11 17:46:03 DEBUG conn.DefaultManagedHttpClientConnection: http-outgoing-0: Close connection
18/12/11 17:46:03 DEBUG internal.SdkSSLSocket: shutting down output of myclouderaraj.s3.amazonaws.com/52.217.1.140:443
18/12/11 17:46:03 DEBUG internal.SdkSSLSocket: closing myclouderaraj.s3.amazonaws.com/52.217.1.140:443
18/12/11 17:46:03 DEBUG conn.PoolingHttpClientConnectionManager: Connection manager shut down
[root@quickstart ~]# I used the same command in both the machine where one is able to connect and the other is giving this issue. Next try: Also, I tried to do the following operations in the working machine for impala using impala-shell https://www.cloudera.com/documentation/enterprise/5-7-x/topics/impala_s3.html Updated [root@quickstart ~]# cd /etc/hadoop/conf/hdfs-site.xml file for access and secret key. Restarted all the services, also tried restarting cloudera-scm-server. Then ran the following from impala-shell [localhost:21000] > create database db_on_hdfs;
[localhost:21000] > use db_on_hdfs;
[localhost:21000] > create table mostly_on_hdfs (x int) partitioned by (year int);
[localhost:21000] > alter table mostly_on_hdfs add partition (year=2013);
[localhost:21000] > alter table mostly_on_hdfs add partition (year=2014);
[localhost:21000] > alter table mostly_on_hdfs add partition (year=2015) location 's3a://impala-demo/dir1/dir2/dir3/t1'; [quickstart.cloudera:21000] > alter table mostly_on_hdfs add partition (year=2015) LOCATION 's3a://myclouderaraj/root'; Query: alter table mostly_on_hdfs add partition (year=2015) LOCATION 's3a://myclouderaraj/root' ERROR: AnalysisException: null CAUSED BY: AmazonClientException: Unable to load AWS credentials from any provider in the chain [quickstart.cloudera:21000] > Can you please help. Thansk & Regards, Rajesh
... View more
12-03-2018
09:40 AM
Hi Seth, Tried the following which are not working [root@quickstart ~]# hdfs dfs -Dfs.s3a.access.key=myaccesskey -Dfs.s3a.secret.key=mysecretkey -DFS.s3a -ls s3a://myclouderaraj/root 18/12/03 18:01:27 INFO Configuration.deprecation: fs.s3a.server-side-encryption-key is deprecated. Instead, use fs.s3a.server-side-encryption.key ls: s3a://myclouderaraj/root: getFileStatus on s3a://myclouderaraj/root: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 78B0773829C23DB5), S3 Extended Request ID: gOFP7ib+VT7lkUo3A/AMjXGKrqNebIPNHSbHRXGsZU9dzkxtw8dceLCNfkkFkypbfPNFN6Pqe1M= [root@quickstart ~]# [root@quickstart ~]# hdfs dfs -Dfs.s3a.access.key= myaccesskey -Dfs.s3a.secret.key= mysecretkey -ls s3a://myclouderaraj/root 18/12/03 18:07:11 INFO Configuration.deprecation: fs.s3a.server-side-encryption-key is deprecated. Instead, use fs.s3a.server-side-encryption.key ls: s3a://myclouderaraj/root: getFileStatus on s3a://myclouderaraj/root: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 4BC4A44FB6133284), S3 Extended Request ID: dD8H3FqvwSrWxqfIWmNbjS+xzC5v8t0yUtQOGwJztzpKtMZ2sZ0k73RbUDxsv3pZzsC5u9Hg4as= [root@quickstart ~]# [cloudera@quickstart ~]$ hdfs dfs -Dfs.s3a.access.key= myaccesskey -Dfs.s3a.secret.key= mysecretkey -DFS.s3a -ls s3a://myclouderaraj/root 18/12/03 18:04:56 INFO Configuration.deprecation: fs.s3a.server-side-encryption-key is deprecated. Instead, use fs.s3a.server-side-encryption.key ls: s3a://myclouderaraj/root: getFileStatus on s3a://myclouderaraj/root: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 9CCAB47BBC71B470), S3 Extended Request ID: Z/N0DveFbFio6AmyEk5EkIn4BfgNnRutSIdQ26xNYy9PDyliY87AadWM8ONEDggd1hRN+XVu1yw= [cloudera@quickstart ~]$ [cloudera@quickstart ~]$ [cloudera@quickstart ~]$ hdfs dfs -Dfs.s3a.access.key= myaccesskey -Dfs.s3a.secret.key= mysecretkey -ls s3a://myclouderaraj/root 18/12/03 18:05:12 INFO Configuration.deprecation: fs.s3a.server-side-encryption-key is deprecated. Instead, use fs.s3a.server-side-encryption.key ls: s3a://myclouderaraj/root: getFileStatus on s3a://myclouderaraj/root: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 75349677808F4357), S3 Extended Request ID: ObgEtqWMjCZQ4R8uMJf/R+x0msuPE5IQiMLnSFFTTJSH2NvZf4AtPUM8m5uncrxLeLqDfe8jEIM= [cloudera@quickstart ~]$ Thanks & Regards, Rajesh
... View more
12-03-2018
03:41 AM
I see I can copy in CDH 5.7, but the error i mentioned is seen in CDH 5.13. To get work in CDH 5.13, do we need to update any thing. Can you please help on this issue. Thanks & Regards, Rajesh
... View more
11-29-2018
11:07 AM
In Hortonworks I was able run the following command with same credentials [root@sandbox ~]# hdfs dfs -Dfs.s3a.access.key=myaccesskey -Dfs.s3a.secret.key=mysecretkey -ls s3a://myclouderaraj/root Found 1 items -rw-rw-rw- 1 root 2121 2018-11-29 18:32 s3a://myclouderaraj/root/temp.json Is there anything specific to Cloudera?
... View more
11-29-2018
10:49 AM
Seth, From AWS CLI i tried the following and it was success C:\Users\rajesh.nukala\Desktop>aws s3 cp temp.json s3://myclouderaraj/root/ upload: .\temp.json to s3://myclouderaraj/root/temp.json Also,I tried the one which was provided by you but seeing the error as mentioned below [root@quickstart ~]# hdfs dfs -ls -Dfs.s3a.access.key=<myaccesskey> -Dfs.s3a.secret.key=<mysecretkey> s3a://myclouderaraj/root -ls: Illegal option -Dfs.s3a.access.key= <myaccesskey> Usage: hadoop fs [generic options] -ls [-C] [-d] [-h] [-q] [-R] [-t] [-S] [-r] [-u] [<path> ...] [root@quickstart ~]# [root@quickstart ~]# hdfs dfs -Dfs.s3a.access.key=<myaccesskey> -Dfs.s3a.secret.key=<mysecretkey> -ls s3a://myclouderaraj/root 18/11/29 19:19:31 INFO Configuration.deprecation: fs.s3a.server-side-encryption-key is deprecated. Instead, use fs.s3a.server-side-encryption.key ls: s3a://myclouderaraj/root: getFileStatus on s3a://myclouderaraj/root: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 062979117C7D7717), S3 Extended Request ID: HKTksk3mPhuVDGDxTfZjE6ElvqYGwO8+a7ryv5IQ14mBF721gfNGI6Xluvv/m0csI05KH0mLO7g= [root@quickstart ~]#
... View more
11-29-2018
07:56 AM
Hello Seth, I tried through CLI and now I am getting the below error I used the below command hadoop distcp -Dfs.s3a.access.key=<myaccesskey> -Dfs.s3a.secret.key=<mysecretkey> /user/cloudera/hdfs_error.xml s3a://myclouderaraj 18/11/29 17:14:32 INFO tools.OptionsParser: parseChunkSize: blocksperchunk false 18/11/29 17:14:36 INFO Configuration.deprecation: fs.s3a.server-side-encryption-key is deprecated. Instead, use fs.s3a.server-side-encryption.key 18/11/29 17:14:36 ERROR tools.DistCp: Invalid arguments: java.lang.IllegalArgumentException: path must be absolute at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88) at org.apache.hadoop.fs.s3a.s3guard.PathMetadata.<init>(PathMetadata.java:63) at org.apache.hadoop.fs.s3a.s3guard.PathMetadata.<init>(PathMetadata.java:55) at org.apache.hadoop.fs.s3a.s3guard.PathMetadata.<init>(PathMetadata.java:51) at org.apache.hadoop.fs.s3a.s3guard.S3Guard.putAndReturn(S3Guard.java:138) at org.apache.hadoop.fs.s3a. Also i tried this command and found the following issue Not sure why this is looking for s3a://myclouderaraj/user/root Copy to S3 error Thanks & Regards, Rajesh
... View more
11-26-2018
06:47 PM
Hello, I am trying to collect metadata and lineage information in Cloudera Navigator for S3. For this I have done the following configuration and while doing HDFS replication I am getting the following error Error Message for HDFS replication ExternalAccounts added in Cloudera Manager Configuration to Access key credentials Configruarion to IAM based Configuration for S3 connector service CDH version used: 5.13.0 Could you please help in creading metadata and lineage for S3 in Cloudera Navigator.
... View more
Labels:
10-22-2018
04:40 AM
Hello,
I can see control flow in lineage diagrams for Hive and Impala operations.
Are there any other sources which can provide the same?
Is it possible to see control flows for jobs executed by Spark or Yarn?
... View more
Labels:
10-22-2018
03:43 AM
Hello Perry, Yes, exactly. I believe replacing everything suspicious is not a proper way of dealing with such a stuff because query can contain asterisk itself...
... View more
10-21-2018
02:17 AM
Thank you Elias. Here is my understanding on Cluster, Cluster Group and Cluster Instances. Please correct me if I am wrong and also please clarify my doubts 1. Cluster Group and Cluster Instances are mostly related to AWS. 2. When you add AWS Cluster as an external account to Cloudera, It will have a Cluster Group and Cluster Instance 3. Cluster Group will have multiple Cluster Instance which are same as different Clusters(AWS AMI's) added as external accounts to Cloudera from same AWS account 4. Suppose, I have a Cloudera Cluster then I have one Cluster Group and Cluster Instance. - When I add AWS as an external account to the Cluster then I will have two Cluster Group and two Cluster Instances where the AWS Cluster Group will have the AWS Cluster Instance. - Similarly if I add another AWS Cluster from the same account, then it will add another Cluster Instance to the same Cluster Group. So, now we will have two Cluster Groups and three Cluster Instances. How we can identify different source types from different Clusters using Cloudera Navigator REST API's? I see the following in Cloudera Navigator UI. Could you please how we can relate this with REST API response. For Operation entities - in metadata it is having Cluster Group http://10.33.6.155:7187/api/v9/entities?query=(sourceType:mapreduce)AND(originalName:wordcount) [ {
"originalName" : "wordcount",
"originalDescription" : null,
"sourceId" : "4692666",
"firstClassParentId" : null,
"parentPath" : null,
"deleteTime" : null,
"extractorRunId" : "4692666##525",
"customProperties" : null,
"name" : "wordcount",
"description" : null,
"tags" : null,
"properties" : null,
"technicalProperties" : null,
"wfIds" : null,
"inputFormat" : null,
"outputFormat" : null,
"outputKey" : "org.apache.hadoop.io.Text",
"outputValue" : "org.apache.hadoop.io.IntWritable",
"mapper" : "org.myorg.WordCount$Map",
"reducer" : "org.myorg.WordCount$Reduce",
"sourceType" : "MAPREDUCE",
"type" : "OPERATION",
"metaClassName" : "mrjobspec",
"deleted" : null,
"packageName" : "nav",
"userEntity" : false,
"identity" : "5431085",
"internalType" : "mrjobspec"
} ] For Operation Execution - in metadata it is having Cluster Instance http://10.33.6.155:7187/api/v9/entities?query=(sourceType:mapreduce)AND(originalName:job_201810202026_0003) [ {
"originalName" : "job_201810202026_0003",
"originalDescription" : null,
"sourceId" : "4692666",
"firstClassParentId" : null,
"parentPath" : null,
"deleteTime" : null,
"extractorRunId" : "4692666##525",
"customProperties" : null,
"name" : null,
"description" : null,
"tags" : null,
"properties" : null,
"technicalProperties" : null,
"started" : "2018-10-20T18:47:20.530Z",
"ended" : "2018-10-20T18:47:50.778Z",
"principal" : "cloudera",
"inputs" : [ "hdfs://quickstart.cloudera:8020/user/cloudera/wordcount/input" ],
"outputs" : [ "hdfs://quickstart.cloudera:8020/user/cloudera/wordcount/output13" ],
"wfInstId" : null,
"errorCodes" : null,
"jobID" : "job_201810202026_0003",
"sourceType" : "MAPREDUCE",
"inputRecursive" : false,
"type" : "OPERATION_EXECUTION",
"metaClassName" : "mrjobinstance",
"deleted" : null,
"packageName" : "nav",
"userEntity" : false,
"identity" : "5435211",
"internalType" : "mrjobinstance"
} ]
... View more
10-20-2018
11:31 AM
Could you please let me know how to submit a case?
... View more
10-18-2018
10:36 AM
Yes, but I still want to have some data to be created with sourceType:Mapreduce. So, I have to execute by mapreduce job not to use Yarn. Please let me know the configuration to achive this.
... View more
10-16-2018
09:03 AM
When I execute a mapreduce job (MR2), it is using YARN and I can see the job/operation details in Cloudera Navigator for sourceType:YARN. But instead I want to execute the mapreduce job(MR2) without using YARN so that the operation details in Cloudera Navigator will be with sourceType:MAPREDUCE. Could you please provide the configuration changes to be done in Cloudera for this scenario? Also, please provide the configuration details to configure MR1. I am using Cloudera version of CDH 5.13.
... View more
Labels:
