Cloudera Community
Nukala
user rank
Contributor
My Accepted Solutions
Show All

Re: Not able to find Cloudera Navigator in clouder...

by Contributor in Support Questions
‎07-23-2019 03:51 AM
‎07-23-2019 03:51 AM
Thanks. It worked. ... View more

Not able to find Cloudera Navigator in cloudera-qu...

by Contributor in Support Questions
‎07-15-2019 11:49 AM
‎07-15-2019 11:49 AM
I am trying to install Cloudera using VMware and docker image and after installtion I am not able to find the Cloudera Navigator and metadataserver running. Could you please quickly help on this. Version used: cloudera-quickstart-vm-5.13.0-0-beta-docker cloudera-quickstart-vm-5.13.0-0-vmware   Thanks in advance.   Regards, Rajesh ... View more

Re: S3 connectivity using impala shell

by Contributor in Support Questions
‎12-13-2018 03:35 AM
‎12-13-2018 03:35 AM
Hello,   Now I was able to run the same query by specifying like this. Not sure does the one which is failed requires any more configuration to be set.   alter table mostly_on_hdfs add partition (year=2015) location 's3a://AWS_ACCESS_KEY_ID:AWS_SECRET_ACCESS_KEY@myclouderaraj/root';   My intension of running these queries to get metadata and lineage information of S3 items in Cloudera Navigator. But I see nothing related to S3 in Cloudera Navigator.     Thanks, Rajesh   ... View more

S3 connectivity using impala shell

by Contributor in Support Questions
‎12-12-2018 06:41 PM
‎12-12-2018 06:41 PM
Hello,   I tried to do the following operations in Cloudera machine for impala using impala-shell https://www.cloudera.com/documentation/enterprise/5-7-x/topics/impala_s3.html   In this Cloudera machine I verified the S3 connectivity using the following command and was able to access hdfs dfs -Dfs.s3a.access.key=myaccesskey -Dfs.s3a.secret.key=mysecretkey -ls s3a://myclouderaraj/root   Updated [root@quickstart ~]# cd /etc/hadoop/conf/hdfs-site.xml file for access and secret key. Restarted all the services, also tried restarting cloudera-scm-server. Then ran the following from impala-shell [localhost:21000] > create database db_on_hdfs; [localhost:21000] > use db_on_hdfs; [localhost:21000] > create table mostly_on_hdfs (x int) partitioned by (year int); [localhost:21000] > alter table mostly_on_hdfs add partition (year=2013); [localhost:21000] > alter table mostly_on_hdfs add partition (year=2014); [localhost:21000] > alter table mostly_on_hdfs add partition (year=2015) location 's3a://impala-demo/dir1/dir2/dir3/t1'; [quickstart.cloudera:21000] > alter table mostly_on_hdfs add partition (year=2015) LOCATION 's3a://myclouderaraj/root'; Query: alter table mostly_on_hdfs add partition (year=2015) LOCATION 's3a://myclouderaraj/root' ERROR: AnalysisException: null CAUSED BY: AmazonClientException: Unable to load AWS credentials from any provider in the chain [quickstart.cloudera:21000] >   Can you please help.   Thanks & Regards, Rajesh ... View more

Re: Metadata and lineage collection for S3

by Contributor in Support Questions
‎12-12-2018 06:35 PM
‎12-12-2018 06:35 PM
Yes, it was successful and i was not able to attach the complete logs.   Yes, I see that there is some change in the request headers/user-agent. Not sure why that was. I have not done any specific changes to different versions of Cloudera and trying to connect the same S3 account.   18/12/11 19:12:54 DEBUG conn.DefaultClientConnection: Sending request: GET /?prefix=root%2F&delimiter=%2F&max-keys=1 HTTP/1.1 18/12/11 19:12:54 DEBUG http.wire: >> "GET /?prefix=root%2F&delimiter=%2F&max-keys=1 HTTP/1.1[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Host: myclouderaraj.s3.amazonaws.com[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:xhHdbA514F67IgOMPolX00Rv1h4=[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Date: Wed, 12 Dec 2018 03:12:54 GMT[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Content-Type: application/x-www-form-urlencoded; charset=utf-8[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Connection: Keep-Alive[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "[\r][\n]" 18/12/11 19:12:54 DEBUG http.headers: >> GET /?prefix=root%2F&delimiter=%2F&max-keys=1 HTTP/1.1 18/12/11 19:12:54 DEBUG http.headers: >> Host: myclouderaraj.s3.amazonaws.com 18/12/11 19:12:54 DEBUG http.headers: >> Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:xhHdbA514F67IgOMPolX00Rv1h4= 18/12/11 19:12:54 DEBUG http.headers: >> User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192 18/12/11 19:12:54 DEBUG http.headers: >> Date: Wed, 12 Dec 2018 03:12:54 GMT 18/12/11 19:12:54 DEBUG http.headers: >> Content-Type: application/x-www-form-urlencoded; charset=utf-8 18/12/11 19:12:54 DEBUG http.headers: >> Connection: Keep-Alive 18/12/11 19:12:54 DEBUG http.wire: << "HTTP/1.1 200 OK[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "x-amz-id-2: n67wc46WhWXcw4ygKZ0I2NEYJXqmyEG2sDyv8+hKbZaGCflTI6YB4gvIpuqQgHb+LCSgCQ33MlU=[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "x-amz-request-id: BE60D884EB5E4D60[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Date: Wed, 12 Dec 2018 03:12:55 GMT[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "x-amz-bucket-region: us-east-1[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Content-Type: application/xml[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Transfer-Encoding: chunked[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Server: AmazonS3[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:54 DEBUG conn.DefaultClientConnection: Receiving response: HTTP/1.1 200 OK 18/12/11 19:12:54 DEBUG http.headers: << HTTP/1.1 200 OK 18/12/11 19:12:54 DEBUG http.headers: << x-amz-id-2: n67wc46WhWXcw4ygKZ0I2NEYJXqmyEG2sDyv8+hKbZaGCflTI6YB4gvIpuqQgHb+LCSgCQ33MlU= 18/12/11 19:12:54 DEBUG http.headers: << x-amz-request-id: BE60D884EB5E4D60 18/12/11 19:12:54 DEBUG http.headers: << Date: Wed, 12 Dec 2018 03:12:55 GMT 18/12/11 19:12:54 DEBUG http.headers: << x-amz-bucket-region: us-east-1 18/12/11 19:12:54 DEBUG http.headers: << Content-Type: application/xml 18/12/11 19:12:54 DEBUG http.headers: << Transfer-Encoding: chunked 18/12/11 19:12:54 DEBUG http.headers: << Server: AmazonS3 18/12/11 19:12:54 DEBUG client.SdkHttpClient: Connection can be kept alive indefinitely 18/12/11 19:12:54 DEBUG amazonaws.requestId: x-amzn-RequestId: not available 18/12/11 19:12:54 TRACE amazonaws.request: Beginning to parse service response XML 18/12/11 19:12:54 DEBUG transform.XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.cloudera.com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler 18/12/11 19:12:54 DEBUG http.wire: << "286[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "<?xml version="1.0" encoding="UTF-8"?>[\n]" 18/12/11 19:12:54 DEBUG http.wire: << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>myclouderaraj</Name><Prefix>root/</Prefix><Marker></Marker><NextMarker>root/bulkInsert.csv</NextMarker><MaxKeys>1</MaxKeys><Delimiter>/</Delimiter><IsTruncated>true</IsTruncated><Contents><Key>root/bulkInsert.csv</Key><LastModified>2018-11-30T10:30:01.000Z</LastModified><ETag>&quot;963e1e40ac0247f23ebf138755ceabfb&quot;</ETag><Size>224</Size><Owner><ID>fdbe33e179958960b9d1fab4b547c1b8ccf7a9ed1b17b3ecc20f39d0866957ab</ID><DisplayName>rajesh210</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents></ListBucketResult>" 18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "0[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com] can be kept alive indefinitely 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection released: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:54 DEBUG transform.XmlResponsesSaxParser: Parsing XML response document with handler: class com.cloudera.com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler 18/12/11 19:12:54 DEBUG transform.XmlResponsesSaxParser: Examining listing for bucket: myclouderaraj 18/12/11 19:12:54 TRACE amazonaws.request: Done parsing service response XML 18/12/11 19:12:54 DEBUG amazonaws.request: Received successful response: 200, AWS Request ID: BE60D884EB5E4D60 18/12/11 19:12:54 DEBUG amazonaws.requestId: AWS Request ID: BE60D884EB5E4D60 18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: Found path as directory (with /): 0/1 18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: Summary: root/bulkInsert.csv 224 18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: listStatus: doing listObjects for directory root/ 18/12/11 19:12:54 DEBUG amazonaws.request: Sending Request: GET https://myclouderaraj.s3.amazonaws.com / Parameters: ({"prefix":["root/"],"delimiter":["/"],"max-keys":["5000"]}Headers: (User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192, Content-Type: application/x-www-form-urlencoded; charset=utf-8, ) 18/12/11 19:12:54 DEBUG internal.S3Signer: Calculated string to sign: "GET application/x-www-form-urlencoded; charset=utf-8 Wed, 12 Dec 2018 03:12:54 GMT /myclouderaraj/" 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection request: [route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection leased: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 0; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:54 DEBUG client.SdkHttpClient: Stale connection check 18/12/11 19:12:54 DEBUG protocol.RequestAddCookies: CookieSpec selected: best-match 18/12/11 19:12:54 DEBUG protocol.RequestAuthCache: Auth cache not set in the context 18/12/11 19:12:54 DEBUG protocol.RequestProxyAuthentication: Proxy auth state: UNCHALLENGED 18/12/11 19:12:54 DEBUG client.SdkHttpClient: Attempt 1 to execute request 18/12/11 19:12:54 DEBUG conn.DefaultClientConnection: Sending request: GET /?prefix=root%2F&delimiter=%2F&max-keys=5000 HTTP/1.1 18/12/11 19:12:54 DEBUG http.wire: >> "GET /?prefix=root%2F&delimiter=%2F&max-keys=5000 HTTP/1.1[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Host: myclouderaraj.s3.amazonaws.com[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:xhHdbA514F67IgOMPolX00Rv1h4=[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Date: Wed, 12 Dec 2018 03:12:54 GMT[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Content-Type: application/x-www-form-urlencoded; charset=utf-8[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Connection: Keep-Alive[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "[\r][\n]" 18/12/11 19:12:54 DEBUG http.headers: >> GET /?prefix=root%2F&delimiter=%2F&max-keys=5000 HTTP/1.1 18/12/11 19:12:54 DEBUG http.headers: >> Host: myclouderaraj.s3.amazonaws.com 18/12/11 19:12:54 DEBUG http.headers: >> Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:xhHdbA514F67IgOMPolX00Rv1h4= 18/12/11 19:12:54 DEBUG http.headers: >> User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192 18/12/11 19:12:54 DEBUG http.headers: >> Date: Wed, 12 Dec 2018 03:12:54 GMT 18/12/11 19:12:54 DEBUG http.headers: >> Content-Type: application/x-www-form-urlencoded; charset=utf-8 18/12/11 19:12:54 DEBUG http.headers: >> Connection: Keep-Alive 18/12/11 19:12:54 DEBUG http.wire: << "HTTP/1.1 200 OK[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "x-amz-id-2: qt25vFz8qCf82h/fM/Pl2kHdk8eWPN12AmczWd25XULPEZbxmvgkoWmnMkDdaK9O13AEcF7V5Fs=[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "x-amz-request-id: 70E4AB92D29F2154[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Date: Wed, 12 Dec 2018 03:12:55 GMT[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "x-amz-bucket-region: us-east-1[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Content-Type: application/xml[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Transfer-Encoding: chunked[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Server: AmazonS3[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:54 DEBUG conn.DefaultClientConnection: Receiving response: HTTP/1.1 200 OK 18/12/11 19:12:54 DEBUG http.headers: << HTTP/1.1 200 OK 18/12/11 19:12:54 DEBUG http.headers: << x-amz-id-2: qt25vFz8qCf82h/fM/Pl2kHdk8eWPN12AmczWd25XULPEZbxmvgkoWmnMkDdaK9O13AEcF7V5Fs= 18/12/11 19:12:54 DEBUG http.headers: << x-amz-request-id: 70E4AB92D29F2154 18/12/11 19:12:54 DEBUG http.headers: << Date: Wed, 12 Dec 2018 03:12:55 GMT 18/12/11 19:12:54 DEBUG http.headers: << x-amz-bucket-region: us-east-1 18/12/11 19:12:54 DEBUG http.headers: << Content-Type: application/xml 18/12/11 19:12:54 DEBUG http.headers: << Transfer-Encoding: chunked 18/12/11 19:12:54 DEBUG http.headers: << Server: AmazonS3 18/12/11 19:12:54 DEBUG client.SdkHttpClient: Connection can be kept alive indefinitely 18/12/11 19:12:54 DEBUG amazonaws.requestId: x-amzn-RequestId: not available 18/12/11 19:12:54 TRACE amazonaws.request: Beginning to parse service response XML 18/12/11 19:12:54 DEBUG transform.XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.cloudera.com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler 18/12/11 19:12:54 DEBUG http.wire: << "64b[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "<?xml version="1.0" encoding="UTF-8"?>[\n]" 18/12/11 19:12:54 DEBUG http.wire: << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>myclouderaraj</Name><Prefix>root/</Prefix><Marker></Marker><MaxKeys>5000</MaxKeys><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Contents><Key>root/bulkInsert.csv</Key><LastModified>2018-11-30T10:30:01.000Z</LastModified><ETag>&quot;963e1e40ac0247f23ebf138755ceabfb&quot;</ETag><Size>224</Size><Owner><ID>fdbe33e179958960b9d1fab4b547c1b8ccf7a9ed1b17b3ecc20f39d0866957ab</ID><DisplayName>rajesh210</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>root/customers</Key><LastModified>2018-11-30T05:57:52.000Z</LastModified><ETag>&quot;8811238ea9271f2441f6e9a96b1393a9&quot;</ETag><Size>15812</Size><Owner><ID>fdbe33e179958960b9d1fab4b547c1b8ccf7a9ed1b17b3ecc20f39d0866957ab</ID><DisplayName>rajesh210</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>root/table_test.csv</Key><LastModified>2018-11-30T10:30:26.000Z</LastModified><ETag>&quot;d41d8cd98f00b204e9800998ecf8427e&quot;</ETag><Size>0</Size><Owner><ID>fdbe33e179958960b9d1fab4b547c1b8ccf7a9ed1b17b3ecc20f39d0866957ab</ID><DisplayName>rajesh210</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>root/temp.json</Key><LastModified>2018-11-29T18:32:55.000Z</LastModified><ETag>&quot;bdd3d8948e659d61ab0438132780e116&quot;</ETag><Size>2121</Size><Owner><ID>fdbe33e179958960b9d1fab4b547c1b8ccf7a9ed1b17b3ecc20f39d0866957ab</ID><DisplayName>rajesh210</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents></ListBucketResult>" 18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "0[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com] can be kept alive indefinitely 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection released: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:54 DEBUG transform.XmlResponsesSaxParser: Parsing XML response document with handler: class com.cloudera.com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler 18/12/11 19:12:54 DEBUG transform.XmlResponsesSaxParser: Examining listing for bucket: myclouderaraj 18/12/11 19:12:54 TRACE amazonaws.request: Done parsing service response XML 18/12/11 19:12:54 DEBUG amazonaws.request: Received successful response: 200, AWS Request ID: 70E4AB92D29F2154 18/12/11 19:12:54 DEBUG amazonaws.requestId: AWS Request ID: 70E4AB92D29F2154 18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: Adding: fi: s3a://myclouderaraj/root/bulkInsert.csv 18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: Adding: fi: s3a://myclouderaraj/root/customers 18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: Adding: fi: s3a://myclouderaraj/root/table_test.csv 18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: Adding: fi: s3a://myclouderaraj/root/temp.json Found 4 items -rw-rw-rw- 1 root 224 2018-11-30 02:30 s3a://myclouderaraj/root/bulkInsert.csv -rw-rw-rw- 1 root 15812 2018-11-29 21:57 s3a://myclouderaraj/root/customers -rw-rw-rw- 1 root 0 2018-11-30 02:30 s3a://myclouderaraj/root/table_test.csv -rw-rw-rw- 1 root 2121 2018-11-29 10:32 s3a://myclouderaraj/root/temp.json 18/12/11 19:12:54 TRACE core.TracerPool: TracerPool(Global): removing tracer Tracer(FsShell/10.33.6.146) 18/12/11 19:12:55 DEBUG conn.PoolingClientConnectionManager: Connection manager is shutting down 18/12/11 19:12:55 DEBUG internal.SdkSSLSocket: shutting down output of myclouderaraj.s3.amazonaws.com/52.216.107.100:443 18/12/11 19:12:55 DEBUG internal.SdkSSLSocket: closing myclouderaraj.s3.amazonaws.com/52.216.107.100:443 18/12/11 19:12:55 DEBUG conn.DefaultClientConnection: Connection 0.0.0.0:51025<->52.216.107.100:443 closed 18/12/11 19:12:55 DEBUG conn.DefaultClientConnection: Connection 0.0.0.0:51025<->52.216.107.100:443 closed 18/12/11 19:12:55 DEBUG conn.PoolingClientConnectionManager: Connection manager shut down [root@quickstart ~]#     Thanks & Regards, Rajesh ... View more

Re: Metadata and lineage collection for S3

by Contributor in Support Questions
‎12-12-2018 05:09 AM
‎12-12-2018 05:09 AM
Seth,   This is the success log   [root@quickstart ~]# hdfs dfs -Dfs.s3a.access.key=myaccesskey -Dfs.s3a.secret.key=mysecretkey -ls s3a://myclouderaraj/root 18/12/11 19:09:45 DEBUG util.Shell: setsid exited with exit code 0 18/12/11 19:10:10 DEBUG conf.Configuration: parsing URL jar:file:/usr/jars/hadoop-common-2.6.0-cdh5.7.0.jar!/core-default.xml 18/12/11 19:10:10 DEBUG conf.Configuration: parsing input stream sun.net.www.protocol.jar.JarURLConnection$JarURLInputStream@ea1a8d5 18/12/11 19:11:30 DEBUG conf.Configuration: parsing URL file:/etc/hadoop/conf.cloudera.yarn/core-site.xml 18/12/11 19:11:31 DEBUG conf.Configuration: parsing input stream java.io.BufferedInputStream@fcd6521 18/12/11 19:12:34 DEBUG core.Tracer: sampler.classes = ; loaded no samplers 18/12/11 19:12:35 TRACE core.TracerId: ProcessID(fmt=%{tname}/%{ip}): computed process ID of "FsShell/10.33.6.146" 18/12/11 19:12:35 TRACE core.TracerPool: TracerPool(Global): adding tracer Tracer(FsShell/10.33.6.146) 18/12/11 19:12:35 DEBUG core.Tracer: span.receiver.classes = ; loaded no span receivers 18/12/11 19:12:35 TRACE core.Tracer: Created Tracer(FsShell/10.33.6.146) for FsShell 18/12/11 19:12:37 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of successful kerberos logins and latency (milliseconds)]) 18/12/11 19:12:37 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of failed kerberos logins and latency (milliseconds)]) 18/12/11 19:12:37 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[GetGroups]) 18/12/11 19:12:37 DEBUG impl.MetricsSystemImpl: UgiMetrics, User and group related metrics Java config name: null Native config name: /etc/krb5.conf Loaded from native config 18/12/11 19:12:38 DEBUG security.Groups: Creating new Groups object 18/12/11 19:12:38 DEBUG security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000; warningDeltaMs=5000 [UnixLoginModule]: succeeded importing info: uid = 0 gid = 0 supp gid = 0 18/12/11 19:12:38 DEBUG security.UserGroupInformation: hadoop login [UnixLoginModule]: added UnixPrincipal, UnixNumericUserPrincipal, UnixNumericGroupPrincipal(s), to Subject 18/12/11 19:12:38 DEBUG security.UserGroupInformation: hadoop login commit 18/12/11 19:12:38 DEBUG security.UserGroupInformation: using local user:UnixPrincipal: root 18/12/11 19:12:38 DEBUG security.UserGroupInformation: Using user: "UnixPrincipal: root" with name root 18/12/11 19:12:38 DEBUG security.UserGroupInformation: User entry: "root" 18/12/11 19:12:38 DEBUG security.UserGroupInformation: UGI loginUser:root (auth:SIMPLE) 18/12/11 19:12:38 DEBUG core.Tracer: sampler.classes = ; loaded no samplers 18/12/11 19:12:38 TRACE core.TracerId: ProcessID(fmt=%{tname}/%{ip}): computed process ID of "FSClient/10.33.6.146" 18/12/11 19:12:38 TRACE core.TracerPool: TracerPool(Global): adding tracer Tracer(FSClient/10.33.6.146) 18/12/11 19:12:38 DEBUG core.Tracer: span.receiver.classes = ; loaded no span receivers 18/12/11 19:12:38 TRACE core.Tracer: Created Tracer(FSClient/10.33.6.146) for FSClient 18/12/11 19:12:40 DEBUG azure.NativeAzureFileSystem: finalize() called. 18/12/11 19:12:40 DEBUG azure.NativeAzureFileSystem: finalize() called. 18/12/11 19:12:41 DEBUG amazonaws.AmazonWebServiceClient: Internal logging succesfully configured to commons logger: true 18/12/11 19:12:44 DEBUG metrics.AwsSdkMetrics: Admin mbean registered under com.cloudera.com.amazonaws.management:type=AwsSdkMetrics 18/12/11 19:12:50 DEBUG auth.AWSCredentialsProviderChain: Loading credentials from BasicAWSCredentialsProvider 18/12/11 19:12:50 DEBUG amazonaws.request: Sending Request: HEAD https://myclouderaraj.s3.amazonaws.com / Headers: (User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192, Content-Type: application/x-www-form-urlencoded; charset=utf-8, ) 18/12/11 19:12:51 DEBUG internal.S3Signer: Calculated string to sign: "HEAD application/x-www-form-urlencoded; charset=utf-8 Wed, 12 Dec 2018 03:12:50 GMT /myclouderaraj/" 18/12/11 19:12:51 DEBUG conn.PoolingClientConnectionManager: Connection request: [route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 0; route allocated: 0 of 15; total allocated: 0 of 15] 18/12/11 19:12:51 DEBUG conn.PoolingClientConnectionManager: Connection leased: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 0; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:51 DEBUG ssl.SdkTLSSocketFactory: socket.getSupportedProtocols(): [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2], socket.getEnabledProtocols(): [TLSv1, TLSv1.1, TLSv1.2] 18/12/11 19:12:51 DEBUG ssl.SdkTLSSocketFactory: TLS protocol enabled for SSL handshake: [TLSv1.2, TLSv1.1, TLSv1] 18/12/11 19:12:51 DEBUG conn.DefaultClientConnectionOperator: Connecting to myclouderaraj.s3.amazonaws.com:443 18/12/11 19:12:51 DEBUG ssl.SdkTLSSocketFactory: connecting to myclouderaraj.s3.amazonaws.com/52.216.107.100:443 18/12/11 19:12:53 DEBUG internal.SdkSSLSocket: created: myclouderaraj.s3.amazonaws.com/52.216.107.100:443 18/12/11 19:12:53 DEBUG protocol.RequestAddCookies: CookieSpec selected: best-match 18/12/11 19:12:53 DEBUG protocol.RequestAuthCache: Auth cache not set in the context 18/12/11 19:12:53 DEBUG protocol.RequestProxyAuthentication: Proxy auth state: UNCHALLENGED 18/12/11 19:12:53 DEBUG client.SdkHttpClient: Attempt 1 to execute request 18/12/11 19:12:53 DEBUG conn.DefaultClientConnection: Sending request: HEAD / HTTP/1.1 18/12/11 19:12:53 DEBUG http.wire: >> "HEAD / HTTP/1.1[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Host: myclouderaraj.s3.amazonaws.com[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:XU3AFfGphmQy0rEhRrWjajyeFbA=[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Date: Wed, 12 Dec 2018 03:12:50 GMT[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Content-Type: application/x-www-form-urlencoded; charset=utf-8[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Connection: Keep-Alive[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "[\r][\n]" 18/12/11 19:12:53 DEBUG http.headers: >> HEAD / HTTP/1.1 18/12/11 19:12:53 DEBUG http.headers: >> Host: myclouderaraj.s3.amazonaws.com 18/12/11 19:12:53 DEBUG http.headers: >> Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:XU3AFfGphmQy0rEhRrWjajyeFbA= 18/12/11 19:12:53 DEBUG http.headers: >> User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192 18/12/11 19:12:53 DEBUG http.headers: >> Date: Wed, 12 Dec 2018 03:12:50 GMT 18/12/11 19:12:53 DEBUG http.headers: >> Content-Type: application/x-www-form-urlencoded; charset=utf-8 18/12/11 19:12:53 DEBUG http.headers: >> Connection: Keep-Alive 18/12/11 19:12:53 DEBUG http.wire: << "HTTP/1.1 200 OK[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "x-amz-id-2: sjybfZSJELaf9eW/QJHWV/FcObyfD5ElzagJj8SqLiPwVogmvp1lih+csHvi73bNoHKuS/OLqSI=[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "x-amz-request-id: 03AD0786BC412E7C[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "Date: Wed, 12 Dec 2018 03:12:54 GMT[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "x-amz-bucket-region: us-east-1[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "Content-Type: application/xml[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "Transfer-Encoding: chunked[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "Server: AmazonS3[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:53 DEBUG conn.DefaultClientConnection: Receiving response: HTTP/1.1 200 OK 18/12/11 19:12:53 DEBUG http.headers: << HTTP/1.1 200 OK 18/12/11 19:12:53 DEBUG http.headers: << x-amz-id-2: sjybfZSJELaf9eW/QJHWV/FcObyfD5ElzagJj8SqLiPwVogmvp1lih+csHvi73bNoHKuS/OLqSI= 18/12/11 19:12:53 DEBUG http.headers: << x-amz-request-id: 03AD0786BC412E7C 18/12/11 19:12:53 DEBUG http.headers: << Date: Wed, 12 Dec 2018 03:12:54 GMT 18/12/11 19:12:53 DEBUG http.headers: << x-amz-bucket-region: us-east-1 18/12/11 19:12:53 DEBUG http.headers: << Content-Type: application/xml 18/12/11 19:12:53 DEBUG http.headers: << Transfer-Encoding: chunked 18/12/11 19:12:53 DEBUG http.headers: << Server: AmazonS3 18/12/11 19:12:53 DEBUG client.SdkHttpClient: Connection can be kept alive indefinitely 18/12/11 19:12:53 DEBUG conn.PoolingClientConnectionManager: Connection [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com] can be kept alive indefinitely 18/12/11 19:12:53 DEBUG conn.PoolingClientConnectionManager: Connection released: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:53 DEBUG amazonaws.requestId: x-amzn-RequestId: not available 18/12/11 19:12:53 DEBUG amazonaws.request: Received successful response: 200, AWS Request ID: 03AD0786BC412E7C 18/12/11 19:12:53 DEBUG amazonaws.requestId: AWS Request ID: 03AD0786BC412E7C 18/12/11 19:12:53 DEBUG s3a.S3AFileSystem: Getting path status for s3a://myclouderaraj/root (root) 18/12/11 19:12:53 DEBUG amazonaws.request: Sending Request: HEAD https://myclouderaraj.s3.amazonaws.com /root Headers: (User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192, Content-Type: application/x-www-form-urlencoded; charset=utf-8, ) 18/12/11 19:12:53 DEBUG internal.S3Signer: Calculated string to sign: "HEAD application/x-www-form-urlencoded; charset=utf-8 Wed, 12 Dec 2018 03:12:53 GMT /myclouderaraj/root" 18/12/11 19:12:53 DEBUG conn.PoolingClientConnectionManager: Connection request: [route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:53 DEBUG conn.PoolingClientConnectionManager: Connection leased: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 0; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:53 DEBUG client.SdkHttpClient: Stale connection check 18/12/11 19:12:53 DEBUG protocol.RequestAddCookies: CookieSpec selected: best-match 18/12/11 19:12:53 DEBUG protocol.RequestAuthCache: Auth cache not set in the context 18/12/11 19:12:53 DEBUG protocol.RequestProxyAuthentication: Proxy auth state: UNCHALLENGED 18/12/11 19:12:53 DEBUG client.SdkHttpClient: Attempt 1 to execute request 18/12/11 19:12:53 DEBUG conn.DefaultClientConnection: Sending request: HEAD /root HTTP/1.1 18/12/11 19:12:53 DEBUG http.wire: >> "HEAD /root HTTP/1.1[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Host: myclouderaraj.s3.amazonaws.com[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:SLVFDzdTaXiUPcomkZCGPdiXP+I=[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Date: Wed, 12 Dec 2018 03:12:53 GMT[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Content-Type: application/x-www-form-urlencoded; charset=utf-8[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Connection: Keep-Alive[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "[\r][\n]" 18/12/11 19:12:53 DEBUG http.headers: >> HEAD /root HTTP/1.1 18/12/11 19:12:53 DEBUG http.headers: >> Host: myclouderaraj.s3.amazonaws.com 18/12/11 19:12:53 DEBUG http.headers: >> Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:SLVFDzdTaXiUPcomkZCGPdiXP+I= 18/12/11 19:12:53 DEBUG http.headers: >> User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192 18/12/11 19:12:53 DEBUG http.headers: >> Date: Wed, 12 Dec 2018 03:12:53 GMT 18/12/11 19:12:53 DEBUG http.headers: >> Content-Type: application/x-www-form-urlencoded; charset=utf-8 18/12/11 19:12:53 DEBUG http.headers: >> Connection: Keep-Alive 18/12/11 19:12:53 DEBUG http.wire: << "HTTP/1.1 404 Not Found[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "x-amz-request-id: 562A66F35BB7E4A1[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "x-amz-id-2: FC+MjpOY8O1AM4iHk4leLkFYxsqUqsOGka5QBXub98ve+SirWRY2PXH8778eWyv9UUFbFwVn0LQ=[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "Content-Type: application/xml[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "Transfer-Encoding: chunked[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "Date: Wed, 12 Dec 2018 03:12:53 GMT[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "Server: AmazonS3[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:53 DEBUG conn.DefaultClientConnection: Receiving response: HTTP/1.1 404 Not Found 18/12/11 19:12:53 DEBUG http.headers: << HTTP/1.1 404 Not Found 18/12/11 19:12:53 DEBUG http.headers: << x-amz-request-id: 562A66F35BB7E4A1 18/12/11 19:12:53 DEBUG http.headers: << x-amz-id-2: FC+MjpOY8O1AM4iHk4leLkFYxsqUqsOGka5QBXub98ve+SirWRY2PXH8778eWyv9UUFbFwVn0LQ= 18/12/11 19:12:53 DEBUG http.headers: << Content-Type: application/xml 18/12/11 19:12:53 DEBUG http.headers: << Transfer-Encoding: chunked 18/12/11 19:12:53 DEBUG http.headers: << Date: Wed, 12 Dec 2018 03:12:53 GMT 18/12/11 19:12:53 DEBUG http.headers: << Server: AmazonS3 18/12/11 19:12:53 DEBUG client.SdkHttpClient: Connection can be kept alive indefinitely 18/12/11 19:12:53 DEBUG conn.PoolingClientConnectionManager: Connection [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com] can be kept alive indefinitely 18/12/11 19:12:53 DEBUG conn.PoolingClientConnectionManager: Connection released: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:53 DEBUG amazonaws.requestId: x-amzn-RequestId: not available 18/12/11 19:12:53 DEBUG amazonaws.request: Received error response: com.cloudera.com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: null; Status Code: 404; Error Code: 404 Not Found; Request ID: 562A66F35BB7E4A1), S3 Extended Request ID: FC+MjpOY8O1AM4iHk4leLkFYxsqUqsOGka5QBXub98ve+SirWRY2PXH8778eWyv9UUFbFwVn0LQ= 18/12/11 19:12:53 DEBUG amazonaws.request: Sending Request: HEAD https://myclouderaraj.s3.amazonaws.com /root/ Headers: (User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192, Content-Type: application/x-www-form-urlencoded; charset=utf-8, ) 18/12/11 19:12:53 DEBUG internal.S3Signer: Calculated string to sign: "HEAD application/x-www-form-urlencoded; charset=utf-8 Wed, 12 Dec 2018 03:12:53 GMT /myclouderaraj/root/" 18/12/11 19:12:53 DEBUG conn.PoolingClientConnectionManager: Connection request: [route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:53 DEBUG conn.PoolingClientConnectionManager: Connection leased: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 0; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:53 DEBUG client.SdkHttpClient: Stale connection check 18/12/11 19:12:53 DEBUG protocol.RequestAddCookies: CookieSpec selected: best-match 18/12/11 19:12:53 DEBUG protocol.RequestAuthCache: Auth cache not set in the context 18/12/11 19:12:53 DEBUG protocol.RequestProxyAuthentication: Proxy auth state: UNCHALLENGED 18/12/11 19:12:53 DEBUG client.SdkHttpClient: Attempt 1 to execute request 18/12/11 19:12:53 DEBUG conn.DefaultClientConnection: Sending request: HEAD /root/ HTTP/1.1 18/12/11 19:12:53 DEBUG http.wire: >> "HEAD /root/ HTTP/1.1[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Host: myclouderaraj.s3.amazonaws.com[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:nWk9yWYYe8oEEGtfxRoqQmfgSK0=[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Date: Wed, 12 Dec 2018 03:12:53 GMT[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Content-Type: application/x-www-form-urlencoded; charset=utf-8[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Connection: Keep-Alive[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "[\r][\n]" 18/12/11 19:12:53 DEBUG http.headers: >> HEAD /root/ HTTP/1.1 18/12/11 19:12:53 DEBUG http.headers: >> Host: myclouderaraj.s3.amazonaws.com 18/12/11 19:12:53 DEBUG http.headers: >> Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:nWk9yWYYe8oEEGtfxRoqQmfgSK0= 18/12/11 19:12:53 DEBUG http.headers: >> User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192 18/12/11 19:12:53 DEBUG http.headers: >> Date: Wed, 12 Dec 2018 03:12:53 GMT 18/12/11 19:12:53 DEBUG http.headers: >> Content-Type: application/x-www-form-urlencoded; charset=utf-8 18/12/11 19:12:53 DEBUG http.headers: >> Connection: Keep-Alive 18/12/11 19:12:53 DEBUG http.wire: << "HTTP/1.1 404 Not Found[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "x-amz-request-id: C5C8989A67B509AC[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "x-amz-id-2: 19gaP2XtwSoU9Bl1Y+jO3A8rPAr+v00XbdZQDD08v6u3XJKiB3uRfLSF+E4vvivzeiwCK74Vy+I=[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "Content-Type: application/xml[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "Transfer-Encoding: chunked[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "Date: Wed, 12 Dec 2018 03:12:53 GMT[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "Server: AmazonS3[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:53 DEBUG conn.DefaultClientConnection: Receiving response: HTTP/1.1 404 Not Found 18/12/11 19:12:53 DEBUG http.headers: << HTTP/1.1 404 Not Found 18/12/11 19:12:53 DEBUG http.headers: << x-amz-request-id: C5C8989A67B509AC 18/12/11 19:12:53 DEBUG http.headers: << x-amz-id-2: 19gaP2XtwSoU9Bl1Y+jO3A8rPAr+v00XbdZQDD08v6u3XJKiB3uRfLSF+E4vvivzeiwCK74Vy+I= 18/12/11 19:12:53 DEBUG http.headers: << Content-Type: application/xml 18/12/11 19:12:53 DEBUG http.headers: << Transfer-Encoding: chunked 18/12/11 19:12:53 DEBUG http.headers: << Date: Wed, 12 Dec 2018 03:12:53 GMT 18/12/11 19:12:53 DEBUG http.headers: << Server: AmazonS3 18/12/11 19:12:53 DEBUG client.SdkHttpClient: Connection can be kept alive indefinitely 18/12/11 19:12:53 DEBUG conn.PoolingClientConnectionManager: Connection [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com] can be kept alive indefinitely 18/12/11 19:12:53 DEBUG conn.PoolingClientConnectionManager: Connection released: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:53 DEBUG amazonaws.requestId: x-amzn-RequestId: not available 18/12/11 19:12:53 DEBUG amazonaws.request: Received error response: com.cloudera.com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: null; Status Code: 404; Error Code: 404 Not Found; Request ID: C5C8989A67B509AC), S3 Extended Request ID: 19gaP2XtwSoU9Bl1Y+jO3A8rPAr+v00XbdZQDD08v6u3XJKiB3uRfLSF+E4vvivzeiwCK74Vy+I= 18/12/11 19:12:53 DEBUG amazonaws.request: Sending Request: GET https://myclouderaraj.s3.amazonaws.com / Parameters: ({"prefix":["root/"],"delimiter":["/"],"max-keys":["1"]}Headers: (User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192, Content-Type: application/x-www-form-urlencoded; charset=utf-8, ) 18/12/11 19:12:53 DEBUG internal.S3Signer: Calculated string to sign: "GET application/x-www-form-urlencoded; charset=utf-8 Wed, 12 Dec 2018 03:12:53 GMT /myclouderaraj/" 18/12/11 19:12:53 DEBUG conn.PoolingClientConnectionManager: Connection request: [route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:53 DEBUG conn.PoolingClientConnectionManager: Connection leased: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 0; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:53 DEBUG client.SdkHttpClient: Stale connection check 18/12/11 19:12:53 DEBUG protocol.RequestAddCookies: CookieSpec selected: best-match 18/12/11 19:12:53 DEBUG protocol.RequestAuthCache: Auth cache not set in the context 18/12/11 19:12:53 DEBUG protocol.RequestProxyAuthentication: Proxy auth state: UNCHALLENGED 18/12/11 19:12:53 DEBUG client.SdkHttpClient: Attempt 1 to execute request 18/12/11 19:12:53 DEBUG conn.DefaultClientConnection: Sending request: GET /?prefix=root%2F&delimiter=%2F&max-keys=1 HTTP/1.1 18/12/11 19:12:53 DEBUG http.wire: >> "GET /?prefix=root%2F&delimiter=%2F&max-keys=1 HTTP/1.1[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Host: myclouderaraj.s3.amazonaws.com[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:VEhMOYpTlsvVdG9Q0F7ehtOuxyg=[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Date: Wed, 12 Dec 2018 03:12:53 GMT[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Content-Type: application/x-www-form-urlencoded; charset=utf-8[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "Connection: Keep-Alive[\r][\n]" 18/12/11 19:12:53 DEBUG http.wire: >> "[\r][\n]" 18/12/11 19:12:53 DEBUG http.headers: >> GET /?prefix=root%2F&delimiter=%2F&max-keys=1 HTTP/1.1 18/12/11 19:12:53 DEBUG http.headers: >> Host: myclouderaraj.s3.amazonaws.com 18/12/11 19:12:53 DEBUG http.headers: >> Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:VEhMOYpTlsvVdG9Q0F7ehtOuxyg= 18/12/11 19:12:53 DEBUG http.headers: >> User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192 18/12/11 19:12:53 DEBUG http.headers: >> Date: Wed, 12 Dec 2018 03:12:53 GMT 18/12/11 19:12:53 DEBUG http.headers: >> Content-Type: application/x-www-form-urlencoded; charset=utf-8 18/12/11 19:12:53 DEBUG http.headers: >> Connection: Keep-Alive 18/12/11 19:12:54 DEBUG http.wire: << "HTTP/1.1 200 OK[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "x-amz-id-2: fcNfT1n+Dhn7lPqVGiKhrTq36arxsAAG1QNTFjpiJ+O8vSqBUumwNFw7Nmv6onHdi4Y0t0wmYGE=[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "x-amz-request-id: 741C65B73A0E800F[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Date: Wed, 12 Dec 2018 03:12:54 GMT[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "x-amz-bucket-region: us-east-1[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Content-Type: application/xml[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Transfer-Encoding: chunked[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Server: AmazonS3[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:54 DEBUG conn.DefaultClientConnection: Receiving response: HTTP/1.1 200 OK 18/12/11 19:12:54 DEBUG http.headers: << HTTP/1.1 200 OK 18/12/11 19:12:54 DEBUG http.headers: << x-amz-id-2: fcNfT1n+Dhn7lPqVGiKhrTq36arxsAAG1QNTFjpiJ+O8vSqBUumwNFw7Nmv6onHdi4Y0t0wmYGE= 18/12/11 19:12:54 DEBUG http.headers: << x-amz-request-id: 741C65B73A0E800F 18/12/11 19:12:54 DEBUG http.headers: << Date: Wed, 12 Dec 2018 03:12:54 GMT 18/12/11 19:12:54 DEBUG http.headers: << x-amz-bucket-region: us-east-1 18/12/11 19:12:54 DEBUG http.headers: << Content-Type: application/xml 18/12/11 19:12:54 DEBUG http.headers: << Transfer-Encoding: chunked 18/12/11 19:12:54 DEBUG http.headers: << Server: AmazonS3 18/12/11 19:12:54 DEBUG client.SdkHttpClient: Connection can be kept alive indefinitely 18/12/11 19:12:54 DEBUG amazonaws.requestId: x-amzn-RequestId: not available 18/12/11 19:12:54 TRACE amazonaws.request: Beginning to parse service response XML 18/12/11 19:12:54 DEBUG transform.XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.cloudera.com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler 18/12/11 19:12:54 DEBUG http.wire: << "286[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "<?xml version="1.0" encoding="UTF-8"?>[\n]" 18/12/11 19:12:54 DEBUG http.wire: << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>myclouderaraj</Name><Prefix>root/</Prefix><Marker></Marker><NextMarker>root/bulkInsert.csv</NextMarker><MaxKeys>1</MaxKeys><Delimiter>/</Delimiter><IsTruncated>true</IsTruncated><Contents><Key>root/bulkInsert.csv</Key><LastModified>2018-11-30T10:30:01.000Z</LastModified><ETag>&quot;963e1e40ac0247f23ebf138755ceabfb&quot;</ETag><Size>224</Size><Owner><ID>fdbe33e179958960b9d1fab4b547c1b8ccf7a9ed1b17b3ecc20f39d0866957ab</ID><DisplayName>rajesh210</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents></ListBucketResult>" 18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "0[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com] can be kept alive indefinitely 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection released: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:54 DEBUG transform.XmlResponsesSaxParser: Parsing XML response document with handler: class com.cloudera.com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler 18/12/11 19:12:54 DEBUG transform.XmlResponsesSaxParser: Examining listing for bucket: myclouderaraj 18/12/11 19:12:54 TRACE amazonaws.request: Done parsing service response XML 18/12/11 19:12:54 DEBUG amazonaws.request: Received successful response: 200, AWS Request ID: 741C65B73A0E800F 18/12/11 19:12:54 DEBUG amazonaws.requestId: AWS Request ID: 741C65B73A0E800F 18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: Found path as directory (with /): 0/1 18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: Summary: root/bulkInsert.csv 224 18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: List status for path: s3a://myclouderaraj/root 18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: Getting path status for s3a://myclouderaraj/root (root) 18/12/11 19:12:54 DEBUG amazonaws.request: Sending Request: HEAD https://myclouderaraj.s3.amazonaws.com /root Headers: (User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192, Content-Type: application/x-www-form-urlencoded; charset=utf-8, ) 18/12/11 19:12:54 DEBUG internal.S3Signer: Calculated string to sign: "HEAD application/x-www-form-urlencoded; charset=utf-8 Wed, 12 Dec 2018 03:12:54 GMT /myclouderaraj/root" 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection request: [route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection leased: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 0; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:54 DEBUG client.SdkHttpClient: Stale connection check 18/12/11 19:12:54 DEBUG protocol.RequestAddCookies: CookieSpec selected: best-match 18/12/11 19:12:54 DEBUG protocol.RequestAuthCache: Auth cache not set in the context 18/12/11 19:12:54 DEBUG protocol.RequestProxyAuthentication: Proxy auth state: UNCHALLENGED 18/12/11 19:12:54 DEBUG client.SdkHttpClient: Attempt 1 to execute request 18/12/11 19:12:54 DEBUG conn.DefaultClientConnection: Sending request: HEAD /root HTTP/1.1 18/12/11 19:12:54 DEBUG http.wire: >> "HEAD /root HTTP/1.1[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Host: myclouderaraj.s3.amazonaws.com[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:/G0a8NrUHFBGmNvT/YV5nkFRbKU=[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Date: Wed, 12 Dec 2018 03:12:54 GMT[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Content-Type: application/x-www-form-urlencoded; charset=utf-8[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Connection: Keep-Alive[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "[\r][\n]" 18/12/11 19:12:54 DEBUG http.headers: >> HEAD /root HTTP/1.1 18/12/11 19:12:54 DEBUG http.headers: >> Host: myclouderaraj.s3.amazonaws.com 18/12/11 19:12:54 DEBUG http.headers: >> Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:/G0a8NrUHFBGmNvT/YV5nkFRbKU= 18/12/11 19:12:54 DEBUG http.headers: >> User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192 18/12/11 19:12:54 DEBUG http.headers: >> Date: Wed, 12 Dec 2018 03:12:54 GMT 18/12/11 19:12:54 DEBUG http.headers: >> Content-Type: application/x-www-form-urlencoded; charset=utf-8 18/12/11 19:12:54 DEBUG http.headers: >> Connection: Keep-Alive 18/12/11 19:12:54 DEBUG http.wire: << "HTTP/1.1 404 Not Found[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "x-amz-request-id: 48EB44080F918683[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "x-amz-id-2: MLddIb2VuN1THhnjZI4diJfaB37g0zGNOzIFrEIfgOCMYeJBJEb2xV16+VQsv/2Z2kGhoQyhLhM=[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Content-Type: application/xml[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Transfer-Encoding: chunked[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Date: Wed, 12 Dec 2018 03:12:54 GMT[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Server: AmazonS3[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:54 DEBUG conn.DefaultClientConnection: Receiving response: HTTP/1.1 404 Not Found 18/12/11 19:12:54 DEBUG http.headers: << HTTP/1.1 404 Not Found 18/12/11 19:12:54 DEBUG http.headers: << x-amz-request-id: 48EB44080F918683 18/12/11 19:12:54 DEBUG http.headers: << x-amz-id-2: MLddIb2VuN1THhnjZI4diJfaB37g0zGNOzIFrEIfgOCMYeJBJEb2xV16+VQsv/2Z2kGhoQyhLhM= 18/12/11 19:12:54 DEBUG http.headers: << Content-Type: application/xml 18/12/11 19:12:54 DEBUG http.headers: << Transfer-Encoding: chunked 18/12/11 19:12:54 DEBUG http.headers: << Date: Wed, 12 Dec 2018 03:12:54 GMT 18/12/11 19:12:54 DEBUG http.headers: << Server: AmazonS3 18/12/11 19:12:54 DEBUG client.SdkHttpClient: Connection can be kept alive indefinitely 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com] can be kept alive indefinitely 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection released: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:54 DEBUG amazonaws.requestId: x-amzn-RequestId: not available 18/12/11 19:12:54 DEBUG amazonaws.request: Received error response: com.cloudera.com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: null; Status Code: 404; Error Code: 404 Not Found; Request ID: 48EB44080F918683), S3 Extended Request ID: MLddIb2VuN1THhnjZI4diJfaB37g0zGNOzIFrEIfgOCMYeJBJEb2xV16+VQsv/2Z2kGhoQyhLhM= 18/12/11 19:12:54 DEBUG amazonaws.request: Sending Request: HEAD https://myclouderaraj.s3.amazonaws.com /root/ Headers: (User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192, Content-Type: application/x-www-form-urlencoded; charset=utf-8, ) 18/12/11 19:12:54 DEBUG internal.S3Signer: Calculated string to sign: "HEAD application/x-www-form-urlencoded; charset=utf-8 Wed, 12 Dec 2018 03:12:54 GMT /myclouderaraj/root/" 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection request: [route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection leased: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 0; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:54 DEBUG client.SdkHttpClient: Stale connection check 18/12/11 19:12:54 DEBUG protocol.RequestAddCookies: CookieSpec selected: best-match 18/12/11 19:12:54 DEBUG protocol.RequestAuthCache: Auth cache not set in the context 18/12/11 19:12:54 DEBUG protocol.RequestProxyAuthentication: Proxy auth state: UNCHALLENGED 18/12/11 19:12:54 DEBUG client.SdkHttpClient: Attempt 1 to execute request 18/12/11 19:12:54 DEBUG conn.DefaultClientConnection: Sending request: HEAD /root/ HTTP/1.1 18/12/11 19:12:54 DEBUG http.wire: >> "HEAD /root/ HTTP/1.1[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Host: myclouderaraj.s3.amazonaws.com[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:bAUfpGUWpSAqbkk2vygScGZO7Qw=[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Date: Wed, 12 Dec 2018 03:12:54 GMT[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Content-Type: application/x-www-form-urlencoded; charset=utf-8[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Connection: Keep-Alive[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "[\r][\n]" 18/12/11 19:12:54 DEBUG http.headers: >> HEAD /root/ HTTP/1.1 18/12/11 19:12:54 DEBUG http.headers: >> Host: myclouderaraj.s3.amazonaws.com 18/12/11 19:12:54 DEBUG http.headers: >> Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:bAUfpGUWpSAqbkk2vygScGZO7Qw= 18/12/11 19:12:54 DEBUG http.headers: >> User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192 18/12/11 19:12:54 DEBUG http.headers: >> Date: Wed, 12 Dec 2018 03:12:54 GMT 18/12/11 19:12:54 DEBUG http.headers: >> Content-Type: application/x-www-form-urlencoded; charset=utf-8 18/12/11 19:12:54 DEBUG http.headers: >> Connection: Keep-Alive 18/12/11 19:12:54 DEBUG http.wire: << "HTTP/1.1 404 Not Found[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "x-amz-request-id: D79FFCC520EEA4D7[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "x-amz-id-2: TZC+frfDHCmwLqED0QD8Cg/lABBzKNkVwRIto+wmg3IVMm6+9KGRE87O3cr/jpQ8LFM3YvfaUuI=[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Content-Type: application/xml[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Transfer-Encoding: chunked[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Date: Wed, 12 Dec 2018 03:12:54 GMT[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Server: AmazonS3[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:54 DEBUG conn.DefaultClientConnection: Receiving response: HTTP/1.1 404 Not Found 18/12/11 19:12:54 DEBUG http.headers: << HTTP/1.1 404 Not Found 18/12/11 19:12:54 DEBUG http.headers: << x-amz-request-id: D79FFCC520EEA4D7 18/12/11 19:12:54 DEBUG http.headers: << x-amz-id-2: TZC+frfDHCmwLqED0QD8Cg/lABBzKNkVwRIto+wmg3IVMm6+9KGRE87O3cr/jpQ8LFM3YvfaUuI= 18/12/11 19:12:54 DEBUG http.headers: << Content-Type: application/xml 18/12/11 19:12:54 DEBUG http.headers: << Transfer-Encoding: chunked 18/12/11 19:12:54 DEBUG http.headers: << Date: Wed, 12 Dec 2018 03:12:54 GMT 18/12/11 19:12:54 DEBUG http.headers: << Server: AmazonS3 18/12/11 19:12:54 DEBUG client.SdkHttpClient: Connection can be kept alive indefinitely 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com] can be kept alive indefinitely 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection released: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:54 DEBUG amazonaws.requestId: x-amzn-RequestId: not available 18/12/11 19:12:54 DEBUG amazonaws.request: Received error response: com.cloudera.com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: null; Status Code: 404; Error Code: 404 Not Found; Request ID: D79FFCC520EEA4D7), S3 Extended Request ID: TZC+frfDHCmwLqED0QD8Cg/lABBzKNkVwRIto+wmg3IVMm6+9KGRE87O3cr/jpQ8LFM3YvfaUuI= 18/12/11 19:12:54 DEBUG amazonaws.request: Sending Request: GET https://myclouderaraj.s3.amazonaws.com / Parameters: ({"prefix":["root/"],"delimiter":["/"],"max-keys":["1"]}Headers: (User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192, Content-Type: application/x-www-form-urlencoded; charset=utf-8, ) 18/12/11 19:12:54 DEBUG internal.S3Signer: Calculated string to sign: "GET application/x-www-form-urlencoded; charset=utf-8 Wed, 12 Dec 2018 03:12:54 GMT /myclouderaraj/" 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection request: [route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection leased: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 0; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:54 DEBUG client.SdkHttpClient: Stale connection check 18/12/11 19:12:54 DEBUG protocol.RequestAddCookies: CookieSpec selected: best-match 18/12/11 19:12:54 DEBUG protocol.RequestAuthCache: Auth cache not set in the context 18/12/11 19:12:54 DEBUG protocol.RequestProxyAuthentication: Proxy auth state: UNCHALLENGED 18/12/11 19:12:54 DEBUG client.SdkHttpClient: Attempt 1 to execute request 18/12/11 19:12:54 DEBUG conn.DefaultClientConnection: Sending request: GET /?prefix=root%2F&delimiter=%2F&max-keys=1 HTTP/1.1 18/12/11 19:12:54 DEBUG http.wire: >> "GET /?prefix=root%2F&delimiter=%2F&max-keys=1 HTTP/1.1[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Host: myclouderaraj.s3.amazonaws.com[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:xhHdbA514F67IgOMPolX00Rv1h4=[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Date: Wed, 12 Dec 2018 03:12:54 GMT[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Content-Type: application/x-www-form-urlencoded; charset=utf-8[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "Connection: Keep-Alive[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: >> "[\r][\n]" 18/12/11 19:12:54 DEBUG http.headers: >> GET /?prefix=root%2F&delimiter=%2F&max-keys=1 HTTP/1.1 18/12/11 19:12:54 DEBUG http.headers: >> Host: myclouderaraj.s3.amazonaws.com 18/12/11 19:12:54 DEBUG http.headers: >> Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:xhHdbA514F67IgOMPolX00Rv1h4= 18/12/11 19:12:54 DEBUG http.headers: >> User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192 18/12/11 19:12:54 DEBUG http.headers: >> Date: Wed, 12 Dec 2018 03:12:54 GMT 18/12/11 19:12:54 DEBUG http.headers: >> Content-Type: application/x-www-form-urlencoded; charset=utf-8 18/12/11 19:12:54 DEBUG http.headers: >> Connection: Keep-Alive 18/12/11 19:12:54 DEBUG http.wire: << "HTTP/1.1 200 OK[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "x-amz-id-2: n67wc46WhWXcw4ygKZ0I2NEYJXqmyEG2sDyv8+hKbZaGCflTI6YB4gvIpuqQgHb+LCSgCQ33MlU=[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "x-amz-request-id: BE60D884EB5E4D60[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Date: Wed, 12 Dec 2018 03:12:55 GMT[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "x-amz-bucket-region: us-east-1[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Content-Type: application/xml[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Transfer-Encoding: chunked[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "Server: AmazonS3[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:54 DEBUG conn.DefaultClientConnection: Receiving response: HTTP/1.1 200 OK 18/12/11 19:12:54 DEBUG http.headers: << HTTP/1.1 200 OK 18/12/11 19:12:54 DEBUG http.headers: << x-amz-id-2: n67wc46WhWXcw4ygKZ0I2NEYJXqmyEG2sDyv8+hKbZaGCflTI6YB4gvIpuqQgHb+LCSgCQ33MlU= 18/12/11 19:12:54 DEBUG http.headers: << x-amz-request-id: BE60D884EB5E4D60 18/12/11 19:12:54 DEBUG http.headers: << Date: Wed, 12 Dec 2018 03:12:55 GMT 18/12/11 19:12:54 DEBUG http.headers: << x-amz-bucket-region: us-east-1 18/12/11 19:12:54 DEBUG http.headers: << Content-Type: application/xml 18/12/11 19:12:54 DEBUG http.headers: << Transfer-Encoding: chunked 18/12/11 19:12:54 DEBUG http.headers: << Server: AmazonS3 18/12/11 19:12:54 DEBUG client.SdkHttpClient: Connection can be kept alive indefinitely 18/12/11 19:12:54 DEBUG amazonaws.requestId: x-amzn-RequestId: not available 18/12/11 19:12:54 TRACE amazonaws.request: Beginning to parse service response XML 18/12/11 19:12:54 DEBUG transform.XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.cloudera.com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler 18/12/11 19:12:54 DEBUG http.wire: << "286[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "<?xml version="1.0" encoding="UTF-8"?>[\n]" 18/12/11 19:12:54 DEBUG http.wire: << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>myclouderaraj</Name><Prefix>root/</Prefix><Marker></Marker><NextMarker>root/bulkInsert.csv</NextMarker><MaxKeys>1</MaxKeys><Delimiter>/</Delimiter><IsTruncated>true</IsTruncated><Contents><Key>root/bulkInsert.csv</Key><LastModified>2018-11-30T10:30:01.000Z</LastModified><ETag>&quot;963e1e40ac0247f23ebf138755ceabfb&quot;</ETag><Size>224</Size><Owner><ID>fdbe33e179958960b9d1fab4b547c1b8ccf7a9ed1b17b3ecc20f39d0866957ab</ID><DisplayName>rajesh210</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents></ListBucketResult>" 18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "0[\r][\n]" 18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]" 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com] can be kept alive indefinitely 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection released: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:54 DEBUG transform.XmlResponsesSaxParser: Parsing XML response document with handler: class com.cloudera.com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler 18/12/11 19:12:54 DEBUG transform.XmlResponsesSaxParser: Examining listing for bucket: myclouderaraj 18/12/11 19:12:54 TRACE amazonaws.request: Done parsing service response XML 18/12/11 19:12:54 DEBUG amazonaws.request: Received successful response: 200, AWS Request ID: BE60D884EB5E4D60 18/12/11 19:12:54 DEBUG amazonaws.requestId: AWS Request ID: BE60D884EB5E4D60 18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: Found path as directory (with /): 0/1 18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: Summary: root/bulkInsert.csv 224 18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: listStatus: doing listObjects for directory root/ 18/12/11 19:12:54 DEBUG amazonaws.request: Sending Request: GET https://myclouderaraj.s3.amazonaws.com / Parameters: ({"prefix":["root/"],"delimiter":["/"],"max-keys":["5000"]}Headers: (User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192, Content-Type: application/x-www-form-urlencoded; charset=utf-8, ) 18/12/11 19:12:54 DEBUG internal.S3Signer: Calculated string to sign: "GET application/x-www-form-urlencoded; charset=utf-8 Wed, 12 Dec 2018 03:12:54 GMT /myclouderaraj/" 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection request: [route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection leased: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 0; route allocated: 1 of 15; total allocated: 1 of 15] 18/12/11 19:12:54 DEBUG client.SdkHttpClient: Stale connection check 18/12/11 19:12:54 DEBUG protocol.RequestAddCookies: CookieSpec selected: best-match 18/12/11 19:12:54 DEBUG protocol.RequestAuthCache: Auth cache not set in the context 18/12/11 19:12:54 DEBUG protocol.RequestProxyAuthentication: Proxy auth state: UNCHALLENGED 18/12/11 19:12:54 DEBUG client.SdkHttpClient: Attempt 1 to execute request 18/12/11 19:12:54 DEBUG conn.DefaultClientConnection: Sending request: GET /?prefix=root%2F&delimiter=%2F&max-keys=5000 HTTP/1.1 18/12/11 19:12:54 DEBUG http.wire: >> "GET /?prefix=root%2F&delimiter=%2F&max-keys=5000 HTTP/1.1[\r][\n]" ... View more

Re: Metadata and lineage collection for S3

by Contributor in Support Questions
‎12-11-2018 09:24 AM
‎12-11-2018 09:24 AM
Hi Seth,   I did as you mentioned and the logs are as memtioned below   [root@quickstart ~]# hdfs dfs -Dfs.s3a.access.key=myaccesskey -Dfs.s3a.secret.key=mysecretkey -ls s3a://myclouderaraj/root 18/12/11 17:45:55 DEBUG util.Shell: setsid exited with exit code 0 18/12/11 17:45:55 DEBUG conf.Configuration: parsing URL jar:file:/usr/lib/hadoop/hadoop-common-2.6.0-cdh5.13.0.jar!/core-default.xml 18/12/11 17:45:55 DEBUG conf.Configuration: parsing input stream sun.net.www.protocol.jar.JarURLConnection$JarURLInputStream@4232c52b 18/12/11 17:45:55 DEBUG conf.Configuration: parsing URL file:/etc/hadoop/conf.cloudera.yarn/core-site.xml 18/12/11 17:45:55 DEBUG conf.Configuration: parsing input stream java.io.BufferedInputStream@4671e53b 18/12/11 17:45:56 DEBUG core.Tracer: sampler.classes = ; loaded no samplers 18/12/11 17:45:56 TRACE core.TracerId: ProcessID(fmt=%{tname}/%{ip}): computed process ID of "FsShell/10.33.6.155" 18/12/11 17:45:56 TRACE core.TracerPool: TracerPool(Global): adding tracer Tracer(FsShell/10.33.6.155) 18/12/11 17:45:56 DEBUG core.Tracer: span.receiver.classes = ; loaded no span receivers 18/12/11 17:45:56 TRACE core.Tracer: Created Tracer(FsShell/10.33.6.155) for FsShell 18/12/11 17:45:56 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of successful kerberos logins and latency (milliseconds)]) 18/12/11 17:45:56 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of failed kerberos logins and latency (milliseconds)]) 18/12/11 17:45:56 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[GetGroups]) 18/12/11 17:45:56 DEBUG lib.MutableMetricsFactory: field private org.apache.hadoop.metrics2.lib.MutableGaugeLong org.apache.hadoop.security.UserGroupInformation$UgiMetrics.renewalFailuresTotal with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Renewal failures since startup]) 18/12/11 17:45:56 DEBUG lib.MutableMetricsFactory: field private org.apache.hadoop.metrics2.lib.MutableGaugeInt org.apache.hadoop.security.UserGroupInformation$UgiMetrics.renewalFailures with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Renewal failures since last successful login]) 18/12/11 17:45:56 DEBUG impl.MetricsSystemImpl: UgiMetrics, User and group related metrics 18/12/11 17:45:56 DEBUG security.SecurityUtil: Setting hadoop.security.token.service.use_ip to true Java config name: null Native config name: /etc/krb5.conf Loaded from native config 18/12/11 17:45:56 DEBUG security.Groups: Creating new Groups object 18/12/11 17:45:57 DEBUG security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000; warningDeltaMs=5000 [UnixLoginModule]: succeeded importing info: uid = 0 gid = 0 supp gid = 0 18/12/11 17:45:57 DEBUG security.UserGroupInformation: hadoop login [UnixLoginModule]: added UnixPrincipal, UnixNumericUserPrincipal, UnixNumericGroupPrincipal(s), to Subject 18/12/11 17:45:57 DEBUG security.UserGroupInformation: hadoop login commit 18/12/11 17:45:57 DEBUG security.UserGroupInformation: using local user:UnixPrincipal: root 18/12/11 17:45:57 DEBUG security.UserGroupInformation: Using user: "UnixPrincipal: root" with name root 18/12/11 17:45:57 DEBUG security.UserGroupInformation: User entry: "root" 18/12/11 17:45:57 DEBUG security.UserGroupInformation: UGI loginUser:root (auth:SIMPLE) 18/12/11 17:45:57 DEBUG core.Tracer: sampler.classes = ; loaded no samplers 18/12/11 17:45:57 TRACE core.TracerId: ProcessID(fmt=%{tname}/%{ip}): computed process ID of "FSClient/10.33.6.155" 18/12/11 17:45:57 TRACE core.TracerPool: TracerPool(Global): adding tracer Tracer(FSClient/10.33.6.155) 18/12/11 17:45:57 DEBUG core.Tracer: span.receiver.classes = ; loaded no span receivers 18/12/11 17:45:57 TRACE core.Tracer: Created Tracer(FSClient/10.33.6.155) for FSClient 18/12/11 17:45:57 DEBUG conf.Configuration: Reloading 3 existing configurations 18/12/11 17:45:58 DEBUG s3a.S3AFileSystem: Propagating entries under fs.s3a.bucket.myclouderaraj. 18/12/11 17:45:58 DEBUG s3a.S3AFileSystem: For URI s3a://myclouderaraj//root, using credentials AWSCredentialProviderList: BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider org.apache.hadoop.fs.s3a.SharedInstanceProfileCredentialsProvider@3f191845 18/12/11 17:45:59 DEBUG azure.NativeAzureFileSystem: finalize() called. 18/12/11 17:45:59 DEBUG s3a.S3AFileSystem: Value of fs.s3a.connection.maximum is 100 18/12/11 17:45:59 DEBUG s3a.S3AFileSystem: Value of fs.s3a.attempts.maximum is 20 18/12/11 17:45:59 DEBUG s3a.S3AFileSystem: Value of fs.s3a.connection.establish.timeout is 5000 18/12/11 17:45:59 DEBUG azure.NativeAzureFileSystem: finalize() called. 18/12/11 17:45:59 DEBUG s3a.S3AFileSystem: Value of fs.s3a.connection.timeout is 200000 18/12/11 17:45:59 DEBUG s3a.S3AFileSystem: Value of fs.s3a.socket.send.buffer is 8192 18/12/11 17:45:59 DEBUG s3a.S3AFileSystem: Value of fs.s3a.socket.recv.buffer is 8192 18/12/11 17:45:59 DEBUG s3a.S3AFileSystem: Using User-Agent: Hadoop 2.6.0-cdh5.13.0 18/12/11 17:45:59 DEBUG amazonaws.AmazonWebServiceClient: Internal logging successfully configured to commons logger: true 18/12/11 17:45:59 DEBUG metrics.AwsSdkMetrics: Admin mbean registered under com.amazonaws.management:type=AwsSdkMetrics 18/12/11 17:46:01 DEBUG s3a.S3AFileSystem: Value of fs.s3a.paging.maximum is 5000 18/12/11 17:46:01 DEBUG s3a.S3AFileSystem: Value of fs.s3a.block.size is 33554432 18/12/11 17:46:01 DEBUG s3a.S3AFileSystem: Value of fs.s3a.readahead.range is 65536 18/12/11 17:46:01 DEBUG s3a.S3AFileSystem: Value of fs.s3a.max.total.tasks is 5 18/12/11 17:46:01 DEBUG s3a.S3AFileSystem: Value of fs.s3a.threads.keepalivetime is 60 18/12/11 17:46:01 DEBUG s3.AmazonS3Client: Bucket region cache doesn't have an entry for myclouderaraj. Trying to get bucket region from Amazon S3. 18/12/11 17:46:01 DEBUG s3a.AWSCredentialProviderList: Using credentials from BasicAWSCredentialsProvider 18/12/11 17:46:01 DEBUG amazonaws.request: Sending Request: HEAD https://myclouderaraj.s3.amazonaws.com / Headers: (User-Agent: Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161, amz-sdk-invocation-id: 9c0b91f3-3175-fc6a-3669-b075a6a89b9a, Content-Type: application/octet-stream, ) 18/12/11 17:46:01 DEBUG auth.AWS4Signer: AWS4 Canonical Request: '"HEAD / amz-sdk-invocation-id:9c0b91f3-3175-fc6a-3669-b075a6a89b9a amz-sdk-retry:0/0/500 content-type:application/octet-stream host:myclouderaraj.s3.amazonaws.com user-agent:Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161 x-amz-content-sha256:UNSIGNED-PAYLOAD x-amz-date:20181211T164601Z amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date UNSIGNED-PAYLOAD" 18/12/11 17:46:01 DEBUG auth.AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256 20181211T164601Z 20181211/us-east-1/s3/aws4_request 767662b5323a076b055a71d9bf21e1b437c5791f56d0efcafab5311d6e510cfd" 18/12/11 17:46:01 DEBUG auth.AWS4Signer: Generating a new signing key as the signing key not available in the cache for the date 1544486400000 18/12/11 17:46:02 DEBUG protocol.RequestAddCookies: CookieSpec selected: default 18/12/11 17:46:02 DEBUG protocol.RequestAuthCache: Auth cache not set in the context 18/12/11 17:46:02 DEBUG conn.PoolingHttpClientConnectionManager: Connection request: [route: {s}->https://myclouderaraj.s3.amazonaws.com:443][total kept alive: 0; route allocated: 0 of 100; total allocated: 0 of 100] 18/12/11 17:46:02 DEBUG conn.PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com:443][total kept alive: 0; route allocated: 1 of 100; total allocated: 1 of 100] 18/12/11 17:46:02 DEBUG execchain.MainClientExec: Opening connection {s}->https://myclouderaraj.s3.amazonaws.com:443 18/12/11 17:46:02 DEBUG conn.DefaultHttpClientConnectionOperator: Connecting to myclouderaraj.s3.amazonaws.com/52.217.1.140:443 18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory: connecting to myclouderaraj.s3.amazonaws.com/52.217.1.140:443 18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory: Connecting socket to myclouderaraj.s3.amazonaws.com/52.217.1.140:443 with timeout 5000 18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory: Enabled protocols: [TLSv1, TLSv1.1, TLSv1.2] 18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory: Enabled cipher suites:[TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV] 18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory: socket.getSupportedProtocols(): [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2], socket.getEnabledProtocols(): [TLSv1, TLSv1.1, TLSv1.2] 18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory: TLS protocol enabled for SSL handshake: [TLSv1.2, TLSv1.1, TLSv1] 18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory: Starting handshake 18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory: Secure session established 18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory: negotiated protocol: TLSv1.2 18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory: negotiated cipher suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA 18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory: peer principal: CN=*.s3.amazonaws.com, O=Amazon.com Inc., L=Seattle, ST=Washington, C=US 18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory: peer alternative names: [*.s3.amazonaws.com, s3.amazonaws.com] 18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory: issuer principal: CN=DigiCert Baltimore CA-2 G2, OU=www.digicert.com, O=DigiCert Inc, C=US 18/12/11 17:46:03 DEBUG internal.SdkSSLSocket: created: myclouderaraj.s3.amazonaws.com/52.217.1.140:443 18/12/11 17:46:03 DEBUG conn.DefaultHttpClientConnectionOperator: Connection established 10.33.6.155:38019<->52.217.1.140:443 18/12/11 17:46:03 DEBUG conn.DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 18/12/11 17:46:03 DEBUG execchain.MainClientExec: Executing request HEAD / HTTP/1.1 18/12/11 17:46:03 DEBUG execchain.MainClientExec: Proxy auth state: UNCHALLENGED 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> HEAD / HTTP/1.1 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Host: myclouderaraj.s3.amazonaws.com 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> x-amz-content-sha256: UNSIGNED-PAYLOAD 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=AKIAIYFZQ7LF7F5MJL4Q/20181211/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=bc733f2ab05759751c4eeaf11095f286eceba2647ee8b014211cab2e03f967db 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> X-Amz-Date: 20181211T164601Z 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> User-Agent: Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> amz-sdk-invocation-id: 9c0b91f3-3175-fc6a-3669-b075a6a89b9a 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Content-Type: application/octet-stream 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Connection: Keep-Alive 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "HEAD / HTTP/1.1[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Host: myclouderaraj.s3.amazonaws.com[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "x-amz-content-sha256: UNSIGNED-PAYLOAD[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=AKIAIYFZQ7LF7F5MJL4Q/20181211/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=bc733f2ab05759751c4eeaf11095f286eceba2647ee8b014211cab2e03f967db[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "X-Amz-Date: 20181211T164601Z[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "User-Agent: Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "amz-sdk-invocation-id: 9c0b91f3-3175-fc6a-3669-b075a6a89b9a[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "HTTP/1.1 403 Forbidden[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "x-amz-bucket-region: us-east-1[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "x-amz-request-id: 93DABB98D9C20FB2[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "x-amz-id-2: GImfViIrJZLJvNPS0MGpxZSfsmJqibcEtOLcBK86sdCo79CpRHxlKQhANkddSqVMPIIBtIGEZho=[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "Transfer-Encoding: chunked[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "Date: Tue, 11 Dec 2018 17:08:34 GMT[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "Server: AmazonS3[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "[\r][\n]" 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << HTTP/1.1 403 Forbidden 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << x-amz-bucket-region: us-east-1 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << x-amz-request-id: 93DABB98D9C20FB2 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << x-amz-id-2: GImfViIrJZLJvNPS0MGpxZSfsmJqibcEtOLcBK86sdCo79CpRHxlKQhANkddSqVMPIIBtIGEZho= 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << Content-Type: application/xml 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << Transfer-Encoding: chunked 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << Date: Tue, 11 Dec 2018 17:08:34 GMT 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << Server: AmazonS3 18/12/11 17:46:03 DEBUG execchain.MainClientExec: Connection can be kept alive for 60000 MILLISECONDS 18/12/11 17:46:03 DEBUG conn.PoolingHttpClientConnectionManager: Connection [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com:443] can be kept alive for 60.0 seconds 18/12/11 17:46:03 DEBUG conn.PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com:443][total kept alive: 1; route allocated: 1 of 100; total allocated: 1 of 100] 18/12/11 17:46:03 DEBUG amazonaws.request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 93DABB98D9C20FB2), S3 Extended Request ID: GImfViIrJZLJvNPS0MGpxZSfsmJqibcEtOLcBK86sdCo79CpRHxlKQhANkddSqVMPIIBtIGEZho= 18/12/11 17:46:03 DEBUG s3.AmazonS3Client: Region for myclouderaraj is us-east-1 18/12/11 17:46:03 DEBUG amazonaws.request: Sending Request: HEAD https://myclouderaraj.s3.amazonaws.com / Headers: (User-Agent: Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161, amz-sdk-invocation-id: fa6a15f3-60e0-0dd4-85de-ce71e5864550, Content-Type: application/octet-stream, ) 18/12/11 17:46:03 DEBUG auth.AWS4Signer: AWS4 Canonical Request: '"HEAD / amz-sdk-invocation-id:fa6a15f3-60e0-0dd4-85de-ce71e5864550 amz-sdk-retry:0/0/500 content-type:application/octet-stream host:myclouderaraj.s3.amazonaws.com user-agent:Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161 x-amz-content-sha256:UNSIGNED-PAYLOAD x-amz-date:20181211T164603Z amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date UNSIGNED-PAYLOAD" 18/12/11 17:46:03 DEBUG auth.AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256 20181211T164603Z 20181211/us-east-1/s3/aws4_request 7af1674e22b74abaf21c562903716022a599fb7ae7c2cd46b493450627c84858" 18/12/11 17:46:03 DEBUG protocol.RequestAddCookies: CookieSpec selected: default 18/12/11 17:46:03 DEBUG protocol.RequestAuthCache: Auth cache not set in the context 18/12/11 17:46:03 DEBUG conn.PoolingHttpClientConnectionManager: Connection request: [route: {s}->https://myclouderaraj.s3.amazonaws.com:443][total kept alive: 1; route allocated: 1 of 100; total allocated: 1 of 100] 18/12/11 17:46:03 DEBUG conn.PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com:443][total kept alive: 0; route allocated: 1 of 100; total allocated: 1 of 100] 18/12/11 17:46:03 DEBUG conn.DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 18/12/11 17:46:03 DEBUG execchain.MainClientExec: Executing request HEAD / HTTP/1.1 18/12/11 17:46:03 DEBUG execchain.MainClientExec: Proxy auth state: UNCHALLENGED 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> HEAD / HTTP/1.1 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Host: myclouderaraj.s3.amazonaws.com 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> x-amz-content-sha256: UNSIGNED-PAYLOAD 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=AKIAIYFZQ7LF7F5MJL4Q/20181211/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=6e8a6bbf0e44be59a0894e30c4645049de3f3c8e2c774077a081885702236aaa 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> X-Amz-Date: 20181211T164603Z 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> User-Agent: Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> amz-sdk-invocation-id: fa6a15f3-60e0-0dd4-85de-ce71e5864550 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Content-Type: application/octet-stream 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Connection: Keep-Alive 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "HEAD / HTTP/1.1[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Host: myclouderaraj.s3.amazonaws.com[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "x-amz-content-sha256: UNSIGNED-PAYLOAD[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=AKIAIYFZQ7LF7F5MJL4Q/20181211/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=6e8a6bbf0e44be59a0894e30c4645049de3f3c8e2c774077a081885702236aaa[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "X-Amz-Date: 20181211T164603Z[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "User-Agent: Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "amz-sdk-invocation-id: fa6a15f3-60e0-0dd4-85de-ce71e5864550[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "HTTP/1.1 403 Forbidden[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "x-amz-bucket-region: us-east-1[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "x-amz-request-id: 0D4E622D9A858BD3[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "x-amz-id-2: dwOjYsPPO7Hn7MlUZPAZ6VVOVIWg6gGbQ0i9JY0YRfLnOV+GEhagAsXJuB1S0dt4bgo4VwZzpYk=[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "Transfer-Encoding: chunked[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "Date: Tue, 11 Dec 2018 17:08:35 GMT[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "Server: AmazonS3[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "[\r][\n]" 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << HTTP/1.1 403 Forbidden 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << x-amz-bucket-region: us-east-1 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << x-amz-request-id: 0D4E622D9A858BD3 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << x-amz-id-2: dwOjYsPPO7Hn7MlUZPAZ6VVOVIWg6gGbQ0i9JY0YRfLnOV+GEhagAsXJuB1S0dt4bgo4VwZzpYk= 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << Content-Type: application/xml 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << Transfer-Encoding: chunked 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << Date: Tue, 11 Dec 2018 17:08:35 GMT 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << Server: AmazonS3 18/12/11 17:46:03 DEBUG execchain.MainClientExec: Connection can be kept alive for 60000 MILLISECONDS 18/12/11 17:46:03 DEBUG conn.PoolingHttpClientConnectionManager: Connection [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com:443] can be kept alive for 60.0 seconds 18/12/11 17:46:03 DEBUG conn.PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com:443][total kept alive: 1; route allocated: 1 of 100; total allocated: 1 of 100] 18/12/11 17:46:03 DEBUG amazonaws.request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 0D4E622D9A858BD3), S3 Extended Request ID: dwOjYsPPO7Hn7MlUZPAZ6VVOVIWg6gGbQ0i9JY0YRfLnOV+GEhagAsXJuB1S0dt4bgo4VwZzpYk= 18/12/11 17:46:03 DEBUG s3a.S3AFileSystem: Value of fs.s3a.multipart.purge.age is 86400 18/12/11 17:46:03 INFO Configuration.deprecation: fs.s3a.server-side-encryption-key is deprecated. Instead, use fs.s3a.server-side-encryption.key 18/12/11 17:46:03 DEBUG s3a.S3AFileSystem: Data is unencrypted 18/12/11 17:46:03 DEBUG s3a.S3AFileSystem: Using SSE-C with null key 18/12/11 17:46:03 DEBUG s3a.S3AFileSystem: Using S3AOutputStream 18/12/11 17:46:03 DEBUG s3guard.S3Guard: Using NullMetadataStore metadata store for s3a filesystem 18/12/11 17:46:03 DEBUG s3a.S3AFileSystem: op_glob_status += 1 -> 1 18/12/11 17:46:03 DEBUG s3a.S3AFileSystem: op_get_file_status += 1 -> 1 18/12/11 17:46:03 DEBUG s3a.S3AFileSystem: Getting path status for s3a://myclouderaraj/root (root) 18/12/11 17:46:03 DEBUG s3a.S3AFileSystem: object_metadata_requests += 1 -> 1 18/12/11 17:46:03 DEBUG s3.AmazonS3Client: Region for myclouderaraj is us-east-1 18/12/11 17:46:03 DEBUG amazonaws.request: Sending Request: HEAD https://myclouderaraj.s3.amazonaws.com /root Headers: (User-Agent: Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161, amz-sdk-invocation-id: 1777bc11-f8c0-c825-58e6-113912bcd9cd, Content-Type: application/octet-stream, ) 18/12/11 17:46:03 DEBUG auth.AWS4Signer: AWS4 Canonical Request: '"HEAD /root amz-sdk-invocation-id:1777bc11-f8c0-c825-58e6-113912bcd9cd amz-sdk-retry:0/0/500 content-type:application/octet-stream host:myclouderaraj.s3.amazonaws.com user-agent:Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161 x-amz-content-sha256:UNSIGNED-PAYLOAD x-amz-date:20181211T164603Z amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date UNSIGNED-PAYLOAD" 18/12/11 17:46:03 DEBUG auth.AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256 20181211T164603Z 20181211/us-east-1/s3/aws4_request ed1bf2a929e630f833c3eb1f6639dbcb84723d570e76ed41cc99ddbc8b33d3a1" 18/12/11 17:46:03 DEBUG protocol.RequestAddCookies: CookieSpec selected: default 18/12/11 17:46:03 DEBUG protocol.RequestAuthCache: Auth cache not set in the context 18/12/11 17:46:03 DEBUG conn.PoolingHttpClientConnectionManager: Connection request: [route: {s}->https://myclouderaraj.s3.amazonaws.com:443][total kept alive: 1; route allocated: 1 of 100; total allocated: 1 of 100] 18/12/11 17:46:03 DEBUG conn.PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com:443][total kept alive: 0; route allocated: 1 of 100; total allocated: 1 of 100] 18/12/11 17:46:03 DEBUG conn.DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 18/12/11 17:46:03 DEBUG execchain.MainClientExec: Executing request HEAD /root HTTP/1.1 18/12/11 17:46:03 DEBUG execchain.MainClientExec: Proxy auth state: UNCHALLENGED 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> HEAD /root HTTP/1.1 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Host: myclouderaraj.s3.amazonaws.com 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> x-amz-content-sha256: UNSIGNED-PAYLOAD 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=AKIAIYFZQ7LF7F5MJL4Q/20181211/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=0e06c4f75551b83e1e122abdbf7092ae2c4413f38419d89572b77ecdc4da1d5d 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> X-Amz-Date: 20181211T164603Z 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> User-Agent: Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> amz-sdk-invocation-id: 1777bc11-f8c0-c825-58e6-113912bcd9cd 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Content-Type: application/octet-stream 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Connection: Keep-Alive 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "HEAD /root HTTP/1.1[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Host: myclouderaraj.s3.amazonaws.com[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "x-amz-content-sha256: UNSIGNED-PAYLOAD[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=AKIAIYFZQ7LF7F5MJL4Q/20181211/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=0e06c4f75551b83e1e122abdbf7092ae2c4413f38419d89572b77ecdc4da1d5d[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "X-Amz-Date: 20181211T164603Z[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "User-Agent: Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "amz-sdk-invocation-id: 1777bc11-f8c0-c825-58e6-113912bcd9cd[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "HTTP/1.1 403 Forbidden[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "x-amz-request-id: 1F1ED2B55D3DBA53[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "x-amz-id-2: oSqyv7R/B05MjjUWQflFaLoOv5Mb2ycprL+PcmqjussFP+11AiJxpKwj+MlDGrfULeumJRHkTjY=[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "Transfer-Encoding: chunked[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "Date: Tue, 11 Dec 2018 17:08:35 GMT[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "Server: AmazonS3[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "[\r][\n]" 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << HTTP/1.1 403 Forbidden 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << x-amz-request-id: 1F1ED2B55D3DBA53 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << x-amz-id-2: oSqyv7R/B05MjjUWQflFaLoOv5Mb2ycprL+PcmqjussFP+11AiJxpKwj+MlDGrfULeumJRHkTjY= 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << Content-Type: application/xml 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << Transfer-Encoding: chunked 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << Date: Tue, 11 Dec 2018 17:08:35 GMT 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << Server: AmazonS3 18/12/11 17:46:03 DEBUG execchain.MainClientExec: Connection can be kept alive for 60000 MILLISECONDS 18/12/11 17:46:03 DEBUG conn.PoolingHttpClientConnectionManager: Connection [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com:443] can be kept alive for 60.0 seconds 18/12/11 17:46:03 DEBUG conn.PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com:443][total kept alive: 1; route allocated: 1 of 100; total allocated: 1 of 100] 18/12/11 17:46:03 DEBUG amazonaws.request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 1F1ED2B55D3DBA53), S3 Extended Request ID: oSqyv7R/B05MjjUWQflFaLoOv5Mb2ycprL+PcmqjussFP+11AiJxpKwj+MlDGrfULeumJRHkTjY= ls: s3a://myclouderaraj/root: getFileStatus on s3a://myclouderaraj/root: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 1F1ED2B55D3DBA53), S3 Extended Request ID: oSqyv7R/B05MjjUWQflFaLoOv5Mb2ycprL+PcmqjussFP+11AiJxpKwj+MlDGrfULeumJRHkTjY= 18/12/11 17:46:03 TRACE core.TracerPool: TracerPool(Global): removing tracer Tracer(FsShell/10.33.6.155) 18/12/11 17:46:03 DEBUG http.IdleConnectionReaper: Reaper thread: java.lang.InterruptedException: sleep interrupted at java.lang.Thread.sleep(Native Method) at com.amazonaws.http.IdleConnectionReaper.run(IdleConnectionReaper.java:181) 18/12/11 17:46:03 DEBUG conn.PoolingHttpClientConnectionManager: Connection manager is shutting down 18/12/11 17:46:03 DEBUG http.IdleConnectionReaper: Shutting down reaper thread. 18/12/11 17:46:03 DEBUG conn.DefaultManagedHttpClientConnection: http-outgoing-0: Close connection 18/12/11 17:46:03 DEBUG internal.SdkSSLSocket: shutting down output of myclouderaraj.s3.amazonaws.com/52.217.1.140:443 18/12/11 17:46:03 DEBUG internal.SdkSSLSocket: closing myclouderaraj.s3.amazonaws.com/52.217.1.140:443 18/12/11 17:46:03 DEBUG conn.PoolingHttpClientConnectionManager: Connection manager shut down [root@quickstart ~]# I used the same command in both the machine where one is able to connect and the other is giving this issue.   Next try:   Also, I tried to do the following operations in the working machine for impala using impala-shell https://www.cloudera.com/documentation/enterprise/5-7-x/topics/impala_s3.html Updated [root@quickstart ~]# cd /etc/hadoop/conf/hdfs-site.xml file for access and secret key. Restarted all the services, also tried restarting cloudera-scm-server. Then ran the following from impala-shell [localhost:21000] > create database db_on_hdfs; [localhost:21000] > use db_on_hdfs; [localhost:21000] > create table mostly_on_hdfs (x int) partitioned by (year int); [localhost:21000] > alter table mostly_on_hdfs add partition (year=2013); [localhost:21000] > alter table mostly_on_hdfs add partition (year=2014); [localhost:21000] > alter table mostly_on_hdfs add partition (year=2015) location 's3a://impala-demo/dir1/dir2/dir3/t1'; [quickstart.cloudera:21000] > alter table mostly_on_hdfs add partition (year=2015) LOCATION 's3a://myclouderaraj/root'; Query: alter table mostly_on_hdfs add partition (year=2015) LOCATION 's3a://myclouderaraj/root' ERROR: AnalysisException: null CAUSED BY: AmazonClientException: Unable to load AWS credentials from any provider in the chain [quickstart.cloudera:21000] >   Can you please help.   Thansk & Regards, Rajesh   ... View more

Re: Metadata and lineage collection for S3

by Contributor in Support Questions
‎12-03-2018 09:40 AM
‎12-03-2018 09:40 AM
Hi Seth,   Tried the following which are not working   [root@quickstart ~]# hdfs dfs -Dfs.s3a.access.key=myaccesskey -Dfs.s3a.secret.key=mysecretkey -DFS.s3a -ls s3a://myclouderaraj/root 18/12/03 18:01:27 INFO Configuration.deprecation: fs.s3a.server-side-encryption-key is deprecated. Instead, use fs.s3a.server-side-encryption.key ls: s3a://myclouderaraj/root: getFileStatus on s3a://myclouderaraj/root: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 78B0773829C23DB5), S3 Extended Request ID: gOFP7ib+VT7lkUo3A/AMjXGKrqNebIPNHSbHRXGsZU9dzkxtw8dceLCNfkkFkypbfPNFN6Pqe1M= [root@quickstart ~]#   [root@quickstart ~]# hdfs dfs -Dfs.s3a.access.key=myaccesskey -Dfs.s3a.secret.key=mysecretkey -ls s3a://myclouderaraj/root 18/12/03 18:07:11 INFO Configuration.deprecation: fs.s3a.server-side-encryption-key is deprecated. Instead, use fs.s3a.server-side-encryption.key ls: s3a://myclouderaraj/root: getFileStatus on s3a://myclouderaraj/root: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 4BC4A44FB6133284), S3 Extended Request ID: dD8H3FqvwSrWxqfIWmNbjS+xzC5v8t0yUtQOGwJztzpKtMZ2sZ0k73RbUDxsv3pZzsC5u9Hg4as= [root@quickstart ~]#     [cloudera@quickstart ~]$ hdfs dfs -Dfs.s3a.access.key=myaccesskey -Dfs.s3a.secret.key=mysecretkey -DFS.s3a -ls s3a://myclouderaraj/root 18/12/03 18:04:56 INFO Configuration.deprecation: fs.s3a.server-side-encryption-key is deprecated. Instead, use fs.s3a.server-side-encryption.key ls: s3a://myclouderaraj/root: getFileStatus on s3a://myclouderaraj/root: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 9CCAB47BBC71B470), S3 Extended Request ID: Z/N0DveFbFio6AmyEk5EkIn4BfgNnRutSIdQ26xNYy9PDyliY87AadWM8ONEDggd1hRN+XVu1yw= [cloudera@quickstart ~]$ [cloudera@quickstart ~]$ [cloudera@quickstart ~]$ hdfs dfs -Dfs.s3a.access.key=myaccesskey -Dfs.s3a.secret.key=mysecretkey  -ls s3a://myclouderaraj/root 18/12/03 18:05:12 INFO Configuration.deprecation: fs.s3a.server-side-encryption-key is deprecated. Instead, use fs.s3a.server-side-encryption.key ls: s3a://myclouderaraj/root: getFileStatus on s3a://myclouderaraj/root: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 75349677808F4357), S3 Extended Request ID: ObgEtqWMjCZQ4R8uMJf/R+x0msuPE5IQiMLnSFFTTJSH2NvZf4AtPUM8m5uncrxLeLqDfe8jEIM= [cloudera@quickstart ~]$   Thanks & Regards, Rajesh ... View more

Re: Metadata and lineage collection for S3

by Contributor in Support Questions
‎12-03-2018 03:41 AM
‎12-03-2018 03:41 AM
I see I can copy in CDH 5.7, but the error i mentioned is seen in CDH 5.13. To get work in CDH 5.13, do we need to update any thing. Can you please help on this issue.   Thanks & Regards, Rajesh ... View more

Re: Metadata and lineage collection for S3

by Contributor in Support Questions
‎11-29-2018 11:07 AM
‎11-29-2018 11:07 AM
In Hortonworks I was able run the following command with same credentials   [root@sandbox ~]# hdfs dfs -Dfs.s3a.access.key=myaccesskey -Dfs.s3a.secret.key=mysecretkey -ls s3a://myclouderaraj/root Found 1 items -rw-rw-rw- 1 root 2121 2018-11-29 18:32 s3a://myclouderaraj/root/temp.json   Is there anything specific to Cloudera? ... View more
Contact Me
Online
Offline
Last Visited
‎01-08-2020 02:20 AM
Community Statistics
Member Since ‎09-05-2018 10:48 PM
Last Visited ‎01-08-2020 02:20 AM
Posts 49
Kudos received 2