Support Questions

Nukala · ‎11-26-2018

Hello,

I am trying to collect metadata and lineage information in Cloudera Navigator for S3. For this I have done the following configuration and while doing HDFS replication I am getting the following error

Error Message for HDFS replicationExternalAccounts added in Cloudera ManagerConfiguration to Access key credentialsConfigruarion to IAM basedConfiguration for S3 connector service

CDH version used: 5.13.0

Could you please help in creading metadata and lineage for S3 in Cloudera Navigator.

Nukala · ‎12-11-2018

Hi Seth,

I did as you mentioned and the logs are as memtioned below

[root@quickstart ~]# hdfs dfs -Dfs.s3a.access.key=myaccesskey -Dfs.s3a.secret.key=mysecretkey -ls s3a://myclouderaraj/root
18/12/11 17:45:55 DEBUG util.Shell: setsid exited with exit code 0
18/12/11 17:45:55 DEBUG conf.Configuration: parsing URL jar:file:/usr/lib/hadoop/hadoop-common-2.6.0-cdh5.13.0.jar!/core-default.xml
18/12/11 17:45:55 DEBUG conf.Configuration: parsing input stream sun.net.www.protocol.jar.JarURLConnection$JarURLInputStream@4232c52b
18/12/11 17:45:55 DEBUG conf.Configuration: parsing URL file:/etc/hadoop/conf.cloudera.yarn/core-site.xml
18/12/11 17:45:55 DEBUG conf.Configuration: parsing input stream java.io.BufferedInputStream@4671e53b
18/12/11 17:45:56 DEBUG core.Tracer: sampler.classes = ; loaded no samplers
18/12/11 17:45:56 TRACE core.TracerId: ProcessID(fmt=%{tname}/%{ip}): computed process ID of "FsShell/10.33.6.155"
18/12/11 17:45:56 TRACE core.TracerPool: TracerPool(Global): adding tracer Tracer(FsShell/10.33.6.155)
18/12/11 17:45:56 DEBUG core.Tracer: span.receiver.classes = ; loaded no span receivers
18/12/11 17:45:56 TRACE core.Tracer: Created Tracer(FsShell/10.33.6.155) for FsShell
18/12/11 17:45:56 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of successful kerberos logins and latency (milliseconds)])
18/12/11 17:45:56 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of failed kerberos logins and latency (milliseconds)])
18/12/11 17:45:56 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[GetGroups])
18/12/11 17:45:56 DEBUG lib.MutableMetricsFactory: field private org.apache.hadoop.metrics2.lib.MutableGaugeLong org.apache.hadoop.security.UserGroupInformation$UgiMetrics.renewalFailuresTotal with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Renewal failures since startup])
18/12/11 17:45:56 DEBUG lib.MutableMetricsFactory: field private org.apache.hadoop.metrics2.lib.MutableGaugeInt org.apache.hadoop.security.UserGroupInformation$UgiMetrics.renewalFailures with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Renewal failures since last successful login])
18/12/11 17:45:56 DEBUG impl.MetricsSystemImpl: UgiMetrics, User and group related metrics
18/12/11 17:45:56 DEBUG security.SecurityUtil: Setting hadoop.security.token.service.use_ip to true
Java config name: null
Native config name: /etc/krb5.conf
Loaded from native config
18/12/11 17:45:56 DEBUG security.Groups:  Creating new Groups object
18/12/11 17:45:57 DEBUG security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000; warningDeltaMs=5000
                [UnixLoginModule]: succeeded importing info:
                        uid = 0
                        gid = 0
                        supp gid = 0
18/12/11 17:45:57 DEBUG security.UserGroupInformation: hadoop login
                [UnixLoginModule]: added UnixPrincipal,
                                UnixNumericUserPrincipal,
                                UnixNumericGroupPrincipal(s),
                         to Subject
18/12/11 17:45:57 DEBUG security.UserGroupInformation: hadoop login commit
18/12/11 17:45:57 DEBUG security.UserGroupInformation: using local user:UnixPrincipal: root
18/12/11 17:45:57 DEBUG security.UserGroupInformation: Using user: "UnixPrincipal: root" with name root
18/12/11 17:45:57 DEBUG security.UserGroupInformation: User entry: "root"
18/12/11 17:45:57 DEBUG security.UserGroupInformation: UGI loginUser:root (auth:SIMPLE)
18/12/11 17:45:57 DEBUG core.Tracer: sampler.classes = ; loaded no samplers
18/12/11 17:45:57 TRACE core.TracerId: ProcessID(fmt=%{tname}/%{ip}): computed process ID of "FSClient/10.33.6.155"
18/12/11 17:45:57 TRACE core.TracerPool: TracerPool(Global): adding tracer Tracer(FSClient/10.33.6.155)
18/12/11 17:45:57 DEBUG core.Tracer: span.receiver.classes = ; loaded no span receivers
18/12/11 17:45:57 TRACE core.Tracer: Created Tracer(FSClient/10.33.6.155) for FSClient
18/12/11 17:45:57 DEBUG conf.Configuration: Reloading 3 existing configurations
18/12/11 17:45:58 DEBUG s3a.S3AFileSystem: Propagating entries under fs.s3a.bucket.myclouderaraj.
18/12/11 17:45:58 DEBUG s3a.S3AFileSystem: For URI s3a://myclouderaraj//root, using credentials AWSCredentialProviderList: BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider org.apache.hadoop.fs.s3a.SharedInstanceProfileCredentialsProvider@3f191845
18/12/11 17:45:59 DEBUG azure.NativeAzureFileSystem: finalize() called.
18/12/11 17:45:59 DEBUG s3a.S3AFileSystem: Value of fs.s3a.connection.maximum is 100
18/12/11 17:45:59 DEBUG s3a.S3AFileSystem: Value of fs.s3a.attempts.maximum is 20
18/12/11 17:45:59 DEBUG s3a.S3AFileSystem: Value of fs.s3a.connection.establish.timeout is 5000
18/12/11 17:45:59 DEBUG azure.NativeAzureFileSystem: finalize() called.
18/12/11 17:45:59 DEBUG s3a.S3AFileSystem: Value of fs.s3a.connection.timeout is 200000
18/12/11 17:45:59 DEBUG s3a.S3AFileSystem: Value of fs.s3a.socket.send.buffer is 8192
18/12/11 17:45:59 DEBUG s3a.S3AFileSystem: Value of fs.s3a.socket.recv.buffer is 8192
18/12/11 17:45:59 DEBUG s3a.S3AFileSystem: Using User-Agent: Hadoop 2.6.0-cdh5.13.0
18/12/11 17:45:59 DEBUG amazonaws.AmazonWebServiceClient: Internal logging successfully configured to commons logger: true
18/12/11 17:45:59 DEBUG metrics.AwsSdkMetrics: Admin mbean registered under com.amazonaws.management:type=AwsSdkMetrics
18/12/11 17:46:01 DEBUG s3a.S3AFileSystem: Value of fs.s3a.paging.maximum is 5000
18/12/11 17:46:01 DEBUG s3a.S3AFileSystem: Value of fs.s3a.block.size is 33554432
18/12/11 17:46:01 DEBUG s3a.S3AFileSystem: Value of fs.s3a.readahead.range is 65536
18/12/11 17:46:01 DEBUG s3a.S3AFileSystem: Value of fs.s3a.max.total.tasks is 5
18/12/11 17:46:01 DEBUG s3a.S3AFileSystem: Value of fs.s3a.threads.keepalivetime is 60
18/12/11 17:46:01 DEBUG s3.AmazonS3Client: Bucket region cache doesn't have an entry for myclouderaraj. Trying to get bucket region from Amazon S3.
18/12/11 17:46:01 DEBUG s3a.AWSCredentialProviderList: Using credentials from BasicAWSCredentialsProvider
18/12/11 17:46:01 DEBUG amazonaws.request: Sending Request: HEAD https://myclouderaraj.s3.amazonaws.com / Headers: (User-Agent: Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161, amz-sdk-invocation-id: 9c0b91f3-3175-fc6a-3669-b075a6a89b9a, Content-Type: application/octet-stream, )
18/12/11 17:46:01 DEBUG auth.AWS4Signer: AWS4 Canonical Request: '"HEAD
/

amz-sdk-invocation-id:9c0b91f3-3175-fc6a-3669-b075a6a89b9a
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:myclouderaraj.s3.amazonaws.com
user-agent:Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20181211T164601Z

amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
UNSIGNED-PAYLOAD"
18/12/11 17:46:01 DEBUG auth.AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20181211T164601Z
20181211/us-east-1/s3/aws4_request
767662b5323a076b055a71d9bf21e1b437c5791f56d0efcafab5311d6e510cfd"
18/12/11 17:46:01 DEBUG auth.AWS4Signer: Generating a new signing key as the signing key not available in the cache for the date 1544486400000
18/12/11 17:46:02 DEBUG protocol.RequestAddCookies: CookieSpec selected: default
18/12/11 17:46:02 DEBUG protocol.RequestAuthCache: Auth cache not set in the context
18/12/11 17:46:02 DEBUG conn.PoolingHttpClientConnectionManager: Connection request: [route: {s}->https://myclouderaraj.s3.amazonaws.com:443][total kept alive: 0; route allocated: 0 of 100; total allocated: 0 of 100]
18/12/11 17:46:02 DEBUG conn.PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com:443][total kept alive: 0; route allocated: 1 of 100; total allocated: 1 of 100]
18/12/11 17:46:02 DEBUG execchain.MainClientExec: Opening connection {s}->https://myclouderaraj.s3.amazonaws.com:443
18/12/11 17:46:02 DEBUG conn.DefaultHttpClientConnectionOperator: Connecting to myclouderaraj.s3.amazonaws.com/52.217.1.140:443
18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory: connecting to myclouderaraj.s3.amazonaws.com/52.217.1.140:443
18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory: Connecting socket to myclouderaraj.s3.amazonaws.com/52.217.1.140:443 with timeout 5000
18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory: Enabled protocols: [TLSv1, TLSv1.1, TLSv1.2]
18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory: Enabled cipher suites:[TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory: socket.getSupportedProtocols(): [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2], socket.getEnabledProtocols(): [TLSv1, TLSv1.1, TLSv1.2]
18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory: TLS protocol enabled for SSL handshake: [TLSv1.2, TLSv1.1, TLSv1]
18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory: Starting handshake
18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory: Secure session established
18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory:  negotiated protocol: TLSv1.2
18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory:  negotiated cipher suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory:  peer principal: CN=*.s3.amazonaws.com, O=Amazon.com Inc., L=Seattle, ST=Washington, C=US
18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory:  peer alternative names: [*.s3.amazonaws.com, s3.amazonaws.com]
18/12/11 17:46:02 DEBUG ssl.SdkTLSSocketFactory:  issuer principal: CN=DigiCert Baltimore CA-2 G2, OU=www.digicert.com, O=DigiCert Inc, C=US
18/12/11 17:46:03 DEBUG internal.SdkSSLSocket: created: myclouderaraj.s3.amazonaws.com/52.217.1.140:443
18/12/11 17:46:03 DEBUG conn.DefaultHttpClientConnectionOperator: Connection established 10.33.6.155:38019<->52.217.1.140:443
18/12/11 17:46:03 DEBUG conn.DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
18/12/11 17:46:03 DEBUG execchain.MainClientExec: Executing request HEAD / HTTP/1.1
18/12/11 17:46:03 DEBUG execchain.MainClientExec: Proxy auth state: UNCHALLENGED
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> HEAD / HTTP/1.1
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Host: myclouderaraj.s3.amazonaws.com
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> x-amz-content-sha256: UNSIGNED-PAYLOAD
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=AKIAIYFZQ7LF7F5MJL4Q/20181211/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=bc733f2ab05759751c4eeaf11095f286eceba2647ee8b014211cab2e03f967db
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> X-Amz-Date: 20181211T164601Z
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> User-Agent: Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> amz-sdk-invocation-id: 9c0b91f3-3175-fc6a-3669-b075a6a89b9a
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Content-Type: application/octet-stream
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Connection: Keep-Alive
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "HEAD / HTTP/1.1[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Host: myclouderaraj.s3.amazonaws.com[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "x-amz-content-sha256: UNSIGNED-PAYLOAD[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=AKIAIYFZQ7LF7F5MJL4Q/20181211/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=bc733f2ab05759751c4eeaf11095f286eceba2647ee8b014211cab2e03f967db[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "X-Amz-Date: 20181211T164601Z[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "User-Agent: Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "amz-sdk-invocation-id: 9c0b91f3-3175-fc6a-3669-b075a6a89b9a[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "HTTP/1.1 403 Forbidden[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "x-amz-bucket-region: us-east-1[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "x-amz-request-id: 93DABB98D9C20FB2[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "x-amz-id-2: GImfViIrJZLJvNPS0MGpxZSfsmJqibcEtOLcBK86sdCo79CpRHxlKQhANkddSqVMPIIBtIGEZho=[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "Transfer-Encoding: chunked[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "Date: Tue, 11 Dec 2018 17:08:34 GMT[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "Server: AmazonS3[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "[\r][\n]"
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << HTTP/1.1 403 Forbidden
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << x-amz-bucket-region: us-east-1
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << x-amz-request-id: 93DABB98D9C20FB2
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << x-amz-id-2: GImfViIrJZLJvNPS0MGpxZSfsmJqibcEtOLcBK86sdCo79CpRHxlKQhANkddSqVMPIIBtIGEZho=
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << Content-Type: application/xml
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << Transfer-Encoding: chunked
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << Date: Tue, 11 Dec 2018 17:08:34 GMT
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << Server: AmazonS3
18/12/11 17:46:03 DEBUG execchain.MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
18/12/11 17:46:03 DEBUG conn.PoolingHttpClientConnectionManager: Connection [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com:443] can be kept alive for 60.0 seconds
18/12/11 17:46:03 DEBUG conn.PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com:443][total kept alive: 1; route allocated: 1 of 100; total allocated: 1 of 100]
18/12/11 17:46:03 DEBUG amazonaws.request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 93DABB98D9C20FB2), S3 Extended Request ID: GImfViIrJZLJvNPS0MGpxZSfsmJqibcEtOLcBK86sdCo79CpRHxlKQhANkddSqVMPIIBtIGEZho=
18/12/11 17:46:03 DEBUG s3.AmazonS3Client: Region for myclouderaraj is us-east-1
18/12/11 17:46:03 DEBUG amazonaws.request: Sending Request: HEAD https://myclouderaraj.s3.amazonaws.com / Headers: (User-Agent: Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161, amz-sdk-invocation-id: fa6a15f3-60e0-0dd4-85de-ce71e5864550, Content-Type: application/octet-stream, )
18/12/11 17:46:03 DEBUG auth.AWS4Signer: AWS4 Canonical Request: '"HEAD
/

amz-sdk-invocation-id:fa6a15f3-60e0-0dd4-85de-ce71e5864550
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:myclouderaraj.s3.amazonaws.com
user-agent:Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20181211T164603Z

amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
UNSIGNED-PAYLOAD"
18/12/11 17:46:03 DEBUG auth.AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20181211T164603Z
20181211/us-east-1/s3/aws4_request
7af1674e22b74abaf21c562903716022a599fb7ae7c2cd46b493450627c84858"
18/12/11 17:46:03 DEBUG protocol.RequestAddCookies: CookieSpec selected: default
18/12/11 17:46:03 DEBUG protocol.RequestAuthCache: Auth cache not set in the context
18/12/11 17:46:03 DEBUG conn.PoolingHttpClientConnectionManager: Connection request: [route: {s}->https://myclouderaraj.s3.amazonaws.com:443][total kept alive: 1; route allocated: 1 of 100; total allocated: 1 of 100]
18/12/11 17:46:03 DEBUG conn.PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com:443][total kept alive: 0; route allocated: 1 of 100; total allocated: 1 of 100]
18/12/11 17:46:03 DEBUG conn.DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
18/12/11 17:46:03 DEBUG execchain.MainClientExec: Executing request HEAD / HTTP/1.1
18/12/11 17:46:03 DEBUG execchain.MainClientExec: Proxy auth state: UNCHALLENGED
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> HEAD / HTTP/1.1
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Host: myclouderaraj.s3.amazonaws.com
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> x-amz-content-sha256: UNSIGNED-PAYLOAD
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=AKIAIYFZQ7LF7F5MJL4Q/20181211/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=6e8a6bbf0e44be59a0894e30c4645049de3f3c8e2c774077a081885702236aaa
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> X-Amz-Date: 20181211T164603Z
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> User-Agent: Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> amz-sdk-invocation-id: fa6a15f3-60e0-0dd4-85de-ce71e5864550
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Content-Type: application/octet-stream
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Connection: Keep-Alive
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "HEAD / HTTP/1.1[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Host: myclouderaraj.s3.amazonaws.com[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "x-amz-content-sha256: UNSIGNED-PAYLOAD[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=AKIAIYFZQ7LF7F5MJL4Q/20181211/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=6e8a6bbf0e44be59a0894e30c4645049de3f3c8e2c774077a081885702236aaa[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "X-Amz-Date: 20181211T164603Z[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "User-Agent: Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "amz-sdk-invocation-id: fa6a15f3-60e0-0dd4-85de-ce71e5864550[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "HTTP/1.1 403 Forbidden[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "x-amz-bucket-region: us-east-1[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "x-amz-request-id: 0D4E622D9A858BD3[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "x-amz-id-2: dwOjYsPPO7Hn7MlUZPAZ6VVOVIWg6gGbQ0i9JY0YRfLnOV+GEhagAsXJuB1S0dt4bgo4VwZzpYk=[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "Transfer-Encoding: chunked[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "Date: Tue, 11 Dec 2018 17:08:35 GMT[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "Server: AmazonS3[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "[\r][\n]"
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << HTTP/1.1 403 Forbidden
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << x-amz-bucket-region: us-east-1
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << x-amz-request-id: 0D4E622D9A858BD3
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << x-amz-id-2: dwOjYsPPO7Hn7MlUZPAZ6VVOVIWg6gGbQ0i9JY0YRfLnOV+GEhagAsXJuB1S0dt4bgo4VwZzpYk=
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << Content-Type: application/xml
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << Transfer-Encoding: chunked
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << Date: Tue, 11 Dec 2018 17:08:35 GMT
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << Server: AmazonS3
18/12/11 17:46:03 DEBUG execchain.MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
18/12/11 17:46:03 DEBUG conn.PoolingHttpClientConnectionManager: Connection [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com:443] can be kept alive for 60.0 seconds
18/12/11 17:46:03 DEBUG conn.PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com:443][total kept alive: 1; route allocated: 1 of 100; total allocated: 1 of 100]
18/12/11 17:46:03 DEBUG amazonaws.request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 0D4E622D9A858BD3), S3 Extended Request ID: dwOjYsPPO7Hn7MlUZPAZ6VVOVIWg6gGbQ0i9JY0YRfLnOV+GEhagAsXJuB1S0dt4bgo4VwZzpYk=
18/12/11 17:46:03 DEBUG s3a.S3AFileSystem: Value of fs.s3a.multipart.purge.age is 86400
18/12/11 17:46:03 INFO Configuration.deprecation: fs.s3a.server-side-encryption-key is deprecated. Instead, use fs.s3a.server-side-encryption.key
18/12/11 17:46:03 DEBUG s3a.S3AFileSystem: Data is unencrypted
18/12/11 17:46:03 DEBUG s3a.S3AFileSystem: Using SSE-C with null key
18/12/11 17:46:03 DEBUG s3a.S3AFileSystem: Using S3AOutputStream
18/12/11 17:46:03 DEBUG s3guard.S3Guard: Using NullMetadataStore metadata store for s3a filesystem
18/12/11 17:46:03 DEBUG s3a.S3AFileSystem: op_glob_status += 1  ->  1
18/12/11 17:46:03 DEBUG s3a.S3AFileSystem: op_get_file_status += 1  ->  1
18/12/11 17:46:03 DEBUG s3a.S3AFileSystem: Getting path status for s3a://myclouderaraj/root  (root)
18/12/11 17:46:03 DEBUG s3a.S3AFileSystem: object_metadata_requests += 1  ->  1
18/12/11 17:46:03 DEBUG s3.AmazonS3Client: Region for myclouderaraj is us-east-1
18/12/11 17:46:03 DEBUG amazonaws.request: Sending Request: HEAD https://myclouderaraj.s3.amazonaws.com /root Headers: (User-Agent: Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161, amz-sdk-invocation-id: 1777bc11-f8c0-c825-58e6-113912bcd9cd, Content-Type: application/octet-stream, )
18/12/11 17:46:03 DEBUG auth.AWS4Signer: AWS4 Canonical Request: '"HEAD
/root

amz-sdk-invocation-id:1777bc11-f8c0-c825-58e6-113912bcd9cd
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:myclouderaraj.s3.amazonaws.com
user-agent:Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20181211T164603Z

amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
UNSIGNED-PAYLOAD"
18/12/11 17:46:03 DEBUG auth.AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20181211T164603Z
20181211/us-east-1/s3/aws4_request
ed1bf2a929e630f833c3eb1f6639dbcb84723d570e76ed41cc99ddbc8b33d3a1"
18/12/11 17:46:03 DEBUG protocol.RequestAddCookies: CookieSpec selected: default
18/12/11 17:46:03 DEBUG protocol.RequestAuthCache: Auth cache not set in the context
18/12/11 17:46:03 DEBUG conn.PoolingHttpClientConnectionManager: Connection request: [route: {s}->https://myclouderaraj.s3.amazonaws.com:443][total kept alive: 1; route allocated: 1 of 100; total allocated: 1 of 100]
18/12/11 17:46:03 DEBUG conn.PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com:443][total kept alive: 0; route allocated: 1 of 100; total allocated: 1 of 100]
18/12/11 17:46:03 DEBUG conn.DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
18/12/11 17:46:03 DEBUG execchain.MainClientExec: Executing request HEAD /root HTTP/1.1
18/12/11 17:46:03 DEBUG execchain.MainClientExec: Proxy auth state: UNCHALLENGED
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> HEAD /root HTTP/1.1
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Host: myclouderaraj.s3.amazonaws.com
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> x-amz-content-sha256: UNSIGNED-PAYLOAD
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=AKIAIYFZQ7LF7F5MJL4Q/20181211/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=0e06c4f75551b83e1e122abdbf7092ae2c4413f38419d89572b77ecdc4da1d5d
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> X-Amz-Date: 20181211T164603Z
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> User-Agent: Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> amz-sdk-invocation-id: 1777bc11-f8c0-c825-58e6-113912bcd9cd
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Content-Type: application/octet-stream
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Connection: Keep-Alive
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "HEAD /root HTTP/1.1[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Host: myclouderaraj.s3.amazonaws.com[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "x-amz-content-sha256: UNSIGNED-PAYLOAD[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=AKIAIYFZQ7LF7F5MJL4Q/20181211/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=0e06c4f75551b83e1e122abdbf7092ae2c4413f38419d89572b77ecdc4da1d5d[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "X-Amz-Date: 20181211T164603Z[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "User-Agent: Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "amz-sdk-invocation-id: 1777bc11-f8c0-c825-58e6-113912bcd9cd[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "HTTP/1.1 403 Forbidden[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "x-amz-request-id: 1F1ED2B55D3DBA53[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "x-amz-id-2: oSqyv7R/B05MjjUWQflFaLoOv5Mb2ycprL+PcmqjussFP+11AiJxpKwj+MlDGrfULeumJRHkTjY=[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "Transfer-Encoding: chunked[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "Date: Tue, 11 Dec 2018 17:08:35 GMT[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "Server: AmazonS3[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "[\r][\n]"
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << HTTP/1.1 403 Forbidden
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << x-amz-request-id: 1F1ED2B55D3DBA53
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << x-amz-id-2: oSqyv7R/B05MjjUWQflFaLoOv5Mb2ycprL+PcmqjussFP+11AiJxpKwj+MlDGrfULeumJRHkTjY=
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << Content-Type: application/xml
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << Transfer-Encoding: chunked
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << Date: Tue, 11 Dec 2018 17:08:35 GMT
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << Server: AmazonS3
18/12/11 17:46:03 DEBUG execchain.MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
18/12/11 17:46:03 DEBUG conn.PoolingHttpClientConnectionManager: Connection [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com:443] can be kept alive for 60.0 seconds
18/12/11 17:46:03 DEBUG conn.PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com:443][total kept alive: 1; route allocated: 1 of 100; total allocated: 1 of 100]
18/12/11 17:46:03 DEBUG amazonaws.request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 1F1ED2B55D3DBA53), S3 Extended Request ID: oSqyv7R/B05MjjUWQflFaLoOv5Mb2ycprL+PcmqjussFP+11AiJxpKwj+MlDGrfULeumJRHkTjY=
ls: s3a://myclouderaraj/root: getFileStatus on s3a://myclouderaraj/root: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 1F1ED2B55D3DBA53), S3 Extended Request ID: oSqyv7R/B05MjjUWQflFaLoOv5Mb2ycprL+PcmqjussFP+11AiJxpKwj+MlDGrfULeumJRHkTjY=
18/12/11 17:46:03 TRACE core.TracerPool: TracerPool(Global): removing tracer Tracer(FsShell/10.33.6.155)
18/12/11 17:46:03 DEBUG http.IdleConnectionReaper: Reaper thread:
java.lang.InterruptedException: sleep interrupted
        at java.lang.Thread.sleep(Native Method)
        at com.amazonaws.http.IdleConnectionReaper.run(IdleConnectionReaper.java:181)
18/12/11 17:46:03 DEBUG conn.PoolingHttpClientConnectionManager: Connection manager is shutting down
18/12/11 17:46:03 DEBUG http.IdleConnectionReaper: Shutting down reaper thread.
18/12/11 17:46:03 DEBUG conn.DefaultManagedHttpClientConnection: http-outgoing-0: Close connection
18/12/11 17:46:03 DEBUG internal.SdkSSLSocket: shutting down output of myclouderaraj.s3.amazonaws.com/52.217.1.140:443
18/12/11 17:46:03 DEBUG internal.SdkSSLSocket: closing myclouderaraj.s3.amazonaws.com/52.217.1.140:443
18/12/11 17:46:03 DEBUG conn.PoolingHttpClientConnectionManager: Connection manager shut down
[root@quickstart ~]#

I used the same command in both the machine where one is able to connect and the other is giving this issue.

Next try:

Also, I tried to do the following operations in the working machine for impala using impala-shell

https://www.cloudera.com/documentation/enterprise/5-7-x/topics/impala_s3.html

Updated [root@quickstart ~]# cd /etc/hadoop/conf/hdfs-site.xml file for access and secret key.

Restarted all the services, also tried restarting cloudera-scm-server.

Then ran the following from impala-shell

[localhost:21000] > create database db_on_hdfs;
[localhost:21000] > use db_on_hdfs;
[localhost:21000] > create table mostly_on_hdfs (x int) partitioned by (year int);
[localhost:21000] > alter table mostly_on_hdfs add partition (year=2013);
[localhost:21000] > alter table mostly_on_hdfs add partition (year=2014);
[localhost:21000] > alter table mostly_on_hdfs add partition (year=2015) location 's3a://impala-demo/dir1/dir2/dir3/t1';

[quickstart.cloudera:21000] > alter table mostly_on_hdfs add partition (year=2015) LOCATION 's3a://myclouderaraj/root';
Query: alter table mostly_on_hdfs add partition (year=2015) LOCATION 's3a://myclouderaraj/root'
ERROR: AnalysisException: null
CAUSED BY: AmazonClientException: Unable to load AWS credentials from any provider in the chain

[quickstart.cloudera:21000] >

Can you please help.

Thansk & Regards,

Rajesh

seth · ‎12-11-2018

Rajesh,

Did you capture debug logging from each environment to see what the other sandbox env is sending to S3 and works vs what quickstart is sending and fails?

Example:

18/12/11 17:46:03 http-outgoing-0 << "x-amz-bucket-region: us-east-1[\r][\n]"
18/12/11 17:46:03 http-outgoing-0 << "x-amz-request-id: 93DABB98D9C20FB2[\r][\n]" <====
18/12/11 17:46:03 http-outgoing-0 << "x-amz-id-2: GImfViIrJZLJvNPS0MGpxZSfsmJqibcEtOLcBK86sdCo79CpRHxlKQhANkddSqVMPIIBtIGEZho=[\r][\n]"
18/12/11 17:46:03 http-outgoing-0 << HTTP/1.1 403 Forbidden

=> Request ID: 93DABB98D9C20FB2

x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20181211T164601Z
amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> x-amz-content-sha256: UNSIGNED-PAYLOAD
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=AKIAIYFZQ7LF7F5MJL4Q/20181211/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=bc733f2ab05759751c4eeaf11095f286eceba2647ee8b014211cab2e03f967db
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "x-amz-content-sha256: UNSIGNED-PAYLOAD[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=AKIAIYFZQ7LF7F5MJL4Q/20181211/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=bc733f2ab05759751c4eeaf11095f286eceba2647ee8b014211cab2e03f967db[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "HTTP/1.1 403 Forbidden[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "x-amz-bucket-region: us-east-1[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "x-amz-request-id: 93DABB98D9C20FB2[\r][\n]"
18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "x-amz-id-2: GImfViIrJZLJvNPS0MGpxZSfsmJqibcEtOLcBK86sdCo79CpRHxlKQhANkddSqVMPIIBtIGEZho=[\r][\n]"
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << HTTP/1.1 403 Forbidden
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << x-amz-bucket-region: us-east-1
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << x-amz-request-id: 93DABB98D9C20FB2
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << x-amz-id-2: GImfViIrJZLJvNPS0MGpxZSfsmJqibcEtOLcBK86sdCo79CpRHxlKQhANkddSqVMPIIBtIGEZho=
18/12/11 17:46:03 DEBUG amazonaws.request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 93DABB98D9C20FB2), S3 Extended Request ID: GImfViIrJZLJvNPS0MGpxZSfsmJqibcEtOLcBK86sdCo79CpRHxlKQhANkddSqVMPIIBtIGEZho=

Nukala · ‎12-12-2018

Seth,

This is the success log

[root@quickstart ~]# hdfs dfs -Dfs.s3a.access.key=myaccesskey -Dfs.s3a.secret.key=mysecretkey -ls s3a://myclouderaraj/root
18/12/11 19:09:45 DEBUG util.Shell: setsid exited with exit code 0
18/12/11 19:10:10 DEBUG conf.Configuration: parsing URL jar:file:/usr/jars/hadoop-common-2.6.0-cdh5.7.0.jar!/core-default.xml
18/12/11 19:10:10 DEBUG conf.Configuration: parsing input stream sun.net.www.protocol.jar.JarURLConnection$JarURLInputStream@ea1a8d5
18/12/11 19:11:30 DEBUG conf.Configuration: parsing URL file:/etc/hadoop/conf.cloudera.yarn/core-site.xml
18/12/11 19:11:31 DEBUG conf.Configuration: parsing input stream java.io.BufferedInputStream@fcd6521
18/12/11 19:12:34 DEBUG core.Tracer: sampler.classes = ; loaded no samplers
18/12/11 19:12:35 TRACE core.TracerId: ProcessID(fmt=%{tname}/%{ip}): computed process ID of "FsShell/10.33.6.146"
18/12/11 19:12:35 TRACE core.TracerPool: TracerPool(Global): adding tracer Tracer(FsShell/10.33.6.146)
18/12/11 19:12:35 DEBUG core.Tracer: span.receiver.classes = ; loaded no span receivers
18/12/11 19:12:35 TRACE core.Tracer: Created Tracer(FsShell/10.33.6.146) for FsShell
18/12/11 19:12:37 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of successful kerberos logins and latency (milliseconds)])
18/12/11 19:12:37 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of failed kerberos logins and latency (milliseconds)])
18/12/11 19:12:37 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[GetGroups])
18/12/11 19:12:37 DEBUG impl.MetricsSystemImpl: UgiMetrics, User and group related metrics
Java config name: null
Native config name: /etc/krb5.conf
Loaded from native config
18/12/11 19:12:38 DEBUG security.Groups: Creating new Groups object
18/12/11 19:12:38 DEBUG security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000; warningDeltaMs=5000
[UnixLoginModule]: succeeded importing info:
uid = 0
gid = 0
supp gid = 0
18/12/11 19:12:38 DEBUG security.UserGroupInformation: hadoop login
[UnixLoginModule]: added UnixPrincipal,
UnixNumericUserPrincipal,
UnixNumericGroupPrincipal(s),
to Subject
18/12/11 19:12:38 DEBUG security.UserGroupInformation: hadoop login commit
18/12/11 19:12:38 DEBUG security.UserGroupInformation: using local user:UnixPrincipal: root
18/12/11 19:12:38 DEBUG security.UserGroupInformation: Using user: "UnixPrincipal: root" with name root
18/12/11 19:12:38 DEBUG security.UserGroupInformation: User entry: "root"
18/12/11 19:12:38 DEBUG security.UserGroupInformation: UGI loginUser:root (auth:SIMPLE)
18/12/11 19:12:38 DEBUG core.Tracer: sampler.classes = ; loaded no samplers
18/12/11 19:12:38 TRACE core.TracerId: ProcessID(fmt=%{tname}/%{ip}): computed process ID of "FSClient/10.33.6.146"
18/12/11 19:12:38 TRACE core.TracerPool: TracerPool(Global): adding tracer Tracer(FSClient/10.33.6.146)
18/12/11 19:12:38 DEBUG core.Tracer: span.receiver.classes = ; loaded no span receivers
18/12/11 19:12:38 TRACE core.Tracer: Created Tracer(FSClient/10.33.6.146) for FSClient
18/12/11 19:12:40 DEBUG azure.NativeAzureFileSystem: finalize() called.
18/12/11 19:12:40 DEBUG azure.NativeAzureFileSystem: finalize() called.
18/12/11 19:12:41 DEBUG amazonaws.AmazonWebServiceClient: Internal logging succesfully configured to commons logger: true
18/12/11 19:12:44 DEBUG metrics.AwsSdkMetrics: Admin mbean registered under com.cloudera.com.amazonaws.management:type=AwsSdkMetrics
18/12/11 19:12:50 DEBUG auth.AWSCredentialsProviderChain: Loading credentials from BasicAWSCredentialsProvider
18/12/11 19:12:50 DEBUG amazonaws.request: Sending Request: HEAD https://myclouderaraj.s3.amazonaws.com / Headers: (User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192, Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
18/12/11 19:12:51 DEBUG internal.S3Signer: Calculated string to sign:
"HEAD

application/x-www-form-urlencoded; charset=utf-8
Wed, 12 Dec 2018 03:12:50 GMT
/myclouderaraj/"
18/12/11 19:12:51 DEBUG conn.PoolingClientConnectionManager: Connection request: [route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 0; route allocated: 0 of 15; total allocated: 0 of 15]
18/12/11 19:12:51 DEBUG conn.PoolingClientConnectionManager: Connection leased: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 0; route allocated: 1 of 15; total allocated: 1 of 15]
18/12/11 19:12:51 DEBUG ssl.SdkTLSSocketFactory: socket.getSupportedProtocols(): [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2], socket.getEnabledProtocols(): [TLSv1, TLSv1.1, TLSv1.2]
18/12/11 19:12:51 DEBUG ssl.SdkTLSSocketFactory: TLS protocol enabled for SSL handshake: [TLSv1.2, TLSv1.1, TLSv1]
18/12/11 19:12:51 DEBUG conn.DefaultClientConnectionOperator: Connecting to myclouderaraj.s3.amazonaws.com:443
18/12/11 19:12:51 DEBUG ssl.SdkTLSSocketFactory: connecting to myclouderaraj.s3.amazonaws.com/52.216.107.100:443
18/12/11 19:12:53 DEBUG internal.SdkSSLSocket: created: myclouderaraj.s3.amazonaws.com/52.216.107.100:443
18/12/11 19:12:53 DEBUG protocol.RequestAddCookies: CookieSpec selected: best-match
18/12/11 19:12:53 DEBUG protocol.RequestAuthCache: Auth cache not set in the context
18/12/11 19:12:53 DEBUG protocol.RequestProxyAuthentication: Proxy auth state: UNCHALLENGED
18/12/11 19:12:53 DEBUG client.SdkHttpClient: Attempt 1 to execute request
18/12/11 19:12:53 DEBUG conn.DefaultClientConnection: Sending request: HEAD / HTTP/1.1
18/12/11 19:12:53 DEBUG http.wire: >> "HEAD / HTTP/1.1[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: >> "Host: myclouderaraj.s3.amazonaws.com[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: >> "Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:XU3AFfGphmQy0rEhRrWjajyeFbA=[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: >> "User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: >> "Date: Wed, 12 Dec 2018 03:12:50 GMT[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: >> "Content-Type: application/x-www-form-urlencoded; charset=utf-8[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: >> "Connection: Keep-Alive[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: >> "[\r][\n]"
18/12/11 19:12:53 DEBUG http.headers: >> HEAD / HTTP/1.1
18/12/11 19:12:53 DEBUG http.headers: >> Host: myclouderaraj.s3.amazonaws.com
18/12/11 19:12:53 DEBUG http.headers: >> Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:XU3AFfGphmQy0rEhRrWjajyeFbA=
18/12/11 19:12:53 DEBUG http.headers: >> User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192
18/12/11 19:12:53 DEBUG http.headers: >> Date: Wed, 12 Dec 2018 03:12:50 GMT
18/12/11 19:12:53 DEBUG http.headers: >> Content-Type: application/x-www-form-urlencoded; charset=utf-8
18/12/11 19:12:53 DEBUG http.headers: >> Connection: Keep-Alive
18/12/11 19:12:53 DEBUG http.wire: << "HTTP/1.1 200 OK[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: << "x-amz-id-2: sjybfZSJELaf9eW/QJHWV/FcObyfD5ElzagJj8SqLiPwVogmvp1lih+csHvi73bNoHKuS/OLqSI=[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: << "x-amz-request-id: 03AD0786BC412E7C[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: << "Date: Wed, 12 Dec 2018 03:12:54 GMT[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: << "x-amz-bucket-region: us-east-1[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: << "Content-Type: application/xml[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: << "Transfer-Encoding: chunked[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: << "Server: AmazonS3[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: << "[\r][\n]"
18/12/11 19:12:53 DEBUG conn.DefaultClientConnection: Receiving response: HTTP/1.1 200 OK
18/12/11 19:12:53 DEBUG http.headers: << HTTP/1.1 200 OK
18/12/11 19:12:53 DEBUG http.headers: << x-amz-id-2: sjybfZSJELaf9eW/QJHWV/FcObyfD5ElzagJj8SqLiPwVogmvp1lih+csHvi73bNoHKuS/OLqSI=
18/12/11 19:12:53 DEBUG http.headers: << x-amz-request-id: 03AD0786BC412E7C
18/12/11 19:12:53 DEBUG http.headers: << Date: Wed, 12 Dec 2018 03:12:54 GMT
18/12/11 19:12:53 DEBUG http.headers: << x-amz-bucket-region: us-east-1
18/12/11 19:12:53 DEBUG http.headers: << Content-Type: application/xml
18/12/11 19:12:53 DEBUG http.headers: << Transfer-Encoding: chunked
18/12/11 19:12:53 DEBUG http.headers: << Server: AmazonS3
18/12/11 19:12:53 DEBUG client.SdkHttpClient: Connection can be kept alive indefinitely
18/12/11 19:12:53 DEBUG conn.PoolingClientConnectionManager: Connection [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com] can be kept alive indefinitely
18/12/11 19:12:53 DEBUG conn.PoolingClientConnectionManager: Connection released: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15]
18/12/11 19:12:53 DEBUG amazonaws.requestId: x-amzn-RequestId: not available
18/12/11 19:12:53 DEBUG amazonaws.request: Received successful response: 200, AWS Request ID: 03AD0786BC412E7C
18/12/11 19:12:53 DEBUG amazonaws.requestId: AWS Request ID: 03AD0786BC412E7C
18/12/11 19:12:53 DEBUG s3a.S3AFileSystem: Getting path status for s3a://myclouderaraj/root (root)
18/12/11 19:12:53 DEBUG amazonaws.request: Sending Request: HEAD https://myclouderaraj.s3.amazonaws.com /root Headers: (User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192, Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
18/12/11 19:12:53 DEBUG internal.S3Signer: Calculated string to sign:
"HEAD

application/x-www-form-urlencoded; charset=utf-8
Wed, 12 Dec 2018 03:12:53 GMT
/myclouderaraj/root"
18/12/11 19:12:53 DEBUG conn.PoolingClientConnectionManager: Connection request: [route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15]
18/12/11 19:12:53 DEBUG conn.PoolingClientConnectionManager: Connection leased: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 0; route allocated: 1 of 15; total allocated: 1 of 15]
18/12/11 19:12:53 DEBUG client.SdkHttpClient: Stale connection check
18/12/11 19:12:53 DEBUG protocol.RequestAddCookies: CookieSpec selected: best-match
18/12/11 19:12:53 DEBUG protocol.RequestAuthCache: Auth cache not set in the context
18/12/11 19:12:53 DEBUG protocol.RequestProxyAuthentication: Proxy auth state: UNCHALLENGED
18/12/11 19:12:53 DEBUG client.SdkHttpClient: Attempt 1 to execute request
18/12/11 19:12:53 DEBUG conn.DefaultClientConnection: Sending request: HEAD /root HTTP/1.1
18/12/11 19:12:53 DEBUG http.wire: >> "HEAD /root HTTP/1.1[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: >> "Host: myclouderaraj.s3.amazonaws.com[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: >> "Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:SLVFDzdTaXiUPcomkZCGPdiXP+I=[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: >> "User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: >> "Date: Wed, 12 Dec 2018 03:12:53 GMT[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: >> "Content-Type: application/x-www-form-urlencoded; charset=utf-8[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: >> "Connection: Keep-Alive[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: >> "[\r][\n]"
18/12/11 19:12:53 DEBUG http.headers: >> HEAD /root HTTP/1.1
18/12/11 19:12:53 DEBUG http.headers: >> Host: myclouderaraj.s3.amazonaws.com
18/12/11 19:12:53 DEBUG http.headers: >> Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:SLVFDzdTaXiUPcomkZCGPdiXP+I=
18/12/11 19:12:53 DEBUG http.headers: >> User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192
18/12/11 19:12:53 DEBUG http.headers: >> Date: Wed, 12 Dec 2018 03:12:53 GMT
18/12/11 19:12:53 DEBUG http.headers: >> Content-Type: application/x-www-form-urlencoded; charset=utf-8
18/12/11 19:12:53 DEBUG http.headers: >> Connection: Keep-Alive
18/12/11 19:12:53 DEBUG http.wire: << "HTTP/1.1 404 Not Found[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: << "x-amz-request-id: 562A66F35BB7E4A1[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: << "x-amz-id-2: FC+MjpOY8O1AM4iHk4leLkFYxsqUqsOGka5QBXub98ve+SirWRY2PXH8778eWyv9UUFbFwVn0LQ=[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: << "Content-Type: application/xml[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: << "Transfer-Encoding: chunked[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: << "Date: Wed, 12 Dec 2018 03:12:53 GMT[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: << "Server: AmazonS3[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: << "[\r][\n]"
18/12/11 19:12:53 DEBUG conn.DefaultClientConnection: Receiving response: HTTP/1.1 404 Not Found
18/12/11 19:12:53 DEBUG http.headers: << HTTP/1.1 404 Not Found
18/12/11 19:12:53 DEBUG http.headers: << x-amz-request-id: 562A66F35BB7E4A1
18/12/11 19:12:53 DEBUG http.headers: << x-amz-id-2: FC+MjpOY8O1AM4iHk4leLkFYxsqUqsOGka5QBXub98ve+SirWRY2PXH8778eWyv9UUFbFwVn0LQ=
18/12/11 19:12:53 DEBUG http.headers: << Content-Type: application/xml
18/12/11 19:12:53 DEBUG http.headers: << Transfer-Encoding: chunked
18/12/11 19:12:53 DEBUG http.headers: << Date: Wed, 12 Dec 2018 03:12:53 GMT
18/12/11 19:12:53 DEBUG http.headers: << Server: AmazonS3
18/12/11 19:12:53 DEBUG client.SdkHttpClient: Connection can be kept alive indefinitely
18/12/11 19:12:53 DEBUG conn.PoolingClientConnectionManager: Connection [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com] can be kept alive indefinitely
18/12/11 19:12:53 DEBUG conn.PoolingClientConnectionManager: Connection released: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15]
18/12/11 19:12:53 DEBUG amazonaws.requestId: x-amzn-RequestId: not available
18/12/11 19:12:53 DEBUG amazonaws.request: Received error response: com.cloudera.com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: null; Status Code: 404; Error Code: 404 Not Found; Request ID: 562A66F35BB7E4A1), S3 Extended Request ID: FC+MjpOY8O1AM4iHk4leLkFYxsqUqsOGka5QBXub98ve+SirWRY2PXH8778eWyv9UUFbFwVn0LQ=
18/12/11 19:12:53 DEBUG amazonaws.request: Sending Request: HEAD https://myclouderaraj.s3.amazonaws.com /root/ Headers: (User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192, Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
18/12/11 19:12:53 DEBUG internal.S3Signer: Calculated string to sign:
"HEAD

application/x-www-form-urlencoded; charset=utf-8
Wed, 12 Dec 2018 03:12:53 GMT
/myclouderaraj/root/"
18/12/11 19:12:53 DEBUG conn.PoolingClientConnectionManager: Connection request: [route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15]
18/12/11 19:12:53 DEBUG conn.PoolingClientConnectionManager: Connection leased: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 0; route allocated: 1 of 15; total allocated: 1 of 15]
18/12/11 19:12:53 DEBUG client.SdkHttpClient: Stale connection check
18/12/11 19:12:53 DEBUG protocol.RequestAddCookies: CookieSpec selected: best-match
18/12/11 19:12:53 DEBUG protocol.RequestAuthCache: Auth cache not set in the context
18/12/11 19:12:53 DEBUG protocol.RequestProxyAuthentication: Proxy auth state: UNCHALLENGED
18/12/11 19:12:53 DEBUG client.SdkHttpClient: Attempt 1 to execute request
18/12/11 19:12:53 DEBUG conn.DefaultClientConnection: Sending request: HEAD /root/ HTTP/1.1
18/12/11 19:12:53 DEBUG http.wire: >> "HEAD /root/ HTTP/1.1[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: >> "Host: myclouderaraj.s3.amazonaws.com[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: >> "Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:nWk9yWYYe8oEEGtfxRoqQmfgSK0=[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: >> "User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: >> "Date: Wed, 12 Dec 2018 03:12:53 GMT[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: >> "Content-Type: application/x-www-form-urlencoded; charset=utf-8[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: >> "Connection: Keep-Alive[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: >> "[\r][\n]"
18/12/11 19:12:53 DEBUG http.headers: >> HEAD /root/ HTTP/1.1
18/12/11 19:12:53 DEBUG http.headers: >> Host: myclouderaraj.s3.amazonaws.com
18/12/11 19:12:53 DEBUG http.headers: >> Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:nWk9yWYYe8oEEGtfxRoqQmfgSK0=
18/12/11 19:12:53 DEBUG http.headers: >> User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192
18/12/11 19:12:53 DEBUG http.headers: >> Date: Wed, 12 Dec 2018 03:12:53 GMT
18/12/11 19:12:53 DEBUG http.headers: >> Content-Type: application/x-www-form-urlencoded; charset=utf-8
18/12/11 19:12:53 DEBUG http.headers: >> Connection: Keep-Alive
18/12/11 19:12:53 DEBUG http.wire: << "HTTP/1.1 404 Not Found[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: << "x-amz-request-id: C5C8989A67B509AC[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: << "x-amz-id-2: 19gaP2XtwSoU9Bl1Y+jO3A8rPAr+v00XbdZQDD08v6u3XJKiB3uRfLSF+E4vvivzeiwCK74Vy+I=[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: << "Content-Type: application/xml[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: << "Transfer-Encoding: chunked[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: << "Date: Wed, 12 Dec 2018 03:12:53 GMT[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: << "Server: AmazonS3[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: << "[\r][\n]"
18/12/11 19:12:53 DEBUG conn.DefaultClientConnection: Receiving response: HTTP/1.1 404 Not Found
18/12/11 19:12:53 DEBUG http.headers: << HTTP/1.1 404 Not Found
18/12/11 19:12:53 DEBUG http.headers: << x-amz-request-id: C5C8989A67B509AC
18/12/11 19:12:53 DEBUG http.headers: << x-amz-id-2: 19gaP2XtwSoU9Bl1Y+jO3A8rPAr+v00XbdZQDD08v6u3XJKiB3uRfLSF+E4vvivzeiwCK74Vy+I=
18/12/11 19:12:53 DEBUG http.headers: << Content-Type: application/xml
18/12/11 19:12:53 DEBUG http.headers: << Transfer-Encoding: chunked
18/12/11 19:12:53 DEBUG http.headers: << Date: Wed, 12 Dec 2018 03:12:53 GMT
18/12/11 19:12:53 DEBUG http.headers: << Server: AmazonS3
18/12/11 19:12:53 DEBUG client.SdkHttpClient: Connection can be kept alive indefinitely
18/12/11 19:12:53 DEBUG conn.PoolingClientConnectionManager: Connection [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com] can be kept alive indefinitely
18/12/11 19:12:53 DEBUG conn.PoolingClientConnectionManager: Connection released: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15]
18/12/11 19:12:53 DEBUG amazonaws.requestId: x-amzn-RequestId: not available
18/12/11 19:12:53 DEBUG amazonaws.request: Received error response: com.cloudera.com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: null; Status Code: 404; Error Code: 404 Not Found; Request ID: C5C8989A67B509AC), S3 Extended Request ID: 19gaP2XtwSoU9Bl1Y+jO3A8rPAr+v00XbdZQDD08v6u3XJKiB3uRfLSF+E4vvivzeiwCK74Vy+I=
18/12/11 19:12:53 DEBUG amazonaws.request: Sending Request: GET https://myclouderaraj.s3.amazonaws.com / Parameters: ({"prefix":["root/"],"delimiter":["/"],"max-keys":["1"]}Headers: (User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192, Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
18/12/11 19:12:53 DEBUG internal.S3Signer: Calculated string to sign:
"GET

application/x-www-form-urlencoded; charset=utf-8
Wed, 12 Dec 2018 03:12:53 GMT
/myclouderaraj/"
18/12/11 19:12:53 DEBUG conn.PoolingClientConnectionManager: Connection request: [route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15]
18/12/11 19:12:53 DEBUG conn.PoolingClientConnectionManager: Connection leased: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 0; route allocated: 1 of 15; total allocated: 1 of 15]
18/12/11 19:12:53 DEBUG client.SdkHttpClient: Stale connection check
18/12/11 19:12:53 DEBUG protocol.RequestAddCookies: CookieSpec selected: best-match
18/12/11 19:12:53 DEBUG protocol.RequestAuthCache: Auth cache not set in the context
18/12/11 19:12:53 DEBUG protocol.RequestProxyAuthentication: Proxy auth state: UNCHALLENGED
18/12/11 19:12:53 DEBUG client.SdkHttpClient: Attempt 1 to execute request
18/12/11 19:12:53 DEBUG conn.DefaultClientConnection: Sending request: GET /?prefix=root%2F&delimiter=%2F&max-keys=1 HTTP/1.1
18/12/11 19:12:53 DEBUG http.wire: >> "GET /?prefix=root%2F&delimiter=%2F&max-keys=1 HTTP/1.1[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: >> "Host: myclouderaraj.s3.amazonaws.com[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: >> "Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:VEhMOYpTlsvVdG9Q0F7ehtOuxyg=[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: >> "User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: >> "Date: Wed, 12 Dec 2018 03:12:53 GMT[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: >> "Content-Type: application/x-www-form-urlencoded; charset=utf-8[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: >> "Connection: Keep-Alive[\r][\n]"
18/12/11 19:12:53 DEBUG http.wire: >> "[\r][\n]"
18/12/11 19:12:53 DEBUG http.headers: >> GET /?prefix=root%2F&delimiter=%2F&max-keys=1 HTTP/1.1
18/12/11 19:12:53 DEBUG http.headers: >> Host: myclouderaraj.s3.amazonaws.com
18/12/11 19:12:53 DEBUG http.headers: >> Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:VEhMOYpTlsvVdG9Q0F7ehtOuxyg=
18/12/11 19:12:53 DEBUG http.headers: >> User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192
18/12/11 19:12:53 DEBUG http.headers: >> Date: Wed, 12 Dec 2018 03:12:53 GMT
18/12/11 19:12:53 DEBUG http.headers: >> Content-Type: application/x-www-form-urlencoded; charset=utf-8
18/12/11 19:12:53 DEBUG http.headers: >> Connection: Keep-Alive
18/12/11 19:12:54 DEBUG http.wire: << "HTTP/1.1 200 OK[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "x-amz-id-2: fcNfT1n+Dhn7lPqVGiKhrTq36arxsAAG1QNTFjpiJ+O8vSqBUumwNFw7Nmv6onHdi4Y0t0wmYGE=[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "x-amz-request-id: 741C65B73A0E800F[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "Date: Wed, 12 Dec 2018 03:12:54 GMT[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "x-amz-bucket-region: us-east-1[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "Content-Type: application/xml[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "Transfer-Encoding: chunked[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "Server: AmazonS3[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]"
18/12/11 19:12:54 DEBUG conn.DefaultClientConnection: Receiving response: HTTP/1.1 200 OK
18/12/11 19:12:54 DEBUG http.headers: << HTTP/1.1 200 OK
18/12/11 19:12:54 DEBUG http.headers: << x-amz-id-2: fcNfT1n+Dhn7lPqVGiKhrTq36arxsAAG1QNTFjpiJ+O8vSqBUumwNFw7Nmv6onHdi4Y0t0wmYGE=
18/12/11 19:12:54 DEBUG http.headers: << x-amz-request-id: 741C65B73A0E800F
18/12/11 19:12:54 DEBUG http.headers: << Date: Wed, 12 Dec 2018 03:12:54 GMT
18/12/11 19:12:54 DEBUG http.headers: << x-amz-bucket-region: us-east-1
18/12/11 19:12:54 DEBUG http.headers: << Content-Type: application/xml
18/12/11 19:12:54 DEBUG http.headers: << Transfer-Encoding: chunked
18/12/11 19:12:54 DEBUG http.headers: << Server: AmazonS3
18/12/11 19:12:54 DEBUG client.SdkHttpClient: Connection can be kept alive indefinitely
18/12/11 19:12:54 DEBUG amazonaws.requestId: x-amzn-RequestId: not available
18/12/11 19:12:54 TRACE amazonaws.request: Beginning to parse service response XML
18/12/11 19:12:54 DEBUG transform.XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.cloudera.com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
18/12/11 19:12:54 DEBUG http.wire: << "286[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
18/12/11 19:12:54 DEBUG http.wire: << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>myclouderaraj</Name><Prefix>root/</Prefix><Marker></M...>"
18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "0[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]"
18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com] can be kept alive indefinitely
18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection released: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15]
18/12/11 19:12:54 DEBUG transform.XmlResponsesSaxParser: Parsing XML response document with handler: class com.cloudera.com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
18/12/11 19:12:54 DEBUG transform.XmlResponsesSaxParser: Examining listing for bucket: myclouderaraj
18/12/11 19:12:54 TRACE amazonaws.request: Done parsing service response XML
18/12/11 19:12:54 DEBUG amazonaws.request: Received successful response: 200, AWS Request ID: 741C65B73A0E800F
18/12/11 19:12:54 DEBUG amazonaws.requestId: AWS Request ID: 741C65B73A0E800F
18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: Found path as directory (with /): 0/1
18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: Summary: root/bulkInsert.csv 224
18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: List status for path: s3a://myclouderaraj/root
18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: Getting path status for s3a://myclouderaraj/root (root)
18/12/11 19:12:54 DEBUG amazonaws.request: Sending Request: HEAD https://myclouderaraj.s3.amazonaws.com /root Headers: (User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192, Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
18/12/11 19:12:54 DEBUG internal.S3Signer: Calculated string to sign:
"HEAD

application/x-www-form-urlencoded; charset=utf-8
Wed, 12 Dec 2018 03:12:54 GMT
/myclouderaraj/root"
18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection request: [route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15]
18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection leased: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 0; route allocated: 1 of 15; total allocated: 1 of 15]
18/12/11 19:12:54 DEBUG client.SdkHttpClient: Stale connection check
18/12/11 19:12:54 DEBUG protocol.RequestAddCookies: CookieSpec selected: best-match
18/12/11 19:12:54 DEBUG protocol.RequestAuthCache: Auth cache not set in the context
18/12/11 19:12:54 DEBUG protocol.RequestProxyAuthentication: Proxy auth state: UNCHALLENGED
18/12/11 19:12:54 DEBUG client.SdkHttpClient: Attempt 1 to execute request
18/12/11 19:12:54 DEBUG conn.DefaultClientConnection: Sending request: HEAD /root HTTP/1.1
18/12/11 19:12:54 DEBUG http.wire: >> "HEAD /root HTTP/1.1[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: >> "Host: myclouderaraj.s3.amazonaws.com[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: >> "Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:/G0a8NrUHFBGmNvT/YV5nkFRbKU=[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: >> "User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: >> "Date: Wed, 12 Dec 2018 03:12:54 GMT[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: >> "Content-Type: application/x-www-form-urlencoded; charset=utf-8[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: >> "Connection: Keep-Alive[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: >> "[\r][\n]"
18/12/11 19:12:54 DEBUG http.headers: >> HEAD /root HTTP/1.1
18/12/11 19:12:54 DEBUG http.headers: >> Host: myclouderaraj.s3.amazonaws.com
18/12/11 19:12:54 DEBUG http.headers: >> Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:/G0a8NrUHFBGmNvT/YV5nkFRbKU=
18/12/11 19:12:54 DEBUG http.headers: >> User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192
18/12/11 19:12:54 DEBUG http.headers: >> Date: Wed, 12 Dec 2018 03:12:54 GMT
18/12/11 19:12:54 DEBUG http.headers: >> Content-Type: application/x-www-form-urlencoded; charset=utf-8
18/12/11 19:12:54 DEBUG http.headers: >> Connection: Keep-Alive
18/12/11 19:12:54 DEBUG http.wire: << "HTTP/1.1 404 Not Found[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "x-amz-request-id: 48EB44080F918683[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "x-amz-id-2: MLddIb2VuN1THhnjZI4diJfaB37g0zGNOzIFrEIfgOCMYeJBJEb2xV16+VQsv/2Z2kGhoQyhLhM=[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "Content-Type: application/xml[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "Transfer-Encoding: chunked[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "Date: Wed, 12 Dec 2018 03:12:54 GMT[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "Server: AmazonS3[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]"
18/12/11 19:12:54 DEBUG conn.DefaultClientConnection: Receiving response: HTTP/1.1 404 Not Found
18/12/11 19:12:54 DEBUG http.headers: << HTTP/1.1 404 Not Found
18/12/11 19:12:54 DEBUG http.headers: << x-amz-request-id: 48EB44080F918683
18/12/11 19:12:54 DEBUG http.headers: << x-amz-id-2: MLddIb2VuN1THhnjZI4diJfaB37g0zGNOzIFrEIfgOCMYeJBJEb2xV16+VQsv/2Z2kGhoQyhLhM=
18/12/11 19:12:54 DEBUG http.headers: << Content-Type: application/xml
18/12/11 19:12:54 DEBUG http.headers: << Transfer-Encoding: chunked
18/12/11 19:12:54 DEBUG http.headers: << Date: Wed, 12 Dec 2018 03:12:54 GMT
18/12/11 19:12:54 DEBUG http.headers: << Server: AmazonS3
18/12/11 19:12:54 DEBUG client.SdkHttpClient: Connection can be kept alive indefinitely
18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com] can be kept alive indefinitely
18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection released: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15]
18/12/11 19:12:54 DEBUG amazonaws.requestId: x-amzn-RequestId: not available
18/12/11 19:12:54 DEBUG amazonaws.request: Received error response: com.cloudera.com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: null; Status Code: 404; Error Code: 404 Not Found; Request ID: 48EB44080F918683), S3 Extended Request ID: MLddIb2VuN1THhnjZI4diJfaB37g0zGNOzIFrEIfgOCMYeJBJEb2xV16+VQsv/2Z2kGhoQyhLhM=
18/12/11 19:12:54 DEBUG amazonaws.request: Sending Request: HEAD https://myclouderaraj.s3.amazonaws.com /root/ Headers: (User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192, Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
18/12/11 19:12:54 DEBUG internal.S3Signer: Calculated string to sign:
"HEAD

application/x-www-form-urlencoded; charset=utf-8
Wed, 12 Dec 2018 03:12:54 GMT
/myclouderaraj/root/"
18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection request: [route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15]
18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection leased: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 0; route allocated: 1 of 15; total allocated: 1 of 15]
18/12/11 19:12:54 DEBUG client.SdkHttpClient: Stale connection check
18/12/11 19:12:54 DEBUG protocol.RequestAddCookies: CookieSpec selected: best-match
18/12/11 19:12:54 DEBUG protocol.RequestAuthCache: Auth cache not set in the context
18/12/11 19:12:54 DEBUG protocol.RequestProxyAuthentication: Proxy auth state: UNCHALLENGED
18/12/11 19:12:54 DEBUG client.SdkHttpClient: Attempt 1 to execute request
18/12/11 19:12:54 DEBUG conn.DefaultClientConnection: Sending request: HEAD /root/ HTTP/1.1
18/12/11 19:12:54 DEBUG http.wire: >> "HEAD /root/ HTTP/1.1[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: >> "Host: myclouderaraj.s3.amazonaws.com[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: >> "Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:bAUfpGUWpSAqbkk2vygScGZO7Qw=[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: >> "User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: >> "Date: Wed, 12 Dec 2018 03:12:54 GMT[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: >> "Content-Type: application/x-www-form-urlencoded; charset=utf-8[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: >> "Connection: Keep-Alive[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: >> "[\r][\n]"
18/12/11 19:12:54 DEBUG http.headers: >> HEAD /root/ HTTP/1.1
18/12/11 19:12:54 DEBUG http.headers: >> Host: myclouderaraj.s3.amazonaws.com
18/12/11 19:12:54 DEBUG http.headers: >> Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:bAUfpGUWpSAqbkk2vygScGZO7Qw=
18/12/11 19:12:54 DEBUG http.headers: >> User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192
18/12/11 19:12:54 DEBUG http.headers: >> Date: Wed, 12 Dec 2018 03:12:54 GMT
18/12/11 19:12:54 DEBUG http.headers: >> Content-Type: application/x-www-form-urlencoded; charset=utf-8
18/12/11 19:12:54 DEBUG http.headers: >> Connection: Keep-Alive
18/12/11 19:12:54 DEBUG http.wire: << "HTTP/1.1 404 Not Found[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "x-amz-request-id: D79FFCC520EEA4D7[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "x-amz-id-2: TZC+frfDHCmwLqED0QD8Cg/lABBzKNkVwRIto+wmg3IVMm6+9KGRE87O3cr/jpQ8LFM3YvfaUuI=[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "Content-Type: application/xml[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "Transfer-Encoding: chunked[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "Date: Wed, 12 Dec 2018 03:12:54 GMT[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "Server: AmazonS3[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]"
18/12/11 19:12:54 DEBUG conn.DefaultClientConnection: Receiving response: HTTP/1.1 404 Not Found
18/12/11 19:12:54 DEBUG http.headers: << HTTP/1.1 404 Not Found
18/12/11 19:12:54 DEBUG http.headers: << x-amz-request-id: D79FFCC520EEA4D7
18/12/11 19:12:54 DEBUG http.headers: << x-amz-id-2: TZC+frfDHCmwLqED0QD8Cg/lABBzKNkVwRIto+wmg3IVMm6+9KGRE87O3cr/jpQ8LFM3YvfaUuI=
18/12/11 19:12:54 DEBUG http.headers: << Content-Type: application/xml
18/12/11 19:12:54 DEBUG http.headers: << Transfer-Encoding: chunked
18/12/11 19:12:54 DEBUG http.headers: << Date: Wed, 12 Dec 2018 03:12:54 GMT
18/12/11 19:12:54 DEBUG http.headers: << Server: AmazonS3
18/12/11 19:12:54 DEBUG client.SdkHttpClient: Connection can be kept alive indefinitely
18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com] can be kept alive indefinitely
18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection released: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15]
18/12/11 19:12:54 DEBUG amazonaws.requestId: x-amzn-RequestId: not available
18/12/11 19:12:54 DEBUG amazonaws.request: Received error response: com.cloudera.com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: null; Status Code: 404; Error Code: 404 Not Found; Request ID: D79FFCC520EEA4D7), S3 Extended Request ID: TZC+frfDHCmwLqED0QD8Cg/lABBzKNkVwRIto+wmg3IVMm6+9KGRE87O3cr/jpQ8LFM3YvfaUuI=
18/12/11 19:12:54 DEBUG amazonaws.request: Sending Request: GET https://myclouderaraj.s3.amazonaws.com / Parameters: ({"prefix":["root/"],"delimiter":["/"],"max-keys":["1"]}Headers: (User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192, Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
18/12/11 19:12:54 DEBUG internal.S3Signer: Calculated string to sign:
"GET

application/x-www-form-urlencoded; charset=utf-8
Wed, 12 Dec 2018 03:12:54 GMT
/myclouderaraj/"
18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection request: [route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15]
18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection leased: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 0; route allocated: 1 of 15; total allocated: 1 of 15]
18/12/11 19:12:54 DEBUG client.SdkHttpClient: Stale connection check
18/12/11 19:12:54 DEBUG protocol.RequestAddCookies: CookieSpec selected: best-match
18/12/11 19:12:54 DEBUG protocol.RequestAuthCache: Auth cache not set in the context
18/12/11 19:12:54 DEBUG protocol.RequestProxyAuthentication: Proxy auth state: UNCHALLENGED
18/12/11 19:12:54 DEBUG client.SdkHttpClient: Attempt 1 to execute request
18/12/11 19:12:54 DEBUG conn.DefaultClientConnection: Sending request: GET /?prefix=root%2F&delimiter=%2F&max-keys=1 HTTP/1.1
18/12/11 19:12:54 DEBUG http.wire: >> "GET /?prefix=root%2F&delimiter=%2F&max-keys=1 HTTP/1.1[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: >> "Host: myclouderaraj.s3.amazonaws.com[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: >> "Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:xhHdbA514F67IgOMPolX00Rv1h4=[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: >> "User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: >> "Date: Wed, 12 Dec 2018 03:12:54 GMT[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: >> "Content-Type: application/x-www-form-urlencoded; charset=utf-8[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: >> "Connection: Keep-Alive[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: >> "[\r][\n]"
18/12/11 19:12:54 DEBUG http.headers: >> GET /?prefix=root%2F&delimiter=%2F&max-keys=1 HTTP/1.1
18/12/11 19:12:54 DEBUG http.headers: >> Host: myclouderaraj.s3.amazonaws.com
18/12/11 19:12:54 DEBUG http.headers: >> Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:xhHdbA514F67IgOMPolX00Rv1h4=
18/12/11 19:12:54 DEBUG http.headers: >> User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192
18/12/11 19:12:54 DEBUG http.headers: >> Date: Wed, 12 Dec 2018 03:12:54 GMT
18/12/11 19:12:54 DEBUG http.headers: >> Content-Type: application/x-www-form-urlencoded; charset=utf-8
18/12/11 19:12:54 DEBUG http.headers: >> Connection: Keep-Alive
18/12/11 19:12:54 DEBUG http.wire: << "HTTP/1.1 200 OK[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "x-amz-id-2: n67wc46WhWXcw4ygKZ0I2NEYJXqmyEG2sDyv8+hKbZaGCflTI6YB4gvIpuqQgHb+LCSgCQ33MlU=[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "x-amz-request-id: BE60D884EB5E4D60[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "Date: Wed, 12 Dec 2018 03:12:55 GMT[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "x-amz-bucket-region: us-east-1[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "Content-Type: application/xml[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "Transfer-Encoding: chunked[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "Server: AmazonS3[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]"
18/12/11 19:12:54 DEBUG conn.DefaultClientConnection: Receiving response: HTTP/1.1 200 OK
18/12/11 19:12:54 DEBUG http.headers: << HTTP/1.1 200 OK
18/12/11 19:12:54 DEBUG http.headers: << x-amz-id-2: n67wc46WhWXcw4ygKZ0I2NEYJXqmyEG2sDyv8+hKbZaGCflTI6YB4gvIpuqQgHb+LCSgCQ33MlU=
18/12/11 19:12:54 DEBUG http.headers: << x-amz-request-id: BE60D884EB5E4D60
18/12/11 19:12:54 DEBUG http.headers: << Date: Wed, 12 Dec 2018 03:12:55 GMT
18/12/11 19:12:54 DEBUG http.headers: << x-amz-bucket-region: us-east-1
18/12/11 19:12:54 DEBUG http.headers: << Content-Type: application/xml
18/12/11 19:12:54 DEBUG http.headers: << Transfer-Encoding: chunked
18/12/11 19:12:54 DEBUG http.headers: << Server: AmazonS3
18/12/11 19:12:54 DEBUG client.SdkHttpClient: Connection can be kept alive indefinitely
18/12/11 19:12:54 DEBUG amazonaws.requestId: x-amzn-RequestId: not available
18/12/11 19:12:54 TRACE amazonaws.request: Beginning to parse service response XML
18/12/11 19:12:54 DEBUG transform.XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.cloudera.com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
18/12/11 19:12:54 DEBUG http.wire: << "286[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
18/12/11 19:12:54 DEBUG http.wire: << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>myclouderaraj</Name><Prefix>root/</Prefix><Marker></M...>"
18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "0[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]"
18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com] can be kept alive indefinitely
18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection released: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15]
18/12/11 19:12:54 DEBUG transform.XmlResponsesSaxParser: Parsing XML response document with handler: class com.cloudera.com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
18/12/11 19:12:54 DEBUG transform.XmlResponsesSaxParser: Examining listing for bucket: myclouderaraj
18/12/11 19:12:54 TRACE amazonaws.request: Done parsing service response XML
18/12/11 19:12:54 DEBUG amazonaws.request: Received successful response: 200, AWS Request ID: BE60D884EB5E4D60
18/12/11 19:12:54 DEBUG amazonaws.requestId: AWS Request ID: BE60D884EB5E4D60
18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: Found path as directory (with /): 0/1
18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: Summary: root/bulkInsert.csv 224
18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: listStatus: doing listObjects for directory root/
18/12/11 19:12:54 DEBUG amazonaws.request: Sending Request: GET https://myclouderaraj.s3.amazonaws.com / Parameters: ({"prefix":["root/"],"delimiter":["/"],"max-keys":["5000"]}Headers: (User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192, Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
18/12/11 19:12:54 DEBUG internal.S3Signer: Calculated string to sign:
"GET

application/x-www-form-urlencoded; charset=utf-8
Wed, 12 Dec 2018 03:12:54 GMT
/myclouderaraj/"
18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection request: [route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15]
18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection leased: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 0; route allocated: 1 of 15; total allocated: 1 of 15]
18/12/11 19:12:54 DEBUG client.SdkHttpClient: Stale connection check
18/12/11 19:12:54 DEBUG protocol.RequestAddCookies: CookieSpec selected: best-match
18/12/11 19:12:54 DEBUG protocol.RequestAuthCache: Auth cache not set in the context
18/12/11 19:12:54 DEBUG protocol.RequestProxyAuthentication: Proxy auth state: UNCHALLENGED
18/12/11 19:12:54 DEBUG client.SdkHttpClient: Attempt 1 to execute request
18/12/11 19:12:54 DEBUG conn.DefaultClientConnection: Sending request: GET /?prefix=root%2F&delimiter=%2F&max-keys=5000 HTTP/1.1
18/12/11 19:12:54 DEBUG http.wire: >> "GET /?prefix=root%2F&delimiter=%2F&max-keys=5000 HTTP/1.1[\r][\n]"

seth · ‎12-12-2018

Looks like different authorization content with S3 in the debug outputs.. was the last one successful ?

Different aws jdk , java version, and user agent.

== notworking ==
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Host: myclouderaraj.s3.amazonaws.com
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=AKIAIYFZQ7LF7F5MJL4Q/20181211/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=bc733f2ab05759751c4eeaf11095f286eceba2647ee8b014211cab2e03f967db
18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> User-Agent: Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161

== working ==
18/12/11 19:12:53 DEBUG http.headers: >> Host: myclouderaraj.s3.amazonaws.com
18/12/11 19:12:53 DEBUG http.headers: >> Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:XU3AFfGphmQy0rEhRrWjajyeFbA=
18/12/11 19:12:53 DEBUG http.headers: >> User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192

I'm not sure what restrictions are on the S3 side for how you are trying to authorize but it is definitely sending 256 encryption vs nothing in second test from "quickstart"

Nukala · ‎12-12-2018

Yes, it was successful and i was not able to attach the complete logs.

Yes, I see that there is some change in the request headers/user-agent. Not sure why that was. I have not done any specific changes to different versions of Cloudera and trying to connect the same S3 account.

18/12/11 19:12:54 DEBUG conn.DefaultClientConnection: Sending request: GET /?prefix=root%2F&delimiter=%2F&max-keys=1 HTTP/1.1
18/12/11 19:12:54 DEBUG http.wire: >> "GET /?prefix=root%2F&delimiter=%2F&max-keys=1 HTTP/1.1[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: >> "Host: myclouderaraj.s3.amazonaws.com[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: >> "Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:xhHdbA514F67IgOMPolX00Rv1h4=[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: >> "User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: >> "Date: Wed, 12 Dec 2018 03:12:54 GMT[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: >> "Content-Type: application/x-www-form-urlencoded; charset=utf-8[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: >> "Connection: Keep-Alive[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: >> "[\r][\n]"
18/12/11 19:12:54 DEBUG http.headers: >> GET /?prefix=root%2F&delimiter=%2F&max-keys=1 HTTP/1.1
18/12/11 19:12:54 DEBUG http.headers: >> Host: myclouderaraj.s3.amazonaws.com
18/12/11 19:12:54 DEBUG http.headers: >> Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:xhHdbA514F67IgOMPolX00Rv1h4=
18/12/11 19:12:54 DEBUG http.headers: >> User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192
18/12/11 19:12:54 DEBUG http.headers: >> Date: Wed, 12 Dec 2018 03:12:54 GMT
18/12/11 19:12:54 DEBUG http.headers: >> Content-Type: application/x-www-form-urlencoded; charset=utf-8
18/12/11 19:12:54 DEBUG http.headers: >> Connection: Keep-Alive
18/12/11 19:12:54 DEBUG http.wire: << "HTTP/1.1 200 OK[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "x-amz-id-2: n67wc46WhWXcw4ygKZ0I2NEYJXqmyEG2sDyv8+hKbZaGCflTI6YB4gvIpuqQgHb+LCSgCQ33MlU=[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "x-amz-request-id: BE60D884EB5E4D60[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "Date: Wed, 12 Dec 2018 03:12:55 GMT[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "x-amz-bucket-region: us-east-1[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "Content-Type: application/xml[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "Transfer-Encoding: chunked[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "Server: AmazonS3[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]"
18/12/11 19:12:54 DEBUG conn.DefaultClientConnection: Receiving response: HTTP/1.1 200 OK
18/12/11 19:12:54 DEBUG http.headers: << HTTP/1.1 200 OK
18/12/11 19:12:54 DEBUG http.headers: << x-amz-id-2: n67wc46WhWXcw4ygKZ0I2NEYJXqmyEG2sDyv8+hKbZaGCflTI6YB4gvIpuqQgHb+LCSgCQ33MlU=
18/12/11 19:12:54 DEBUG http.headers: << x-amz-request-id: BE60D884EB5E4D60
18/12/11 19:12:54 DEBUG http.headers: << Date: Wed, 12 Dec 2018 03:12:55 GMT
18/12/11 19:12:54 DEBUG http.headers: << x-amz-bucket-region: us-east-1
18/12/11 19:12:54 DEBUG http.headers: << Content-Type: application/xml
18/12/11 19:12:54 DEBUG http.headers: << Transfer-Encoding: chunked
18/12/11 19:12:54 DEBUG http.headers: << Server: AmazonS3
18/12/11 19:12:54 DEBUG client.SdkHttpClient: Connection can be kept alive indefinitely
18/12/11 19:12:54 DEBUG amazonaws.requestId: x-amzn-RequestId: not available
18/12/11 19:12:54 TRACE amazonaws.request: Beginning to parse service response XML
18/12/11 19:12:54 DEBUG transform.XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.cloudera.com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
18/12/11 19:12:54 DEBUG http.wire: << "286[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
18/12/11 19:12:54 DEBUG http.wire: << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>myclouderaraj</Name><Prefix>root/</Prefix><Marker></M...>"
18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "0[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]"
18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com] can be kept alive indefinitely
18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection released: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15]
18/12/11 19:12:54 DEBUG transform.XmlResponsesSaxParser: Parsing XML response document with handler: class com.cloudera.com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
18/12/11 19:12:54 DEBUG transform.XmlResponsesSaxParser: Examining listing for bucket: myclouderaraj
18/12/11 19:12:54 TRACE amazonaws.request: Done parsing service response XML
18/12/11 19:12:54 DEBUG amazonaws.request: Received successful response: 200, AWS Request ID: BE60D884EB5E4D60
18/12/11 19:12:54 DEBUG amazonaws.requestId: AWS Request ID: BE60D884EB5E4D60
18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: Found path as directory (with /): 0/1
18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: Summary: root/bulkInsert.csv 224
18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: listStatus: doing listObjects for directory root/
18/12/11 19:12:54 DEBUG amazonaws.request: Sending Request: GET https://myclouderaraj.s3.amazonaws.com / Parameters: ({"prefix":["root/"],"delimiter":["/"],"max-keys":["5000"]}Headers: (User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192, Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
18/12/11 19:12:54 DEBUG internal.S3Signer: Calculated string to sign:
"GET

application/x-www-form-urlencoded; charset=utf-8
Wed, 12 Dec 2018 03:12:54 GMT
/myclouderaraj/"
18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection request: [route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15]
18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection leased: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 0; route allocated: 1 of 15; total allocated: 1 of 15]
18/12/11 19:12:54 DEBUG client.SdkHttpClient: Stale connection check
18/12/11 19:12:54 DEBUG protocol.RequestAddCookies: CookieSpec selected: best-match
18/12/11 19:12:54 DEBUG protocol.RequestAuthCache: Auth cache not set in the context
18/12/11 19:12:54 DEBUG protocol.RequestProxyAuthentication: Proxy auth state: UNCHALLENGED
18/12/11 19:12:54 DEBUG client.SdkHttpClient: Attempt 1 to execute request
18/12/11 19:12:54 DEBUG conn.DefaultClientConnection: Sending request: GET /?prefix=root%2F&delimiter=%2F&max-keys=5000 HTTP/1.1
18/12/11 19:12:54 DEBUG http.wire: >> "GET /?prefix=root%2F&delimiter=%2F&max-keys=5000 HTTP/1.1[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: >> "Host: myclouderaraj.s3.amazonaws.com[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: >> "Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:xhHdbA514F67IgOMPolX00Rv1h4=[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: >> "User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: >> "Date: Wed, 12 Dec 2018 03:12:54 GMT[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: >> "Content-Type: application/x-www-form-urlencoded; charset=utf-8[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: >> "Connection: Keep-Alive[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: >> "[\r][\n]"
18/12/11 19:12:54 DEBUG http.headers: >> GET /?prefix=root%2F&delimiter=%2F&max-keys=5000 HTTP/1.1
18/12/11 19:12:54 DEBUG http.headers: >> Host: myclouderaraj.s3.amazonaws.com
18/12/11 19:12:54 DEBUG http.headers: >> Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:xhHdbA514F67IgOMPolX00Rv1h4=
18/12/11 19:12:54 DEBUG http.headers: >> User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192
18/12/11 19:12:54 DEBUG http.headers: >> Date: Wed, 12 Dec 2018 03:12:54 GMT
18/12/11 19:12:54 DEBUG http.headers: >> Content-Type: application/x-www-form-urlencoded; charset=utf-8
18/12/11 19:12:54 DEBUG http.headers: >> Connection: Keep-Alive
18/12/11 19:12:54 DEBUG http.wire: << "HTTP/1.1 200 OK[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "x-amz-id-2: qt25vFz8qCf82h/fM/Pl2kHdk8eWPN12AmczWd25XULPEZbxmvgkoWmnMkDdaK9O13AEcF7V5Fs=[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "x-amz-request-id: 70E4AB92D29F2154[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "Date: Wed, 12 Dec 2018 03:12:55 GMT[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "x-amz-bucket-region: us-east-1[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "Content-Type: application/xml[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "Transfer-Encoding: chunked[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "Server: AmazonS3[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]"
18/12/11 19:12:54 DEBUG conn.DefaultClientConnection: Receiving response: HTTP/1.1 200 OK
18/12/11 19:12:54 DEBUG http.headers: << HTTP/1.1 200 OK
18/12/11 19:12:54 DEBUG http.headers: << x-amz-id-2: qt25vFz8qCf82h/fM/Pl2kHdk8eWPN12AmczWd25XULPEZbxmvgkoWmnMkDdaK9O13AEcF7V5Fs=
18/12/11 19:12:54 DEBUG http.headers: << x-amz-request-id: 70E4AB92D29F2154
18/12/11 19:12:54 DEBUG http.headers: << Date: Wed, 12 Dec 2018 03:12:55 GMT
18/12/11 19:12:54 DEBUG http.headers: << x-amz-bucket-region: us-east-1
18/12/11 19:12:54 DEBUG http.headers: << Content-Type: application/xml
18/12/11 19:12:54 DEBUG http.headers: << Transfer-Encoding: chunked
18/12/11 19:12:54 DEBUG http.headers: << Server: AmazonS3
18/12/11 19:12:54 DEBUG client.SdkHttpClient: Connection can be kept alive indefinitely
18/12/11 19:12:54 DEBUG amazonaws.requestId: x-amzn-RequestId: not available
18/12/11 19:12:54 TRACE amazonaws.request: Beginning to parse service response XML
18/12/11 19:12:54 DEBUG transform.XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.cloudera.com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
18/12/11 19:12:54 DEBUG http.wire: << "64b[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
18/12/11 19:12:54 DEBUG http.wire: << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>myclouderaraj</Name><Prefix>root/</Prefix><Marker></M...>"
18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "0[\r][\n]"
18/12/11 19:12:54 DEBUG http.wire: << "[\r][\n]"
18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com] can be kept alive indefinitely
18/12/11 19:12:54 DEBUG conn.PoolingClientConnectionManager: Connection released: [id: 0][route: {s}->https://myclouderaraj.s3.amazonaws.com][total kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15]
18/12/11 19:12:54 DEBUG transform.XmlResponsesSaxParser: Parsing XML response document with handler: class com.cloudera.com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
18/12/11 19:12:54 DEBUG transform.XmlResponsesSaxParser: Examining listing for bucket: myclouderaraj
18/12/11 19:12:54 TRACE amazonaws.request: Done parsing service response XML
18/12/11 19:12:54 DEBUG amazonaws.request: Received successful response: 200, AWS Request ID: 70E4AB92D29F2154
18/12/11 19:12:54 DEBUG amazonaws.requestId: AWS Request ID: 70E4AB92D29F2154
18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: Adding: fi: s3a://myclouderaraj/root/bulkInsert.csv
18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: Adding: fi: s3a://myclouderaraj/root/customers
18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: Adding: fi: s3a://myclouderaraj/root/table_test.csv
18/12/11 19:12:54 DEBUG s3a.S3AFileSystem: Adding: fi: s3a://myclouderaraj/root/temp.json
Found 4 items
-rw-rw-rw- 1 root 224 2018-11-30 02:30 s3a://myclouderaraj/root/bulkInsert.csv
-rw-rw-rw- 1 root 15812 2018-11-29 21:57 s3a://myclouderaraj/root/customers
-rw-rw-rw- 1 root 0 2018-11-30 02:30 s3a://myclouderaraj/root/table_test.csv
-rw-rw-rw- 1 root 2121 2018-11-29 10:32 s3a://myclouderaraj/root/temp.json
18/12/11 19:12:54 TRACE core.TracerPool: TracerPool(Global): removing tracer Tracer(FsShell/10.33.6.146)
18/12/11 19:12:55 DEBUG conn.PoolingClientConnectionManager: Connection manager is shutting down
18/12/11 19:12:55 DEBUG internal.SdkSSLSocket: shutting down output of myclouderaraj.s3.amazonaws.com/52.216.107.100:443
18/12/11 19:12:55 DEBUG internal.SdkSSLSocket: closing myclouderaraj.s3.amazonaws.com/52.216.107.100:443
18/12/11 19:12:55 DEBUG conn.DefaultClientConnection: Connection 0.0.0.0:51025<->52.216.107.100:443 closed
18/12/11 19:12:55 DEBUG conn.DefaultClientConnection: Connection 0.0.0.0:51025<->52.216.107.100:443 closed
18/12/11 19:12:55 DEBUG conn.PoolingClientConnectionManager: Connection manager shut down
[root@quickstart ~]#

Thanks & Regards,

Rajesh

seth · ‎12-13-2018

Ok, you can mark as resolved if this is no longer an issue.

I've responded in the other new threads as well.

Cloudera Community

Support Questions

Metadata and lineage collection for S3