Be careful with setting ParallelGCThreads to 8 as it will decrease threads if the system has more processing power. => https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/parallel.html On a machine with N hardware threads where N is greater than 8, the parallel collector uses a fixed fraction of N as the number of garbage collector threads. The fraction is approximately 5/8 for large values of N. At values of N below 8, the number used is N. On selected platforms, the fraction drops to 5/16. The specific number of garbage collector threads can be adjusted with a command-line option (which is described later). On a host with one processor, the parallel collector will likely not perform as well as the serial collector because of the overhead required for parallel execution (for example, synchronization). However, when running applications with medium-sized to large-sized heaps, it generally outperforms the serial collector by a modest amount on machines with two processors, and usually performs significantly better than the serial collector when more than two processors are available. ... View more

Ok, you can mark as resolved if this is no longer an issue. I've responded in the other new threads as well. ... View more

Looks like different authorization content with S3 in the debug outputs.. was the last one successful ?   Different aws jdk , java version, and user agent.   == notworking ==  18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Host: myclouderaraj.s3.amazonaws.com 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=AKIAIYFZQ7LF7F5MJL4Q/20181211/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=bc733f2ab05759751c4eeaf11095f286eceba2647ee8b014211cab2e03f967db 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> User-Agent: Hadoop 2.6.0-cdh5.13.0, aws-sdk-java/1.11.134 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12/1.8.0_161 == working  == 18/12/11 19:12:53 DEBUG http.headers: >> Host: myclouderaraj.s3.amazonaws.com 18/12/11 19:12:53 DEBUG http.headers: >> Authorization: AWS AKIAIYFZQ7LF7F5MJL4Q:XU3AFfGphmQy0rEhRrWjajyeFbA= 18/12/11 19:12:53 DEBUG http.headers: >> User-Agent: aws-sdk-java/1.10.6 Linux/2.6.32-573.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.192-b12/1.8.0_192   I'm not sure what restrictions are on the S3 side for how you are trying to authorize but it is definitely sending 256 encryption vs nothing in second test from "quickstart" ... View more

Rajesh,    Did you capture debug logging from each environment to see what the other sandbox env is sending to S3 and works vs what quickstart is sending and fails?   Example:   18/12/11 17:46:03 http-outgoing-0 << "x-amz-bucket-region: us-east-1[\r][\n]" 18/12/11 17:46:03 http-outgoing-0 << "x-amz-request-id: 93DABB98D9C20FB2[\r][\n]"     <==== 18/12/11 17:46:03 http-outgoing-0 << "x-amz-id-2: GImfViIrJZLJvNPS0MGpxZSfsmJqibcEtOLcBK86sdCo79CpRHxlKQhANkddSqVMPIIBtIGEZho=[\r][\n]" 18/12/11 17:46:03  http-outgoing-0 << HTTP/1.1 403 Forbidden   => Request ID: 93DABB98D9C20FB2   x-amz-content-sha256:UNSIGNED-PAYLOAD x-amz-date:20181211T164601Z amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> x-amz-content-sha256: UNSIGNED-PAYLOAD 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=AKIAIYFZQ7LF7F5MJL4Q/20181211/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=bc733f2ab05759751c4eeaf11095f286eceba2647ee8b014211cab2e03f967db 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "x-amz-content-sha256: UNSIGNED-PAYLOAD[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=AKIAIYFZQ7LF7F5MJL4Q/20181211/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=bc733f2ab05759751c4eeaf11095f286eceba2647ee8b014211cab2e03f967db[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "HTTP/1.1 403 Forbidden[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "x-amz-bucket-region: us-east-1[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "x-amz-request-id: 93DABB98D9C20FB2[\r][\n]" 18/12/11 17:46:03 DEBUG http.wire: http-outgoing-0 << "x-amz-id-2: GImfViIrJZLJvNPS0MGpxZSfsmJqibcEtOLcBK86sdCo79CpRHxlKQhANkddSqVMPIIBtIGEZho=[\r][\n]" 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << HTTP/1.1 403 Forbidden 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << x-amz-bucket-region: us-east-1 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << x-amz-request-id: 93DABB98D9C20FB2 18/12/11 17:46:03 DEBUG http.headers: http-outgoing-0 << x-amz-id-2: GImfViIrJZLJvNPS0MGpxZSfsmJqibcEtOLcBK86sdCo79CpRHxlKQhANkddSqVMPIIBtIGEZho= 18/12/11 17:46:03 DEBUG amazonaws.request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 93DABB98D9C20FB2), S3 Extended Request ID: GImfViIrJZLJvNPS0MGpxZSfsmJqibcEtOLcBK86sdCo79CpRHxlKQhANkddSqVMPIIBtIGEZho= ... View more

Rajesh,    You are passing a different key or a different configuration on the connection attempt to s3a:// in working and not working environment.    To check you could enable debug logger and see what working/sandbox shows vs not-working/quickstart is passing to the S3a endpoint   [quickstart] # export HADOOP_ROOT_LOGGER=TRACE,console # export HADOOP_JAAS_DEBUG=true # export HADOOP_OPTS="-Dsun.security.krb5.debug=true"   # hdfs dfs -Dfs.s3a.access.key=myaccesskey -Dfs.s3a.secret.key=mysecretkey -ls s3a://myclouderaraj/root   ** note that the values of each parameter like "myaccesskey" need to be correct or connection will fail.   Thanks Seth ... View more

Rajesh,    Try including the -DFS.s3a flags prior to the -ls flags, as this works in your example from other HortonWorks cluster.   - not working [quickstart]- hdfs dfs -ls -Dfs.s3a.access.key=<myaccesskey> -Dfs.s3a.secret.key=<mysecretkey> s3a://myclouderaraj/root -ls: Illegal option -Dfs.s3a.access.key=<myaccesskey>   - working [sandbox] - hdfs dfs -Dfs.s3a.access.key=myaccesskey -Dfs.s3a.secret.key=mysecretkey -ls s3a://myclouderaraj/root   Try from Cloudera [quickstart]: hdfs dfs -Dfs.s3a.access.key=myaccesskey -Dfs.s3a.secret.key=mysecretkey -ls s3a://myclouderaraj/root   If successful, you will need to add these parameters to *-site.xml to work. => https://www.cloudera.com/documentation/enterprise/5-15-x/topics/cdh_admin_distcp_data_cluster_migrate.html#distcp_and_s3   S3 credentials can be provided in a configuration file (for example, core-site.xml): <property> <name>fs.s3a.access.key</name> <value>...</value> </property> <property> <name>fs.s3a.secret.key</name> <value>...</value> </property>   Let me know if you are successful.   Thanks, Seth   ... View more

Nukula,    You can validate with AWS but you need a deeper path to the s3 bucket for this to work.  Confirm with the s3 tools from the vendor and use that path your user has access to test.   You can also test with this expression    hdfs dfs -ls -Dfs.s3a.access.key=myAccessKey -Dfs.s3a.secret.key=mySecretKey s3a://myBucket/mydata_backup ... View more

Nukala,    From error generated in BDR, there is a problem accessing the path used S3 destination path with credentials provided.   You can validate if credentails are correct via cli   hadoop distcp -Dfs.s3a.access.key=myAccessKey -Dfs.s3a.secret.key=mySecretKey /user/hdfs/mydata s3a://myBucket/mydata_backup     Let us know if this resolves the issue and results if not.   LINKS: [1] https://www.cloudera.com/documentation/enterprise/5-15-x/topics/cdh_admin_distcp_data_cluster_migrate.html#distcp_and_s3     ... View more