About Sarthak

Sarthak · ‎03-20-2018

I am trying to copy data from hdfs to s3 While using the distcp command, the command works for individual files. So, hadoop distcp /user/username/file.txt s3a://xxxxx works fine But when I try to copy the entire director structure it fails to create the directory giving the error: Error: java.io.IOException: mkdir failed for s3a://bucket****/ Error Code: 403 Forbidden; Request ID: 447400E9C5995ED9), S3 Extended Request ID: T0hsw+XaBMrkMUhDcJBKGIRtSF58dKedZdCH2qC32v9uVkwR94SGiI7Xxe8lqaFaDyjwS3oCpkg= Even if I do a simple mkdir command in s3 it fails giving the same 403 forbidden issue. So not sure what is the root cause. I am able to copy the files but not able to create directories.

Sarthak · ‎11-21-2015

Hi, I am trying to schedule a spark job using cron. I have made a shell script and it executes well on the terminal. However, when I execute the script using cron it gives me insufficient memory to start JVM thread error. Every time I start the script using terminal there is no issue. This issue comes when the script starts with cron. Kindly if you could suggest something.

Online	Offline
Last Visited	‎09-06-2018 01:49 AM

Member Since	‎11-21-2015 05:39 AM
Last Visited	‎09-06-2018 01:49 AM
Posts	7

Cloudera Community

Unable to copy data from HDFS to S3

Re: Scheduling Spark with Crontab