S3 loading into HDFS

Question

Hi All

Cloudera suggests as best practice using S3 storage only for initial and final storage. The intermediate files will need to be stored in HDFS... In that case, we are still using HDFS but the cluster will only run during the batch ETL and then tore off daily.

How we can pull S3 data into HDFS for each batch ETL job? and then store back the final results to S3?

If Cloudera means to use distcp, how that might work for ETL batch jobs each time? It did not make sense to me using distcp...

Thanks

CK

Harsh J · Accepted Answer

You can apply the queries directly on that external table. Hive will use
HDFS for any transient storage it requires as part of the query stages.

Of course, if it is a set of queries overall, you can also store all the
intermediate temporary tables on HDFS in the way you describe, but the
point am trying to make is that you do not need to copy the original data
as-is, just allow Hive to read off of S3/write into S3 at the points that
matter.

Harsh J · Answer

You do not need to pull files into HDFS as a step in your processing, as
CDH provides inbuilt connectors to pull input/write output directly from S3
storage (s3a:// URIs, backed by some configurations that provide
credentials and targets).

This page is a good starting reference to setting up S3 access over Cloud
installations:
https://www.cloudera.com/documentation/director/latest/topics/director_s3_object_storage.html
-
make sure to checkout the page links from the opening paragraph too.

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.