question Efficient ways to store many images files in Support Questions

Efficient ways to store many images files

vincentD — Fri, 16 Sep 2022 14:23:51 GMT

We have ten millions image and video files, are looking for efficient ways to store them in Hadoop (HDFS ...), and analyze them with tools available in the Hadoop ecosystem. I understand HDFS prefer big files. These image files are small, they are under ten megabytes. Please advise. Thanks very much!

Re: Efficient ways to store many images files

Harsh J — Mon, 20 May 2019 01:17:15 GMT

You can do this via two methods: Container files, or HBase MOBs. Which is
the right path depends on your eventual, dominant read pattern for this
data.

If your analysis will require loading up only a small range of images out
of the total dataset, or individual images, then HBase is a better fit with
its key based access model, columnar storage and caches.

If instead you will require processing these images in bulk, then large
container files (such as Sequence Files (with BytesWritable or equivalent),
Parquet Files (with BINARY/BYTE_ARRAY types), etc. that can store multiple
images into a single file, and allow for fast, sequential reads of all
images in bulk.

Re: Efficient ways to store many images files

vincentD — Fri, 24 May 2019 01:48:30 GMT

Thanks a lot for your reply Harsh. These sound great. Can you give some pointers to some learning materials on both methods, i.e. examples, blogs, URLs or books etc?

Re: Efficient ways to store many images files

Harsh J — Fri, 24 May 2019 03:46:39 GMT

For HBase MOBs, this can serve as a good starting point as most of the changes are administrative and the writer API remains the same as regular cells: https://www.cloudera.com/documentation/enterprise/latest/topics/admin_hbase_mob.html

For SequenceFiles, a good short snippet can be found here: https://github.com/sakserv/sequencefile-examples/blob/master/test/main/java/com/github/sakserv/sequencefile/SequenceFileTest.java#L65-L70 and for Parquet: https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/example/ExampleParquetWriter.java

More general reading for the file formats: https://blog.cloudera.com/blog/2011/01/hadoop-io-sequence-map-set-array-bloommap-files/ and https://parquet.apache.org/documentation/latest/