We currently have a Pig script that just loads in image (blob) data using AvroStorage with a predefined Avro schema, then stores into HBase with HBaseStorage specifying which columns to use. Each row from the original DB consists of a row ID and 5 image columns, although any number of the image columns could be empty/NULL. e.g. KEY IMG1 IMG2 IMG3 1 null blob blob 2 blob null blob In the HBase table, the column family for the images is i, with column names i1, i2, etc. It was my understanding that any cells containing a NULL value would automatically not be stored in HBase, however those cells are being stored as a key-value pair with a full key and an empty value '', rather than not existing. Is this the normal/expected behaviour? If not, what is the best way to get around it? Do I have to project the key with each of the five columns, filter out nulls and store them individually (i.e. store 5 times)? Or could it be related to a difference in the way Avro, Pig and HBase all represent null values? Is there a simple type conversion I could do on all the image columns so that they would automatically not be stored if they are empty? Versions: we are running plain HDP 2.1, soon to be upgraded to 2.4
... View more
