'%1s' is a java format specifier that I use to dynamically add the HBase table name in the catalog for testing purpose. The other issue (multiple columns) you mentioned is not applicable in my case since I don't create a new table using the option 'HBaseTableCatalog.newTable -> “5”'. I use an existing pre split table. This is the code from 'Utils$.toBytes()'. If one of the fields in the Data Frame is Null, the method toBytes() gets the first argument 'input' as Null and Null is not instance of anything. So, it eventually goes to 'label323' and throws that error. So, the only workaround at this stage is to remove the null fields from the data frame or to populate them with something which is not always feasible. public byte[] toBytes(Object input, Field field) { if (field.schema().isDefined()); Object record; Object localObject1 = input; Object localObject2; if (localObject1 instanceof Boolean) { boolean bool = BoxesRunTime.unboxToBoolean(localObject1); localObject2 = Bytes.toBytes(bool); } else if (localObject1 instanceof Byte) { int i = BoxesRunTime.unboxToByte(localObject1); localObject2 = new byte[] { i }; } else if (localObject1 instanceof byte[]) { byte[] arrayOfByte = (byte[])localObject1; localObject2 = arrayOfByte; } else if (localObject1 instanceof Double) { double d = BoxesRunTime.unboxToDouble(localObject1); localObject2 = Bytes.toBytes(d); } else if (localObject1 instanceof Float) { float f = BoxesRunTime.unboxToFloat(localObject1); localObject2 = Bytes.toBytes(f); } else if (localObject1 instanceof Integer) { int j = BoxesRunTime.unboxToInt(localObject1); localObject2 = Bytes.toBytes(j); } else if (localObject1 instanceof Long) { long l = BoxesRunTime.unboxToLong(localObject1); localObject2 = Bytes.toBytes(l); } else if (localObject1 instanceof Short) { short s = BoxesRunTime.unboxToShort(localObject1); localObject2 = Bytes.toBytes(s); } else if (localObject1 instanceof UTF8String) { UTF8String localUTF8String = (UTF8String)localObject1; localObject2 = localUTF8String.getBytes(); } else { if (!(localObject1 instanceof String)) break label323; String str = (String)localObject1; localObject2 = Bytes.toBytes(str); } return ((record = field.catalystToAvro().apply(input)) ? AvroSedes..MODULE$.serialize(record, (Schema)field.schema().get()) : (field.sedes().isDefined()) ? ((Sedes)field.sedes().get()).serialize(input) : localObject2); label323: throw new Exception(new StringContext(Predef..MODULE$.wrapRefArray((Object[])new String[] { "unsupported data type ", "" })).s(Predef..MODULE$.genericWrapArray(new Object[] { field.dt() }))); } ... View more

Thanks! I totally agree with you and we are already using Avro in other cases. But sometimes, Avro doesn't seem like a better option when you only need to read a few columns. Storing all the fields as Avro binary in one HBase column kind of forces you to read all the columns even when you need only a few. ... View more

This error occurs in 'org.apache.spark.sql.execution.datasources.hbase.Utils$.toBytes()' when the value of a String field in the DF is 'null'. The value doesn't really fall anywhere in the If-Else chain and eventually throws this error. I think this can be handled by checking for 'null' and just putting a blank String in hbase. ... View more