question Re: ConvertRecord fails for some files in Support Questions

ConvertRecord fails for some files

tono425 — Thu, 16 Jan 2025 08:06:49 GMT

Hello all,

I'm trying to convert many records from json to parquet with ConvertRecord processor.
Most succeed, but convert fails for some files with this error.

org.apache.parquet.schema.InvalidSchemaException: Cannot write a schema with an empty group: optional group pop_pools

I assume this is because some json files contain the following field.

"pop_pools": {}

As source record is log data, we can't modify it.
Is there any way to avoid this error and convert the records to parquet format?

Thanks,

Re: ConvertRecord fails for some files

cloude — Fri, 17 Jan 2025 06:28:46 GMT

Hi @tono425 ,

You cannot write a empty struct in parquet.

This is due to the way the parquet format works, a parquet file only consists of leaf field data, the intermediate structure is not stored and can be inferred using the schema and the repetition levels and definition levels of the written leaf fields. An empty struct (which is written as a group) has no leaf fields and that is why parquet fails to write this, I would suggest to change the format or filter the value before converting.

Re: ConvertRecord fails for some files

tono425 — Mon, 20 Jan 2025 08:20:20 GMT

@cloude
Thank you for your answer.
Now I understand that is expected behavior.
I'll consider solution.

Thanks,