question Re: Problem when trying to convert parquet file in Support Questions

Problem when trying to convert parquet file

alecssander — Tue, 26 Nov 2024 13:20:14 GMT

hello,
I'm trying to read a parquet file using the ConvertRecord processor and I'm getting the error:

ConvertRecord[id=e599cd8f-9a1d-3134-4daf-af7bc91cdd57] Failed to process FlowFile[filename=objectTable_tract_5074_DC2_2_2i_runs_DP0_2_v23_0_1_PREOPS-905_step3_31_20220314T212509Z-part0_output.parquet]; will route to failure: org.apache.avro.SchemaParseException: Illegal initial character: 0

In my file the columns are numeric and the first one starts with 0 (zero).

Re: Problem when trying to convert parquet file

DianaTorres — Tue, 26 Nov 2024 15:11:20 GMT

@alecssander Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our NiFi experts @MattWho @SAMSAL who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

Re: Problem when trying to convert parquet file

SAMSAL — Tue, 26 Nov 2024 18:52:27 GMT

Hi ,

Can you provide more explanation\screenshot of your dataflow and the configuration set on each processor\controller service? Also if you can provide sample data that can be converted to parquet which can then reproduce the error that would be helpful as well.

Thanks

Re: Problem when trying to convert parquet file

alecssander — Tue, 26 Nov 2024 20:37:25 GMT

The process is simple, I take a parquet file from a bucket and try to insert it into a postresql database:

My file has 301 columns ranging from 0 to 300 with more than 280 lines:

Re: Problem when trying to convert parquet file

SAMSAL — Tue, 26 Nov 2024 22:04:28 GMT

It seems like whenever dealing with parquet reader\writer services , those services are trying to use Avro schema, possibly to make sense of the data when passing it along to the target processors ( like PutDatabaseRecord ) since parquet is in binary format. The problem with this is that Avro has limitation on how fields should be called. Actually this is reported as a bug in Jira but it doesnt seem to have been resolved. According to the ticket Avro fields should only start with the following characters [A-Za-z_] . Given this , it seems you have to think of some workaround to address this issue since Nifi doesnt provide a solution out of the box. you can check my answer to this post as an option. Basically, you can use python to read the parquet content and transfer to another format (such as CSV as an example) then pass the CSV to the PutDatabaseRecord. This should work as I have tested it. Since you seem to be using Nifi 2.0 , you can develop python extension processor for this instead of ExecuteStreamCommand mentioned in the post.

Hope that helps. If it does, please accept the solution.

Thanks

Re: Problem when trying to convert parquet file

alecssander — Wed, 27 Nov 2024 17:20:46 GMT

Thanks for the support

Re: Problem when trying to convert parquet file

SAMSAL — Wed, 27 Nov 2024 19:12:42 GMT

Sure, If you come up with a solution different than what I suggested please do post about it so it can help others who might run into similar situation. good luck

Re: Problem when trying to convert parquet file

DianaTorres — Tue, 10 Dec 2024 01:02:48 GMT

@alecssander Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks.