question Re: Trouble Indexing data to elasticsearch using NIFI in Support Questions

Trouble Indexing data to elasticsearch using NIFI

eylon — Sun, 28 Jan 2024 20:05:20 GMT

Hey,

i'm using NIFI 1.22 and preparing NIFI flow for upgrade to 2.0.0 version.
Currently, using PutElasticsearchHttpRecord in order to index flowfiles which contain array of nested jsons. This processor sends each flowfile as whole, does not require splits to single jsons or defining schema.

This processor is deprecated and it suggested to move to PutElasticsearchRecord.

My data is very much complex and variant so I tried to avoid defining schema and to use the Inffer Schema strategy of JsonTreeReader in PutElasticsearchRecord.
But the processor is failing on type conversion of the first field in the first json of the array. It seems that it cannot handle nested jsons in array (while splitting it to single jsons worked).

In addition, my implemetation need to deal the following limitation:
1. Processing a large dataset (TBs), so it need to be fast.
2. Avoiding spliting the array of jsons into a single json per flow file, as it cause content claim issues (too many large files and small files in the flow).
3. Avoiding managing schema, as the data is much variant.

Love to hear how is best to Index this data to Elasticsearch.

Re: Trouble Indexing data to elasticsearch using NIFI

VidyaSargur — Mon, 29 Jan 2024 10:48:52 GMT

@eylon, Welcome to our community! To help you get the best possible answer, I have tagged in our NiFi experts @SAMSAL @cotopaul @MattWho @TimothySpann who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.

Re: Trouble Indexing data to elasticsearch using NIFI

eylon — Tue, 30 Jan 2024 07:18:15 GMT

Thanks for the reply.
Adding more details about my flow configuration and example for the input filesputElasticsearchRecord_errorputElasticsearchRecord_configjsonTreeReader_configinputFile smaple

Re: Trouble Indexing data to elasticsearch using NIFI

Green_ — Thu, 18 Sep 2025 13:32:25 GMT

Hey @eylon , did you ever find a solution to this problem? We are facing the exact same issue right now where we'd like to write to elastic without records due to a changing schema, yet our files contain batches of jsons and so we can't use PutElasticsearchJson as we'd like.

Re: Trouble Indexing data to elasticsearch using NIFI

mburgess — Sat, 20 Sep 2025 16:15:37 GMT

If "Infer Schema" isn't working this is likely a bug. Could you provide an example JSON and the error message that happens during schema inference?