question PutBigQuery not evaluating flowfile attributes in Support Questions

PutBigQuery not evaluating flowfile attributes

jisaitua — Mon, 29 May 2023 15:30:19 GMT

I'm attempting to transition from using PutBigQueryBatch to PutBigQuery. However, the latter doesn't seem to evaluate the dataset and table attributes as expected.

In the provided example, one processor has the values hardcoded, which works correctly, while the other processor relies on retrieving those values from the flowfile's attributes. However, it appears that the attributes are not being properly evaluated.

The example flow (using Nifi 1.21.0):

The flowFile used:

The left PutBigQuery processor has hardcoded values, and it works:

The right PutBigQuery processor fails because it seems it's using the EL definition but it's not evaluating it:

Any help would be appreciated.

Re: PutBigQuery not evaluating flowfile attributes

DianaTorres — Mon, 29 May 2023 15:31:43 GMT

@jisaitua Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our NiFi experts @MattWho @cotopaul who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

Re: PutBigQuery not evaluating flowfile attributes

jisaitua — Mon, 29 May 2023 15:36:51 GMT

Thank you @DianaTorres for your comment. Let's see what the experts have to say.

Regards!

Re: PutBigQuery not evaluating flowfile attributes

steven-matison — Tue, 30 May 2023 13:02:51 GMT

@jisaitua Thank you for making such a well written post with all of the right screen shots, etc.

I am working on duplicating this issue and so far i have confirmed the same experience.

INVALID_ARGUMENT: Invalid project resource name projects/${test}; Project id: ${test}

So this is a known JIRA bug:

https://issues.apache.org/jira/projects/NIFI/issues/NIFI-11608?filter=allissues

I will work on getting some traction here!!

Re: PutBigQuery not evaluating flowfile attributes

jisaitua — Tue, 30 May 2023 13:58:45 GMT

Thank you for helping with this, @steven-matison!

Re: PutBigQuery not evaluating flowfile attributes

steven-matison — Tue, 30 May 2023 18:05:33 GMT

Some progress:

 resource 'projects/gcp-se-cdp-sandbox-env/datasets/dataset/tables/tablename'

With datasets = ${bq.dataset} tables = ${bq.table.name}

Re: PutBigQuery not evaluating flowfile attributes

steven-matison — Tue, 30 May 2023 18:42:55 GMT

@jisaitua I am not sure how long its going to take, but i should have a github commit ready w/ a new PutBigQuery nar that you can use with 1.21. I will update again tomorrow morning.

Re: PutBigQuery not evaluating flowfile attributes

steven-matison — Wed, 31 May 2023 11:38:02 GMT

Pull Request is in!!

https://github.com/apache/nifi/pull/7316

This will be fixed in future releases of NiFi.

@jisaitua send me a DM with your email if you need to use the new processor right away.

Re: PutBigQuery not evaluating flowfile attributes

jisaitua — Thu, 01 Jun 2023 15:35:13 GMT

Wow, that was fast, thank you very much @steven-matison!