Skip to main content
jisaitua
jisaitua
Silver ⭐️⭐️
May 29, 2023
Solved

PutBigQuery not evaluating flowfile attributes

I'm attempting to transition from using PutBigQueryBatch to PutBigQuery. However, the latter doesn't seem to evaluate the dataset and table attributes as expected.

In the provided example, one processor has the values hardcoded, which works correctly, while the other processor relies on retrieving those values from the flowfile's attributes. However, it appears that the attributes are not being properly evaluated.

 

The example flow (using Nifi 1.21.0):

jisaitua_0-1685372232049.png

The flowFile used:

jisaitua_5-1685372977756.png

 

The left PutBigQuery processor has hardcoded values, and it works:

jisaitua_2-1685372286142.png

The right PutBigQuery processor fails because it seems it's using the EL definition but it's not evaluating it:

jisaitua_3-1685372299255.png

jisaitua_4-1685372442211.png

 

Any help would be appreciated.

steven-matison
steven-matisonAnswer
Platinum ⭐️⭐️
May 30, 2023

@jisaitua Thank you for making such a well written post with all of the right screen shots, etc.  

I am working on duplicating this issue and so far i have confirmed the same experience. 

INVALID_ARGUMENT: Invalid project resource name projects/${test}; Project id: ${test}

 

 

So this is a known JIRA bug:

 

https://issues.apache.org/jira/projects/NIFI/issues/NIFI-11608?filter=allissues

 

I will work on getting some traction here!!  

View in thread

8 replies

DianaTorres
Community Manager
DianaTorres
Community Manager
May 29, 2023

@jisaitua Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our NiFi experts @MattWho @cotopaul  who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

Translate
Regards, Diana Torres, Senior Community Moderator
jisaitua
jisaituaAuthor
Silver ⭐️⭐️
May 29, 2023

Thank you @DianaTorres for your comment. Let's see what the experts have to say.

 

Regards!

Translate
steven-matison
Platinum ⭐️⭐️
steven-matisonAnswer
Platinum ⭐️⭐️
May 30, 2023

@jisaitua Thank you for making such a well written post with all of the right screen shots, etc.  

I am working on duplicating this issue and so far i have confirmed the same experience. 

INVALID_ARGUMENT: Invalid project resource name projects/${test}; Project id: ${test}

 

 

So this is a known JIRA bug:

 

https://issues.apache.org/jira/projects/NIFI/issues/NIFI-11608?filter=allissues

 

I will work on getting some traction here!!  

Translate
    jisaitua
    jisaituaAuthor
    Silver ⭐️⭐️
    May 30, 2023

    Thank you for helping with this, @steven-matison!

    Translate
    steven-matison
    Platinum ⭐️⭐️
    steven-matison
    Platinum ⭐️⭐️
    May 30, 2023

    Some progress:

    resource 'projects/gcp-se-cdp-sandbox-env/datasets/dataset/tables/tablename'

    With datasets = ${bq.dataset} tables = ${bq.table.name}

    Translate
    steven-matison
    Platinum ⭐️⭐️
    steven-matison
    Platinum ⭐️⭐️
    May 30, 2023

    @jisaitua I am not sure how long its going to take, but i should have a github commit ready w/ a new PutBigQuery nar that you can use with 1.21.   I will update again tomorrow morning.

    Translate
      steven-matison
      Platinum ⭐️⭐️
      steven-matison
      Platinum ⭐️⭐️
      May 31, 2023

      Pull Request is in!!   


      https://github.com/apache/nifi/pull/7316

       

      This will be fixed in future releases of NiFi.

       

      @jisaitua send me a DM with your email if you need to use the new processor right away.

       

      Translate
      jisaitua
      jisaituaAuthor
      Silver ⭐️⭐️
      June 1, 2023

      Wow, that was fast, thank you very much @steven-matison!

      Translate
      Terms & ConditionsAccessibility statement