About cotopaul

cotopaul · ‎07-12-2023

the question is how? 😀 I just tested again and I was able to extract data from (and into) BigQuery without any errors. It seems that something you have defined is not 100% correct. Circling back to my initial post, you would need those defined as: Database Connection URL: jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=your-project-if-here;OAuthType=0;OAuthServiceAcctEmail=your-email-for-service-account-here;OAuthPvtKeyPath=path_on_nifi_server_where_the_service_account_json_is_located; Database Driver Class Name: com.simba.googlebigquery.jdbc.Driver Database Driver Location: full_path_to_jars_location/ Where the Database driver location has to be the full path to the folder where you have extracted the JDBC (not ODBC) ZIP File... and you only have to point to the folder, not to a specific JAR file. In addition, pay attention to how the connection URL is configured and make sure that you follow the same structure. Plus, make sure that your SA has all the necessary rights for your actions.

cotopaul · ‎07-12-2023

@MWM, If you want to read line by line (record by record), you should use SplitRecord to split the CSV record by record and send them further into processing. You will define and RecordReader - CSV in your case and a RecordWriter - JSON. Next, using EvaluateJsonPath, you can extract the content of each FlowFile and add them as attributes. Lastly, you can send those attributes to create the rest query. But please take note that if you are working with CSVs containing far too many lines, you will have some performance issues and you might want to rethink your option. PS: there are other options to perform the task so you can wait for other responses and choose the best one for you 🙂

cotopaul · ‎07-11-2023

A screenshot of what? I have no template which I can export so i cannot share any XML Files 🙂

cotopaul · ‎07-10-2023

@steven-matisonthanks for you answer:) You can download a template here: download Instead of GenerateFlowFile, I have another processing section, but nevertheless, the relevant part starts with AttributesToJson going up until the PutBigQuery Processors 🙂

DianaTorres · ‎07-07-2023

@devanand007 Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks.

c3turner7 · ‎07-05-2023

Oh god. One of my chrome extensions was causing this issue. Dark Reader. Issue solved.

Wpq · ‎06-26-2023

The issue ended up being that ip is a reserved word (a default function) so changing the name of the attribute from ip to extractedIp solved it.

cotopaul · ‎06-26-2023

@Carson, Like @joseomjr wrote (but did not gave the entire link by mistake), you should take a look at the following Article as it describes exactly what you need --> https://community.cloudera.com/t5/Community-Articles/ExecuteScript-Cookbook-part-3/ta-p/249148 Basically, you add the property in your NiFi Processor and reference it to your parameter value and afterwards you call it in your script: myValue1 = myProperty1.getValue()

cotopaul · ‎06-21-2023

@drewski7, in this case, have a look at @steven-matison 's answer because that is your solution to your problem.

Samya · ‎06-20-2023

It just gave this information in DEBUG. "PutAzureDataLakeStorage DataLakeServiceClient is not found in the cache with the given credentials. Creating it."

Online	Offline
Last Visited	‎03-14-2024 06:37 AM

Member Since	‎01-27-2023 08:25 AM
Last Visited	‎03-14-2024 06:37 AM
Posts	229
Kudos received	73

Cloudera Community

Re: About mergecontent question

Re: how can get the content of Json record and val...

Re: DBCP Connection Pool can't connect to "Progres...

Re: terminate kafka connection if publish kafka pr...

Re: Not able to delete an inifinite loop built wit...

Re: How to load data from Google Big query table u...

Re: reading csv file with nifi

Re: How to populate audit columns using Nifi?

Re: PutBigQuery inserts null, whereas PutBigQueryB...

Re: Error with ParseEvtx processor

Re: Stuck loading after login

Re: Why is there an error on the colon in ${attrib...

Re: Is it possible to reference parameters within ...

Re: NiFi Optimization Record Oriented Processors

Re: create ADLS directory using PutZaureDataLakeSt...