How to load data from Google Big query table using Apache Nifi?

Question

I need to load the data from big query table using nifi.

I used executesql nifi processor to load data from big query table.

when i was executing processor, i got following error.

java.lang.NoSuchMethodError: 'void com.google.api.gax.rpc.RequestUrlParamsEncoder.(com.google.api.gax.rpc.RequestParamsExtractor)'

configure DBCPConnectionPooling Service in following class name - com.simba.googlebigquery.jdbc42.Driver

I am using nifi1.21.0 version. download latest version of big query jdbc driver(version 4.2) and other all dependencies and put into nifi/lib folder

I also used correct connection url when configuring pooling service.

How to solve this error?

cotopaul · Accepted Answer

the question is how? 😀I just tested again and I was able to extract data from (and into) BigQuery without any errors. It seems that something you have defined is not 100% correct.Circling back to my initial post, you would need those defined as:Database Connection URL: jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=your-project-if-here;OAuthType=0;OAuthServiceAcctEmail=your-email-for-service-account-here;OAuthPvtKeyPath=path_on_nifi_server_where_the_service_account_json_is_located; Database Driver Class Name: com.simba.googlebigquery.jdbc.DriverDatabase Driver Location: full_path_to_jars_location/Where the Database driver location has to be the full path to the folder where you have extracted the JDBC (not ODBC) ZIP File... and you only have to point to the folder, not to a specific JAR file.In addition, pay attention to how the connection URL is configured and make sure that you follow the same structure. Plus, make sure that your SA has all the necessary rights for your actions.

cotopaul · Answer

@Sivaluxan,I am not quite sure you have the correct Database Driver Class Name. I am extracting data our of BigQuery using the combination of GenerateTableFetch and ExecuteSQLRecord and I receive no error message at all.In terms of configurations I have the following:Database Connection URL: jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=your-project-if-here;OAuthType=0;OAuthServiceAcctEmail=your-email-for-service-account-here;OAuthPvtKeyPath=path_on_nifi_server_where_the_service_account_json_is_located; Database Driver Class Name: com.simba.googlebigquery.jdbc.DriverDatabase Driver Location: full_path_to_jars_location/

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.