i do not see any processors to connect to Google Big Query and execute queries.
is there a way where i can get results from google big query using NiFi.?
Unfortunately, there is no specialized processor to connect to Google Big Query and execute queries. There has been some discussions about a set of new processors to support various Google Cloud services, but those processors are still to be planned into a release.
Until then you can use ExecuteScript processor.
Here is an example on how to write a script using Python: https://cloud.google.com/bigquery/create-simple-app-api#bigquery-simple-app-print-result-python . At https://cloud.google.com/bigquery/create-simple-app-api you can see other examples using other languages also supported by ExecuteScript processor.
Obviously, there is always the possibility to develop your own processor leveraging the Java example provided by Google doc. Example of how to build NiFi custom processor:
If this response addressed reasonably your question, please vote and accept answer.
The community processor mentioned by Tim is a good example on how to write a custom processor. It is limited to Put action and quite old. You would have to rebuild it using more up-to-date libraries.
Community processors are not supported by Hortonworks.
You can use their API to download the data from those tables. Those examples show how to select the data. However, you may deal with a lot of data. You may want to extract it from BigQuery, store it in Google Cloud Storage Bucket (GCS) and connect NiFi to GCS which is supported nicely with GCS processors to list, fetch, put, delete from GCS. That is the most efficient way.
Look at this reference to see how to extract the data: https://cloud.google.com/bigquery/docs/exporting-data
You can schedule a job to extract and put to GCS bucket and NiFi will just pick it up.
My file names will change everyday like datafile_yyyymmdd.json , so we still need to create a scheduled job using Nifi and their RESTAPI to move the file to GCS.? Or is there a way that we can separate this .?