watsonx_bearer_token- IBM's IAM Token that you retrieved earlier in the prerequisites.
Extra Small NiFi node size is enough for this data ingestion.
After deployment is done, you would be able to see the flow in Dashboard.
All NiFi Flow parameters can be updated while the flow is running, from Deployment Manager. As soon as you Apply Changes, running processors that are impacted by the Parameter changes will automatically be restarted.
Step #2 - Setup Cloudera Data Warehouse (CDW)
Go to CDW user interface. Ensure CDW service is activated in your CDP environment, and a Database Catalog & a Virtual Warehouse compute cluster are available for use.
In Hue editor, executequery.sql. This query creates an external table that points to your S3 Bucket's output path.Please change AWS S3 location in the query before executing it.
After the query execution is successful, you will seemodel_responsetable under default database.
Step #3 - Execute
Ensure your Cloudera Watsonx Flow is started. If it's not, do the following to start it -- CDF Dashboard >> Deployment Manager >> Action >> Start Flow.
Drop files in your S3 Bucket's input path. A couple of sample input files are provided inassetsdirectory for reference.
After a few seconds, notice the output in your S3 Bucket's output path.
You can also go in Hue and query the table -SELECT * FROM default.model_response;.
In the end, a notification email goes out to the user acknowledging the receipt of the document.