can I keep the JSON file as a constatnts in nifi itself? so that I can avoid the invokeHTTP call. every time I received a flow data, I just need to check the id from the flow data and using that i need to find the correspondent entry from the conastant json file and append those in to the flow data. ... View more

NiFi can be a powerful tool for orchestrating data flows relevant to B2B data building. While it excels at transferring data between CDP instances using processors like 'ListHDFS' and 'PutHDFS,' additional configuration might be needed for handling data deletion specific to B2B sources. For B2B data building, you'll likely be acquiring data from external sources, not HDFS. NiFi's capabilities can still be leveraged, but the specific processors used would depend on the data format and source. However, the challenge of handling data deletion remains. Tools like 'GetHDFSEvents' might not be applicable for B2B data sources. ... View more

To retrieve data records after crm enrichment added since the last time you polled an Impala table using a Date/Timestamp column, you can use a SQL query with a WHERE clause filtering for records with timestamps greater than the last timestamp you retrieved. Here's a basic example assuming your timestamp column is named timestamp_column: SELECT * FROM your_table WHERE timestamp_column > 'last_poll_timestamp';   Replace 'last_poll_timestamp' with the actual timestamp value you stored from your last poll. Make sure the timestamp format matches the format stored in your table. Here's a step-by-step guide: Store the timestamp of the last poll in your source system. Use this timestamp to construct your SQL query, ensuring you're retrieving records with timestamps greater than the last poll timestamp. Execute the SQL query against your Impala table to retrieve the new records. ... View more

Yes, you can achieve the same lead data enrichment  using the RestLookupService. The format of the data that the API should return depends on the specific requirements of your use case, but it typically should be in a format that can be easily processed and integrated into your data flow. For example, if you are looking to perform simple key-value lookups, the API could return data in JSON format with a structure similar to the following: { "key1": "value1", "key2": "value2", ... } You can then use the values returned from the API to update or add fields in your data flow by mapping the keys in the API response to fields in your data. It's important to note that the RestLookupService can handle a variety of data formats, including JSON, XML, or even plain text. The choice of format depends on the specific requirements of your use case and the capabilities of the API you are using. ... View more

To perform a data lookup using MS SQL as the datasource in NiFi, you can use the LookupRecord processor along with the DBCPConnectionPool and ScriptedLookupService. Here is an outline of the steps you can follow: Create a DBCPConnectionPool service: Go to the NiFi UI and click on the gear icon to access the Controller Services. Click on the "+" button and select "DBCPConnection Pool" Configure the connection pool with the details of your MS SQL database, such as the driver class name, URL, username, password, etc. Create a ScriptedLookupService: In the same Controller Services screen, click on the "+" button and select "ScriptedLookupService". In the ScriptedLookupService, you need to write a Javascript code that will be used to perform the lookup. You can refer to the example code provided in the forum topic How to get DBCP service inside ScriptedLookupService. Configure the LookupRecord processor: Add a LookupRecord processor to your data flow. In the Properties tab, set the "Lookup Service" to the ScriptedLookupService you created in step 2. In the same Properties tab, set the "Connection Pool" to the DBCPConnectionPool you created in step 1. Connect the LookupRecord processor to your data flow, with the input flow file containing the data that you want to enrich.   ... View more