@YasBHK HDP is no longer supported, there will be no releases and no upgrades.   I highly recommend not solutioning or training on the HDP Sandbox.   I would recommend that you investigate the modernized alternatives to HDP, such as CDP Private Cloud Base or CDP Public Cloud. ... View more

@udayabaski  This sounds like a job for DistributedMapCache.  Using this, you can set key/value pairs for start and end time at one point in the flow, and access them in other parts of the flow.   Here are some important docs to get you started :   https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-distributed-cache-services-nar/1.20.0/org.apache.nifi.distributed.cache.server.map.DistributedMapCacheServer/index.html https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.20.0/org.apache.nifi.processors.standard.PutDistributedMapCache/index.html https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.20.0/org.apache.nifi.processors.standard.FetchDistributedMapCache/index.html         ... View more

 @cotopaul   - Thanks for being such a community superstar!! @swanifi in addition to above, In the future please be sure to include screen shot of the processor property and configurations. One thing that i think you could achieve is some type of an upstream gate on the flow files, so that they are released in blocks that satisfy the expectation of 1000 records per operation.    I am not aware if this is configureable within the processor.    ... View more

@bennour This error is an issue with your database endpoint ssl.   My first suggestion is to use the following arugment in your sqoop command:   useSSL=false   then if this doesn't work, focus on getting a proper ssl certificate for the host.     ... View more

The file you need to send or upload to the api, should be the content of your flowfile that routes into InvokeHttp.   So some upstream process should read the file, and send the content through success relationship to InvokeHttp.   For example if this file is json,  I use GenerateFlowFile processor, put the json contents in there, then send to InvokeHttp. ... View more

@Techie123 to add headers in invokeHttp just click the + sign to add a new dynamic property, and set key to "Content-Type" and value to "multipart/form-data".   You will also need to make sure the rest of your invokeHttp is confirmed correctly.  Depending on your version, you can also create and send an attribute called "Content-Type" of the same value.  Newest versions of NiFi will see a default property "Request Content-Type" which can also be set to attribute $mime-type which you can also define to suit your API.     One other suggestion,  make sure you are testing remote API Calls with something like postman to confirm all the required values.   Then, work with NiFi after you have a known operational understanding for the api. ... View more

@Fahmihamzah84 This appears to be an issue with your schema.  The BigQuery error is suggesting an issue trying to cast a string into a collection (array/list/ect).   It's hard to tell which array may be causing the issue as there are many.   My suggestion is to set the processor to log level DEBUG and see if you can get more verbose error.   This will help you figure out which field or fields is the culprit.      Keep in mind it could be one of the empty arrays too.     I do not suggest the following as a solution just as path to figuring out where the problem is.  Sometimes when i have issues with type casting,  i make everything a string temporarily and for development.  If you do this carefully one at a time, when the error goes away, you can determine which field it is.   This also helps you identify a working state for your flow and allow you to work from that operational base to find solution for the end schema being the format you need. ... View more

@anton123 Please check these references, there is important information here that everyone needs to master in order to work with ExecuteScript: https://community.hortonworks.com/articles/75032/executescript-cookbook-part-1.html https://community.hortonworks.com/articles/75545/executescript-cookbook-part-2.html https://community.hortonworks.com/articles/77739/executescript-cookbook-part-3.html   Some suggestions: Make sure you have all the imports you need Make sure you are using the correct variable names for "filter".  You have this defined but its not used. ... View more

Awesome news, +2 solutions here.   ... View more