Member since
07-30-2019
3406
Posts
1622
Kudos Received
1008
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 136 | 12-17-2025 05:55 AM | |
| 197 | 12-15-2025 01:29 PM | |
| 137 | 12-15-2025 06:50 AM | |
| 260 | 12-05-2025 08:25 AM | |
| 421 | 12-03-2025 10:21 AM |
10-14-2019
08:37 AM
2 Kudos
@bha This can be done via a single long complex NiFi Expression Language (EL) statement or via a couple smaller EL statements within the "Advanced" Ui of the UpdateAttribute processor. Before you can use NiFI EL to solve this query, you will need to get your date in to a FlowFile attribute. Since i have no idea where the date 10/12/2019 originates from, I can't help much here. If it is on the content, perhaps extractText processor can be used. Lets assume we have a FlowFile with the following FlowFile attribute on it: FlowFile attribute name: FlowFile Attribute value: mydate 10/12/2019 We can pass this FlowFile through an UpdateAttribute processor where we create a new attribute with the date of the subsequent Sunday. (My EL assumes that if input date is already Sunday, it will calculate the next Sunday instead of just reporting that today is Sunday) Solution1 (one long NiFi EL statement): The full EL is below (all one line): ${mydate:toDate('MM/dd/yyyy'):format('u'):lt('7'):ifElse("${literal('7'):minus(${mydate:toDate('MM/dd/yyyy'):format('u')}):multiply('86400000'):plus(${mydate:toDate('MM/dd/yyyy'):toNumber()}):format('EEE,MM/dd/yyyy')}","${literal('604800000'):plus(${mydate:toDate('MM/dd/yyyy'):toNumber()}):format('EEE,MM/dd/yyyy')}")} NOTE: Be careful with copy paste as it may screw up the single and double quote marks in the above EL making the EL statement invalid. What above does is calculate day of week (1 - 7)... IF - Then if that number is less then 7: --- it subtracts it from 7 to determine how many day until next Sunday remain. --- Then multiples that number by number of millseconds in a day --- Then adds that number to the millisecond value of your input date. --- Then converts that new millisecond value back in to a date again (format here is Sun,10/13/2019) ELSE - When number is not less then 7 --- It adds 7 days in milliseconds to number to the millisecond value of your input date. --- Then converts that new millisecond value back in to a date again (format here is Sun,10/13/2019) So essential you have tow EL statements handled by an ifElse function. One covers days 1 -6 and the other covers day 7. Solution 2 (Using "Advanced" UI of UpdateAttribute to break apart above long EL): The Advanced Ui allows you to construct rules. Only the rules where the "conditions" resolve to "true" will apply the actions. So we simply create two rules. One for days 1-6 and the other for when day of week is 7. Day less than 7: Day equal 7: Using the "Advanced"UI makes handling complex EL statements easier. Hope this helps, Matt
... View more
10-11-2019
08:42 AM
@littlesea374 There is no capability to externalize these rules used in the UpdateAttribute processor. There are numerous "lookup" processors within NiFi; however, none of them integrate with git. I think you may need a custom processor to do what you are looking for. Contributions to the Apache NiFi community are always welcome. NiFi also offers numerous scripting processors which allow you to write your own scripts. Matt
... View more
10-11-2019
07:38 AM
Hello @shrujana The ListenHTTP processor setups an internal Jetty server that listens for inbound http connections on the configured port. It is not used to establish connections to external services. It may e helpful if you provide more detail about your use case(s) along with your current dataflow design specifics. How do you interface with your micro-services when not using NIFi?
... View more
10-11-2019
06:58 AM
Hello @littlesea374 The PutS3Object processor supports NiFi Expression Language on the majority of its properties. https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-aws-nar/1.5.0/org.apache.nifi.processors.aws.s3.PutS3Object/index.html So rather then routing and creating a separate putS3Object processor for each unique setup of FlowFiles, you may consider using the Advanced UI ( ) in the UpdateAttribute to set FlowFile attributes to unique values based on your routing rules. You essentially create a new "Rule" for each of your unique match criteria. For each Rule you create an expression. If the expression for a rule matches (resolves to "true"), all the actions will be applied to the matching FlowFile. Those actions would simply be setting FlowFile attributes for the putS3Object processor properties. Designing your dataflow in this manor allows you to scale up by simply adding new rules to the UpdateAttribute processor. The only time you would need a different putS3Object processor is if some match requires a different property configured that does not support NiFi Expression Language (for example: different AWS credentials or region) Hope this helps, Matt
... View more
10-10-2019
10:51 AM
1 Kudo
Hello @Gcima009 The flow.xml.gz includes all passwords set in the components you added within the NiFi UI. Every passwords would have be encrypted using the sensitive properties password set in the nifi.properties file: nifi.sensitive.props.key= The stack trace you shared is telling us that this NiFi is unable to decrypt the sensitive properties encrypted in the flow.xml.gz file provided to this NiFi using the sensitive properties password configured in the nifi.properties file. In other words the NiFi were you copied this flow.xml.gz file from is using a different sensitive properties key. Edited the nifi.properties file on this NiFi so that it is using the same password used by the NiFi were you copied the flow.xml.gz file from. Hope this helps, Matt
... View more
10-10-2019
10:15 AM
Hello @girish6 Your first EL statement would work as well once you correct the single quote marks around the Z. Many outside text editors end up messing up the single quotes, so the NiFi EL ends up being incorrect when you copy/paste. NiFi's EL editor built in to the processor property text blocks can help identify when the EL format is wrong. In above you can see your NiFi EL statement as copied and pasted on line 3. You'll notice the color coding indicates an issue with the function's input parameters. Line 1 has the properly formatted version of that same NiFi EL statement. Be careful when using copy paste from some external editors. Thank you, Matt
... View more
10-10-2019
09:58 AM
1 Kudo
Hello @girish6, NiFi Expression language does have a function that will increment a number, but it does not reset back to zero (unless you restart NiFi) and is incremented globally across call calls to that function. So while this function will give each FlowFile a unique filename per NiFi node, there is no guarantee that they will be sequential or will start at 0 for new source zip file you extract. If this works for your use case the NiFi Expression language statement would look like this in the UpdateAttribute processor: If you are just looking for filename uniqueness, a better solution may be to instead append the FlowFile's uniquely assigned UUID to the filename instead as follows: Tips: 1. There is an expression language guide in the embedded help of your NiFi installation found by clicking on the global menu icon in upper right corner. 2. Within a NiFi processor property that support NiFi EL you can display a list of functions with details by typing "${" (signifies start of an EL statement) and then hold "Control" while clicking space bar. This also works if you are mid statement and type ":"then "control" + click spacebar. Examples: file-${ (then cntrl+spacebar) ${uuid: (then cntrl+spacebar) Hope this helps, Matt
... View more
10-09-2019
10:47 AM
Hello David, The ReplaceTextWithMapping processor has been around for some time now and its design is rather simplistic in its current implementation. It is designed to match a simple regex that will map one group of non-whitespace characters to another group of characters (can have white space and back references). in your example you are matching on a a set of characters that contains whitespaces. If you were to add a replaceText processor to your flow to replace all the whitespace with a non-whitespace character like "-", then it will work. sample ReplaceText processor configuration: Your mapping file would of course then be based off this change: account-sing-up asu
account-sign-in asi
account-sign-out aso the resulting mapping would then work: Thank you, Matt
... View more
05-10-2019
07:55 PM
@sho aa I strongly recommend starting a new question in the community versus adding new questions and setup scenarios under an existing question. You current "Answer" is a new question which is not directly related to the issue expressed in the original question of this post. Creating a new question gets that question exposed to more users of this forum. I am not an HBase Subject Matter Expert (SME). As far as NiFi is concerned, how the NiFi Hbase processor within HDF's NiFi and the Hbase processor in Apache NIFi is no different. The Error you are showing in the screenshot states that the CS is unable to resolve the hostname "sandbox-hdp.hortonworks.com" which means it can no figure out what the IP address is. If it cannot resolve the hostname, it is not going to be able to communicate with that endpoint. While the config come from the provided site.xml files to the CS, there may be challenges accessing ports with the sandbox from an external resources. I am also not an expert with the HDP sandbox. If above tips do not help, best option is to start a new question that will engage more forum users. Thank you, Matt
... View more
05-08-2019
08:51 PM
@3nomis MergeContent should be using Defragment. There is no default value for Max Bin Age, so not sure what you set there. If left blank, processor will wait for ever to merge a bin unless you run out of bins. Also make sure you adjust the object and size thresholds on the connections feeding the MergeContent processors so that they are large enough to accommodate the number of splits that need to be merged. Considering the size of the FlowFiles being merged, it may take time to merge all of them. as far as bins, try setting 21 of them. Thanks, Matt
... View more