Member since
07-30-2019
3471
Posts
1642
Kudos Received
1020
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 150 | 06-03-2026 06:06 PM | |
| 460 | 05-06-2026 09:16 AM | |
| 830 | 05-04-2026 05:20 AM | |
| 498 | 05-01-2026 10:15 AM | |
| 624 | 03-23-2026 05:44 AM |
10-29-2019
01:34 PM
1 Kudo
@big-f The work to support bundles is part of NiFi-Registry 0.4.0 https://nifi.apache.org/docs/nifi-registry-docs/html/administration-guide.html#bundle-persistence-providers https://nifi.apache.org/docs/nifi-registry-docs/html/user-guide.html#manage-bundles https://issues.apache.org/jira/browse/NIFIREG-211
... View more
10-29-2019
11:04 AM
1 Kudo
@big-f NiFi-Registry storage of version controlled flows is only the beginning use case for NiFi-registry. 1. Additional features of NiFi-Registry in the pipe line include: - Storing of FlowFile parameters (NiFi parameters can be used by any property in any component in NiFi.) - Storing of NiFi nars (NiFi has grown to be a very large application. Rather then installing every single component that exists, users can set up local registries that store nars. Then these nars can be pulled and dynamically loaded during runtime in NiFi as needed by the flow which has been built.) 2. The only method of deploying flows other than through NiFi-Registry is the legacy templates. but even here you have an existing NiFi canvas (blank or with other existing flows). You then import and instantiate templates on to that existing canvas somewhere, Downsides to this legacy method include no version control and templates are stored in heap memory. Even if a template is not instantiated to the canvas, it is held in heap memory. More templates equals more heap usage. 3. I think this question is answered by point number two in answer 1. A NiFi FlowFile is what moves from processor to processor via a connection. Not sure what you are referring to when you say "flow file"? Are you talking about the flow.xml.gz?
... View more
10-28-2019
11:54 AM
@Dale1979 It would appear you have a multi-node NiFi cluster that is using the NiFi CA to issue the certificate for each NiFi node. Within a NiFi cluster, one node will be elected cluster coordinator. While you can access your NiFi cluster from any node in the cluster, any user requests (including displaying the UI) need to be replicated to the cluster coordinator. This replication request is performed by the node which you are authenticated in to on behalf of you as the user. Thus requiring that the node itself has been authorized to act as a proxy for your user. Within your authorizers.xml file you should find a file-user-group-provider which should have been configured a separate "Initial User Identity <num>" property for each of your NiFi nodes. The file-user-group-provider is used add the initial users in the users.xml file. The value would match the DN of the certificate found in each node's keystore.jks file. Using you output as an example, you should see a line like this: <property name="Initial User Identity 2">CN=server.domain:port, OU=NIFI</property> Above would be followed by additional similar lines with new number for each of your other NiFi nodes. Note: Having a port number in your CN name is unexpected. IMPORTANT: NiFi will only generate the users.xml and authorizations.xml files from the configuration in the authorizers.xml file the very first time. If you make edits to either the file-user-group-provider (build users.xml) or File-access-provider (builds authorizations.xml), those changes will not be reflected in the already existing users.xml and authorizations.xml files. You must delete these files so they are recreated. The expectation is that following successful securing of NiFi, all additional users and authorizers are added directly via the NiFi UI. Hope this helps, Matt
... View more
10-18-2019
06:13 AM
@maybegoodman NiFi is designed to operate on data which has been consumed by NiFi and placed in FlowFiles. That being said, it sounds like what you are trying to do here is zip all the files on the target SFTP server before consuming the zipped file by NiFi. Perhaps you could use the ExecuteStreamCommand processor to use ssh to manually zip the contents of the target SFTP server based on a configured scheduling cron. Then have your listSFTP processor only list .zip file instead. if the ExecuteStreamCommand is not adequate, there are several Script based processors perhaps you can use to write a custom script that handles step 1 fo connecting to SFTO server and zipping the files. Hope this helps, Matt
... View more
10-17-2019
06:54 AM
@ilyal What version of NiFi are you running? Is it Apache NiFi 1.8.x? There are numerous bugs with the new load balanced connections feature. The good number of these known bugs have been addressed between NiFi 1.9.0 and NiFi 1.9.2. https://issues.apache.org/jira/browse/NIFI-5745 https://issues.apache.org/jira/browse/NIFI-5919 https://issues.apache.org/jira/browse/NIFI-5663 https://issues.apache.org/jira/browse/NIFI-5771 https://issues.apache.org/jira/browse/NIFI-6017 There are still some additional bugs that are fixed in NiFi 1.10.0 https://issues.apache.org/jira/browse/NIFI-6353 https://issues.apache.org/jira/browse/NIFI-6760 https://issues.apache.org/jira/browse/NIFI-6517 https://issues.apache.org/jira/browse/NIFI-6736 https://issues.apache.org/jira/browse/NIFI-6285 https://issues.apache.org/jira/browse/NIFI-6759 I strongly recommend upgrading to Apache NiFi 1.10 as a first step upgrading (releasing soon). Hope this helps, Matt
... View more
10-15-2019
06:14 AM
1 Kudo
@Gcima009 It appears you are running CFM 1.0.0 which has a known issue with the sensitive props key. If a sensitive props key is not set in CFM NiFi configs, Cm randomly creates one for you. The bug here is that a different random sensitive props key is generated on each node. Since NiFi requires that all nodes in the flow have the exact same flow.xml.gz, the sensitive props key used to decrypt passwords within that flow.xml.gz must also be the same. This bug was addressed in CFM 1.0.1 Anytime you manually edit the files in /var/run/cloudera-scm-agent/process/<num>-nifi-NIFI_NODE and then restart that service via CM, it will detect a change and build new configs in a new folder. To get around this issue... 1. On one of the nodes cp the "config.zip" file out of the most recent <num>-nifi-NIFI_NODE to a tmpt directory (/tmp/config) 2. Extract the nifi-globals.xml file ( unzip config.zip staging/nifi-globals.xml ) 3. Extract the random NiFi sensitive props value ( grep -A 1 random.nifi.sensitive staging/nifi-globals.xml ) 4. Shutdown your NiFi cluster via CM. 5. Use the password obtained via step 3 from one node to manually set the password in CM NiFi configs 6. copy flow.xml.gz frorn same node where you obtained the random password to all other nodes in your cluster. 7. start your NiFi cluster. Now all nodes are running with same flow.xml.gz using same sensitive props key password. Thanks, Matt
... View more
10-14-2019
08:37 AM
2 Kudos
@bha This can be done via a single long complex NiFi Expression Language (EL) statement or via a couple smaller EL statements within the "Advanced" Ui of the UpdateAttribute processor. Before you can use NiFI EL to solve this query, you will need to get your date in to a FlowFile attribute. Since i have no idea where the date 10/12/2019 originates from, I can't help much here. If it is on the content, perhaps extractText processor can be used. Lets assume we have a FlowFile with the following FlowFile attribute on it: FlowFile attribute name: FlowFile Attribute value: mydate 10/12/2019 We can pass this FlowFile through an UpdateAttribute processor where we create a new attribute with the date of the subsequent Sunday. (My EL assumes that if input date is already Sunday, it will calculate the next Sunday instead of just reporting that today is Sunday) Solution1 (one long NiFi EL statement): The full EL is below (all one line): ${mydate:toDate('MM/dd/yyyy'):format('u'):lt('7'):ifElse("${literal('7'):minus(${mydate:toDate('MM/dd/yyyy'):format('u')}):multiply('86400000'):plus(${mydate:toDate('MM/dd/yyyy'):toNumber()}):format('EEE,MM/dd/yyyy')}","${literal('604800000'):plus(${mydate:toDate('MM/dd/yyyy'):toNumber()}):format('EEE,MM/dd/yyyy')}")} NOTE: Be careful with copy paste as it may screw up the single and double quote marks in the above EL making the EL statement invalid. What above does is calculate day of week (1 - 7)... IF - Then if that number is less then 7: --- it subtracts it from 7 to determine how many day until next Sunday remain. --- Then multiples that number by number of millseconds in a day --- Then adds that number to the millisecond value of your input date. --- Then converts that new millisecond value back in to a date again (format here is Sun,10/13/2019) ELSE - When number is not less then 7 --- It adds 7 days in milliseconds to number to the millisecond value of your input date. --- Then converts that new millisecond value back in to a date again (format here is Sun,10/13/2019) So essential you have tow EL statements handled by an ifElse function. One covers days 1 -6 and the other covers day 7. Solution 2 (Using "Advanced" UI of UpdateAttribute to break apart above long EL): The Advanced Ui allows you to construct rules. Only the rules where the "conditions" resolve to "true" will apply the actions. So we simply create two rules. One for days 1-6 and the other for when day of week is 7. Day less than 7: Day equal 7: Using the "Advanced"UI makes handling complex EL statements easier. Hope this helps, Matt
... View more
10-11-2019
08:42 AM
@littlesea374 There is no capability to externalize these rules used in the UpdateAttribute processor. There are numerous "lookup" processors within NiFi; however, none of them integrate with git. I think you may need a custom processor to do what you are looking for. Contributions to the Apache NiFi community are always welcome. NiFi also offers numerous scripting processors which allow you to write your own scripts. Matt
... View more
10-11-2019
07:38 AM
Hello @shrujana The ListenHTTP processor setups an internal Jetty server that listens for inbound http connections on the configured port. It is not used to establish connections to external services. It may e helpful if you provide more detail about your use case(s) along with your current dataflow design specifics. How do you interface with your micro-services when not using NIFi?
... View more
10-11-2019
06:58 AM
Hello @littlesea374 The PutS3Object processor supports NiFi Expression Language on the majority of its properties. https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-aws-nar/1.5.0/org.apache.nifi.processors.aws.s3.PutS3Object/index.html So rather then routing and creating a separate putS3Object processor for each unique setup of FlowFiles, you may consider using the Advanced UI ( ) in the UpdateAttribute to set FlowFile attributes to unique values based on your routing rules. You essentially create a new "Rule" for each of your unique match criteria. For each Rule you create an expression. If the expression for a rule matches (resolves to "true"), all the actions will be applied to the matching FlowFile. Those actions would simply be setting FlowFile attributes for the putS3Object processor properties. Designing your dataflow in this manor allows you to scale up by simply adding new rules to the UpdateAttribute processor. The only time you would need a different putS3Object processor is if some match requires a different property configured that does not support NiFi Expression Language (for example: different AWS credentials or region) Hope this helps, Matt
... View more