Member since
07-30-2019
2243
Posts
1230
Kudos Received
634
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
62 | 05-16-2022 08:31 AM | |
104 | 03-09-2022 11:30 AM | |
176 | 03-09-2022 09:18 AM | |
99 | 03-07-2022 09:06 AM | |
374 | 01-26-2022 06:49 AM |
05-17-2022
07:54 PM
Thanks @ckumar @MattWho for your responses. I enabled session affinity on LB and it resolved the issue. I am able to login using LB endpoint. What @ckumar said makes sense but why isn't it causing any issue for my cluster? Is it because I am using just basic LB and not Reverse proxy?
... View more
05-17-2022
06:09 AM
Can you share a screenshoot of current users listed on NiFI UI? Are the node listed? Also provide the related failed authorization messages from nifi-user.log please.
... View more
05-16-2022
12:30 PM
@Eric_B CFM 2.1.4 has been released. https://docs.cloudera.com/cfm/2.1.4/release-notes/topics/cfm-whats-new.html Thanks, Matt
... View more
05-16-2022
09:24 AM
@Pypro Just to add to what @gtorres asked you to try. Make sure that you are performing the action from the NiFi host as the NiFi service user. All components execute as the NiFi service user. And in this case the NiFi service user is trying to connect to the FTP endpoint and then pass the supplied credentials. The target FTP server may not be in the NiFi service users known_hosts file. Did you also try using the ListSFTP processor? Thanks, Matt
... View more
05-06-2022
12:17 PM
@sam_s0ni Verify that all 3 NiFi nodes are: Running the exact same version of NiFI. Running the exact same version of Java 8 or 11 (only supported by newest releases) Contents of the NiFi lib directory, extensions directory, and any custom lib directories contain exact same content. Try removing the NiFi work directory from all three nodes. On restart NiFi will rebuild the contents of the work directory from above lib and extensions folders. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
05-06-2022
12:05 PM
@Shanoj You really do not want to be using the Apache NiFi 0.x line anymore. That release version is more than 6 years old. Many many security bug fixes and improvements have been made since that time. Not to mention that the Cluster Manager was a single point of failure. If it goes down you lose all access to your NiFi. The Apache NiFi 1.x line introduced 0 master clustering allowing users to access NiFi from any node in the cluster. While I strongly encourage the use of an external zookeeper with Apache NiFi 1.x line, NiFi does offer and embedded zookeeper option. https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#embedded_zookeeper I encourage you to read through the following walkthrough documentation: https://nifi.apache.org/docs/nifi-docs/html/walkthroughs.html It includes sections on installing NiFi, Securing it, and deploying a cluster which even covers using the embedded zookeeper. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
05-06-2022
11:50 AM
@Ghilani While I agree that using record based processors so you can work with single FlowFiles with multiple records in them to make more efficient dataflows, what you are doing here should be possible with a ReplaceText processor in the interim using "Literal Replace": Here we are searching for the pattern _" and replacing it with just ". If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
05-06-2022
11:20 AM
@Brenigan The ExtractText processor will support 1 to 40 capture groups in a Java regular expression. The user added property defines the attribute in to which the value from capture group one will be placed. The processor creates additional attribute by capture group number. so in your case you added a new property with: This is a single capture group which reads 4 digits. So in you example (9999, text) this would result in creating attributes: number = 9999 <-- alway contains value from capture group 1. number.1 = 9999 <-- the ".1" signifies the capture group the value came from. number.0 contains the entire matching java regular expression. This attribute is controlled by this property: Setting to false will stop this one from being added to your FlowFiles. To help understand this better, let's look at another example: Suppose your java regular expression looked like this with 2 capture groups instead: Also assume we had "Include Capture Group 0" set to "true" Now with same source text of "9999, text", we would expect to see these attributes added: number = 9999 <-- alway contains value from capture group 1. number.0 = 9999, text <-- The complete match from the java regular expression. number.1 = 9999 <-- The ".1" signifies the capture group the value came from number.2 = text <-- the ".2" signifies the capture group the value came from. Setting "false" for "Include Capture Group 0" would have resulted in "number.0" not being created; however, number, number.1, and number.2 would have still been created. This functionality allows this processor component to handle multiple use cases. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
04-25-2022
11:05 PM
Hi All, we need to increase open file limit for nifi as a service not user , so add the below line in your nifi.service file and restart the server , it should resolve this issue LimitNOFILE=999999
... View more
04-20-2022
04:27 AM
@Naresh_n as this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post.
... View more
03-28-2022
08:35 AM
By checking the status of https://issues.apache.org/jira/browse/NIFIREG-313, it seems that the feature of authenticating using OIDC for nifi registry is already resolved as of 9/14/2020. But In nifi registry admin guide at https://nifi.apache.org/docs/nifi-registry-docs/index.html, it still mentions that only LDAP and Kerberos are supported. Is the document up-to-date?
... View more
03-25-2022
03:05 AM
Thanks @MattWho . It helped
... View more
03-16-2022
01:41 AM
Thinks for all your suggestion. By increasing HEAP size issue resloved. Thanks, Suryakant
... View more
03-14-2022
11:21 PM
This is now sorted. Batching using record based processor helped us to improve performance in multifold. didn't know that that the excess of flowfile might result in slowness in NIFI.
... View more
03-09-2022
11:30 AM
1 Kudo
@sachin_32 You can accomplish by utilizing the "Advanced UI" capability found in the UpdateAttribute processor. The advanced UI allows you to create Rules (think if these as an IF/Then capability). So you would setup 3 rules: 1. If current date falls on Mon - Fri, do X 2. if current date falls on Sat, do nothing 3. if current date falls on Sun, do Y Expression Language guide Below you can see I have created 3 rules (Day1-5, Day6, and Day7) Once you create a Rule, you need to provide a Condition (This is your boolean if statement) In this case I am using it to figure out what the current day of the week with 1= Monday and 7 = Sunday and seeing if the day of the week is prior to Sat or after Sat in the current week. If a rules condition (if statement) resolves to a boolean "true", then the configured "Actions" (then statement) are evaluated. For my "Day1-5" rule, I set: Condition: ${now():format('u'):lt(6)} Action: ${now():toNumber():minus(${now():format('u'):plus(1):multiply(86400000)}):toDate():format("EEE, dd MMM yyyy")} For my "Day6" rule, I set: Condition: ${now():format('u'):equals(6)} Action: ${now():format('EEE, dd MMM yyyy')} For my "Day7" rule, I set: Condition: ${now():format('u'):gt(6)} Action: ${now():toNumber():minus(86400000):toDate():format("EEE, dd MMM yyyy")} About above: - The "now()" function returns the current date. - 86400000 is the number of milliseconds in 1 day. - So first I get the current date and convert it to milliseconds using the "toNumber()" function. - Then for Day1-5, I am subtracting based on current day of week a multiple of days worth of milliseconds. - For Day6, I am doing nothing other than reformatting the current days date. - For Day7, I am just subtracting one day or 86400000 milliseconds No matter which rule is applied the final date format i choose to write to an attribute named "PreviousSaturday" on the FlowFile is formatted using java simple date format "EEE, dd MMM yyyy" Example: " Sat, 05 Mar 2022" If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
03-07-2022
09:18 AM
@nifideveloper You will need to use a NiFi Expression Language (NEL) statement to check if the date within the filename matches today's date. So first thing is isolate from the filename just the date portion and then compare it the output from NEL "now()" function output formatted in same way as source filename date pattern. Without a sample filename to work with, it would be difficult for me to provide an example NEL statement. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
03-07-2022
09:06 AM
2 Kudos
@richG Your observations are correct. The conversion script code does convert those properties. https://github.com/apache/nifi/blob/main/minifi/minifi-toolkit/minifi-toolkit-configuration/src/main/java/org/apache/nifi/minifi/toolkit/configuration/dto/RemotePortSchemaFunction.java I generated the following Apache NiFi jira: https://issues.apache.org/jira/browse/NIFI-9772 If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
02-04-2022
01:46 PM
Late response but... Are you running in FIPS mode?
... View more
02-01-2022
09:52 AM
@raghav130593 Where "*" indicates every value, the "?" can mean i don't care about the value. So you want to have corn execute at 17:30:00 every day, So i would recommend a "*" instead of "?" in the day of week field. A good example of when to use a ? in a supporting field would be this cron: 0 30 17 5 * ? The above says execute at 17:30:00 on the 5th of every month. The day of week field here is a "?" mark because the 5th of every month could fall on any one of the days of the week (will vary from month to month). Thanks, Matt
... View more
01-31-2022
10:06 PM
hi @MattWho , i checked with advance option also but wrong result is coming can you check this formula is it ok or not? ${mydate:toDate('MM/dd/yyyy'):format('u'):lt('6'):ifElse("${literal('6'):minus(${mydate:toDate('MM/dd/yyyy'):format('u')}):multiply('86400000'):minus(${mydate:toDate('MM/dd/yyyy'):toNumber()}):format('EEE,MM/dd/yyyy')}","${literal('518400000'):minus(${mydate:toDate('MM/dd/yyyy'):toNumber()}):format('EEE,MM/dd/yyyy')}")}
... View more
01-31-2022
02:29 PM
@rafy You'll want to read up on the documentation on the Apache MiNiFi page here: https://nifi.apache.org/minifi/index.html Since MiNiFi does not provide a UI from which you can construct a NiFi dataflow, you will need to build the dataflow that you will use on your MiNiFi using a NiFi installation. The converter toolkit is what you can then use to change your NiFi dataflow template into the necessary MiNiFi yaml file. You may also find these community posts helpful: https://community.cloudera.com/t5/Support-Questions/How-send-data-from-nifi-to-minifi-same-config/td-p/325183 https://community.cloudera.com/t5/Community-Articles/Ingesting-Log-data-using-MiNiFi-NiFi/ta-p/248154 If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
01-31-2022
02:05 PM
@vk21 This question is not related to the original question in this post. I recommend starting a new question, so as to avoid confusion via a new conversation when this post already has an accepted solution. Thanks, Matt
... View more
01-28-2022
12:54 AM
Hello, I'm a learner & i would like to use the method you made mentioned here to collect logs in a remote server & send to Nifi. Please, can you put me through because i have been battling with how to build a msi before the real implementation. Thank you so much.
... View more
01-25-2022
01:28 PM
@RonMilne I recommend taking your initial CSV record file and partitioning by the " salesRepId" in to multiple new JSON records. This can be accomplished using the PartitionRecord processor utilizing a CSVReader and a JsonRecordSetWriter. Your PartitionRecord processor configuration would look like this: Your CSVReader would be configured something like this (you'll need to modify it for your specific record's Schema: Note: Pop-out shows the "Schema Text" property and don't forget to set "Treat First Line as Header" property to "true" The JsonRecordSetWriter would need to be configured to produce the desired JSON record output format. However, just leaving default configuration will out put a separate FlowFile for each unique "SalesRepId". If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
01-24-2022
08:33 AM
Thanks @MattWho , actually found a way to filter/search process groups by name using the Summary option in the top right menu. This is very useful to find all the ETL pipelines once we give proper names and then by entering a part of the name we can show all matching Process Groups. Being a newbie I am trying to compare Streamsets UI to Nifi UI so I can work the same way. Streamsets provides an initial list of all ETL pipelines to filter by name etc. I guess if just after Nifi login if we saw two links: Summary and Canvas then users can click intuitively on the summary screen and review all their Process groups and click on specific PG they want to work with. This would make it similar to other ETL tools like Streamsets.
... View more
01-19-2022
06:01 AM
@OliverGong I would avoid dataflow design when possible where you are extracting the entire contents of a FlowFile to FlowFile attribute(s). While FlowFile content only exists on disk (unless read in to memory by a processor during processing), FlowFile attributes are held in NiFi's JVM heap memory all the time (There is per connection swapping that happens when a specific connection reaches the swap threshold set in the nifi.properties file). FlowFiles with lots of attributes and/or large attribute values will consume considerable amounts of JVM heap which can lead to JVM Out Of Memory (OOM) exceptions, long stop-the-world JVM Garbage Collection (GC) events, etc... When options exist that avoid adding large attributes, those should be utilized. Thanks, Matt
... View more
01-19-2022
05:45 AM
@Wisdomstar Thank you, I appreciate that and glad I could help. Matt
... View more
01-18-2022
10:43 AM
@LuisLeite Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks.
... View more
01-18-2022
08:34 AM
@Kilynn So as i mentioned in my last response, once memory usage go to high, OS level OOM Killer was most likely killing the NiFi service to protect the OS. The NiFi bootstrap process would have detected the main process died and started it again assuming OOM killer did not kill the parent process.
... View more
01-18-2022
08:23 AM
@oopslemon NiFi only encrypts and obscures values in properties that support sensitive properties (so those properties which are specifically coded as sensitive properties like "password" properties). So there is no way at this time to encrypt all or portions of property values not coded as sensitive. Keep in mind it is not just what is visible in the UI, your unencrypted passwords will be in plaintext with the NiFi flow.xml.gz file as well. My recommendation to you is to use mutual TLS based authentication instead. You can create a clientAuth certificate to use in your rest API calls. Then you need to make sure that your clientAuth certificate is authorized to perform the actions the rest-api call is making. This is not going to be possible while using the single user login mode as it does not allow you to setup additional users and authorizations. This single users authentication and authorization providers where added to protect users from unprotected access to their NiFis. It was not meant to be the desired choice when securing your NiFi. It is one step above an unsecured default setup that existed prior to NiFi 1.14. It protects you, but also has limitations that go with its very basic functionality. So step one is to switch to another method of authentication and authorization to you NiFi. TLS is always enabled for authentication as soon as NiFi is configured for HTTPS. You can configure additional authentication methods like ldap/AD. https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#user_authentication The authorizer configured in the authorizers.xml file allows you to establish policies that control user/client permissions. https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#multi-tenant-authorization Then you can configure your invokeHTTP processor to simply use a SSLContextService that you would configure with your clientAuth certificate keystore and a truststore. The password fields in this controller service would be encrypted. No more need to constantly get a new bearer token. All you need to worry about is getting a new client certificate before the old one expires which is typically every 2 years, but that is configurable when you create it and get it signed. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more