Member since
07-30-2019
3471
Posts
1642
Kudos Received
1020
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 143 | 06-03-2026 06:06 PM | |
| 458 | 05-06-2026 09:16 AM | |
| 821 | 05-04-2026 05:20 AM | |
| 493 | 05-01-2026 10:15 AM | |
| 620 | 03-23-2026 05:44 AM |
04-25-2017
05:25 PM
@Simon Jespersen Posted answer to above question here: https://community.hortonworks.com/questions/98384/listsftp-failed-to-obtain-connection-to-remote-hos.html
... View more
03-16-2017
04:01 PM
@Matt Clarke : thanks 🙂
... View more
03-16-2017
02:06 PM
1 Kudo
@Thangarajan Pannerselvam
If your GetFTP processor is configured with "delete original" set to false, every time this processor runs it will pull all th the files it finds including those pulled in the last run of the getFP processor. The ListFTP processor maintains state unlike the GetFTP processor. so if you replace your GetFTP with both ListFTP and FetchFTP processors, you will not see the same files pulled twice unless the timestamp on the files on the FTP server are updated. Thanks, Matt
... View more
03-13-2017
04:30 PM
Thank you!! That worked perfectly! Feel free to check out my NiFi template where this is in use! original-content-and-attributes-in-xmlv4.xml
... View more
05-10-2017
12:09 PM
Hi Matt, I'm trying to do the same flow, but I need to get the newest file from an FTP server. I tried to set the time to 600000ms but the flow did not work. ${file.lastModifiedTime
:toDate("yyyy-MM-dd'T'HH:mm:ssZ")
:toNumber()
:ge(${now():minus(21600000)})
} How can I get the newest file? Thanks in advance, Thais , Hi Matt! I'm trying to do the same flow, but I need to get the newest file from an FTP. Ho can I do that? I tried to use 600000 (1 minute) but it did not work. ${file.lastModifiedTime
:toDate("yyyy-MM-dd'T'HH:mm:ssZ")
:toNumber()
:ge(${now():minus(6000000)})
} Can you help?
... View more
03-10-2017
05:49 AM
@Matt Clarke This is what i found in nifi-user.log I had created server certificate, client certificate and a CAcert holding signature of both from tinycert.org. I had mentioned server certificate and CAcert path in nifi.properties and was loading client certificate into my browser. I was using same certificate for both the nifi instances. Like you said I had added the client cert DN under the users but nifi was using server cert DN (which i got to know after seeing the nifi-user.log) for site-to-site authentication. So, i added the server DN under the user and added the same under 'retrieve site-to-site details' policy. Now the issue is fixed. The RPG is now able to establish connection. Thanks, Harshith
... View more
02-24-2017
01:51 PM
Can you take a thread dump and provide the output here? ./bin/nifi.sh dump /path/to/output/dump.txt
... View more
02-23-2017
07:06 PM
5 Kudos
There is a two part process before any access to NiFi UI is possible: 1. Authentication: By default NiFi will use a user/server's SSL certificate when provided in the connection to authenticate. When NO user/server certificate is presented, NiFi will then look for a Kerberos TGT (If Spnego has been configured in NiFi). Finally, if neither of the above where present in the connection, NiFi will use the login identity provider (if configured). Login identity providers include either ldap or kerberos. With both of these options, NiFi will present users with a login screen. 2. Authorization: Authorization is the mechanism that controls what features and components authenticated users are granted access. The default authorizer NiFi will use is the internal file based authorizer. There is an option to configure NiFi to use Ranger as the authorizer instead. The intent of this article is not to discuss how to setup NiFi to use any of the Authentication or Authorizer options. This article covers how to modify what identity is passed two the Authorizer after any one of the authentication mechanism is successful. What is actually passed to the authorizer varies depending on which Authentication method is in use. SSL certificates: Default, always enabled, and always checked first NiFi uses the full DN from the certificate. Spnego (kerberos): Always on when enabled and only used if a SSL Certificate was not present in connection. NiFi uses the full user principal. ldap-provider (option in login-identity-providers): Always on once configured and only used if both SSL certificate and TGT (if Spnego was enabled) are not present in connection. Default configuration of ldap-provider will use the full DN returned by LDAP upon successful authentication. (USE_DN Identity Strategy) Can be configured to pass the username used to login instead. (USE_USERNAME Identity Strategy) Kerberos-provider (option in login-identity-providers): Always on once configured and only used if both SSL certificate and TGT (if Spnego was enabled) are not present in connection. The kerberos-provider will use the use the user full principal upon successful authentication. (USE_DN Identity Strategy) Whether you choose to use the built in file based authorizer or optional configure you NiFi to use Ranger instead, users must be added and granted various access policies. Adding users using either full a DN or users principal can be both annoying and prone to errors since the authorizer is case sensitive and white spaces are valid characters. This is where NiFi's identity mapping optional configurations come in to play. Identity mapping takes place after successful authentication and before authorization occurs. It gives you the ability to take the returned value from all four of the authentication methods and pass them through 1 or more mappings to produce a simple resulting value which is then passed to your authorizer. The identity mapping properties are configured in NiFi's nifi.properties file and consist of two parts to each mapping you define: nifi.security.identity.mapping.pattern.<user defined>=
nifi.security.identity.mapping.value.<user defined>= The mapping pattern takes a java regular expression as input with the expectation that one of more capture groups are defined in that expression. One or more of those capture groups are then used in the mapping value to create the desired final result that will be passed to your configured authorizer. **** Important note: If you are implementing pattern mapping on a existing NiFi cluster that is already running securely, the newly added mappings will be run against the DNs from the certificates created for your nodes and the Initial Admin Identity value you originally configured. If any of your mapping match, a new value is going to passed to your authorizer which means you may lose access to your UI. Before adding any mapping make sure you have added the new mapped value users to your NiFi and authorized them so you do not lose access. By default NiFi includes 2 example identity mappings commented out in the NiFi properties file: You can add as many Identity mapping pattern and value as you like to accommodate all your various user/server authentication types. Each must have a unique identifier. In the above examples the unique identifiers are "dn" and "kerb". You could add for example "nifi.security.identity.mapping.pattern.dn2=" and "nifi.security.identity.mapping.value.dn2=" If you are using Ambari to install and manage your NiFi cluster (HDF 2.x version), you can find the 2 sample identity mapping properties under "Advanced nifi-properties": If you want add additional mappings beyond the above 2 via ambari, these would be added via the "Custom nifi-properties" config section. Simply click the "Add Property..." link to add your new mappings. The result of any successful authentication is run through all configured identity mapping until a match is found. If no match is found the full DN or user principal is passed to the authorizer. Let's take a look at a few examples: User/server DN or Principal Identity Mapping Pattern Identity Mapping Value Result passed to authorizer CN=nifi-server-01.openstacklocal, OU=NIFI ^CN=(.*?), OU=(.*?)$ $1 nifi-server-01 CN=nifi-01, OU=SME, O=mycp, L=Fulton, ST=MD, C=US ^CN=(.*?), OU=(.*?), O=(.*?), L=(.*?), ST=(.*?), C=(.*?)$ $1@$2 nifi-01@SME nifi/instance@MY.COMPANY.COM ^(.*?)/instance@(.*?)$ $1@$2 nifi@MY.COMPANY.COM cn=nifi-user1,ou=SME,dc=mycp,dc=com ^cn=(.*?),ou=(.*?),dc=(.*?),dc=(.*?)$ $1 nifi-user1 JohnDoe@MY.COMPANY.COM ^(.*?)@(.*?)$ $1 JohnDoe ^EMAILADDRESS=none@none.com, CN=nifi-user2, OU=SME, O=mycp, L=Fulton, ST=MD, C=US ^EMAILADDRESS=(.*?), CN=(.*?), OU=(.*?), O=(.*?), L=(.*?), ST=(.*?), C=(.*?)$ $2 nifi-user2 As you can see from the above examples, using NiFi's pattern mapping ability with simplify authorizing new users via either NiFi's default file based authorizer or using Ranger.
... View more
Labels:
02-24-2017
02:48 PM
Thanks for writing this up @Matt Clarke - very helpful. Do you have a rule of thumb for a maximum heap size? Is there a limit where garbage collection will surely cause more problems than any gains from further increases in heap?
... View more
02-22-2017
01:54 PM
@mayki wogno No problem... If one of the answers help drive you to a solution to your question, please accept that answer to help drive this community forward.
... View more