Member since
07-30-2019
2909
Posts
1443
Kudos Received
846
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
53 | 04-23-2024 05:56 AM | |
26 | 04-22-2024 06:13 AM | |
150 | 04-17-2024 11:30 AM | |
117 | 04-16-2024 05:36 AM | |
77 | 04-15-2024 05:31 AM |
10-05-2021
01:07 PM
Hello @edoS Welcome to the community! NiFi's provides so many option for user authentication and authorization, setting up exactly what you need can be overwhelming at times. This is certainly something the Cloudera support could walk you through if you have a support contract with us that covers the NiFi service. At a high level, here is what you need to understand about the authentication and authorization process in NiFi. Authentication happens first and must be successful before any authorization is verified. NiFi supports numerous ways to authenticate users/clients (TLS, Kerberos, LDAP, openID, etc...). No matter which method is used, the end result of any authentication is a user string that identifies the successfully authenticated user/client. That user string is then evaluated against the identity mappings [1] you may have configured in the nifi.properties file. These identity mappings are used to normalize the user strings. for example: Trim the CN from the full DN in a user/client certificate Trim the user name from a kerberos principal convert the user string to all uppercase or lowercase The resulting user/client string is then passed to the authorizer to verify that user/client is authorized for the NiFi Resource Identifier being requested. NiFi authorizers.xml is where this configuration is setup. This file is easiest to read from the bottom up. At the bottom of the authorizers.xml you will find your authorizer which you have setup as the "Ranger-Provider". It is important to understand how this authorizer works. NiFi runs a background thread that checks in with Ranger to see if there is a new policy definition for the NiFi service. If so, the new definition is downloaded by NiFi. What Ranger provides to NiFi in this downloaded policy definition are all the polices setup in Ranger. For each there will be the "NiFi Resource Identifier(s)" along with the user strings and group strings that have been assigned "Read" and/or "Write" permissions. Now remember up to this point all NiFi knows about the authenticated user is the user string. NiFi has no idea yet what groups that user string may belong to. Within the Ranger-Provider, you will find a property name with "User Group Provider". The value set here tell the authorizer where to check to see if the user string passed from authentication has any known user to group associations. Search your authorizers.xml for configured User Group Provider [2]. There are numerous options that can be configured for determining user to group associations. Some of the available providers allow you to configured multiple providers. While the authorizer "ranger-provider" can only point at 1, it may point at a "composite-configurable-user-group-provider" [3] for example that can be setup to reference multiple user-group-providers. The key here is making sure you have added 1 or more user group providers that will return all the user to group associations you need. Based on the log output you shared from the nifi-user.log. We know that none of the user group providers you may have setup returned any group strings associated to your user string (identity[18330301],groups[] ). This is why "groups [ ]" is empty. The "file-user-group-provider" [4] allows you to create user string to group string associations manually via the NiFi UI directly. The commonly used "ldap-user-group-provider" [5] determines user and group associations via user and/or group syncs with ldap/AD. Now that NiFi knows what groups the authenticated user string is associated with, the user and the groups can be checked against the downloaded policies to see if the user is authorized for the action being performed or the end-point trying to be accessed. [1] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#identity-mapping-properties [2] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#authorizers-setup [3] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#composite-implementations [4] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#fileusergroupprovider [5] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#ldapusergroupprovider If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
10-05-2021
12:16 PM
@CodeLa I setup a dataflow using the exact example you shared: After the ReplaceText, I see the content is now: Can you share your sample xml file that is not working? How is your split being done? Thanks, Matt
... View more
10-05-2021
12:06 PM
@Phanikondeti Does the host or IP you configured in this property match with what is assigned to the host? The following command will show you your NIFi host's hostname: hostname The following command will show you the IP addresses associated with your network interfaces on the host: ifconfig
ip address show The following command will allow you to see if some process is already binding to your configured port in the nifi.web.http(s).port= property in the nifi.properties file: netstat -anop|grep 8075|grep LISTEN If you get a return from above, it will include a Process Id (pid) that you can lookup using: ps -ef|grep <pid> The latest exception you shared is different form the first: "NiFi has started, but the UI is not available on any host". NiFi throws this WARN log line when the NiFi code returns no URLs post starting the NiFi JettyServer. In this setup, I would guess that you set the nifi.web.http(s).host= set to either blank or 0.0.0.0. So NiFi passes the earlier checks and start the JettyServer, but then when it tries to bind to all the network interfaces it finds known and throws the above WARN exception. This points at a setup issue on your Amazon EC2 setup and not an issue with NiFi. I'd use the above command to verify that your ec2 shows properly setup interfaces. If you find issues with your ec2 unbuntu interface setup, you may need to reach out to Amazon to help there. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
10-04-2021
12:42 PM
@dansteu Due to size constraints in Apache, the Apache NiFI distribution do not ship with all components to keep the size under the max allowed. The community removes from the default distro those components less commonly used or deprecated by newer components that do the same job better. The ReportLineageToAtlas is one of those less commonly used components that was removed to reduce size, but it can easily be downloaded from Maven and added in to your NiFi 1.14.0 install. https://mvnrepository.com/artifact/org.apache.nifi/nifi-atlas-nar/1.14.0 Look for the "Files" line and download the "nar" file (58 MB for Apache NiFi 1.1.4.0 release). Place this file in the NiFi 1.1.4.0 lib directory with the rest of the included nar files and restart your NiFi instance. This nar will get unpacked to NiFi work directory on startup and then be available for use through the UI. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
10-04-2021
12:03 PM
@CodeLa You can accomplish this via the ReplaceText processor using a multi line approach to your Java Regular Expression (regex). Search Value: Replacement Value: The downside to this approach is that you need to configure this processor with an Evaluation Mode of "Entire text", Evaluation Mode of "All", and make sure the configured buffer size is large enough to fit the entire text. This in turn means a higher heap memory utilization when this processor is executing against your FlowFile. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
10-04-2021
11:18 AM
@Phanikondeti Thanks for sharing the log output. Your NiFi is not up, It failed to start because it was unable to bind to the IP and port shown in the logs. That IP address would correlate to what you have set in either of the following properties in the nifi.properties file: (if unsecured) nifi.web.http.host= (if secured) nifi.web.https.host= You could use ifconfig command to see if your server has a network interface with that IP assigned to it. If it was a port issue, I'd expect you to see a log message about port already being in use or that you're trying to launch NiFi as a privileged user and tried to use a port number below 1024. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
10-04-2021
11:12 AM
@Phanikondeti When NiFi is the bootstrap process starts a child process that may take a little time to fully start depending on size fo flowfile_repository and the size of the flow.xml.gz being loaded. You will want to search the nifi-app.log for the following lines: 2021-10-04 18:05:57,212 INFO [main] org.apache.nifi.web.server.JettyServer NiFi has started. The UI is available at the following URLs:
2021-10-04 18:05:57,212 INFO [main] org.apache.nifi.web.server.JettyServer https://<nifi-hostname or IP>:<nifi port>/nifi
2021-10-04 18:05:57,216 INFO [main] org.apache.nifi.BootstrapListener Successfully initiated communication with Bootstrap Until you see these lines, NiFi is still coming up and the UI will not yet be reachable. If you do see these lines, you will want to make sure that your host where you have launched your browser can reach the hostname/IP logged in the above message. You should also check to see which network interface your NiFi bound to on startup if you have multiple interfaces available. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
10-04-2021
10:04 AM
@JelenaS Making a bucket public only only control whether an unauthenticated and authorized user can import flows from a bucket on to the NiFi instance. So ONLY make it public if you want anonymous users to be able to use your version controlled flows in that bucket. Users must still be authenticated and authorized in order to commit new flows to a public bucket. As far as the global polices you set up for your "CN=<domainname>.net, OU=NiFi", that looks correct (don't need "write" on buckets), but is only correct if that string matches exactly what is coming from the certificates used on your secured NiFi instance(s) post and identity mapping happening on the NiFi-Registry server. So check your nifi-registry.properties file for any configured Identity Mapping Properties: https://nifi.apache.org/docs/nifi-registry-docs/html/administration-guide.html#identity-mapping-properties For example: nifi.registry.security.identity.mapping.pattern.dn=^CN=(.*?), OU=(.*?)$
nifi.registry.security.identity.mapping.value.dn=$1
nifi.registry.security.identity.mapping.transform.dn=NONE with above and "CN=<domainname>.net, OU=NiFi", the string that would get checked for authorization in NiFi-Registry would be only "<domainname>.net" and thus be the string that would need to be authorized in instead of the full DN. When you are authenticated in to your NiFi instance as your nifi_admin user, what exact string is displayed in the upper right corner of the NiFi UI? Is it "nifi_admin" or "CN=nifi_admin, OU=NiFi" because whatever displayed there is going to be the exact user string that gets proxied to the NiFi-Registry. Also keep in mind that USER/CLIENT strings are case sensitive in both NIFi and NiFi-Registry. Mapping transforms can be used to convert strings to all uppercase (UPPER) or all lowercase (LOWER). Hope this helps, Matt
... View more
10-04-2021
09:48 AM
@Theoo Nice job on your path to solving the authorization issues, but you left out a few pieces - When NiFi cluster nodes or a standalone NiFi instance communicates with a secured NiFi-Registry that communication MUST be authenticated and authorized in the NiFi-Registry side. The established connection between NiFi and NiFi-Registry only supports authentication via a mutual TLS handshake (Client is identified via the certificate shared to the NiFi-Registry from NiFi). Both NiFi and NiFi-Registry have identity mapping properties that can be added to the nifi-properties/nifi-registry.properties file that are used to manipulated the DN that comes from the client certificate. For example a NiFi host certificate with DN of "CN=nifi-node01, OU=NIFI" could be manipulated so the client string is only "nifi-node-01". Users and NiFi nodes/instances to both NiFi and NiFi-Registry are just clients, there is no distinction between the two. What matters is what each client is uniquely authorized to do within each service. Whatever the client string happens to be, The NiFi nodes/instance must be authorized for the following global policies in NiFi-Registry: "Can proxy user requests" (/proxy) with "Read, Write, and Delete" - This allows the NiFi nodes/instance to proxy some request made by the user authenticated in NiFi to perform some authorized request against NiFi-Registry (start version control, commit a new version of a version controlled Process Group (PG), etc.) since the NiFi user is not actually authenticating in to NiFi-Registry from NiFi. This does mean that the NiFi user string must exist as a user in NiFi-Registry and be authorized for the action they are trying perform. "Can Manage Buckets" (/buckets) with "Read" - This policy is needed by the NiFi nodes/instance so that the NiFi background thread that occasionally communicates with NiFi-Registry to see if newer version of a version controlled PG is available or so NiFi can display a list of available buckets). This request is not done on behalf of the user authenticated into NiFi. ---- When it comes to the NiFi user, the policies needed in NiFi-Registry vary based on what you want that user to be able to do through NiFi or directly via the NiFi-Registry UI. In order for a user who is currently authenticated and authorized in to NiFi to interact with NiFi-Registry, that user string would need to be authorized in NiFi-Registry for the following: - A NiFi-Registry admin user would need to create a bucket and authorize the NiFi user on that bucket so it can be used by the NiFi user. - "READ" on the bucket would allow the user to import and existing version controlled flow from Nifi-Registry on the NiFi UI. - "WRITE" on the bucket would allow the user to start version control or change the version of the versioned PG in NiFi. - "Delete" on the bucket would allow a user who can authenticate in to NiFi-Registry to delete flows within that bucket. -------- As far as authentication of users in to NiFi and/or NiFi-Registry, you can create certificates for each fo your users, but the most commonly used method is LDAP/AD based authentication. You can add users in NiFi-Registry's authorizer so that those user string can be associated to authorization policies without those user even being able to authenticate and be authorized directly in to the NiFi-Registry's UI. They simply need to exist for the proxied request that come from NiFi on that user's behalf. Hope this exposes all that is needed in this thread. Thanks, Matt
... View more
09-30-2021
01:39 PM
1 Kudo
@VagnerBelfort From your example, it appears you are looking to modify only the first line of your input file and your modification seems pretty simple. In that case, one possible solution is simply to use the ReplaceText processor to modify that first line to match you new modified structure. Here is a ReplaceText processor configuration I used to accomplish your desired output: Search Value (Java regular Expression): ^"(.*?)":"(.*?)":(.*?)\{ Above contains 3 capture groups to capture the unique parts of your input we want to reuse. Replacement Value: {
"hash":"$1:$2:$3", Make note of the added line return. All characters are literals except the $1, $2, and $3 which get replaced with the string form each of the three capture groups from the Java Regex. Replacement Strategy: Regex Replace Evaluation Mode: Line-by-Line Line-by-Line Evaluation Mode: First-Line If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more