Member since
07-30-2019
3472
Posts
1642
Kudos Received
1020
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 209 | 06-03-2026 06:06 PM | |
| 502 | 05-06-2026 09:16 AM | |
| 961 | 05-04-2026 05:20 AM | |
| 566 | 05-01-2026 10:15 AM | |
| 675 | 03-23-2026 05:44 AM |
09-16-2022
12:32 PM
@double_z NiFi does not allow users to create locally managed user accounts (meaning creating a username and password directly in NiFi. NOTE: The latest versions of NiFi provide a single user login provider just so that NiFi can by default be launched securely. This single user would have complete access to do everything and you cannot create additional users. Ideally while this provides some security, it is not a proper recommended deployment method. The good news here is it sounds like you have successfully setup a ldap-provider in your login-identity.providers.xml. This provider provides a method by which a user can be authenticated via LDAP. Authentication via ldap does not control authorization which is handled within NiFi via the authorizers.xml. When you login to NiFi, the resulting user identity string (case sensitive) resulting from your login is evaluated against any identity mapping patterns you may have configured in your nifi.properties file. After that the resulting identity string (if pattern match was found) or unmodified identity string from login provider is passed to the NiFi authorization process. While NiFi loads the providers from the authorizers.xml from the top down, it is easer for a user to read it from bottom up. - Your are using the "managed-authorizer" which is calling the "file-access-policy-provider" - The "File-access-policy-provider" is responsible for authorizations.xml file and seeding it with an initial set of authorization policies needed for your NiFi nodes (multi-node nifi cluster) and an initial admin user. You need an initial admin so that the admin user can set additional authorization from with in the NiFi UI. - In your "file-access-policy-provider" you have told the provider to create the authorization policies for a user identity string "freeipa". So the file-access-policy-provider" must first check to see if that user is known to this NiFi. For that is is configured to use the "composite-configurable-user-group-provider". - The "composite-configurable-user-group-provider" is then configured to get users and associated groups from the "file-user-group-provider" and "ldap-user-group-provider". You can NOT have to user-group-provider return the exact same user identity string. - Your "ldap-user-group-provider" has ben configured to sync user and group identity strings from your LDAP. One of those user being returned is "freeipa". - Your "file-user-group-provider" has been configured to create a local user identity that has the same user identity string of "freeipa". So now you have two user-group-provider returning the same user identity string, so NiFi has not idea which is correct to use and throws the exception you see about two providers providing same user identity. - What you have shared above also shows the "file-user-group-provider" twice. You can't have same provider defined twice in this file. - Steps to move forward. 1. The file-user-group-provider and the file-access-policy-provider will only create the users.xml file and authorizations.xml file if they do NOT already exist. So if these two files exist, delete them (authorizers.xml and authorizations.xml are two different files, make sure you delete the correct one) 2. Make sure you ldap-provider is configured to USE_USERNAME and not USE_DN if not already set this way. 3. Unset the "Initial User Identity 1" in the file-user-group-provider. We don't want this provider creating the freeipa user in the users.xml since your ldap-user-group-provider will be providing this user identity. 4. Leave the initial admin "freeipa" set in the file-access-policy-provider. 5. start your NiFi, it will create a new users.xml and authorizations.xml during startup. At login UI, provide your ldap "freeipa" username and password. Once in the UI, your freeipa user will have all the authorization policies needed to act as and admin. This does not mean this user has all authorizations, but does have ability to grant additional authorizations to itself or other users. NiFi global menu (upper right corner) --> users (will allow you to see all users and groups long with their associations to one another synced from LDAP). It will also show any local users identities you may define (locally defined identifies will show an edit and delete icon next to them). local user and group identities are only used to set authorizations, they are not able to be used to authenticate in to NiFi. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
09-16-2022
11:56 AM
@noekmc I was not clear that when you accessed the NiFi Web address you were skipping the login window completely. This means that your browser provided and alternative method of client/user authentication. When you access the NiFi web address, NiFi will always negotiate a mutual TLS handshake. This is necessary because this is how NiFi nodes authenticate with one another. If no other methods of client authentication have been configured, the mutual TLS handshake "Requires" a client certificate. When other methods of authentication are configured in NiFi, the mutual TLS handshake will "WANT" a client certificate. If no client certificate is presented, then NiFi will move on to the next configured authentication method which would spnego. Spnego based authentication is enabled when the following properties have been configured in the nifi.properties file: Make sure these two properties are clear to disable spnego auth challenge to your browser. If Spnego auth challenge is not successful, NiFi moves on to next auth method such as a configured login provider like the ldap-provider you have setup. The first step is figuring out which method (TLS client certificate or Spnego) is authenticating your user. Typically a browser will prompt you when either if these methods are invoked the first time. If you ack instead of cancel, the browser will remember that choice going forward. For TLS client auth to work, your browser must have a client certificate loaded in to it that your NiFi's truststore file is capable of trusting. For Spengo to work, Spnego must be configured in your browser. Step one: - Open an incognito browser tab (it will not have any retained cookies that would auto use a certificate or spnego) and provide the NiFi UI address. Does it redirect you immediately to the login UI. If so, you now know one of these other methods are being used. - Clear the two Spnego properties if configured in the nifi.properties file. (if already blank, then we know a TLS certificate is what is being used. - Clear browser cache and cookies. Access NiFi UI address, when prompted via browser for certifcate, cancel and you should get redirected to login window. There is not configuration change that can be made in NiFi to stop a browser from doing this. However, your decision to cancel and continue to URL without providing your certifcate should be cached by your browser so it does not ask you each time afterwards. - Try a different browser. While your certificate maybe loaded in one browser, it may not be loaded in another. Same goes for Spnego, it may not be enabled in all browsers on your client. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
09-15-2022
12:36 PM
@noekmc The UI you are seeing is telling you that your ldap user credentials have successfully been authenticated; however, your user identity is not authorized within NiFi to "view the UI". NiFi Access Policies The ldap-provider configured in the login-identity-providers.xml handles the authentication process. The configuration within the authorizers.xml handles the authorizing of those authenticated user identities. You can tail the nifi-user.log while you login to see that your user identity that is resulting from your successful authentication. You will also then see the not authorized log output with the missing access policy. The following section of the Apache Documentation can help setting up authorization for the first time: multi-tenant-authorization If you were to share the log lines from your nifi-user.log specific to your login attempt along with the contents of your authorizers.xml file, it may be easier to provide guidance on your setup. The multi-tenant-authorization setup in the authorizers.xml has many configuration options and providers to choose from. The very basic setup would use a managed-provider that uses the file-access-policy-provider and file-user-group-provider. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
09-15-2022
12:23 PM
@leandrolinof Understand that NiFi wraps all ingested content in to a NiFi FlowFile. A NiFi FlowFile consists of FlowFile meatdata/attributes (key/value pairs written to the flowfile_repository. NiFi processors can be used to add and modify existing FlowFile Attributes) and FlowFile content (bytes written to a claim within the NiFi content_repository). Since NiFi is data format agnostic, the FlowFile model allows NiFi to ingest any data format. This does not mean that NiFi can read/edit all data formats. It is the responsibility of an individual processor that must be designed/coded to operate against specific data types. While NiFi does not have a purpose built processor specifically for reading PDF, It does ship with scripting processors that can be used to execute custom script code that can extract content from a PDF in to FlowFile attributes. Reference example: https://gist.github.com/mattyb149/48e72a26d0f62f330e30 https://stackoverflow.com/questions/55169492/nifi-extract-from-pdf-to-text I assume since that directory contains multiple PDF, each of those PDF have a unique name? Once you have ingested your PDFs, the above example groovy script may allow you to create attributes on your FlowFile which you can use later via NiFi Expression Language statements to dynamically set configuration properties on the putEmail processor uniquely by FlowFile it executes upon. When you look a processor's documentation for each configuration property, you will see if it supports NiFi Expression language and to what extent it is supported Supports Expression Language: true (will be evaluated using flow file attributes) Example above states NiFi Expression language is supported and you can use FlowFile attributes which would come from each FlowFile that processor is executing against. If you can extract the needed text from each PDF in to additional attributes on each FlowFile (one FlowFile created for each PDF ingested) with buyer name, then you can use those to dynamically define properties per FlowFile in the PutEmail processor. If you desire is to consume every PDF only once, you should be using the ListFile --> FetchFile processors. The ListFile can be configured to maintain state so that the same FlowFiles are not listed more than once. (ListFile must be configured for "primary node" execution only in a multi-node NiFi cluster setup to avoid data duplication). The message body of an email is going to expect that message to ASCII text. The "Content Type" property of the PutEmail processor is expected to be set to the content' mime type (for a PDF file the mime.type would be "application/pdf"); however, i suspect the putEmail processor may have issue with that mime.type. If that is the case, you'll need to send you PDF as attachments to the email only. I am not sure what the "buyer's ID" gets you. Is that buyer's ID in the PDF filename? How does the buyer's ID get you the buyer's email address to which you will send the PDF via PutEmail? to answer you question about is it possible, I believe so. It is about collecting/creating the needed metadata/attributes from 1 or more sources and adding them to your FlowFile with the PDF content needed to successfully send your email. I hope above gets you moving along that path. NiFi offers so many components, there is often more than one may to solve a use case. First step is laying out a step by step processor for how you would solve this use case outside NiFi manually and then translate each of those steps in to automated NiFi dataflow. Then you can narrow down to the specific step in that use case where help is needed further. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
09-14-2022
02:23 PM
@PratikParekh The putEmail processor supports the adding of dynamic configuration properties as outlined in the linked documentation. https://javaee.github.io/javamail/docs/api/com/sun/mail/smtp/package-summary.html It would be helpful to collect the full stack trace written to the nifi-app.log rather then sharing the partial shown via the processor bulletin produced on the NiFi UI. From that stack trace output you may be able to determine what dynamic properties need to be added. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
09-14-2022
02:10 PM
@leandrolinof I am having trouble understanding your query clearly. The PutEmail processor ONLY supports talking the content of the source FlowFIle and adding it as the email message or adding it as an attachment to the email. The putEmail processor is not involved at all with how the content was added or modified in the source FlowFile that comes from the upstream ExecuteSQL processor. So what exactly are you trying to pass through the PutEmail processor? The BASE64_ENCODE you are retrieving from Oracle? You can write that to a FlowFile attribute. The "Message" configuration property of the PutEmail processor, as well as many others, support using NiFi Expression Language (NEL). That means you could read from the FlowFiles attributes anything you have placed there and insert it into the email message body. If you can provide more detail it may be helpful. It sounds like you want to send an email with aPDF file attachment, but sounds like you haven't even ingested the PDF into NiFi yet. Thank you, Matt
... View more
09-13-2022
01:29 PM
1 Kudo
@myzard If you are positive the username and password entered are valid, this likely points to an issue in your login-identity-providers.xml ldap-provider configuration with the manager DN and/or manager password. Since this is an xml file, the first question to ask is if the password contains any XML special characters (<, >, ", `, or &): < replace with <
> replace with >
" replace with "
ˋ replace with '
& replace with & If so, you'll need to escape them by using above substitutions. Next is to use ldapsearch to verify the manager DN and manager password works using same configuration set in the ldap-provider (minus xml substitutions if any) against the same username and password. Make sure the results from your ldapsearch on returns one matching user. I have seen setups where ldap had same username multiple times under different DNs. This will not work with NiFi login as NiFi would be unable to determine which is the actual user being authenticated. In this scenario, adjust your search base so that it only returns one user entry. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
09-13-2022
01:19 PM
@Olwe How is zabbix authenticating it self with your NiFi? When NiFi is secured, all clients must be authenticated and authorized. An HTTP 401 means the client was not authorized. NiFi does not use sessions. So when a user logs in via a username and password, if the authentication was successful, a bearer token is issued for the user for that specific NiFi node. That bearer token will remain valid for the duration configured in your NiFi login provider or until the logout endpoint is invoked. The client is expected to include that bearer token in all subsequent requests. If you hit a NiFi rest-api endpoint like .../nifi-api/flow/cluster/summary without a bearer token or a client certificate, the client will be treated as "anonymous" and not authorized. You should see this unauthorized endpoint request logged in the nifi-user.log. When you access rest-api endpoints via the browser which you used to login, the browser takes care of including the bearer token. I am not familiar with Zabbix, but same requirements exist (get bearer token and include bearer token in future rest-api calls) For example, obtaining a token using a login provider: curl 'https://<nifi-hostname>:<nifi-port>/nifi-api/access/token' \
--data-raw 'username=<username>&password=<password>' \
--compressed \
--insecure Above will return the <bearer token>. Then you would use that bearer token in your future rest-api requests: curl 'https://<nifi-hostname>:<nifi-port>/nifi-api/flow/cluster/summary' \
-H 'Authorization: Bearer <bearer token>' \
--compressed \
--insecure If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
09-13-2022
12:50 PM
@emaxwell A few suggested additions to this excellent article: 1. The keystore password is not required to list the keystore via the keytool command. When prompted for a password, you can just hit "enter" without entering a password. The password would be needed if you want to export the actual privateCertEntry from the keystore which is not necessary in this procedure. 2. The ldap-provider configured in the login-identity-providers.xml file has two options for how to identify the authenticated ldap user: <property name="Identity Strategy">USE_DN</property> Above will use the full Distinguished Name (DN) as you mention in the article. <property name="Identity Strategy">USE_USERNAME</property> Above will use the string entered by user at the login window as the user identity. 3. Identity mapping properties in the nifi.properties file have three properties: nifi.security.identity.mapping.pattern.<unique string>=<Java regex>
nifi.security.identity.mapping.value.<unique string>=<Resulting string when java regex matches, Java Regex capture groups can be used here>
nifi.security.identity.mapping.transform.<unique string>=<NONE, LOWER, UPPER> The <unique string> can be anything but must be same exact string for all three properties. Also patterns are checked in alphanumeric order against the identity strings. First matching pattern has its value and transform applied. So you want to make sure the more complex patterns come first to avoid having wrong pattern's value and transform being applied. So a pattern like "^(.*)$" should definitely come last. The "transform" property allows you to apply an all uppercase, all lowercase, or no transform to the resulting value string. Thanks, Matt
... View more
09-12-2022
01:52 PM
@ajignacio User and group identity strings much match identically. Your ldap-user-group-provider is syncing users and groups by the identity string found in the CN AD attribute. This is why you are seeing only the CN username and CN groupname strings in the users UI within NiFi. However, when you are logging in to NiFi to authenticate you user via the ldap-provider, the resulting user identity sting is the users full AD Distinguished Name (DN). NiFi treats different strings as different users. The ldap-provider can be changed to use the user identity string typed in the username field instead of using the full DN. This is done by changing the following property: <property name="Identity Strategy">USE_DN</property> change it to : <property name="Identity Strategy">USE_USERNAME</property> Upon successful authentication the resulting user identity is evaluated against any identity mapping patterns that may be configured in the nifi.properties file. The resulting mapped value is then passed to the configured authorizer (managed-authorizer in your setup). There the authorizers is looking up that user identity string (case sensitive) against the user strings synced by your configured users group providers. If an exact match is found both the user string and the now learned group string(s) are checked against the configured NiFi policies to determine authorization. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more