Member since
05-05-2017
16
Posts
12
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
3742 | 12-05-2017 03:36 PM |
04-01-2019
01:41 PM
@Samar Aarkotti I think to have it work reliably you need to put your krb5.conf settings in the default location. Note that the settings above are just useful for development and testing, which is why those settings for custom locations are needed. Hope this helps.
... View more
12-06-2018
05:28 PM
I do not think the long sync time period is caused by the group search. I think it is caused by the user search. You are correct that it looks like reducing the number of users to search to would require restructuring the AD. I understand that is not always possible / desire-able. To answer your questions: - Are you sure the NiFi and NiFi Registry settings are identical? What version of NiFi and NiFi Registry are you using? The LDAP sync procedure should be the same, so I would be surprised if there were very different behaviors. - No, it is not possible. The algorithm basically pulls all users that match the user search settings then pulls all groups matching the group search settings, then iterates over the users/groups it has synced in order to build the mapping of users<->groups. Changing this sync algorithm would require a re-write of the code or implementing a custom LdapUserGroupProvider. - It is not possible to specify two OUs for the User Search Base. You could fork the code and implement a modified/custom LdapUserGroupProvider that takes an array of search bases and does multiple searches (one for each base), but that capability is not something currently supported. - For large directories, I've seen people have good luck specifying a page sizes of 100-500. Kevin
... View more
12-05-2018
07:24 PM
@Christian Ludwig This is a known issue with NiFi and NiFi Registry when syncing users and groups from very large external directories. It just takes a long time. My recommendation would be the following: Limit the User Search Base and Group Search Base to be as tight as possible, and use the User Search Filter and Group Search Filter to further limit the number of users and groups that need to be synced. Tune the LdapUserGroupProvider 'Page Size' and 'Sync Interval' properties. Hope this helps, Kevin
... View more
04-20-2018
12:13 AM
@Saikiran Parepally It is not recommended to start multiple servers. Some data is cached in memory, so in addition to the data corruption potential you noted, multiple instances pointed to the shared storage location would become out of sync over time. A better approach for now might be to focus on backing up the data to another location for recovery or for spinning up a cold-spare if your NiFi Registry goes offline. This is not yet a feature of NiFi Registry, so you would have to design and build such a mechanism yourself on top of NiFi Registry. The JIRA @Andrew Lim linked to, NIFIREG-150, is a good collection of ideas for possibilities. If you need backup or HA features, I would recommend commenting on existing JIRAs or open new feature requests. This service is still new so we are looking for feedback from users to prioritize next steps.
... View more
12-05-2017
03:36 PM
1 Kudo
Hi @Scott Aslan, I have seen this behavior before. Those steps to re-initialize your KDC environment are correct. However, it appears on some Mac machines the native library that is used by browsers for kerberos/spnego authentication can get into a bad state, possibly caching values. For me, the current version of Chrome seems to be especially finicky with Kerberos, but relaunching the browsers seems to fix this issue. Try this: 1. Closing all browsers (Chrome, Firefox, Safari, etc) 2. In the shell that you ran $(./kdc shellinit), relaunch your browser from the terminal using the `open` command: open /Applications/Firefox.app or open /Applications/Google\ Chrome.app This is because your krb5.conf file is not in the default location, so launching the browser from the shel that has the KDC environment variables should point the browser to the correct configuration file and keytab. Hope this helps!
... View more
08-29-2017
06:41 PM
1 Kudo
From the RouteText processor documentation: Each line in an incoming FlowFile is compared against the values specified by user-defined Properties. The mechanism by which the text is compared to these user-defined properties is defined by the 'Matching Strategy'. The data is then routed according to these rules, routing each line of the text individually. Is this what you want to accomplish, i.e., routing each line of text? If so, then you don't have to use the NiFi Expression language. Use a Java Regex Pattern string and it will be evaluated for each line in your text file. If instead you wish to route the entire FlowFile if any line matches, there are probably many ways you could do that. One method would be to use RouteOnContent: Applies Regular Expressions to the content of a FlowFile and routes a copy of the FlowFile to each destination whose Regular Expression matches. Regular Expressions are added as User-Defined Properties where the name of the property is the name of the relationship and the value is a Regular Expression to match against the FlowFile content. Again, this uses a Java Regex Pattern String. Does this answer your question?
... View more
06-08-2017
07:02 PM
I know this is an old post, but in case if future search results turn up here and someone is unclear on how to do this part of @Bryan Bende's answer: Connecting your IDE debugger to the Java process will be specific to which IDE you are using. I wanted to share a link to a detailed tutorial I put together for how to do this in IntelliJ IDEA: https://community.hortonworks.com/content/kbentry/106931/nifi-debugging-tutorial.html
... View more
06-08-2017
06:52 PM
7 Kudos
This is a beginner's guide for debugging a running NiFi instance. It covers two important aspects of debugging: adjusting the log levels of what is written to the NiFi application logs and attaching a graphical debugger to a running NiFi instance. Requirements Apache NiFi source code (this tutorial uses NiFi 1.3.0)
If you have never build and run NiFi from source, follow the instructions in the NiFi Development Quick Start Guide: https://nifi.apache.org/quickstart.html In this tutorial, {NIFI_ROOT} refers to the location of your NiFi source directory, i.e., your git clone or uncompressed source directory. IntelliJ IDEA (this tutorial was made using IntelliJ IDEA Community 2107.01) Available from: https://www.jetbrains.com/idea/download If you are using other version of Apache NiFi or IntelliJ IDEA, the steps should be very similar though some of the details and example screen shots might vary slightly. Enabling Debug Logging By default, the log level for NiFi is set to INFO, meaning log lines set to DEBUG and below are suppressed. If you are developing or troubleshooting NiFi, you may find it useful to adjust the logging level to show more verbose logging output. If you are only interested in attaching a debugger, you can skip this section. The log level is controlled by the logback configuration file, which, after you have built from source, is located at: {NIFI_ROOT}/nifi-assembly/target/nifi-1.3.0-bin/nifi-1.3.0/conf/logback.xml In this file, you will find a section of XML that looks like:
<logger name="org.apache.nifi" level="INFO"/>
<logger name="org.apache.nifi.processors" level="WARN"/>
<logger name="org.apache.nifi.processors.standard.LogAttribute" level="INFO"/>
<logger name="org.apache.nifi.processors.standard.LogMessage" level="INFO"/>
<logger name="org.apache.nifi.controller.repository.StandardProcessSession" level="WARN" />
...
<logger name="com.sun.jersey.spi.container.servlet.WebComponent" level="ERROR"/>
<logger name="com.sun.jersey.spi.spring" level="ERROR"/>
<logger name="org.springframework" level="ERROR"/>
If there are specific packages or classes you want to enable debug logging for, simply add an override: <logger name="org.apache.nifi.PACKAGE_OF_INTEREST" level="DEBUG"/> This will be applied recursively to all sub-packages, unless a sub-package has an override.
If you want to mute a noisy class in a package for which you enable debug logging, you can add: <logger name="org.apache.nifi.PACKAGE_OF_INTEREST.NoisyClass" level="WARN"/> Note, the logger/class name of origin is included in each log line, so you can use that to determine what to mute.
If you don't know what you are looking for specifically, you can also change to overall log level for 'org.apache.nifi' to DEBUG: <logger name="org.apache.nifi" level="DEBUG"/> If you do this, I recommend also adding these two overrides at a minimum to avoid a ton of log output: <logger name="org.apache.nifi.engine.FlowEngine" level="INFO"/>
<logger name="org.apache.nifi.controller.repository" level="INFO"/> When you are done making your changes, save and close this file. Note, after modifying logback.xml, you do not have to restart NiFi. After a few minutes, the configuration changes will be picked up and the logger's behavior will reflect the new settings. Attaching a Debugger to NiFi In order to enable remote debugging for NiFI, we need to edit the bootstrap configuration located here: {NIFI_ROOT}/nifi-assembly/target/nifi-1.3.0-bin/nifi-1.3.0/conf/bootstrap.conf Open this file in your text editor of choice and uncomment this line: java.arg.debug=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000 Make a note of the "address=XXXX" value, which in this example is the default '8000'. This is the port the debugger will connect to.
After uncommenting the config line that enables remote debugging, start (or restart) NiFi using the instructions in the Quick Start Guide.
Verify you can access your local NiFi instance by going to this URL in a web browser: http://localhost:8080/nifi/ Now let's setup an IntelliJ IDEA Project, which we will use for debugging NiFi. First thing to do, if you have not already, is create an IntelliJ IDEA Project from our cloned repository: Launch IntelliJ IDEA, and in the "Welcome to IntelliJ IDEA" screen, select "Import." Navigate to the root directory of your NiFi clone, i.e. what I have been referring to as {NIFI_ROOT} NiFi uses the Maven build systems, so IntelliJ IDEA will be able to import it to create an IDEA project: Choose "Import project from external model" Select "Maven" as the external model type Click Next. For the remainder of the wizard, you can accept the default settings by clicking "Next" through the remaining steps.
You will end in the primary IDE view with your project viewer. Note, it may take several minutes to index and fully load. (This is normally done in the background, with progress shown in the status bar at the bottom.) Next, let's create a Run/Debug Configuration that tells IntelliJ how to attach to our NiFi instance: From the Menu: Run > Edit Configurations... Click the '+' button and select 'Remote' from the drop down Give the new configuration a name such as "Debug Local NiFi" Change the "Port" field to the port that matched the address=XXXX line in bootstrap.conf above (in my case, 8000). Optional step: Setup the IntelliJ console to display NiFi logs for this configuration. Switch from the "Configuration" tab to the "Logs" Click the '+' button in the middle of the panel to add a log file Choose whatever alias you want Set the log file location: {NIFI_ROOT}/nifi-assembly/target/nifi-1.3.0-bin/nifi-1.3.0/logs/nifi-app.log Save this configuration. You can now run the configuration to attach to your local NiFi instance whenever NiFi is running on localhost:8080. To launch the debugger, follow the steps in these screenshots: That's it! NiFi's logging output, set to the proper level of debug output, is available in a table in the IntelliJ IDEA console and we can set a breakpoints in the NiFi source code and pause / step-through lines of code from your running NiFi instance. Happy debugging!
... View more
Labels: