Member since
07-30-2019
2283
Posts
1247
Kudos Received
644
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
35 | 07-01-2022 08:57 AM | |
58 | 06-29-2022 11:12 AM | |
90 | 06-21-2022 08:37 AM | |
118 | 06-17-2022 05:56 AM | |
98 | 06-15-2022 11:39 AM |
01-25-2022
01:28 PM
@RonMilne I recommend taking your initial CSV record file and partitioning by the " salesRepId" in to multiple new JSON records. This can be accomplished using the PartitionRecord processor utilizing a CSVReader and a JsonRecordSetWriter. Your PartitionRecord processor configuration would look like this: Your CSVReader would be configured something like this (you'll need to modify it for your specific record's Schema: Note: Pop-out shows the "Schema Text" property and don't forget to set "Treat First Line as Header" property to "true" The JsonRecordSetWriter would need to be configured to produce the desired JSON record output format. However, just leaving default configuration will out put a separate FlowFile for each unique "SalesRepId". If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
01-25-2022
12:53 PM
@vk21 You are running in to a known bug in that version of Apache NiFi. The bug was addressed in Apache NiFi 1.14. Upgrading Apache NiFi 1.5 to 1.14 would solve the issue for you. Here are the bugs that addressed the issue: https://issues.apache.org/jira/browse/NIFI-8281 https://issues.apache.org/jira/browse/NIFI-8630 If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
01-24-2022
06:54 AM
@ebeb You can name each of your process groups uniquely by name that identifies the ETL pipeline it contains. Then you can use the Search box in the upper right corner of the UI to search for specific ETL process groups. You are correct that having 100 PG on a single canvas, if zoomed out far enough to see them all, would result in PG boxes rendered without the visible names on them. So if the PG were not named making them easy to search for. the only option would be to zoom in and pan around looking for a PG you want to access. There is no way to enter some search which would hide some rendered items and highlight others. All items are always visible, so unique naming when dealing with large dataflows is very important from a usability perspective. Every component when created is assigned a Unique ID (UUID) which can be used to search on as well. If each of your pipelines does the same thing, but on different data, creating a separate dataflow for each would not be the most efficient use of resources. Maybe better to tag your data accordingly and share pipeline processing where it makes sense.
... View more
01-19-2022
06:01 AM
@OliverGong I would avoid dataflow design when possible where you are extracting the entire contents of a FlowFile to FlowFile attribute(s). While FlowFile content only exists on disk (unless read in to memory by a processor during processing), FlowFile attributes are held in NiFi's JVM heap memory all the time (There is per connection swapping that happens when a specific connection reaches the swap threshold set in the nifi.properties file). FlowFiles with lots of attributes and/or large attribute values will consume considerable amounts of JVM heap which can lead to JVM Out Of Memory (OOM) exceptions, long stop-the-world JVM Garbage Collection (GC) events, etc... When options exist that avoid adding large attributes, those should be utilized. Thanks, Matt
... View more
01-19-2022
05:45 AM
@Wisdomstar Thank you, I appreciate that and glad I could help. Matt
... View more
01-18-2022
08:34 AM
@Kilynn So as i mentioned in my last response, once memory usage go to high, OS level OOM Killer was most likely killing the NiFi service to protect the OS. The NiFi bootstrap process would have detected the main process died and started it again assuming OOM killer did not kill the parent process.
... View more
01-18-2022
08:23 AM
@oopslemon NiFi only encrypts and obscures values in properties that support sensitive properties (so those properties which are specifically coded as sensitive properties like "password" properties). So there is no way at this time to encrypt all or portions of property values not coded as sensitive. Keep in mind it is not just what is visible in the UI, your unencrypted passwords will be in plaintext with the NiFi flow.xml.gz file as well. My recommendation to you is to use mutual TLS based authentication instead. You can create a clientAuth certificate to use in your rest API calls. Then you need to make sure that your clientAuth certificate is authorized to perform the actions the rest-api call is making. This is not going to be possible while using the single user login mode as it does not allow you to setup additional users and authorizations. This single users authentication and authorization providers where added to protect users from unprotected access to their NiFis. It was not meant to be the desired choice when securing your NiFi. It is one step above an unsecured default setup that existed prior to NiFi 1.14. It protects you, but also has limitations that go with its very basic functionality. So step one is to switch to another method of authentication and authorization to you NiFi. TLS is always enabled for authentication as soon as NiFi is configured for HTTPS. You can configure additional authentication methods like ldap/AD. https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#user_authentication The authorizer configured in the authorizers.xml file allows you to establish policies that control user/client permissions. https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#multi-tenant-authorization Then you can configure your invokeHTTP processor to simply use a SSLContextService that you would configure with your clientAuth certificate keystore and a truststore. The password fields in this controller service would be encrypted. No more need to constantly get a new bearer token. All you need to worry about is getting a new client certificate before the old one expires which is typically every 2 years, but that is configurable when you create it and get it signed. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
01-18-2022
08:08 AM
@Wisdomstar You should be able to use the JoltTransformJson NiFi processor to accomplish this. There is a thread here where a jolt specification example exists fro doing what you are trying to do: https://stackoverflow.com/questions/54696540/jolt-transformation-lowercase-all-keys Add the Jolt specification in to the "Jolt Specification" property in the JoltTransformJson NiFi processor. Set "Pretty Print" to true if you still want the nice multi-line formatted json to be produced. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
01-14-2022
11:09 AM
@sandip87 It would be difficult to point our the specific misconfiguration without seeing your nifi.properties, login-identity-providers.xml, and authorizers.xml files. Also sharing the complete stack trace thrown during NiFi service startup would be helpful as well. There should be more than what you shared that was logged in the nifi-app.log. Thanks, Matt
... View more
01-14-2022
10:49 AM
@samarsimha Did you make any recent changes before you restarted your NiFi nodes to things like hostnames or ports? You could try stopping you NiFi nodes, removing the NiFi local state directory on all nodes, and then restarting NiFi again. You can check the state-management.xml configuration file to see where each node is keeping local state. The default for Apache NiFi is " ./state/local". If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
01-14-2022
10:37 AM
1 Kudo
@gm There are a few changes to you command you will need to make. All configuration changes need to include the current revision. You can get the current revision by first executing the following: curl 'https://<hostname>:<port>/nifi-api/processors/5964ef54-017e-1000-0000-0000219f4de1' \
-H 'Authorization: Bearer <BT>' \
--compressed \
--insecure Then to make a change you can then use: curl 'https://<hostname>:<port>/nifi-api/processors/5964ef54-017e-1000-0000-0000219f4de1' \
-X 'PUT' \
-H 'Authorization: Bearer <BT>' \
-H 'Content-Type: application/json' \
-d '{"component":{"id":"5964ef54-017e-1000-0000-0000219f4de1","config":{"properties":{"bootstrap.servers":"${MIL_KAFKA_KERB_BROKERS}"}}},"revision":{"version":<revision number>}}' \
-i \
-k Things to take careful note of: 1. The user friendly property names shown in processors on the NiFi UI may not always match with the actual property name being modified. Above is a perfect example since the consume and publish Kafka processor displays "Kafka Brokers"; however the actual kafka property being set is "bootstrap.servers". 2. It might be safer to use --data-raw in stead of just -d since the content may have = and @ used in it. 3. Start with '{" instead of '" only. 4. Be carful when copying from a text editor as the ' and " may get altered/changed by the editor. 5. All changes require a correct revision number. The first command I provided will return you the current revision for the component. Then use that revision number as shown in above example when you PUT the change. Making use of the "Developer tools" provided within your browser will make it easier when trying to troubleshoot NiFi rest-api requests. Simply open developer tools, make change to property, click "apply" on the component and observer the call made in the Network tab of the developer tools. In most developer tools you can right click on the call and select "Copy as curl". Then paste that copied command in your editor of choice for review. Keep in mind the what you copy will have some additional unnecessary headers. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
01-14-2022
08:09 AM
@Kilynn I see that you have set some very high values (330 GB) for your NiFi heap space. This is really not recommended. Can also see from what you have shared that the particular NiFi instance has been up for ~2.5 hours and within that 2.5 hour timeframe it has been in a stop-the-world state due to Java Garbage Collection (GC) for ~20 minutes. This means that your NiFi is spending ~13% of its uptime doing nothing by stop-the-world garbage collection. Is there a reason you set your heap this high? Setting large heap simply because your server has the memory available is not a good reason. Java GC kicks in when heap usage is around 80% usage. When an object in heap is no longer being used it is not actually cleaned out of heap and space reclaimed at that time. Space is only reclaimed via GC. The larger the heap the longer the GC events are going to take. Seeing as how you current heap usage is ~53%, I am guessing with each GC event a considerable amount of heap space is being released. Long GC pauses (stop-the-world) can result in node disconnection because NiFi node heartbeats are not being sent. When NiFi seems to be restarting with no indication at all in the nifi-app.log, you may want to take a look at your systems /var/log/messages file for any indications of the Out Of Memory (OOM) killer being executed. With your NiFi instance being the largest consumer of memory on the host, it would be the top pick process by the OS OOM Killer when available OS memory gets too low. In most case you should not need more the 16GB of heap to run most dataflows. If you are building dataflows that utilize a lot of heap, I'd recommend taking a close look at your dataflow designs to see if there are better design choices. This that can lead to large heap usage is creating very large FlowFile attributes (for example, extracting large amount of a FlowFile's content in to a FlowFile attribute). FlowFile attributes all live inside the JVM heap. Some other dataflow designs elements that can lead to high heap usage include: - Merging a very large number of FlowFiles in a single Merge processor. Better to use multiple merge processors in series with first merging up 10,000 to 20,000 FlowFiles and then second merging those in to even larger files - Splitting a very large file in a single split processor resulting in a lot of FlowFiles produced in a single transaction. Better to use multiple or even look at ways of processing the dat without needing to split the file in to multiple files (look at using the available "record" based processors) - Reading in entire content of a large FlowFile in to memory to perform action on it. Like ExtractText processor configured for entire text instead of line-by-line mode. While I 100% agree that you should be looking in to your "thread" allocation choices in your nifi.properties file. Many of them seem unnecessarily high for a 7 node cluster. You should understand that each node in your cluster executes independently of the others, so the settings applied pertain to each node individually. Following the guidance in the NiFi app guide (can be found in NIFi UI under help or on Apache NiFi site) is strongly advised. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
01-14-2022
07:24 AM
@LejlaKM Sharing your dataflow design and processor component configurations may help get you more/better responses to your query. Things you will want to look at before and when you run this dataflow: 1. NiFi heap usage and general memory usage on the host 2. Disk I/O and Network I/O 3. NiFi Host CPU Utilization (If your flow consumes 100% of the CPU(s) during execution, this can lead to what you are observing. Does UI functionality return once copy is complete?) 4. Your dataflow design implementation including components used, configurations, concurrent tasks etc. While most use cases can be accomplished through dataflow implementations within NiFi, not all use cases are a good fit for NiFi. IN this case your description points at copying a large Table from One Oracle DB to another. You made not mention of any filtering, modifying, enhancing, etc being done to the Table data between this move which is where NiFi would fit in. If your use case is a straight forward copying from A to B, then NiFi may not be the best fit for this specific us case as it will introduce unnecessary overhead to the process. NiFi ingest content and writes it a content_repository and creates FlowFiles with attributes/metadata about the ingested data stored in a FlowFile_repository. Then it has to read that content as it writes ti back out to a destination. For simple copy operations where not intermediate manipulation or routing of the DB contents needs to be done, a tool that directly streams from DB A to DB B would likely be much faster. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
01-12-2022
06:11 AM
@LuisLeite First think to note is the age of the NiFi version being used. HDF 3.1.2 was released back in 2018. There have been many improvements and advancements within NiFi since then including to the record based processors and controller services. 1. Does your source AVRO being produced via the ExecuteSQL processor contain the avro schema embedded in the content? 2. The exception you shared deals with writing the Avro Schema which is a function of the record writer and not the ConvertRecord or Record reader. I am assuming you are using the CSVRecordSetWriter in your use case. What do you have the "Schema Write Strategy" set to? Do you get same exception if you set this to " Set 'avro.schema' Attribute" instead? If above does not help, sharing the exact configuration of your convertRecord processor and controller services being utilized may help those in the community offer additional guidance. Of course a sample source file example would also be helpful. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
01-11-2022
08:37 AM
@Neil_1992 I strongly recommend not setting your NiFi heap to 200GB. Java reserves the XMS space and grows to the XMX space as space is requested. Java Garbage Collection (GC) execution to reclaim heap no longer being used does not kick in to ~80% of heap is used. That means GC in your case would kick in at around 160+ GB of used heap. All GC execution is stop-the-world activity which means your NiFi will stop doing anything until GC completes. This can lead to long pauses resulting node disconnections, issues with timeouts with dataflows to external services ,etc. When it comes to flow.xml.gz file, you are correct that it is uncompressed and loaded into heap memory. The flow.xml.gz contains: Everything you add via the NiFi UI to the canvas (processors, RPG, input/output ports, funnels. labels, PGs, controller services, reporting tasks, connections, etc.). This includes all the configuration of fro each of those components. NiFi Templates are also stored in the flow.xml.gz, uncompressed and loaded in to heap as well. Once a NiFi template is created, it should be downloaded, stored outside of NiFi, and local copy of template inside of NiFi deleted. As far as your specific flow.xml.gz, I see a closing tag "</property>" following that indicates that some component has a property which typically consists of a "name" and "value" with the huge null laced null strings in the value field. I'd scroll up to see which component this property belongs to and then check why this value was set. Maybe it was a copy paste issue? Maybe this is just part of some template that was created with this large string for some purpose? Nothing here says with any certainty that there is a bug. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
01-10-2022
01:31 PM
@techNerd I think your scenario may need a bit more detail to understand what you are doing and what it is doing versus what you want the flow to do. The ListFile only listed information about file(s) found in the target directory. It then generates a one of more FlowFiles from the listing that was performed. A corresponding FetchFile processor would actually retrieve the content for each of the listed files. From the sounds of your scenario, you have instituted a 20 sec delay somehow between that ListFile and FetchFile processor? Or you have configured the run schedule on the ListFile processor to "20 secs"? Setting the run schedule only tells the processor how often it should request a thread from the NiFi controller that can be used to execute the processor code. Once the processor gets its thread, it will execute. The ListFile processor will list all files present in the target source directory based on the configured file and path filters. For each File listed it will produce a FlowFile. Run schedule does not mean it executes for a full 20 seconds continuously checking the input directory to see if new files arrive. The run schedule also not impacted by how long it takes a listing to complete. It will request a thread every 20 seconds (00:00:20, 00:00:40, 00:01:00, etc...). The configured "concurrent tasks" controls whether the processor can execute multiple listing in parallel. Let say the thread that was executed at 00:01:00 was still executing 20 seconds later. Since that thread is still using the default 1 concurrent task, the listFile would not be allowed to request another thread from the controller at that time. Since the run schedule is independent of the thread execution duration, there is no way to dynamically alter the schedule. There is also no way for a new file to get listed at same time as a previous file (unless both were already present at time of listing) within the same thread execution. The listFile use the configured "Listing Strategy" to control how it handles listing of files. A "tracking" strategy is used to prevent the ListFile processor from listing the same file twice by recording some information in a state provider or a cache. If "No Tracking" is configured, the listFile will list all found files every time it executes. ListFile does not remove the source file from the directory. Removal of the source file is a function optionally handled by the corresponding FetchFile processor. If this is not clear, share more details around your use case and flow design specific so I can provide more direct feedback. Here is the documentation around processor scheduling (works the same no matter which processor is being used): https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#scheduling-tab If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
01-10-2022
05:58 AM
@Griggsy The "Advanced" UI capability in the UpdateAttribute processor allows you to setup rules to accomplish exactly what you are describing here in your use case. When you right click on the UpdateAttribute processor you will see the "Advanced" button in the lower left corner. From the Advanced UI you can configure conditional rules. 1. Click "+" to the right of "Rules" to create your first rule. 2. Then click "+" to right of "Conditions" to create your boolean NiFi Expression Language (NEL) statement(s). When using multiple statements, all statements must result in true before the corresponding "Action(s)" would be applied. 3. Then click "+" to right of "Actions" to set create one or more actions that are applied if the condition(s) for this rule are true. The thing is that your example or condition does not make sense to me. You are returning the value from a FlowFile attribute named "hostname" and checking if it contains the string "Mickey" in both NEL statements. However, for each you want to return a different output value? Since both would always return either "true" or "false", which output would be desired? Here is example where i check if "Hostname" attribute contains "Mickey" and if that returns true I set FlowFile attribute "hive_database" to "cartoon". I then have a second rule that checks the "Hostname" FlowFile attribute to see if it contains "Bond" and iid that returns true, I set "hive_database" FlowFile" attribute to "movie". The "conditions" and "actions" look similar for the "Bond" rule except "Mickey" is changed to "Bond" and cartoon is set to "movie" instead of "cartoon". So above rules give you an If/then capability. Another feature of using the advanced UI is the "else" capability. Outside the "Advanced" UI set a attribute for "hive_database": If I were to define actions that set the "hive_database" FlowFile attribute and upon evaluating a FlowFile none of the configured rules that set the "hive_database" attribute are applied, the update Attribute processor will set apply the value set outside the advanced UI. If a rule sets the the attribute "hive_database", then value defined outside advanced UI is ignored/not set. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
01-10-2022
05:24 AM
@Elmer What you described here is the use case for the NiFi Parameters capability. Parameters allow you to set configurations values within NiFi components thaT are unique per environment. Parameters can be used in both non-sensitive and sensitive property ( password) fields. Then in each unique environment you set up a parameter context that uses the parameter name, but with the unique value for that specific environment. Then when you port over your flow definition or template, you can configure the NiFi process group in which contains your ported over flow to use the cluster specific parameter context. No need to go to each component and reconfigure them anymore. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt [1]https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#Parameters
... View more
12-14-2021
12:03 PM
1 Kudo
@gvkarthik93 The HandleHTTPRequest processor only support user based authentication via a mutual TLS handshake. But there is no authorization built in to the processor. So if the truststore configured in the configured SSLContextService contains the needed trustedCertEntries to trust a user/client certificate presented in the handshake, that user would be allowed to send traffic to this listener. The processor could then route the success relationship via a connection to a routeOnAttribute processor that checks the dn set in the " http.subject.dn" FlowFile attribute created by the HandleHttpRequest processor to see if matches a list of DN. Based in that outcome decide to either route the FlowFile to a HandleHTTPResponse processor that responds with not authorized or route down an authorized path and respond accordingly. I want to check if the username and password is correct or not Check against what? NiFi has no local users with passwords for authentication, nor do any NiFi processors have any integration in to an external user authenticator like ldap. Even if you were to pass a username and password via a header on the request, there is no native processor that could take that username and password and verify it. Maybe you could use a scripting processor to validate the username and password against an external service like ldap. But between HandleHTTPRequest and that scripting processor, that user's name and password would be in plaintext within the FlowFiles attributes which is not ideal. Also keep in mind that if you do not use TLS at all, anything you send over the connection is not encrypted and in plain text. Hope this helps answer your query, Matt
... View more
12-14-2021
11:32 AM
@Saraali I'd love to help, but I have never done any scripting to convert to excel format. All I can tell you is there is no native processor available in NiFi that does this conversion. I'd recommend raising a new feature request jira in apache for NiFi: https://issues.apache.org/jira/projects/NIFI/issues/ Perhaps others in the community may be looking for the same capability who are willing to contribute code? Thanks, Matt
... View more
12-13-2021
01:02 PM
@armensano Just to make sure we are clear on your ask here. You have a secured NiFi and NiFi-Registry installation that uses user/client certificates to authenticate your user via the mutual TLS handshake? If that is the case, you should not be seeing a NiFi/NiFi-registry login window. Can you share screenshots? What do you see in the nifi-user.log when you attempt to access NiFi? Thanks, Matt
... View more
12-13-2021
12:14 PM
@sunny_de NiFi can only support a single "authorizer". Your authorizers.xml file attached shows that you have two configured: 1. managed-authorizer 2. file-provider <-- (legacy and deprecated in favor of above) Which authorizer is actually being used by NiFi's core is determined by what you have configured in the following property in the nifi.properties file: nifi.security.user.authorizer= So make sure above is set to "managed-authorizer" This provider then references the: file-access-policy-provider This provider is responsible for generating the "authorizations.xml" file which contains all the authorizations created for your users. It will initially seed this file with the minimum authorizations needed for the configured "initial admin identity" string value (Case sensitive), NOTE: The authorizations.xml file is only generated if it does not already exist. If this file already exists and changes made to the file-access-policy-provider will not be reflected in that file. The expectation is once the initial admin and node policies have been seeded, all additional authorization are granted from within the NiFi UI. The "file-access-policy-provider" then references the: file-user-group-provider This provider is designed to generate the users.xml file if it does not exist and seed it with the initial user identities (initial admin and NiFi nodes if clustered). Seeing as how you're able to see the NiFi UI, that tells me that your initial admin was successfully created and given at least "view the user interface" policy permissions. The initial policy setup will only grant polciy permission related to the canvas if a flow.xml.gz was already present when NiFi was secured or started for the first time. In this case, it appears that you did not have a flow.xml.gz and NiFi created one on first startup which happens after the initial admin was created and authorized. You will however have admin policies assigned for your initial admin that will allow you to setup any additional policy you need. I can see from the "operate" panel in the left hand side of the canvas that the following icon is NOT greyed out: This means that your authenticated user is authorized to set policies on the current selected component (in this case the root process group). Click on this icon and grant yourself at a minimum the following policies: 1. view the component 2. modify the component 3. view the data 4. modify the data 5. view provenance After applying above, the icon across the top of the UI should no longer be greyed out since you are now authorized to make modification to the root process group. So now you can drag and drop items to the canvas. When you add new components (processors, input/output ports, funnels, child process groups, etc) to the canvas they will inherit there policies from the parent process group, but you can always select any component and specifically set policies per component. In a multi-team use NiFi, it is common to create a child process group for each team or person and authorized on the desired members to each process group. This prevents user1 from modifying user 2's process group and components within it and vice versa. For more information on setting users, groups, and policies, here is the link to the NiFi documentation that talks about this topic: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#config-users-access-policies If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
12-13-2021
11:42 AM
@sam_001 It appears you are hitting a known bug in Apache NiFi 1.12: https://issues.apache.org/jira/browse/NIFI-7856 This provenance related bug was addressed in Apache NiFi 1.13. I recommend you upgrade. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
12-13-2021
11:27 AM
@Saraali NiFi on contains components for reading XLXS files and converting them to CSV. There are no native components included with NiFi that can convert the other direction. That being said, NiFi offers numerous processors that allow you to execute your own scripts or external commands against the content of a NiFi FlowFile. Here is an example where someone converted from CSV to Excel by passing the FlowFile content to an externally executed python script: https://stackoverflow.com/questions/68937735/how-to-convert-csv-to-excel-using-python-with-pandas-in-apache-nifi If you found this response assisted with your query, please take a moment to login and click on " Accept as Solution " below this post. Thank you, Matt
... View more
12-13-2021
10:57 AM
@Eristocrat @Eristocrat Logging varies by class. Some classes don't produce anything beyond INFO level, while others will produce even TRACE level logging. With TRACE level being the highest level, you will get all possible logging by using it. Do you have a screenshot of the exception you are seeing? At time of exception, did you find anything logged in the NiFi user and/or app logs? Thanks, Matt
... View more
12-09-2021
12:27 PM
@Eristocrat The "nifi.web.https.host" value on each node should be set to the unique hostname assigned to each of your NiFi nodes. Or are you saying that all the three instances of NiFi in your cluster are all installed on the same server/host? You should not have all your NiFi cluster nodes trying to bind to the same host unless they are all installed on sam host which would be extremely unusual and not recommended. is useful when the client to NiFi presents a different hostname than what was specified in the browser URL. For example, you enter https://nifi-proxy:443/nifi and that directs to a proxy that then routes to one of the NiFi cluster host urls https://nifi-node1:443/nifi. What happens here is NiFi receives a request for connecting to nifi-proxy, but nifi-node1 is not nifi-proxy so it fails. That is because it looks like someone made a request to access server nifi-proxy, but that request landed on nifi-node1. So adding nifi-proxy to the nifi.web.proxy.host list tells NiFi to accept request on nifi-node1 that were destined for nifi-proxy. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
12-09-2021
11:55 AM
1 Kudo
@MarcioMarchiori Depending on which NiFi Expression Language (NEL) function you are using, either of the following two NEL statements should work for you: ${path:replaceAll("\\\\",'/')}/${filename} In above you are using the replaceAll function which expects a java regular expression. Since '\" is an escape character and NiFi is a java application, the first \ escapes the second \ and the third \ escapes the forth \. So \\\\ results in a regex of \\ being applied against the value present in the "path" variable. or ${path:replace('\\','/')}/${filename} In above you are using the replace function which does not path the first argument to be evaluated as a java regular expression but rather takes a string literal. And since '\' is an escape character you need \\ which tells java to treat the second \ as the literal string value to search for. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
12-08-2021
10:42 AM
@IAJ Specific to your case we see the following difference: Local Fingerprint (This is fingerprint of flow from node trying to join cluster): s.email.ConsumeIMAPNO_VALUEorg.apache.nifinifi-email-nar1.13.210 sec30 sec1 secWARNfalseTIMER_DRIVENPRIMARY0Mark Messages as Read=truefolder=Inboxhost=outlook.office365.commail.imap.socketFactory.clas Cluster Fingerprint (This is fingerprint of flow currently being used by the cluster): s.email.ConsumeIMAPNO_VALUEorg.apache.nifinifi-email-nar1.13.210 sec30 sec1 secWARNfalseTIMER_DRIVENALL0Mark Messages as Read=truefolder=Inboxhost=outlook.office365.commail.imap.socketFactory.clas If we look closely we see that you have a consumeIMAP processor in your cluster flow configured to execute on "ALL" nodes. On the node trying to connect the same consumeIMAP processor is configured to execute on "Primary" node only. A common scenario where this can happen is if the node was "Disconnected" from the cluster and a user edited the configuration of this processor at that time. Now that node can not rejoin the cluster because of the mismatch. As other have said the flow loaded from the flow.xml.gz in to NiFi JVM Heap memory must match across all nodes. Since all nodes run the same flow.xml.gz, you can copy the this file form any node connected to the cluster to the node failing to connect. NOTE: Understand that by this, you will lose those local changes made while that node was disconnected. Another scenario is the configuration change was made while both nodes were connected, which means that changed was replicated to all nodes in the cluster. If in your 2 node cluster only one node responded to that replicated request that the change was made, the node(s) that did not would get disconnected. Failure to respond to a replication request can happen most often because of server resource issues and/or network issues. Thank you, Matt
... View more
12-07-2021
09:24 AM
@TN08 The InvokeHTTP processor supports HTTP methods for connecting to a target server. The InvokeHTTP processor does not establish a listener for inbound http connections. The properties for Basic Authentication Username and Password on this processor are used when the target endpoint supports basic auth and the username and password would need to provided by that target endpoint. If basic auth is not supported by target endpoint, TLS is also supported. If you want to setup a listener in NiFi for inbound HTTP you would use either ListenHTTP processor or the newer HandleHttpRequest/HandleHttpResponse processors. NiFi processors do not support basic auth for inbound connections. So to connect with these processors the source systems would need to pass a client/user certificate in a mutual TLS handshake as a way to identify the client. This means the processor would need to have a StandardRestrictedSSLContextService controller service that is configured with a keystore (containing a serverAuth certificate the client system can trust) and truststore (containing the complete trust chain for the clientAuth certificate presented by your client). If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
12-03-2021
05:39 AM
@dontiffjr The exception in your browser ERR_CERT_AUTHORITY_INVALID typically means that the trust chain for your NiFi's serverAuth certificate is not trusted by your browser. You should see an option in the browser to "proceed to ...". If you click on that, can you get to the NiFi UI? You can also use openssl command to inspect the server hello coming from your NiFi and obtain the public cert for your NiFi server's certificate. You can load those public certificates into you browser trust. openssl s_client -connect <nifi-hostname>:<nifi-port> -showcerts Next thing to look at would be the contents of your certificate. <path to java>/bin/keytool -v -list -keystore <path to>/keystore.jks You'll want to make sure it contains: 1. A DN that does not contain wildcards 2. ExtendedKeyUsage (EKU) with both clientAuth and serverAuth 3. SubjectAlternativeName (SAN) with entry that matches the hostname of the server on which it is being used. 4. verify issue and expiration dates for certificate and that server clock and your local client machine where you are using browser has same date and time. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more