Solved

Hi, We've installed the Query Processor service within the cluster. Everything is working great but I'm having problems changing the log location. Unlike other services (Hive, Impala, etc) the configuration does not have a log dir setting. I was hoping that I might be able to override this directly in the log4j2 config via the hue-query-processor-log4j2.yml_role_safety_valve however this option does not seem to be resulting in any change to the generated config. Grateful for any pointers? 😀

My team and I are using Apache Nifi to sync data between databases. Currently we are working through attempting to connect both out Production Nifi service and our Development Nifi service to our Nifi registry service.The production Nifi service and the Nifi registry are running on one EC2 server and the development Nifi service is running on another. We have them all set up to be secure using LDAP and users/groups are managed in Windows server AD. We are having some issues trying to get the Nifi services connected to the Registry service and were hoping we could get some feedback or guidance on what we may be missing or doing incorrectly.The Nifi production instance is available from the host https://nifi.example.com/nifi and the Nifi registry is available at https://nifi-registry.example.com/nifi-registry. We have followed the documentation and set the registry in Nifi to the correct endpoint, but are getting the following error:"Unable to obtain listing of buckets: javax.net.ss

hello.I am a beginner in NIFI.Currently, there are two CENTOS7-based Linux servers with NIFI version 1.27.0 installed.Clustering two serversIt is linked with external zookeepers.The nifi setup was completed with the ssl authentication setup using the jks file.In this state, I changed the "nifi.security.user.authorizer" option to "managed-authorizer".At this time, I would like to create and use multiple users in the users.xml file without LDAP, Kerberos, etc(only users.xml file and authorizations.xml file).Example user.xml file and authorizations.xmlI'm curious about the file example.Also if possibleI would like to see again how to configure these settings. After searching, I found that there is no way to dynamically add users without restarting the service.So ultimately what I want to do isWithout LDAP or Kerberos,After creating multiple users in users.xml and restarting the service,I want to log in with those accounts.

Hello, everyone.I have an issue with NiFi Registry 1.20 -- UI button "Add user" is inactive with Initial Admins account.But, if I check the user permissions, it looks good enough:The Initial Admin user is LDAP user and also member of "nifi admins" LDAP group.My authorizers.xml file:<?xml version="1.0" encoding="UTF-8" standalone="yes"?><authorizers><userGroupProvider> <identifier>ldap-user-group-provider-1</identifier> <class>org.apache.nifi.registry.security.ldap.tenants.LdapUserGroupProvider</class> <property name="Authentication Strategy">LDAPS</property> <property name="Manager DN">user</property> <property name="Manager Password">password</property> <property name="TLS - Keystore">./conf/keystore.jks</property> <property name="TLS - Keystore Password">pass</property> <property name="TLS - Keystore Type">JKS</property

I want to filter rows of a CSV file based on a regular expression, but so far I didn't get any row as a match.CSV:F;0;00003;;;SA;30012025;012025;;0456;3790;4940;0;0;150.00;"- Sachbezug (sonstiger)";;;EURF;0;00003;;;SA;30012025;012025;;0461;3790;4949;0;0;50.00;"- Abzug Gutschein stfr.";;;EURG;0;00003;;;SA;30012025;012025;;0486;3790;3725;0;0;970.00;"- sonstiger Abzug";;;EURG;0;00003;;;SA;30012025;012025;;0487;3790;6301;0;0;50.00;"- Abzug Arbeitskleidung";;;EURF;0;00003;;;SA;30012025;012025;;1160;3790;6010;0;0;-25.91;"Ausgl. Zuschl.b.Krankheit";;;EURF;0;00003;;;SA;30012025;012025;;1161;3790;6010;0;0;-63.36;"Ausgl. Zuschl. Feiertag";;;EURNifi-Workflow:Because I don't have any permission to upload images I need to describe what the workflow looks like - sorry.I have a GetFile processor to read the CSV file. This processor is connected directly with the 'RouteText' processor. This processor is connected via the 'unmatches' output to a disabled 'Wait' processor. An output called 'MatchedG' ha

Hello, everyone!We have a three-node Apache NiFi cluster with LDAP auth and now, users must enter node name to access NiFi UI, e.g.:https://nifi-node-01.company.com:9443/nifiI'd like to do it more friendly for them and create some LB with HAProxy/Nginx etc. You know, to access NiFi with address like that:https://nifi-service.company.comI've read a lot of articles about LB, but all of them talk about LB for data flow, not for UI. So, what do you think - it is OK? Are where any potential pitfalls?

Can NiFi be used to send AS2 messages for EDI file ?If Yes, please provide some information.Thanks

Hi Team,Could you please confirm whether the CountText processor is a standard NiFi processor? I couldn't find it listed under the "Supported NiFi Processors" section in the Cloudera Flow Management 2.1.7 release notes:https://docs.cloudera.com/cfm/2.1.7/release-notes/topics/cfm-supported-processors.htmlYour insights would be appreciated.

Hi We've an existing HDP cluster with version 2.6.5.0-292 with Ranger service setup. Will this Ranger work with MySql 8.4 (without upgrading HDP itself)? From the documentation, it looks like it doesn't but wanted to confirm.Thanks

Hi PutFile adds extra slash at the end, which create a unwanted dir with the filename.Tried using EL to remove the extra slash at the end, but its not working├── abc101c.csv│   └── abc101c.csv├── abc.txt│   └── abc.txt├── abcxyz.txt│   └── abcxyz.txt├── demo│   └── xyz.txt├── fex│   └── test999.txt├── folder│   ├── folder1.txt│   └── folder.txt Thanks

Hello,I am attempting to unzip a series of simple .zip files in my NiFi flow. All of my flow executes with no issues except when I try to unpack. There is nothing special about these zip files (i.e. no passwords, etc...). But no matter which way I try to unpack the zip files, I get the error that the zip file does not contain any entries. I have verified that the zip files do contain entries. I can easily unzip with WinZip. I can also easily unzip with our old custom C# application that we are replacing with NiFi.I have tried adjusting the UnpackContent settings and even have added an UpdateAttribute to add the application/zip mime type as well. Still keep getting the failure. I have also tried the CompressContent processor and still get these errors on simple WinZip files.I am attaching my flow, my processor settings and error message. 

Hello.We have ORC tables loaded from Hive (Tez) and queried from Impala. Due to their number of partitions, we had to increase the TEZ AM task's heap, and the note https://my.cloudera.com/knowledge/Tez-AMtask-fails-with-OOME-because-of-huge-impala-stats-chunk?id=348603 seems to match our issueWe'd like to reduce the AM memory footprint, but the solution doesn't work in our 7.1.9 and 7.3.1 environments:* After setting the option in the hive-site.xml, the "impala_intermediate_stats_chunk*" messages are still present in the Hive's plans.* Setting the option at the session level returns the error "Cannot modify hive.plan.mapwork.serialization.skip.properties at runtime. It is not in the list of params that are allowed to be modified at runtime (state=42000,code=1)".* https://github.com/apache/hive/pull/2620 shows that the option should be allowed to be set at the session level, as the HIVE-25501 was implemented in the CDP-7.1.7.After some research we came across the Jira https://

Hi all,I'm working with Cloudera Hive for the first time, seen that my company chose to use this database.I received the JDBC connectionString and I created a little method in java.Here the scriptpublic String[] getEmployee(){ String[] idcod ={"ABC123", "FLS163","XYZ001","PLE456", "ERV021", and so on......}; String[] names= new String[idcod.length]; int ncount=0; long start1, elapsedTimequery; try{ Statement s = conn.createStatement(); System.out.println("Fetch size: "+s.getFetchSize()); ResultSet r=null; for(int i=0; i<idcod.length; i++) { String query=""; query+=" SELECT code, surname, name FROM mytable where code='"+idcod[i]+"'"; ncount=0; start1 = System.nanoTime(); r = s.executeQuery(query); elapsedTimequery = System.nanoTime() - start1; System.out.println((double)(elapsedTimequery/1000/1000)/1000+" seconds"); while(r.next()){ names[i]=r.getString("surname") + " " +r.getString("name"); ncount++; } System.out.println(ncount+"--"+idcod[i])

Hi all,I have the requirement that to do the unit testing for each nifi processors while send hl7 message via a python script. The following are the needs:1. After sent the HL7 test message it should receive the ACK/NACK data=  Done2. After processed the validation check this flow file will move into another upstream processors = [ Here I need the status of the flow file on each upstream processors whether it successful / fails.For that I am trying to extract or fetch the provenance data through nifi rest api, I am using NiFi version 2.0.0 with the python script I sent the payload aspayload = {"provenance": {"request": {"componentId": "f6ca227e-0196-1000-13c2-d66cf56e69f2","sort": "eventTime","maxResults": 1}}}with url  https://localhsot:port/nifi-api/provenanceI am getting this output ::clipboard::clipboard: Showing 1 recent provenance events::three_o_clock: 05/14/2025 15:48:24.041 IST:wrench: Component: ExecuteGroovyScript (ExecuteGroovyScript):page_facing_up: Type: CLONE🧾

I'm encountering persistent issues setting up NiFi 2.4.0 on Kubernetes (deployed via a Helm chart, celtic) with OIDC authentication (Keycloak) for an initial admin user. My goal is to have a specific OIDC user (e.g., 'nifi-admin') fully permissioned to manage flows and processors upon deployment.When I first start up the pod, the users and authorizations file look like the following.<?xml version="1.0" encoding="UTF-8" standalone="yes"?><authorizations> <policies> <policy identifier="f99bccd1-a30e-3e4a-98a2-dbc708edc67f" resource="/flow" action="R"> <user identifier="0482ca97-7afc-3d5a-8a3f-21803898bde0"/> </policy> <policy identifier="b8775bd4-704a-34c6-987b-84f2daf7a515" resource="/restricted-components" action="W"> <user identifier="0482ca97-7afc-3d5a-8a3f-21803898bde0"/> </policy> <policy identifier="627410be-1717-35b4-a06f-e9362b89e0b7" resource="/tenants" action="R

Hi Team,I'm currently using a custom NiFi processor to convert JSON file content into flow file attributes, and I'm looking to replace it with a standard NiFi processor if possible.Sample JSON Input:The JSON files look like this (keys may vary per file):{"SourceFilePath": "/root/rdw/SrcFiles","SourceFilePattern": "Sales_data","SourceFile": "Sales_data_20241211.csv"}Current Setup:I'm using a custom processor of type CreateFlatJSON (Bundle: NestedJSONToFlatJSON) to extract the key-value pairs from the JSON and assign them to attributes.After processing, the flow file attributes look like:SourceFilePath = /root/rdw/SrcFilesSourceFile = Sales_data_20231111164536.csvSourceFilePattern = Sales_dataThese attributes are then used in downstream processors like so:${SourceFilePath}/${SourceFile}Current Flow:FetchSFTP → ExtractText → CreateFlatJSON (Custom) → Downstream ProcessorsFetchSFTP: Retrieves the JSON file.ExtractText: Reads the content into the flow file.CreateFlatJSON: Parses JSON and co

Hello!I'm trying to set up Nifi-Registry OIDC authentication with Windows ADFS as oidc provider, but have no luck. Using self-signed sertificates for SSL connection i'm able to LOGIN in web interface with initial admin and create some groups with different priveleges, for examle nifi-reg-admins with full rights. In Active Directory this group is also present and the user nifi-admin-2@blackboks.ru is member of this group. In ADFS Management a have setup template Server application accessing a web API with issuance transform rule as "Send Groupmembership as a Claim".  Respectively i choose nifi-reg-admins group as User group, Outgoing claim type - Group and  Outgoing claim value - nifi-reg-admins. I can login with  nifi-admin-2@blackboks.ru into web interface, but there is no admin priveleges at all and in logs i see this:INFO [NiFi Registry Web Server-39] o.a.n.r.w.m.AccessDeniedExceptionMapper identity[nifi-admin-2@blackboks.ru], groups[] does no

My NiFi pipeline fetches and processes files from an SFTP server.When the number of files to fetch does not exceed 500, everything works well.However, sometimes the number of files the pipeline has to list at once exceeds 2,000,000, which critically impacts performance.I’m trying to find a way to throttle the file stream so that no more than 500 files are processed at any given time.First Attempt: GetSFTP ProcessorFirst, I looked into the GetSFTP processor. It has a convenient property called "Remote Poll Batch Size".However, this deprecated processor has several disadvantages, the most significant being that it does not maintain state.Because of this, when I set "Delete original" to False, it fetches files infinitely.Setting "Delete original" to True is not desirable because my pipeline’s guaranteed delivery requirements mandate that files should only be deleted after successful processing (

HiI'm runningApache Nifi 2.3.0MS SQL EXPRESSJDBC Driver mssql-jdbc-12.8.1.jre11.jarI've configured Controller Service (DBCPConnectionPool 2.3.0). It was successfully enabled.Problem: When I try to configure any processor that uses DBCP, I don't see the connection pool. I tried different connection pools but facing the same issue. No sure what is causing this.Appreciate any help   

Hi everyone,I'm currently working with Apache NiFi 2.1.0 versions, and I'm wondering:Is it possible to create a custom Flow Analysis Rule, similar to how we can create custom processors?Thanks in advance for your help!

Hey, everyone!Could you help me with next problem, please? I have a test NiFi instance, where I turned on Content Storage archive. It was works good, but one time I decided to increase value of "nifi.content.repository.archive.max.usage.percentage" from default 50% to 70%.So, NiFi utilized space to 70% of total as I expected.But, after that I disabled archiving and expected that NiFi releases all data used by archive, but it doesn't happens.Why so? I've saw messages, that archived data is never cleanup if "nifi.content.repository.archive.enabled" set to "false" after it has been "true". Is that truth?My current settings:nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepositorynifi.content.claim.max.appendable.size=1 MBnifi.content.repository.directory.repo0=/mnt/nifi/repos/content_repositorynifi.content.repository.archive.max.retention.period=6 hoursnifi.content.repository.archive.max.usage.percentage=60%nifi.content.repository.archive.backpressu

Hello,We have set up Apache nifi 2.1.0 with 2 nodes with cluster mode, secure mode and single user access enabled. When we access by the application web behind a load balancer and sticky session active all it's ok. When we use API Rest by programmatic mode we ask NiFi "/access/token" and we get a valid token immediately we send a request with authorization bearer token and JWT when the request is attended by the same node which service the JWT all right but when the request is attended by the other node, in the same cluster, response with 401 error.  ¿How can we make NiFi to accept the JWT in all nodes?

I set up a 4-node NiFi cluster and generated certificates. When I try to log in using my certificate user nifi_admin, I get the error:"An unexpected error has occurred. Please check the logs for additional details."When I checked the logs, I saw that the nodes in the cluster were encountering the following errors when trying to connect to the coordinator node:The node I connect to the interface:  https://vtmrt3anifit01.x.com:8443/nifi/The error received in the nifi-user.log file on this machine is as follows:ERROR [NiFi Web Server-221] o.a.nifi.web.api.config.ThrowableMapper An unexpected error has occurred: java.io.UncheckedIOException: Read Current User Entity failed. Returning Internal Server Error response.java.io.UncheckedIOException: Read Current User Entity failedat org.apache.nifi.web.api.FlowResource.readReplicatedCurrentUserEntity(FlowResource.java:446)at org.apache.nifi.web.api.FlowResource.getCurrentUser(FlowResource.java:421)at java.base/jdk.internal.reflect.DirectMet

I have have two request formdata param. One param is file which is multipart and another is acno which is string. Here is the postman image: Postman request is working fine. This postman request, I want to process using Apache Nifi. Here I have tried to use below image in apache nifi:But when start above aldiagram file & acno not foundHere is screen steps:1. GetFile screen :2. Update Attribute screen:3. AttributeToJson screen:4. InvokeHTTP screen:I follow steps above screen --> Getfile->UpdateAtribute->AttributeToJson->InvokHttpWhat is the wrong of my configuration?Please help me...

I am running few yarn jobs and sometimes I am seeing the heap utilisation going beyond 80%.As of now the heap for RM is set as 4GB.Let me know how to decide the RM heap ?