Ask the Community

I'm encountering persistent issues setting up NiFi 2.4.0 on Kubernetes (deployed via a Helm chart, celtic) with OIDC authentication (Keycloak) for an initial admin user. My goal is to have a specific OIDC user (e.g., 'nifi-admin') fully permissioned to manage flows and processors upon deployment.When I first start up the pod, the users and authorizations file look like the following.<?xml version="1.0" encoding="UTF-8" standalone="yes"?><authorizations> <policies> <policy identifier="f99bccd1-a30e-3e4a-98a2-dbc708edc67f" resource="/flow" action="R"> <user identifier="0482ca97-7afc-3d5a-8a3f-21803898bde0"/> </policy> <policy identifier="b8775bd4-704a-34c6-987b-84f2daf7a515" resource="/restricted-components" action="W"> <user identifier="0482ca97-7afc-3d5a-8a3f-21803898bde0"/> </policy> <policy identifier="627410be-1717-35b4-a06f-e9362b89e0b7" resource="/tenants" action="R

Hi,We're upgrading some Custom Processors directly from 1.23.2 to 2.4.0, but the newest artifact for nifi-nar-bundles we can find is 2.0.0-M2. Has this artifact been deprecated? And if so, what dependency/ies should we use instead?thanks,C.

Hi,What is the steps to create Python Custom Processor in NiFi 2.3.0 Version. I have created it but its not showing in NiFi GUI.Can you please help me with the process/steps to create python custom processor in NiFi.Thank you 

Hi NIFI Team, We are fairly new to apache NIFI and planning to onboard it into our production environment soon. And we are actually in need of a dedicated professional from the Apache NIFI team who can support or troubleshoot in case of any issues. Please let us know where I can find the appropriate information for professional services provided by NIFI and its pricing. Thanks in advance!  

Hello,I have a 3-node NiFi cluster, and I want to manage it using NiFi Registry. I configured both NiFi and NiFi Registry with a single certificate using the TLS Toolkit. I also set up LDAP integration. I can successfully connect to both NiFi and NiFi Registry individually using my LDAP users.However, the LDAP user that I added and authorized in the Registry does not appear in NiFi. With the certificate user, I can view the bucket in NiFi Registry from NiFi and perform flow version control. But I cannot do this with my LDAP user.NOTE: Even if I generate separate certificates for NiFi and NiFi Registry and trust each certificate independently, the certificate user does not have permission to view the bucket. This is because the certificate user from the Registry is also not created in NiFi. For this reason, I generated both from the same certificate.Nifi / Nifi Registry version: 1.28.1Nifi Registry nifi-registry.properties# Licensed to the Apache Software Foundation (ASF) under one or m

Hi Team,I'm currently using a custom NiFi processor to convert JSON file content into flow file attributes, and I'm looking to replace it with a standard NiFi processor if possible.Sample JSON Input:The JSON files look like this (keys may vary per file):{"SourceFilePath": "/root/rdw/SrcFiles","SourceFilePattern": "Sales_data","SourceFile": "Sales_data_20241211.csv"}Current Setup:I'm using a custom processor of type CreateFlatJSON (Bundle: NestedJSONToFlatJSON) to extract the key-value pairs from the JSON and assign them to attributes.After processing, the flow file attributes look like:SourceFilePath = /root/rdw/SrcFilesSourceFile = Sales_data_20231111164536.csvSourceFilePattern = Sales_dataThese attributes are then used in downstream processors like so:${SourceFilePath}/${SourceFile}Current Flow:FetchSFTP → ExtractText → CreateFlatJSON (Custom) → Downstream ProcessorsFetchSFTP: Retrieves the JSON file.ExtractText: Reads the content into the flow file.CreateFlatJSON: Parses JSON and co

Hello!I'm trying to set up Nifi-Registry OIDC authentication with Windows ADFS as oidc provider, but have no luck. Using self-signed sertificates for SSL connection i'm able to LOGIN in web interface with initial admin and create some groups with different priveleges, for examle nifi-reg-admins with full rights. In Active Directory this group is also present and the user nifi-admin-2@blackboks.ru is member of this group. In ADFS Management a have setup template Server application accessing a web API with issuance transform rule as "Send Groupmembership as a Claim".  Respectively i choose nifi-reg-admins group as User group, Outgoing claim type - Group and  Outgoing claim value - nifi-reg-admins. I can login with  nifi-admin-2@blackboks.ru into web interface, but there is no admin priveleges at all and in logs i see this:INFO [NiFi Registry Web Server-39] o.a.n.r.w.m.AccessDeniedExceptionMapper identity[nifi-admin-2@blackboks.ru], groups[] does no

My NiFi pipeline fetches and processes files from an SFTP server.When the number of files to fetch does not exceed 500, everything works well.However, sometimes the number of files the pipeline has to list at once exceeds 2,000,000, which critically impacts performance.I’m trying to find a way to throttle the file stream so that no more than 500 files are processed at any given time.First Attempt: GetSFTP ProcessorFirst, I looked into the GetSFTP processor. It has a convenient property called "Remote Poll Batch Size".However, this deprecated processor has several disadvantages, the most significant being that it does not maintain state.Because of this, when I set "Delete original" to False, it fetches files infinitely.Setting "Delete original" to True is not desirable because my pipeline’s guaranteed delivery requirements mandate that files should only be deleted after successful processing (

Dear Community,I've faced with an issue when i try to send the following query to Apache Hive using ODBC driver:"-- Request ID: 4b9a2e0a-e250-45b7-b9cc-0915f20f848fSELECT *FROM (-- Request ID: 4b9a2e0a-e250-45b7-b9cc-0915f20f848fselect version()) AS Subquery__9LIMIT 1"then i get an empty response.However, if i remove the commented out fragments with Request Id and its uuid then it starts working as expected by returning the version.What am i missing here or is it an ODBC driver bug?PS: it's not reproducible when i do the same using JDBC.

HiI'm runningApache Nifi 2.3.0MS SQL EXPRESSJDBC Driver mssql-jdbc-12.8.1.jre11.jarI've configured Controller Service (DBCPConnectionPool 2.3.0). It was successfully enabled.Problem: When I try to configure any processor that uses DBCP, I don't see the connection pool. I tried different connection pools but facing the same issue. No sure what is causing this.Appreciate any help   

Hi everyone,I'm currently working with Apache NiFi 2.1.0 versions, and I'm wondering:Is it possible to create a custom Flow Analysis Rule, similar to how we can create custom processors?Thanks in advance for your help!

Hey, everyone!Could you help me with next problem, please? I have a test NiFi instance, where I turned on Content Storage archive. It was works good, but one time I decided to increase value of "nifi.content.repository.archive.max.usage.percentage" from default 50% to 70%.So, NiFi utilized space to 70% of total as I expected.But, after that I disabled archiving and expected that NiFi releases all data used by archive, but it doesn't happens.Why so? I've saw messages, that archived data is never cleanup if "nifi.content.repository.archive.enabled" set to "false" after it has been "true". Is that truth?My current settings:nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepositorynifi.content.claim.max.appendable.size=1 MBnifi.content.repository.directory.repo0=/mnt/nifi/repos/content_repositorynifi.content.repository.archive.max.retention.period=6 hoursnifi.content.repository.archive.max.usage.percentage=60%nifi.content.repository.archive.backpressu

Hello,We have set up Apache nifi 2.1.0 with 2 nodes with cluster mode, secure mode and single user access enabled. When we access by the application web behind a load balancer and sticky session active all it's ok. When we use API Rest by programmatic mode we ask NiFi "/access/token" and we get a valid token immediately we send a request with authorization bearer token and JWT when the request is attended by the same node which service the JWT all right but when the request is attended by the other node, in the same cluster, response with 401 error.  ¿How can we make NiFi to accept the JWT in all nodes?

Hi everyone, I would like some clarification. Thank you.Embedded Container Service 1.5.1, CML installation keeps getting stuck and eventually fails with a timeoutcheck the CML pod:  /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml -n cml get po  We found that the status of the s2i-builder pod is CrashLoopBackOff.check log:  /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml -n cml logs s2i-builder-97d946b65-46fhcDefaulted container "fluent-bit" out of: fluent-bit, s2i-builder, update-ca-trust (init), chowndockersock (init), chowndockercert (init)Fluent Bit v1.9.10* Copyright (C) 2015-2022 The Fluent Bit Authors* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd* https://fluentbit.io[2024/09/26 08:19:29] [ info] [fluent bit] version=1.9.10, commit=, pid=1[2024/09/26 08:19:29] [ info] [storage] version=1.3.0, type=memory-only, sync=normal, checksum=disabled, max_chunks_up=128[2024/09/26 08:19:29]

We have a number of PostHTTP processors sending files to ListenHTTP processors that chug along day after day happy as a clam. Then all of a sudden, there will be a spate of these errors: "Cannot send . . . to . . . because the destination does not accept FlowFiles and this processor is configured to delivery FlowFiles; routing to failure."For one of the PostHTTP processors, I created additional routes on success/failure so that I could look at and compare the files. These files are JSON, and after downloading the files to both success and failure, both are correctly formed JSON. We have a retry-loop flow here, too, that on failure retries the post five times. So these additional routes also allowed me to see in provenance one of the files that failed with the above error eventually went through.I'm looking for any insight on why this might be happening.About = niagarafiles 11.31.0 based on Apache NiFi 1.26.0a

I set up a 4-node NiFi cluster and generated certificates. When I try to log in using my certificate user nifi_admin, I get the error:"An unexpected error has occurred. Please check the logs for additional details."When I checked the logs, I saw that the nodes in the cluster were encountering the following errors when trying to connect to the coordinator node:The node I connect to the interface:  https://vtmrt3anifit01.x.com:8443/nifi/The error received in the nifi-user.log file on this machine is as follows:ERROR [NiFi Web Server-221] o.a.nifi.web.api.config.ThrowableMapper An unexpected error has occurred: java.io.UncheckedIOException: Read Current User Entity failed. Returning Internal Server Error response.java.io.UncheckedIOException: Read Current User Entity failedat org.apache.nifi.web.api.FlowResource.readReplicatedCurrentUserEntity(FlowResource.java:446)at org.apache.nifi.web.api.FlowResource.getCurrentUser(FlowResource.java:421)at java.base/jdk.internal.reflect.DirectMet

I have have two request formdata param. One param is file which is multipart and another is acno which is string. Here is the postman image: Postman request is working fine. This postman request, I want to process using Apache Nifi. Here I have tried to use below image in apache nifi:But when start above aldiagram file & acno not foundHere is screen steps:1. GetFile screen :2. Update Attribute screen:3. AttributeToJson screen:4. InvokeHTTP screen:I follow steps above screen --> Getfile->UpdateAtribute->AttributeToJson->InvokHttpWhat is the wrong of my configuration?Please help me...

I am running few yarn jobs and sometimes I am seeing the heap utilisation going beyond 80%.As of now the heap for RM is set as 4GB.Let me know how to decide the RM heap ?

Hello,After cloudera cluster upgrade, the hive service becomes unhealthy after a few hours.What i noticed is that the symlinks from /etc/alternatives are still pointing to the old parcel, but not sure if this is what makes the service unhealthy because I keep seeing the following error in the logs:ERRORS:Failed to get database cloudera_manager_metastore_canary_test_catalog_hive_HIVEMETASTORE_*NoSuchObjectException(message:No catalog cloudera_manager_metastore_canary_test_catalog_hive_HIVEMETASTORE_*Failed to delete hdfs://*/user/hue/.cloudera_manager_hive_metastore_canary/cloudera_manager_metastore_canary_test_catalog_hive_HIVEMETASTORE_* Thank you!

Hello all,I could use some help — what’s the current process for obtaining a developer license?We have an older link to a Google request form that we’ve used in the past, but it doesn’t seem to work anymore.Thank you.

Hi, We've developed custom Python-based processors (GetCheckpoint, UpdateCheckpoint) for NiFi 2.0.0-M2 to handle checkpoint creation and updates and put it in  python/extensions/GetCheckpoint.pypython/extensions/UpdateCheckpoint.py These processors worked correctly in 2.0.0-M2. However, after upgrading to NiFi 2.1.0, we’re encountering the following issues:     All custom processor properties now display the message:    “Sensitive of the parameter does not match the sensitivity of the property”    — even though the properties were not marked as sensitive in the code.     When adding a new instance of the processor, properties appear correct, but state handling breaks.     We modified and rebuilt the processors and, when adding them freshly to a flow, they seem to work correctly without any errros. However, existing GetCheckpoint /UpdateCheckpoint still show the sensitivity mismatch warning.  

I need to extract a  parent JSON column present in Hadoop:This is an example of my JSOn.Here I need to extract only values products class but not of offer. I cant use index of product as the the number of occurrences might change in future.Have used the following regex regexp_replace(regexp_replace(get_json_object(xml_data, '$.customer.products'),'\\}\\,\\{','\\}\\\n\\{'),'\\[|\\]','') AS prd_JSONBut is is only giving all values in a row , not in multiple rows.Have also tried something like below, but it doesnt give correct result:regexp_replace(regexp_replace(get_json_object(xml_data, '$.customer.products'),'\\}\\,\\{','\\}\\\n\\{'),'\\[|\\]','') AS prd_JSONfrom prd_table)select get_json_object(prd_item,'$.productCode') as productCode,get_json_object(prd_item,'$.type') as Ttypefrom prdLATERAL VIEW OUTER EXPLODE(split(prd_JSON, ';')) p AS prd_item) a {"Customer": {"CustomerId": "19900101","appl": [{***Some values}]"products": [{"ProductCOde": ABC,"Type": C1,"Offer": 1657

Hi Team i want to acees the hive ACID table using the spark.sql but when i am trying that i am just getting the table columns noit the data, now i tried using HWC but i am not sure as its showing in the docs that LLAP deamon shold be there in order to work with HWC can anyoine help me i am using spark 3.2.1 what is the correct way to acess the hive acoid table using spark.sql ?what are the prerequsite in order to work with HWC?can someone share a code sniped so that i can also access the hiv acid table using spark.sql ?spark = (            SparkSession.builder.appName("ReadHiveTable")            .config("hive.metastore.uris", hive_metastore_uri)            .config("spark.sql.warehouse.dir", warehouse_dir).enableHiveSupport().getOrCreate() df = spark.sql(f"SELECT * FROM {table_name}").show() 

Hello everyone,I try to create a custom PutSFTP processor to add different failure Relationships to further improve my error handling and go different routes if an error occurs.Im using NiFi-2.3.0 and a Java 21 shaded JAR for my custom processorsmy issue is that i get java.lang.NoClassDefFoundError: org/apache/nifi/processors/standard/PutSFTP message when loading my custom processor in Nifi. I already tried:adding the standard processors to my shaded jar but that only made things worse and some standard processors stopped workingadding nifi-file-transfer dependency to shaded jar but then the default PutSFTP stopped workinguse extends PutFileTransfer<SFTPTransfer> instead of PutSFTP but again NoClassDefFound only this time for PutFileTransferIs there a way to add the missing Class without breaking anything else?I really want to avoid rebuilding the whole PutSFTP to a custom PutSFTP when i only need to change small parts of it. 

@Cloudera We have received below mentioned vulnerability from our security team. Could you please check and let us know if our Hadoop cluster is impacted by this vulnerability.If yes the can you please provide a fix for this.CVE-2025-3884     Cloudera Hue Directory Traversal Information Disclosure Vulnerability CM version   :  Cloudera Enterprise 6.3.4 (#21561749)CDH Version :  6.3.4-1.cdh6.3.4.p5552.32087246