Ask the Community

Hi,All applications within our portfolio utilize PgBouncer as the database connection pool mechanism, and consequently, any application-level pooling is to be disabled. Apache NiFi is one of the applications required to adhere to this standard. We are seeking guidance on the options available to disable database connection pooling in NiFi. Could you please advise on how we can accomplish this? Our current nifi version is 2.1Thanks for your help and let me know if more details are required. 

I am having an issue using the ExecuteSQLRecord processor and writing to JSON. For some reason the results returns a NULL value for one of my fields in the FlowFile results. However, if I execute the query directly in SQL Server, I get all my results with no NULL fields. I checked and there are no special characters, single quotes are escaped and nothing looks to be out of the ordinary. SQL Select Statement:Select emailTo = REPLACE(e.Email_Recipients, ';', ','), emailFrom = e.Email_Sender, emailSubject = e.Email_Subject, emailMessage = CASE WHEN c.ValidationCount = 0 THENREPLACE(e.Email_Message, '{0}','${filename}') WHEN c.ValidationCount = 1 THENCONCAT(REPLACE(e.Email_Message, '{0}','${filename}'), '<p>There is 1 validation error in this file</p>', v.Validations)ELSECONCAT(REPLACE(e.Email_Message, '{0}','${filename}'), '<p>There are ', c.ValidationCount, ' validation errors in this file</p>', v.Validations)END,    v.

Hello community,I have a PowerBi application with some dashboards. The dashboards are updated according to the configuration at least 2 times a day. The dashboards connected to Impala sometimes do not update and I get this error (see below). It does not happen all the time but many times. I have to update it manually every time I get the error. I use cloudera ODBC driver (64 bit) and basic gateway (PowerBi Cloud): on-premise data gateway.Can you help me to solve the update problems?Thank you,Translated with DeepL.com (free version)Error message:  {"error":{"code":"DM_GWPipeline_Gateway_MashupDataAccessError","pbi.error":{"code":"DM_GWPipeline_Gateway_MashupDataAccessError","parameters":{},"details":[{"code":"DM_ErrorDetailNameCode_UnderlyingErrorCode","detail":{"type":1,"value":"-2147467259"}}, {"code":"DM_ErrorDetailNameCode_UnderlyingErrorMessage","detail":{"type":1,"value":"ODBC: ERROR [HY000] [Microsoft][DriverSupport] (1170) Unexpected response received from server. Plea

Hi, We've installed the Query Processor service within the cluster. Everything is working great but I'm having problems changing the log location. Unlike other services (Hive, Impala, etc) the configuration does not have a log dir setting. I was hoping that I might be able to override this directly in the log4j2 config via the hue-query-processor-log4j2.yml_role_safety_valve however this option does not seem to be resulting in any change to the generated config. Grateful for any pointers? 😀

My team and I are using Apache Nifi to sync data between databases. Currently we are working through attempting to connect both out Production Nifi service and our Development Nifi service to our Nifi registry service.The production Nifi service and the Nifi registry are running on one EC2 server and the development Nifi service is running on another. We have them all set up to be secure using LDAP and users/groups are managed in Windows server AD. We are having some issues trying to get the Nifi services connected to the Registry service and were hoping we could get some feedback or guidance on what we may be missing or doing incorrectly.The Nifi production instance is available from the host https://nifi.example.com/nifi and the Nifi registry is available at https://nifi-registry.example.com/nifi-registry. We have followed the documentation and set the registry in Nifi to the correct endpoint, but are getting the following error:"Unable to obtain listing of buckets: javax.net.ss

When I create a new Windows Forms App project in c# a file called Resources.resx is added automatically to the project properties.I added an image to this file.But how do I access this resource ? It doesn't seem to be included in the publish solution.What do I need to do to have the resources in my application and how can I access the image in this file ?

Does anyone have experience using SQL value in statement.type for PutDatabaseRecord processor in Nifi?It is undocumented as to how to format/parameterize this query correctly.  Use of ? or record path values both fail e.g.INSERT INTO radius.sim_usage(id,sim_id,"ICCID",sim_month,sim_usage,"updatedAt",supplier_id)VALUES(/id,/sim_id,/iccid,/sim_month,/sim_usage,/updatedAt,/supplier_id)ON CONFLICT(id)DO UPDATE SETsim_usage =CASE WHEN sim_month <> '${nowDate}'THEN sim_usage = sim_usage - /sim_usageELSE sim_usage = /sim_usageEND INSERT INTO radius.sim_usage(id,sim_id,"ICCID",sim_month,sim_usage,"updatedAt",supplier_id)VALUES(?,?,?,?,?,?,?)ON CONFLICT(id)DO UPDATE SETsim_usage =CASE WHEN sim_month <> '${nowDate}'THEN sim_usage = sim_usage - ?ELSE sim_usage = ?END

hello.I am a beginner in NIFI.Currently, there are two CENTOS7-based Linux servers with NIFI version 1.27.0 installed.Clustering two serversIt is linked with external zookeepers.The nifi setup was completed with the ssl authentication setup using the jks file.In this state, I changed the "nifi.security.user.authorizer" option to "managed-authorizer".At this time, I would like to create and use multiple users in the users.xml file without LDAP, Kerberos, etc(only users.xml file and authorizations.xml file).Example user.xml file and authorizations.xmlI'm curious about the file example.Also if possibleI would like to see again how to configure these settings. After searching, I found that there is no way to dynamically add users without restarting the service.So ultimately what I want to do isWithout LDAP or Kerberos,After creating multiple users in users.xml and restarting the service,I want to log in with those accounts.

Hello, everyone.I have an issue with NiFi Registry 1.20 -- UI button "Add user" is inactive with Initial Admins account.But, if I check the user permissions, it looks good enough:The Initial Admin user is LDAP user and also member of "nifi admins" LDAP group.My authorizers.xml file:<?xml version="1.0" encoding="UTF-8" standalone="yes"?><authorizers><userGroupProvider> <identifier>ldap-user-group-provider-1</identifier> <class>org.apache.nifi.registry.security.ldap.tenants.LdapUserGroupProvider</class> <property name="Authentication Strategy">LDAPS</property> <property name="Manager DN">user</property> <property name="Manager Password">password</property> <property name="TLS - Keystore">./conf/keystore.jks</property> <property name="TLS - Keystore Password">pass</property> <property name="TLS - Keystore Type">JKS</property

Due to data masking, I can't read tables directly using 'vanilla' Spark. The workaround is connecting Spark to Impala via JDBC and the problem is: when I use reserved words or some operations like `+ INTERVAL 1 DAY` Impala returns the column names as values in the DataFrame.That's how I start the Spark session:spark = ( SparkSession .builder .config("spark.jars", "/home/cdsw/ImpalaJDBC42.jar") .getOrCreate())and how I query data:( spark .read .format("jdbc") .option("driver", "com.cloudera.impala.jdbc.Driver") .option("url", "jdbc:impala://MY_IMPALA_HOST:443/default;AuthMech=3;transportMode=http;httpPath=cliservice;ssl=1") .option("PWD", "MY_PASSWORD") .option("UID", "MY_USERNAME") .option("query", "SELECT 'a' AS index FROM MY_TABLE") .load() .show())That's what I get:+-----+|index|+-----+|index||index||index||index||index||index||index||index||index||index||index||index||index||index||index||index|+-----+Other errors are derived from this one.

I want to filter rows of a CSV file based on a regular expression, but so far I didn't get any row as a match.CSV:F;0;00003;;;SA;30012025;012025;;0456;3790;4940;0;0;150.00;"- Sachbezug (sonstiger)";;;EURF;0;00003;;;SA;30012025;012025;;0461;3790;4949;0;0;50.00;"- Abzug Gutschein stfr.";;;EURG;0;00003;;;SA;30012025;012025;;0486;3790;3725;0;0;970.00;"- sonstiger Abzug";;;EURG;0;00003;;;SA;30012025;012025;;0487;3790;6301;0;0;50.00;"- Abzug Arbeitskleidung";;;EURF;0;00003;;;SA;30012025;012025;;1160;3790;6010;0;0;-25.91;"Ausgl. Zuschl.b.Krankheit";;;EURF;0;00003;;;SA;30012025;012025;;1161;3790;6010;0;0;-63.36;"Ausgl. Zuschl. Feiertag";;;EURNifi-Workflow:Because I don't have any permission to upload images I need to describe what the workflow looks like - sorry.I have a GetFile processor to read the CSV file. This processor is connected directly with the 'RouteText' processor. This processor is connected via the 'unmatches' output to a disabled 'Wait' processor. An output called 'MatchedG' ha

Hi All, I am new to Apache NIFI ,I did the setup on AWS ECS using latest NIFI image and with LDAP authentication. Everything is fine, but when Admin or any other user logs in, then all of the control sections are hidden (processor, input port, output port, funnel).Not sure where I am doing wrong. Pleaes assist me

Hello, everyone!We have a three-node Apache NiFi cluster with LDAP auth and now, users must enter node name to access NiFi UI, e.g.:https://nifi-node-01.company.com:9443/nifiI'd like to do it more friendly for them and create some LB with HAProxy/Nginx etc. You know, to access NiFi with address like that:https://nifi-service.company.comI've read a lot of articles about LB, but all of them talk about LB for data flow, not for UI. So, what do you think - it is OK? Are where any potential pitfalls?

Can NiFi be used to send AS2 messages for EDI file ?If Yes, please provide some information.Thanks

Hi Team,Could you please confirm whether the CountText processor is a standard NiFi processor? I couldn't find it listed under the "Supported NiFi Processors" section in the Cloudera Flow Management 2.1.7 release notes:https://docs.cloudera.com/cfm/2.1.7/release-notes/topics/cfm-supported-processors.htmlYour insights would be appreciated.

Hi We've an existing HDP cluster with version 2.6.5.0-292 with Ranger service setup. Will this Ranger work with MySql 8.4 (without upgrading HDP itself)? From the documentation, it looks like it doesn't but wanted to confirm.Thanks

Hi PutFile adds extra slash at the end, which create a unwanted dir with the filename.Tried using EL to remove the extra slash at the end, but its not working├── abc101c.csv│   └── abc101c.csv├── abc.txt│   └── abc.txt├── abcxyz.txt│   └── abcxyz.txt├── demo│   └── xyz.txt├── fex│   └── test999.txt├── folder│   ├── folder1.txt│   └── folder.txt Thanks

Legal question about using Cloudera ODBC Driver (Connector) for Apache Hive: would it be possible to include its package into my software package (i.e. to bundle them) and install them altogether?

Hello,I am attempting to unzip a series of simple .zip files in my NiFi flow. All of my flow executes with no issues except when I try to unpack. There is nothing special about these zip files (i.e. no passwords, etc...). But no matter which way I try to unpack the zip files, I get the error that the zip file does not contain any entries. I have verified that the zip files do contain entries. I can easily unzip with WinZip. I can also easily unzip with our old custom C# application that we are replacing with NiFi.I have tried adjusting the UnpackContent settings and even have added an UpdateAttribute to add the application/zip mime type as well. Still keep getting the failure. I have also tried the CompressContent processor and still get these errors on simple WinZip files.I am attaching my flow, my processor settings and error message. 

Hello.We have ORC tables loaded from Hive (Tez) and queried from Impala. Due to their number of partitions, we had to increase the TEZ AM task's heap, and the note https://my.cloudera.com/knowledge/Tez-AMtask-fails-with-OOME-because-of-huge-impala-stats-chunk?id=348603 seems to match our issueWe'd like to reduce the AM memory footprint, but the solution doesn't work in our 7.1.9 and 7.3.1 environments:* After setting the option in the hive-site.xml, the "impala_intermediate_stats_chunk*" messages are still present in the Hive's plans.* Setting the option at the session level returns the error "Cannot modify hive.plan.mapwork.serialization.skip.properties at runtime. It is not in the list of params that are allowed to be modified at runtime (state=42000,code=1)".* https://github.com/apache/hive/pull/2620 shows that the option should be allowed to be set at the session level, as the HIVE-25501 was implemented in the CDP-7.1.7.After some research we came across the Jira https://

Hi all,I'm working with Cloudera Hive for the first time, seen that my company chose to use this database.I received the JDBC connectionString and I created a little method in java.Here the scriptpublic String[] getEmployee(){ String[] idcod ={"ABC123", "FLS163","XYZ001","PLE456", "ERV021", and so on......}; String[] names= new String[idcod.length]; int ncount=0; long start1, elapsedTimequery; try{ Statement s = conn.createStatement(); System.out.println("Fetch size: "+s.getFetchSize()); ResultSet r=null; for(int i=0; i<idcod.length; i++) { String query=""; query+=" SELECT code, surname, name FROM mytable where code='"+idcod[i]+"'"; ncount=0; start1 = System.nanoTime(); r = s.executeQuery(query); elapsedTimequery = System.nanoTime() - start1; System.out.println((double)(elapsedTimequery/1000/1000)/1000+" seconds"); while(r.next()){ names[i]=r.getString("surname") + " " +r.getString("name"); ncount++; } System.out.println(ncount+"--"+idcod[i])

Hey all,We recently upgrade to nifi 2.4.0 from 1.x and have noticed something weird about the parameter provider in 2.x. From time to time, for some reason, the parameter provider will complain about invalid revision given when fetching for parameters with nifi-cli. Specifically we are using AzureKeyvaultParameterProvider. And the only way to get rid of it is the re-creation of the parameter provider.See the screenshot And it looks like the revision manager was messed up that every time the new revision is always one version below, we are running nifi in a standalone mode and consecutively triggering of the parameter fetch will constantly get the invalid revision(each time the revision is up by 1) until the parameter provider is recreated, even restarting nifi won't help. I'm also quite sure there is only one person sending request to fetch for parameter and attempt to change the parameter provider at the same time so no concurrent operation.  And once this problem happe

Hi all,I have the requirement that to do the unit testing for each nifi processors while send hl7 message via a python script. The following are the needs:1. After sent the HL7 test message it should receive the ACK/NACK data=  Done2. After processed the validation check this flow file will move into another upstream processors = [ Here I need the status of the flow file on each upstream processors whether it successful / fails.For that I am trying to extract or fetch the provenance data through nifi rest api, I am using NiFi version 2.0.0 with the python script I sent the payload aspayload = {"provenance": {"request": {"componentId": "f6ca227e-0196-1000-13c2-d66cf56e69f2","sort": "eventTime","maxResults": 1}}}with url  https://localhsot:port/nifi-api/provenanceI am getting this output ::clipboard::clipboard: Showing 1 recent provenance events::three_o_clock: 05/14/2025 15:48:24.041 IST:wrench: Component: ExecuteGroovyScript (ExecuteGroovyScript):page_facing_up: Type: CLONE🧾

Please help me,I got an issue with the posted job by Livy with the error " table not found", even though the table exists.this script for run job Livycurl -X POST --data '{ "file": "hdfs:///scripts/get_count.py", "conf": { "spark.executor.memory": "17g", "spark.executor.cores": "5", "spark.executor.instances": "23", "spark.driver.memory": "17g", "spark.driver.cores": "5", "spark.default.parallelism": "110", "spark.driver.maxResultSize": "17g", "spark.driver.memoryOverhead": "1740m", "spark.executor.memoryOverhead": "1740m", "spark.dynamicAllocation.enabled": "false", "spark.sql.adaptive.enabled": "true", "spark.sql.extensions": "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions", "spark.sql.catalog.spark_catalog": "org.apache.iceberg.spark.SparkCatalog", "spark.sql.catalog.spark_catalog.type": "hive", "spark.sql.catalog.spark_catalog.uri": "thrift://master1.com:9083,thrift://master2.com:9083", "spark.sql.catalog.spa

Spark job is writing  event log to  hdfs://namenode:8021/spark-history , but our job creates many events within  10  to 12 minutes.As per the document spark.eventLog.buffer.kb  =100kbDoes it mean when event writes to /spark-history/application_xxxxxx_xx/xx  ,  whenever buffer gets 100kb?  It means every time it is going to call fsync? spark.eventLog.buffer.kb100kBuffer size to use when writing to output streams, in KiB unless otherwise specified.