Solved

Hi,We have deployed Cloudera Flow Management through the Kubernetes Operator on a k8s cluster. What's the best practice how to add custom processor NARs in this environment?On a plain (not containerized) NiFi deployment I would put the NAR into the extensions sub folder of the NiFi deployment and it gets unpacked during startup of NiFi.How does it work with the Cloudera Kubernetes Operator version of NiFi? We have a NFS share available that could be mapped to a container's sub folder.Has anyone done this successfully?Thanks,-karsten

I can able to connect the impala using isql when I use to connect the same using pyodbc I am getting this error Error: ('H000', '[H000] [ (50404) (SQLDriverConnect)')

I’m currently migrating from Apache NiFi 1.21 to NiFi 2.6.0. In my existing flow, I was using the InvokeGRPC processor, but I noticed that this processor has been removed in NiFi 2.x versions.I have a few questions:What is the exact reason for deprecating and removing InvokeGRPC in NiFi 2.x?Was it due to architectural changes, performance issues, or security concerns?If we take a fork of the NiFi 1.21 gRPC NAR and use it in NiFi 2.6.0, will that work?Are there any known performance risks with this approach?Could this introduce security vulnerabilities in the current or future releases?Is there an alternative recommended approach for gRPC integration in NiFi 2.x?For example, should we use ExecuteScript, custom processors, or any official replacement?Any insights or official guidance would be greatly appreciated. Thanks in advance!

Hello, All!I am sorry if the questions seems stupid, but I am a newbie. Is it possible to install cloudera on premises free trial version for on a single laptop for learning purposes?Or i need a cluster of laptops to be able to run cloudera and hadoop services? 

Hi, I am looking how to handle cases when json format may vary between different client responses. On response{"data": [{"ID": "123","Name": "Queen"},{"ID": "456","Name": "King"}]}we apply following jolt transform:[{"operation": "shift","spec": {"data": {"*": {"ID": {"456": {"@(2,Name)": "Name"}}}}}}]but some times response is coming without leading "{"data":and our transformation fails. So, generally, how can we query json with JOLT if we know for instance that Name and ID are next to each other somewhere in the structure?  

Hello dear community,I haven’t been able to find a solution for the following setup:I have Apache NiFi 2.5.0 and NiFi Registry 2.5.0 installed on the same EC2 instance (Linux 2023). Both NiFi and NiFi Registry are configured with SSL and served through an Application Load Balancer (ALB).External access is only available via the ALB: https://alb...com.Internally, the services can be reached at:NiFi: https://nifi_ec2server.com:8443/nifiNiFi Registry: https://nifi_ec2server.com:18443/nifi-registryAuthentication is handled via Active Directory (LDAP).Accessing NiFi through https://alb.../nifi works fine — I can log in as an LDAP user. The same applies to NiFi Registry under https://alb.../nifi-registry.The problem:As an LDAP user, I cannot access the NiFi Registry from within NiFi, and no buckets are displayed.The error message is attached below.[NiFi Registry Web Server-50] o.a.n.r.w.s.a.IdentityFilter Attempting to extract user credentials using X509IdentityProvider[NiFi Registry Web Ser

Hello Everyone,                          As you know that , it is recommended to use Parameter rather than variables in Nifi as it is gonna get deprecated soon.                         I have two process groups PG1 and PG1A. PG1A is the child process group to PG1.                        I have ParameterContext PC1 which has database and other common application configuration parameters and it is assigned to PG1.                        I would like to create new ParameterContext PC1A which has its own parameters specific to the process group PG1A and it should also inherit the common application parameters from PC1. How it can be achieved? Since we can't assign two different Parameter contexts(PC1 & PC1A) to one process group (PG1A)

Hi everyone, I have a issue on Nifi 1.15.3 InvokeScriptedProcessor (Groovy), I set a custom property in sensitive in my processor, put for value a context parameter which is sensitive too, but when i'm trying to context.getProperty, it return the expression #{blahblah} instead of the value.This problem is "new", the flow was working, and one day, the token from sensitive value was not evaluated anymore... My property declaration :def externalRequestTokenDescriptor = new PropertyDescriptor.Builder().name("External request token")        .description("External request token")        .required(true)        .sensitive(true)        .addValidator(Validator.VALID)        .expressionLanguageSupported(true)        .build();Context param setup : Sensitive property setup :  How i get the property :context.getProperty("External request token").getValue();

HelloI need to ask if its okey to upgrade our postgresssql DB (from 13.22 to 16 version) that we are using on our Cloudera cluster, we are using 7.11.3  cloudera manager with 7.1.9 SP1 runtime and RHEL 8.10. From support matrix it seems like we can but https://docs.cloudera.com/cdp-private-cloud-base/7.1.9/administering-hue/topics/hue-download-install-postgres.htmlRHEL and CentOS 7 include PostgreSQL version 9.2. However, on these OS versions, Hue is compatible only with PostgreSQL versions 10, 11.x, 12, 13 and 14. Therefore, you must download and install the PostgreSQL distribution separately.Here is only maximum of 14 version

In Ranger, if a user is both an admin and an auditor then the auditor role is chosen.  I would prefer that the highest privilege is chosen so the user should be an admin.Is there a way to make Ranger pick Admin over Auditor?

I have setup an ExecuteSQLRecord processor to get a list of files that were processed daily at 5pm and send this list via email. The process flow works as expected, except for the CRON schedule. My CRON expression is as follows:0 0 17 * * ?I believe this is correct to execute daily at 17:00 (5pm).However, the process flow executes daily at 16:07 (4:07pm) and 17:00 (5pm).I am scratching my head trying to figure out why the odd execution at 4:07pm.Does anyone have any insights to where I can check to see why this is happening?

Since the later versions of NiFi no longer provide the detailed reason an XML FlowFile fails validation against a schema (just a generic Validation Failed), I have been trying to create my own using the ExecuteGroovyScript processor. I am new to NiFI and Groovy, but I am an experienced .Net developer in C#.Below is my Groovy script in the procesor:import groovy.xml.XmlUtilimport javax.xml.transform.stream.StreamSourceimport javax.xml.validation.SchemaFactoryimport javax.xml.XMLConstantsdef flowFile = session.get()if (!flowFile) returndef xmlContent = flowFile.read().getText("UTF-8")def schemaFile = new File("My_XML_Schema.xsd")def schemaFactory = SchemaFactory.newInstance(XMLConstants.W3C_XML_SCHEMA_NS_URI)def schema = schemaFactory.newSchema(schemaFile)def validator = schema.newValidator()def validationErrors = new StringBuilder()validator.setErrorHandler(new org.xml.sax.helpers.DefaultHandler() {@Overridevoid warning(org.xml.sax.SAXParseException e) throws org.xml.sax.SAXException {v

Hello.I am on CDP CDH-7.3.1-1.cdh7.3.1.p400.67986116, using Spark 3.5.4.7.3.1.400-100 and no longer able to to find HWC classes under com.hortonworks.spark.sql.hive.llap.*. Is HWC deprecated and/or unsupported on CDP 7.3.1+?Thank you.David.

Hi , This is my current NIFI flow where i am consuming data in NIFI using ConsumeKafka_1_0 and publishing using PublishKafka_1_0 . Now HiveMQ (MQTT) → Kafka Topic A → NiFi consumes → NiFi publishes → Kafka Topic BMy requirement is send data in real time , i want to do real time data streams like kafka streams .How to achieve that using NIFI . Is kafka streams available in NIFI ?iam not doing and data transformation or any other operation to data. Iam just simply consuming and publishing data , but now i want to do it in real time .

Hello,We have problems with parcels in our PROD cluster (CDP Runtime 7.1.7), we tried to remove old parcels (CDH 6.3.3, CFM 1.0.1.0, CFM 1.1.0.0) on host dallas20- After the manual delete, Cloudera Manager still shows some parcels with status Distributed.- The parcel folders on disk are not correct now, probably because of the manual cleanup.- Also, the cluster is slow and some services presents an error like this in Parcels tab on CM:Error when distributing to dallas20.metromadrid.net: [Errno 20] Not a directory: /opt/cloudera/parcels/STREAMS_MESSAGING_MANAGER-2.1.0.2.0.1.0-2/meta/parcel.jsonWe need help with these points:How can we clean the old parcels in a safe way so CM shows the correct status?How to fix the error [Errno 20 Not a directory] and make parcel distribution work again?Is there a supported procedure to reset parcel states in Cloudera Manager?This problem is blocking us to free space in /opt/cloudera/parcels/ and manage parcels normally.We are using CDP 7.1.7 (whic

Hi AllUnder CDP 7.3.1, our Impala's log is flooding with messages likeORC read request to already read range. Falling back to readRandom. offset: 16248424 length: 105200 colrange_offset: 15988393 colrange_length: 365231 colrange_pos: 16353624 typeId: 94 kind: data filename: .......from queries reading ACID ORC tables created from Hive.We've been searching about that and the only thing we've found is the code that triggers that message.void HdfsOrcScanner::ScanRangeInputStream::read(void* buf, uint64_t length, uint64_t offset) { Status status; if (scanner_->IsInFooterRange(offset, length)) { status = scanner_->ReadFooterStream(buf, length, offset); } else { ColumnRange* columnRange = scanner_->FindColumnRange(length, offset); if (columnRange == nullptr) { status = readRandom(buf, length, offset); } else if (offset < columnRange->current_position_) { VLOG_QUERY << Substitute( "ORC read request to already read range. Falling back t

I an trying to configuring NiFi regoistry (2.5.0) with LDAP, have successfully configured so I can login using AD credential. However I am not able to see users or groups from AD in the admininstration sections.I have checked the user  and group sections authorizers.xml) are the same as NiFI, as I can see users and groups when I go to Users or Policies section.I am not using LDAPS (use LDAP in the dev envirobment), and this works fine in NiFi 2.5, and just cant see why it doesnt work with NiFI registry using teh same LDAP usergroup provider settings. Any thoughts?

I manage NiFi installations for various clients within our company. On any servers or client systems running a Windows-based operating system, and on any version of NiFi I currently maintain (1.11.4, 1.16.3, 1.19.1), NiFi does not roll over logs using the default logback.xml. Example:  <appender name="APP_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender"> <file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app.log</file> <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy"> <!-- For daily rollover, use 'app_%d.log'. For hourly rollover, use 'app_%d{yyyy-MM-dd_HH}.log'. To GZIP rolled files, replace '.log' with '.log.gz'. To ZIP rolled files, replace '.log' with '.log.zip'. --> <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern>

i have a JSON payload which contains latest and old orders. The data should be compared and output New, update, deleted. How can do it do that in NIFI?.  "latest_orders": [    { "order_id": "1001", "customer": "Alice", "amount": 250 },    { "order_id": "1002", "customer": "Bob", "amount": 300 },    { "order_id": "1004", "customer": "Diana", "amount": 150 }  ],  "old_orders": [    { "order_id": "1001", "customer": "Alice", "amount": 200 },    { "order_id": "1002", "customer": "Bob", "amount": 300 },    { "order_id": "1003", "customer": "Charlie", "amount": 400 }  ]}Output:    { "order_id": "1001", "customer": "Alice", "amount": 250, "Action":"UPDATE" },    { "order_id": "1002", "customer": "Bob", "amount": 300 ,"Action":"UPDATE"},    { "order_id": "1004", "customer": "Diana", "amount": 150,"Action":"NEW" },    { "o

Hi team,I am currently facing issue connecting to Hive database using python libraries (pyodbc, pyhive, impyla), the Cloudera driver and Apache Knox. While the network level connection to the relevant host and port is successful. (verified via ncat, telnet), I am unable to complete the connection and authenticate through these tools. Connection details:Driver=/opt/cloudera/hiveodbc/lib/64/libclouderahiveodbc64.so;""Host=nonprodm01.cdslindia.com;""Port=8443;""Schema=default;""TransportMode=http;""HTTPPath=/gateway/default/hive""SSL=0;"  # Enable SSL if Knox uses HTTPS"AuthMech=3;"  # For username/password authentication; use 1 for no auth or 2 for Kerberos"UID=username;""PWD=password;"Knox gateway is used to access HiveServer2Error message: Connection failed: "('HY000', '[HY000] [Cloudera][DriverSupport] (1110) Unexpected response received from server. Please ensure the server host and port specified for the connection are correct and confirm if SSL should be enabled for the c

Hello!Following the advice from this post: https://community.cloudera.com/t5/Support-Questions/Accessing-NIFI-Metrics-endpoint-for-Prometheus-without/m-p/399710#M250535I'm trying to set up certificate authentication/authorization from prometheus server to nifi, but getting 403 Forbidden. In Nifi user log i can see that authentication CN of certificate is successful, further  comes the 403 error:Prometheus scrape config: Can you please tell what is wrong here?

We've encountered an issue within Cloudera Object Store (powered by Apache Ozone) (1.4.0 and 1.4.1) which we believe are related to HDDS-9967A fix for this particular issue has now been backported to the 'apache:ozone-1.4' project as part of pull request https://github.com/apache/ozone/pull/9066How do we update the Cloudera Runtime or Ozone parcel version (1.4.0.7.1.9.1054-4) to include this fix? Upgrading to the latest 1.4.2 would be ideal.Thanks Stephen

Hi Team, We have the following issue, we are using nifi work flow, in which we are trying to execute impala, before implementing nifi  execute script, we tested the script by using jython to test the feasibility of the connections we established is successful and able to get the count in impala, however when using the same connection configuration in nifi execute script processor, we get the java class path not found, the impala jdbc connection is placed in a customized location, impalajdbc41.jar file is not a nar, file, and the customized location is readable, as confirmed by the support and if placed in the nar file location will Please find the logs for reference as an attachment.2025-09-29 15:31:08,495 ERROR org.apache.nifi.processors.script.ExecuteScript: ExecuteScript[id=49498cfb-70fc-1ebf-ffff-fffff0fa2dc8] Processing failedorg.apache.nifi.processor.exception.ProcessException: javax.script.ScriptException: java.lang.ClassNotFoundException: java.lang.ClassNotFoundException:

Hello everyone,Is it possible to limit the memory for executing queries for an LDAP group in CDP 7.1.7? This would allow creating a memory pool for each group so that if a user in that group launches a memory-intensive query, it does not impact other users in the cluster.BR!

Hi,I read from the release note that the encrypt-config.sh tool from nifi-toolkit has been removed in NiFi 2.0 due to various considerations(https://issues.apache.org/jira/browse/NIFI-13414). I am wondering is there any alternatives to this tool? We heavily relied on this encrypt-config.sh tool to encrypt plain-text passwords in nifi.properties, like nifi.sensitive.props.key, nifi.security.keyPasswd. Now with this tool gone, do we just have to accept the fact that plain-text passwords will remain plain-text?Thanks in advance!!