Ask the Community

I want to validate my JSON objects against an schema like this:{ "type": "object", "required": [ "apple", "banana", "orange" ], "properties": { "apple": { "type": "object", "required": [ "header", "enableAlert" ], "properties": { "header": { "type": "string" }, "enableAlert": { "type": [ "boolean", "string" ], "enum": [ true, false, "true", "false" ] } }, "additionalProperties": true }, "orange": { "type": "object", "required": [ "test1", "test2" ], "properties": {

 nifi is hosted in AKS cluster  of node count 3. occasionally we are receiving below errors and node-0  always gets disconnectedCould you please give some inputs or help possiblelogs:2025-11-01 01:01:12,444 INFO [Reconnect to Cluster] o.a.n.f.s.StandardVersionedComponentSynchronizer Updated UpdateAttribute[id=9b95428d-2972-38fa-b7cd-20d759b4e749]2025-11-01 01:01:12,445 INFO [Reconnect to Cluster] o.a.n.f.s.StandardVersionedComponentSynchronizer Updated ExecuteSQLRecord[id=07e8f117-fcdf-39a9-a989-3deb3b467263]2025-11-01 01:01:12,445 INFO [Reconnect to Cluster] o.a.n.f.s.StandardVersionedComponentSynchronizer Updated UpdateAttribute[id=2be213fa-1965-3303-9e4f-ebbb8e3dec37]2025-11-01 01:01:12,445 INFO [Reconnect to Cluster] o.a.n.f.s.StandardVersionedComponentSynchronizer Updated StandardProcessGroup[identifier=cd3be53f-130d-3a1c-8e5a-ed86de708fd8,name=fullload-process-configuration]2025-11-01 01:01:12,445 INFO [Reconnect to Cluster] o.a.n.f.s.StandardVersionedComponentSync

I try get json from API with help InvokeHTTP.In area Request OAuth2 Access Token Providers i used StandardOauth2AccessTokenProvider08:52:01 UTCERRORStandardOauth2AccessTokenProvider[id=2e9e7bd4-019a-1000-f83f-1ef5f5d2d56d] OAuth2 access token request failed [HTTP 400], response:{"error":"unsupported_grant_type","error_description":"The authorization grant type is not supported by the authorization server.","hint":"Check that all required parameters have been provided","message":"The authorization grant type is not supported by the authorization server."}

Hello!I have a NiFi 1.28 instance running on a Oracle Linux Server 9.6. I'm trying to connect to a Microsoft SQL Server database using a DBCPConnectionPool 1.28.1 controller, with a mssql-jdbc-13.2.1.jre11.jar driver, but I'm getting a very confusing error:Failed to establish Database Connection: java.sql.SQLException: Cannot create PoolableConnectionFactory ("encrypt" property is set to "false" and "trustServerCertificate" property is set to "true" but the driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption: Error: Certificates do not conform to algorithm constraints. Here is what I've already tried to fix this error:- Import the self signed certificate (from the database machine) into the $JAVA_HOME truststore- change the default cypto policy with: update-crypto-policies --set DEFAULT:SHA1- Relax the java security restrictions by going into the java.security file and commenting the jdk.certpath.disa

Hi everyone, I have a issue on Nifi 1.15.3 InvokeScriptedProcessor (Groovy), I set a custom property in sensitive in my processor, put for value a context parameter which is sensitive too, but when i'm trying to context.getProperty, it return the expression #{blahblah} instead of the value.This problem is "new", the flow was working, and one day, the token from sensitive value was not evaluated anymore... My property declaration :def externalRequestTokenDescriptor = new PropertyDescriptor.Builder().name("External request token")        .description("External request token")        .required(true)        .sensitive(true)        .addValidator(Validator.VALID)        .expressionLanguageSupported(true)        .build();Context param setup : Sensitive property setup :  How i get the property :context.getProperty("External request token").getValue();

I have a cdp 7.3.1 instance, and I want to hookup okta authentication to the impala service, so that when tableau connects to the impala, it will trigger authentication / call back with a okta server(I already have admin controls of it, and have scope/application etc).on this instance, I already enabled kerberos, knox and not sure if I need to enable ssl as well, I can't find any documentation for it, can some one walk me through the setup? I mostly rely on chapgpt, and had some issues.  

我的服务器由于硬盘故障导致服务器强制重启，重启之后再开启 DataNode 节点就报着错误，打开datanode日志，错误日志如下： 2025-10-27 09:57:18,577 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: supergroup = supergroup2025-10-27 09:57:18,610 ERROR org.apache.hadoop.jmx.JMXJsonServlet: getting attribute DatanodeNetworkCounts of Hadoop:service=DataNode,name=DataNodeInfo threw an exceptionjavax.management.RuntimeMBeanException: java.lang.NullPointerExceptionat com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:839)at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:852)at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:651)at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)at org.apache.hadoop.jmx.JMXJsonServlet.writeAttribute(JMXJsonServlet.java:346)at org.apache.hadoop.jmx.JMXJsonServlet.listBeans(JMXJsonServlet.java:324)at org.apache.hadoop.jm

Hello everyone,I am working with Apache NiFi and I encountered the following issue:The file I need to get is CaryIL-ASTUDENTDI-2025-03-18-083011.csv. The number of files in the list will change daily, and my goal is to get the file with the latest timestamp in the format yyyy-MM-dd-HHmmss.Specifically, I have 5 queues containing FlowFiles, and I want to merge all the FlowFiles into a single queue. After collecting the FlowFiles from these queues, I want to select the file with the most recent date based on the file name (which follows the yyyy-MM-dd-HHmmss.csv format), and move the FlowFile with the latest date into one queue for further processing. Please refer to the attached file.I also tried passing them through an ExecuteGroovyScript, but it seems that it processes each queue sequentially, not merging the 5 queues into one and then finding the Latest Date time. Below is the ExecuteGroovyScript code snippet.  def flowFiles = session.get()if (flowFiles != null && f

HelloI need to ask if its okey to upgrade our postgresssql DB (from 13.22 to 16 version) that we are using on our Cloudera cluster, we are using 7.11.3  cloudera manager with 7.1.9 SP1 runtime and RHEL 8.10. From support matrix it seems like we can but https://docs.cloudera.com/cdp-private-cloud-base/7.1.9/administering-hue/topics/hue-download-install-postgres.htmlRHEL and CentOS 7 include PostgreSQL version 9.2. However, on these OS versions, Hue is compatible only with PostgreSQL versions 10, 11.x, 12, 13 and 14. Therefore, you must download and install the PostgreSQL distribution separately.Here is only maximum of 14 version

In Ranger, if a user is both an admin and an auditor then the auditor role is chosen.  I would prefer that the highest privilege is chosen so the user should be an admin.Is there a way to make Ranger pick Admin over Auditor?

Hi all, I have been seeing this problem increasingly now that the data gets stuck in between two different processors -- the data being as small as few KBswhat could be the cause of it? The ExecuteSQL contains an SQL insert script. Moreover, this behavior is random -- occurs randomly on any processor. Does my machine specs count in this? I see my CPU usage and memory are not choking as well.Any comments on this will be appreciated. Thanks

I have setup an ExecuteSQLRecord processor to get a list of files that were processed daily at 5pm and send this list via email. The process flow works as expected, except for the CRON schedule. My CRON expression is as follows:0 0 17 * * ?I believe this is correct to execute daily at 17:00 (5pm).However, the process flow executes daily at 16:07 (4:07pm) and 17:00 (5pm).I am scratching my head trying to figure out why the odd execution at 4:07pm.Does anyone have any insights to where I can check to see why this is happening?

Since the later versions of NiFi no longer provide the detailed reason an XML FlowFile fails validation against a schema (just a generic Validation Failed), I have been trying to create my own using the ExecuteGroovyScript processor. I am new to NiFI and Groovy, but I am an experienced .Net developer in C#.Below is my Groovy script in the procesor:import groovy.xml.XmlUtilimport javax.xml.transform.stream.StreamSourceimport javax.xml.validation.SchemaFactoryimport javax.xml.XMLConstantsdef flowFile = session.get()if (!flowFile) returndef xmlContent = flowFile.read().getText("UTF-8")def schemaFile = new File("My_XML_Schema.xsd")def schemaFactory = SchemaFactory.newInstance(XMLConstants.W3C_XML_SCHEMA_NS_URI)def schema = schemaFactory.newSchema(schemaFile)def validator = schema.newValidator()def validationErrors = new StringBuilder()validator.setErrorHandler(new org.xml.sax.helpers.DefaultHandler() {@Overridevoid warning(org.xml.sax.SAXParseException e) throws org.xml.sax.SAXException {v

Hello.I have configured the HSM.However, testing per Validating Key HSM Settings fails.Settings----------# sudo service keyhsm settingskeyHsm Server Configuration information:keyhsm.management.address : 127.0.0.1keyhsm.server.port : 9090keyhsm.management.port : 9899keyhsm.service.port : 19791keyhsm.jvm.heap.mx.gb : 2keyhsm.hardware : ncipherModule OCS Passwordthales.ocs_password : V6DvCGbrimcD7UPA6QnoOhJb37mFOuoScY30LfWjDwvCybL4E99eT5SKUOcZdu6pq5y66iROKZboNagXzCRxl4x7+N3C3ypKzUJV5UwV3hBjaNS2/qpbyUQD+UUgCoOkm6CxuiOFbOu9CmhnlHBC2UwxqjtnMrtzCR7XMI/Vegm6iZGwR9YWFSeTRRjPkQ/Rhce81hTIqmk7U0+LGHEK+niuARmVt6EG7nmDvZMQufqhOoG2yd4FlYKv2Lv9dDKEKTByv/xoT+/Qh/+Y+8ZbuZHDbEPPzJrq6K848jXhV2wBGTt4RJeKayBzUjwix2LREonTcOctgDf/oJhuIbS2dA==Status--------[root@cloudera-manager ~]# sudo service keyhsm statusKey HSM is running as of Thursday October 23rd, 2025 (10:32 AM), (Started : Thursday October 23rd, 2025 (10:29 AM)) Validation failed --------------------What is port 11371?# curl -k https://loca

Hello.I am on CDP CDH-7.3.1-1.cdh7.3.1.p400.67986116, using Spark 3.5.4.7.3.1.400-100 and no longer able to to find HWC classes under com.hortonworks.spark.sql.hive.llap.*. Is HWC deprecated and/or unsupported on CDP 7.3.1+?Thank you.David.

Unable to start Nifiregistry after Enable Ranger Dependency

I am using NIFI to read data from FTP and push it to HDFS. My business process is to schedule workflows for subsequent computing tasks through dophinschedule. One detail here is that my workflow will execute Alter for corresponding analysis before running.I have noticed that the Spark task occasionally encounters a FileNotFound error during runtime, which may cause the task to fail. It is speculated that this is because partition information has already been added. At this time, when the task is running, NIFI is still writing data to the corresponding partition, and the file being written will cause this error. How to optimize this problem? 

Hi All, I am trying to fetch data from the simple wttr api. The below works fine from curl or postman at the same time: curl http://www.wttr.in/paris?format=j1 But accessing same from invokeHTTP gives me connection refused or connection reset. I have tried by putting higher timeouts and adding user agents, but no resolutionDo I need to open the internet connectivity somewhere or something?.  invokeHTTP Image 

Hi , This is my current NIFI flow where i am consuming data in NIFI using ConsumeKafka_1_0 and publishing using PublishKafka_1_0 . Now HiveMQ (MQTT) → Kafka Topic A → NiFi consumes → NiFi publishes → Kafka Topic BMy requirement is send data in real time , i want to do real time data streams like kafka streams .How to achieve that using NIFI . Is kafka streams available in NIFI ?iam not doing and data transformation or any other operation to data. Iam just simply consuming and publishing data , but now i want to do it in real time .

Hi! I have flows on NIFi and in many places, I have to save data to the database. I mainly use PutSql or PutdatabaseRacord processors. So far, everything was working fine, but today we wanted to test our data and issued 15 messages at once. Unfortunately, in several places, during data saving or updating, an error appeared - error.code 1205 error.message Transaction (Process ID 76) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction. error.sql.state 40001 - after which some data was saved or updated correctly, others unfortunately were not. How can I resolve this situation, and how can I protect myself against something like this? 

Hi!I am getting the following error while restarting the Ranger KMS service after configuring the HSM.Please advise.Dalier.Thu Oct 16 15:36:41 EDT 2025JAVA_HOME=/usr/lib/jvm/java-openjdkUsing -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh as CSD_JAVA_OPTSUsing /var/run/cloudera-scm-agent/process/1546345804-ranger_kms-RANGER_KMS_SERVER as conf dirUsing scripts/control.sh as process scriptCONF_DIR=/var/run/cloudera-scm-agent/process/1546345804-ranger_kms-RANGER_KMS_SERVERCMF_CONF_DIR=Thu Oct 16 15:36:42 EDT 2025: [I] Using : /opt/cloudera/cm/lib/postgresql-42.5.1.jar and org.postgresql.Driver for postgresql database typeERROR: Hadoop KMS could not be startedREASON: java.lang.NullPointerException: Cannot invoke "org.apache.hadoop.crypto.key.kms.server.KeyAuthorizationKeyProvider$KeyACLs.startReloader()" because "this.implKeyACLs" is nullStacktrace:---------------------------------------------------java.lang.NullPointerException: Cannot invoke "org.apache.hadoop

Hi guys,i've just started to work with parquet files, all is running with database own export logics, but they are not traceable - use NiFi was the Idea. Now im just annoyed how bad i am to handle this ... seems no default export available for this, install extensible-bundles ... NoClass here and there etc...Did anybody of you solved to add Parquet to NiFi 2.5.0?I've downloaded and provided nifi-parquet-nar-2.5.0.nar, nifi-hadoop-nar-2.5.0.nar and nifi-hadoop-libraries-nar-2.5.0.nar still NoClassDefFoundErrors in this order of log (single named)- org/apache/nifi/serialization/RecordSetWriterFactory- org/apache/nifi/processors/hadoop/AbstractFetchHDFSRecord- org/apache/nifi/processors/hadoop/AbstractPutHDFSRecord- org/apache/nifi/serialization/RecordReaderFactory- org/apache/nifi/serialization/RecordSetWriterFactory- org/apache/parquet/io/OutputFile- org/apache/parquet/io/InputFileAnybody who can helpt me?

Hello I need help how to make a flow which can handle 2 pieces of data from user. Let us consider a scenario-> User submits a form having some fields like name, age, address and also a file having something say a list of his hobbies(just for example). How do we process these together and return a response back?I have my flow like this HandleHttpRequest->ExecuteScript (Groovy)-> HandleHttpResponseI am able to do one logic only that either handle the json body say {"name":"Test Name","age":"25","address":"Test address"} or the file which gets sent.How to process both inside my Groovy Script? 

Error: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for user/myname/.cm/distcp-staging/2025-09-21-05-14-47-c4f9/intermediate.1 at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:447) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:152) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:133) at org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.merge(SequenceFile.java:3566) at org.apache.hadoop.io.SequenceFile$Sorter.merge(SequenceFile.java:3360) at org.apache.hadoop.io.SequenceFile$Sorter.mergePass(SequenceFile.java:3336) at org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:2899) at org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:2938) at com.cloudera.enterprise.distcp.util.DistCpUtils.sortListing(DistCpUtils.java:427) at com.cloudera.enterprise.distcp.mapred.StatusReducer.lambda

Hi everyone, I would like some clarification. Thank you.I have a question about accessing the CDP Private Cloud Data Services's web UI and understanding which pods or services are related to the web UI.The URL for the CDP Private Cloud Data Services is:https://console-[namespace].apps.am.ocp.poc/I would like to know how we can determine which pod this web page is running on and under which service?Additionally, if we adopt this method to install the Cloud Data Service:https://docs.cloudera.com/cdp-private-cloud-data-services/1.5.2/installation/topics/cdppvc-installation-steps.htmlIf our OCP is installed in a private network, do you have any suggestions on how to allow the web page to be accessed externally? (e.g., load balancer, ingress, or external gateway)