Solved

I'm trying to create a new Nifi custom processor.I'm following instructions given on 4 differents tutorials.I only create the structure of the processor, I don't even try to modifiy the code and I'm always having the same mvn clean install [ERROR].Can you please help me to resolve this ?A huge thanks!!!OS : Windows 10 HomeNifi version : 2.0.0-M2Maven : Apache Maven 3.9.6JDK : Java version: 21.0.2, vendor: Oracle Corporation, runtime: C:\Program Files\Java\jdk-21Me version : Java and Maven newbi (0.0.1)Maven is installed and I have added its bin to the path env var.Jdk in installed and I have added its bin to the path env var.JAVA_HOME env var is defined as C:\Program Files\Java\jdk-21, where my jdk is installed.The error:[ERROR] Failed to execute goal on project nifi-helloworld-nar: Could not resolve dependencies for project org.apache.nifi:nifi-helloworld-nar:nar:1.0-SNAPSHOT: The following artifacts could not be resolved: org.apache.nifi:nifi-standard-services-api-nar:nar:1.0-SNAPSHO

Hi community,Our NIFI version: 1.23.0.We are converting JSON message to AVRO format and posting to KAFKA topic using  PublishKafkaRecord_2_0 1.23.0:JSON Tree Reader:AVRO Record Set Writer:Confluent Schema Registry:We are not able to post  the message with the error below.We are able to see the AVRO schema in the KAFKA registry through POSTMAN and using browser URL and we are able to connect to that KAFKA topic with the same parameters in NIFI .Could you please help us to resolve this issue.Error: 

Hello, we need access to Cloudera archive in order to be able to setup our cdh docker image to test our cdh jar for h2o-3.https://archive.cloudera.com/cdh5/ubuntu/trusty/amd64/cdhHow can we request access to the repo?Thank you.Adam Valenta (adam.valenta@h2o.ai)

Can NiFi handle multiple users hitting a request? Say I have HandleHttpRequest or a ListenHttp processor and my requirement is to handle say 100 concurrent users. Can we do it in NiFi? 

Can we setup the Iceberg table to use ranger authorization on top of spark and how to configure that?

Hi Team,I am trying to develop a flow to update the Parameter context on the fly using the InvokeHTTP Processor. Based on the screenshot i have configured the same. However i am getting unAuthorized Error.  My URL format is like : https://<nifi-host>:<port>/nifi-api/parameter-contexts/<context-id>Error Screenshot.  

Hello Cloudera Community,I'm working on a project where I need to provide users with temporary access to call specific Cloudera AI API endpoints, particularly for interacting with deployed models.My goal is to avoid granting users long-lived API V2 keys and instead generate short-lived, scoped access credentials that expire after a set time (e.g., 2 hours).After exploring the CML API v2 and the cmlapi library, I'm finding it challenging to identify a direct method for generating these temporary, scoped access keys.My questions are:Is there a supported mechanism within the CML API v2 or cmlapi to generate temporary access keys with a specified expiration time?If not, are there recommended alternative approaches for providing users with limited, time-bound access to CML API endpoints?Are there plans to introduce temporary access key generation functionality in future CML releases?Is there a best practice to follow for this kind of implementation?I'm aiming for a solution that prioritizes

That pretty much sums it up. We had a database server go down over the weekend, and I have a suspicion that things got backed up, but I can't get to the UI to try to unravel it. I have tried setting autoResumeState to false and setting everything in flow.xml.gz to "Disabled", but the UI still won't come up because of this error. Unfortunately, all of this is in an isolated environment, so sharing full logs is not impossible but very, very difficult. This vm has 23GB of RAM and Xms=4096m and Xmx=8192m. niagarafiles-community-11.6.3

Hi Everyone, I am Emmanuel Katto Dubai, United Arab Emirates (UAE) It appears you're encountering an issue when attempting to run an INSERT INTO query on a Hive table with the Tez execution engine. You mentioned that the query works fine when inserting data into integer columns but fails when inserting into string columns. Additionally, the issue seems related to the Kryo serialization in the logs, where Hive is unable to deserialize the required classes. Here's a breakdown and a few suggestions to troubleshoot the issue:   Error Analysis: Error Type: The main error seems to be related to a serialization issue KryoException: Encountered unregistered class ID: 112     This indicates that Hive, during execution, is attempting to deserialize an object that hasn't been registered with the Kryo serializer. This might be related to how the Tez engine handles serialization during the execution of the query. YARN Logs: The error trace shows a failure at the map

How do I get the Client Id to make any processor update using REST PUT call ?

Hello Experts,I have a flow where I use 'SplitRecord' processors to split json records and then in downstream I am merging them back to single file using defragment strategy.Flow is dividing into 2 route downstream so I use merge in 2 places, where merge works fine in one place and does not work in another place.Flow diagram how it is setup - So on the left side MergeContent its merging fine with defragment strategy. But on the right side (red marked) its giving below error (since another api call is in between the flow file arrival to merge record processor might be not in order.)(Example Error : "Cannot defragment flow files with fragment Id XXXXXbecause the expected number of fragment is 5 but found only 3.")I am sending 20 request to end point (HandleHttpRequest) and each request has 5 json records so each request gets split into 5 FF. so it will be come total 100 flow files.(I am sending all 20 request one after other or parallely then this is the issue)Note : If I send

I was able to setup GitLabFlowRegistry in controller settings successfully (which is new feature in Nifi 2.0)From Nifi UI, I am able to import Nifi flow directly from Gitlab repository.  After modifications, I was able to successfully commit into Gitlab repo, from Nifi UI.Do we have any toolkit commands to perform these operations?  Currently we are using toolkit commands to deploy Nifi flow changes via CICD pipeline.  Can I achieve the same with new GitLabFlowRegistry?  Appreciate any help/pointers. Thanks  

Hello, I am working on importing multiple Atlas entities using Atlas API calls. After completing execution of each curl command, the message I get is "connection is left intact". Is there a way to kill / terminate this connection to Atlas server? Thankssnm1523

I have configured an InvokeHttp Processor with the 'Retry' relationship enabled to capture HTTP status codes between 500 and 599 and initiate retries. I connected the 'Retry' relationship back to the InvokeHTTP processor to enable automatic reattempts. The number of retry attempts is set to 3. However, after all retry attempts are exhausted, the process does not complete and ends up queuing in the Retry relationship, as shown below:Could you please help how to fix it?

If my job is running in Yarn and Either Resource Manager or Node Manager goes down. Will it affect my job.

Hello eveyone  we’re experiencing with consuming Kafka from a third-party (Qradar) system outside of CDP. In their configuration, the only SASL mechanisms available for configuration are:PLAINSCRAM-SHA-256SCRAM-SHA-512However, all of these mechanisms are resulting in authentication errors. how we can connect Qradar to kafa to be able to consume the data from kafka Thanks in advance  

knox service after enabling ssl is failing to start and hive beeline string used oot@invgcscdp71901 jks]# beeline -u "jdbc:hive2://invgcscdp71901.informatica.com:8443/;ssl=true;sslTrustStore=/opt/cloudera/security/jks/truststore.jks;trustStorePassword=hadoop;transportMode=http;httpPath=gateway/cdp-proxy-api/hive"SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.9-1.cdh7.1.9.p1000.55406660/jars/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.9-1.cdh7.1.9.p1000.55406660/jars/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]WARNING: Use "yarn jar" to launch YARN applications.SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/opt/

Do we have any experience on typical disk usage rations for each of the repositories (Flow file, content, and provenance)? E.g. if Content requires 200 GB of storage, the provenance and flow file would require 20 GB (for typical flows)? Trying to use this information to decide how best to slice of a NiFi server which has 12 local drives. E.g. 8 drives allocated for Content, 2 for flow file, and 2 for provenance. Appreciate any thoughts!

We recently observed an unusual scenario in NiFi where timestamps like '20251229043001', '20251230043001', and '20251231043001' were generated for December 29, 2024, December 30, 2024, and December 31, 2024. These timestamps correspond to future dates, which seems abnormal. I suspect that this issue may have occurred due to a change in the timezone settings of the NiFi server.Could you please assist in investigating the root cause of this discrepancy?  

Hi All,I am trying to delete my nifi existing process group through nifi-api curl but unfortunately it gives me error that "The revision must be specified" but everything specified as per the documentation but still not working, below is the curl command what I used for delete the process group(flow) to my target system and also I can able to delete the same through UI but not works with rest api.curl -k -X DELETE "https://localhost:28443/nifi-api/process-groups/836e216e-0195-1000-d3b8-771b257f1fe6" \-H "Content-Type: application/json" \-H "Authorization: Bearer  Token" \--data '{"revision": {"version": 0},"disconnectedNodeAcknowledged": false}'ERROR: 400 - The revision must be specified.Also find the below is the output of the process_groups:{'permissions': {'canRead': True, 'canWrite': True}, 'processGroupFlow': {'id': '187acb23-0195-1000-1908-1543c761f376', 'uri': 'https://localhost:28443/nifi-api/flow/process-groups/187acb23-0195-1000-1908-1543c761f376', 'parameterContext': {'

Hi AllI have a list of users that i have in an array.I am trying to create a job in shell script that creates a process-group for each user and apply policy to that particular process group so that only that user and nifi's super user can access or operate in it here is how the polices config json looks like       '[{"globalAccessPolicies":[{"resource":"flow","action":"read"},{"resource":"provenance","action":"read"}],"componentLevelAccessPolicies":[{"resource":"process-groups","action":"read"},{"resource":"process-groups","action":"write"},{"resource":"operation/process-groups","action":"write"},{"resource":"provenance-data/process-groups","action":"write"},{"resource":"provenance-data/process-groups","action":"read"},{"resource":"data/process-groups","action":"read"},{"resource":"data/process-groups","action":"write"},{"resource":"policies/process-groups","action":"read"},{"resource":"policies/process-groups","action":"write"},{"resource":"data

Running Spring Boot application running in Docker where the application is having a jive jdbc connection and writing to hive db.The application is working fine in standalone. The issue happens in the docker container. The docker container starts , gets connected to the hive db but throws exceptionexecuting query with Exception IllegalStateException: nested jar URLs not supported ..MalformedURLException..on the Hive driver

HI All, we configured three node nifi cluster by name node1. node2 and node3, with external zookeeper z1,z2, & Z3, in a Azure Virtual machine,Now Require to add the Storage resources to these nifi nodes, we are planning for below storage to choose, But require your help in choose to best fitment.1. Azure Blob Storage2. Azure NFS Storage3. Azure external storage (Standard HDD, Standard SSD)our use case to store the below storage,a. Nifi Prevenance Registryb. Nifi content Registryc. Nifi  flow file RegistryRequest you to suggest us best fit in these case.Data expected to grow 3TB to 5TB.        

Hi,I am using JoltTransformJson to convert a plain json to nested json in Nifi and getting null in the output. The same i/p and spec is working in Jolt demo. Below are the input and the spec Input:{"ts" : "2024-09-22T12:03:13.000Z","sourceDeviceId" : 4321,"sourceName" : "light1","sourceType" : "light","locationLat" : 201.34,"locationLon" : 101.5,"locationId" : 1,"recordDate" : "2024-09-22T12:03:13.000Z","_class" : "com.prutech.smartcity.entity.LightEntity","_id" : "66f27c487bd8652abecdb9c7","status" : "INACTIVE","power" : 0,"current" : 0,"intensity" : 100,"voltage" : 0}Jolt Spec:{"operation": "shift","spec":{"ts":"ts","sourceDeviceId":"source.deviceId","sourceName":"source.name","sourceType":"source.type","locationLat":"location.lat","locationLon":"location.lon","locationId":"location.id",         "recordDate":"recordDate","_class":"_class","status":"status","power":"power","current":"current","intensity":"intensity","voltage":"voltage"}}Please help.

I have a CDP On prem cluster on 719 parcels , I am interested to get the HDFS service status of last 7 days . I know one way is to get from CM UI But I am interested to get from DB and want to plot a graph once I have the status of last 7 days . Its not only for HDFS service but for all the services in the cluster, Can you give me table details and query to fetch that data ?