Solved

Hello.In Yarn Queue Manager, there is 4 queues defined :but when we submit queries (using Hue or SAS Software to load data), only the default queue is used.And we can go very high :From what we undestand, Default queue is defined for using 20% of the cluster but can use up to 100%, so 70% is normal.But our cluster is 365 GB / 15 vCores and some applications can use more :7 / 7 / 6 / 5 / 3 vCores in the graph = 28 vCore !How is it possible ?Thanks for your help 

I purchased an exam a few days ago and I could not provide information for billing with which I can refund my exam costs through my company. Can you provide me with some actual help to provide me with a bill? I tried emailing the certification@cloudera.com email a few days (every day from that) but got no reply and it is getting frustrating...

Hi,Input is getting from jolt transforms and data is something like :{"dv_sys_id" : [ "23ec0223138d73006877", "983f853edb2c77007166f", "6a3d0a23138d73006877b3", "805d5339db8dd050f66" ]}output should go in invokehttp processor somthing like: dv_sys_id: 23ec0223138d73006877, 983f853edb2c77007166f, 6a3d0a23138d73006877b3, 805d5339db8dd050f66 remote url : https://<url>${dv_sys_id} How can i achieve this requirement, Pls help me out.    

Hi, I'm struggling with the error below while trying to run Oozie workflow containing ssh action:[FNF], Message [FNF: Required Local file /tmp/oozie-oozi2214367252678167938.dir/ssh/ssh-base.sh not present Can anyone please suggest a potential cause and resolution?  Thank you!

Hello experts,We are facing an issue, where preprod Nifi server is able to read a kafka topic correctly.It is able to read all 3 partitions data.At the same time UAT Nifi server is connecting to same topic with different consumer group id and which is reading only the partition 0 & partition 1 but not partition 2.Screen shot from Lenses abt the same consumer group which is facing issue Any idea what could be the possible reasons?ThanksMahendra

we have HDP Hadoop cluster with two name-node services ( one active name-node and the secondary is the standby name-node )due of unexpected electricity failure , the standby name-node failed to start with the flowing exception , while the active name-node starting successfully2024-02-02 08:47:11,497 INFO common.Storage (Storage.java:tryLock(776)) - Lock on /hadoop/hdfs/namenode/in_use.lock acquired by nodename 36146@master1.delax.com2024-02-02 08:47:11,891 INFO namenode.FSImage (FSImage.java:loadFSImageFile(745)) - Planning to load image: FSImageFile(file=/hadoop/hdfs/namenode/current/fsimage_0000000052670667141, cpktTxId=0000000052670667141)2024-02-02 08:47:11,897 ERROR namenode.FSImage (FSImage.java:loadFSImage(693)) - Failed to load image from FSImageFile(file=/hadoop/hdfs/namenode/current/fsimage_0000000052670667141, cpktTxId=0000000052670667141)java.io.IOException: Premature EOF from inputStreamat org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:204)at org.apache.hadoop.hdfs.se

I have enabled https on nifi registry properties and generated key store and trust store using tls toolkit and added in the properties file. I am able to start nifi registry on port 18443 and host 0.0.0.0 as it showing in logs that jetty server started on https://0.0.0.0:18443/nifi registrymy nifi registry is hosted as aks pod. when I enable ingress for it. It’s returning me as 502 bad request.are there any changes to be done to make my nifi registry run https. Goal : i want to enable 2 way ssl authentication so that nifi could connect to nifi registry.

Hi,recently I have been trying to track my spark application using SparkListener.Now I have a problem:          There is a application, the result of its execution is SUCCESSED on Yarn WebUI,but it's actually a failed application.         And my listener class can not get the error message,the application log only show the Start events for failed jobs and tasks.Some can help me ?  

I am trying to pull the records from the Hbase based on the row key in pyspark using the following details  but the records which is returning showing all the records (i,e without filter), can you please help me to resolving the issue 

hi cloudera, I need to use Spark on a host that is not part of the Cloudera cluster to run Spark jobs on the Cloudera cluster.Is it possible to use it this way? If yes, how to configure?what I've already tried:1. Download "https://www.apache.org/dyn/closer.lua/spark/spark-3.3.4/spark-3.3.4-bin-hadoop3.tgz"2. Copy the "conf" files from the Cloudera cluster and send them to the new Spark directory3. exported the variables "HADOOP_CONF_DIR" and "SPARK_CONF_DIR" and "SPARK_HOME" using the new spark directory "spark-3.3.4-bin-hadoop3" with the files4. When trying to run spark-shell as an example, nothing happens, it hangs as shown below:Setting default log level to "WARN".To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).Welcome to____ __/ __/__ ___ _____/ /___\ \/ _ \/ _ `/ __/ '_//___/ .__/\_,_/_/ /_/\_\ version 3.3.4/_/Using Scala version 2.13.8 (Java HotSpot(TM) 64-Bit Server VM, Java 11.0.16.1)Type in expressions to have them evaluated.Typ

Hi - I've got everything working with NIFI 2.0.0 M2 when the the python command line is commented out but I'd like to get it up and running with the python command line uncommented.nifi.properties file linenifi.python.command=/usr/bin/python3.11 nifi-app.log info:Caused by: java.lang.IllegalStateException: Failed to communicate with Python Controllerat org.apache.nifi.controller.FlowController.<init>(FlowController.java:581)at org.apache.nifi.controller.FlowController.createStandaloneInstance(FlowController.java:411)at org.apache.nifi.spring.FlowControllerFactoryBean.getObject(FlowControllerFactoryBean.java:83)at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:178)... 95 common frames omittedCaused by: java.io.IOException: Failed to create Python Environment ./work/python/controller: process existed with code 1at org.apache.nifi.py4j.PythonProcess.setupEnvironment(PythonProcess.java:223)at org.ap

From Hue security browser,  i have a RBAC role (admin), which was given all permission on all databases.I have this Database A which consists of table1,2,3I want to deny this admin role from dropping Database A -> Table1 and Table2.Can you show me how this can be done?

Hi Team,I am using Nifi (3-node cluster) version 1.23.2 in my environment. My doubt, is there any case if Nifi processor (any processor within a process group) stops suddenly due to load/any other issue? If that happens, then what will be the solution? Does it lead to data loss?  

We try to use PutHDFS process ,automatically the owner of file which created by PutHDFS in Cloudera  will be same domain user which NIFI running.The OS of NIFI is window 10 and the server of Hive is Cloudera docker.how we can update owner of file from domain user to Cloudera user.

Good evening, I am having a lot of trouble getting an instance of NiFi running with Mutual TLS. I have different errors depending on whether the effort involves version 1.xx or version 2.xx. I'm hoping to get some help. This is all being attempted via Docker.My YAML:```version: "3.9"services:  nifi:    image: apache/nifi:1.24.0    # hostname: nifi    container_name: nifi    ports:      # - 8443:8443      - "9449:9449"    environment:      - NIFI_WEB_HTTPS_PORT=9449      - NIFI_WEB_HTTPS_HOST=0.0.0.0      - NIFI_WEB_PROXY_HOST=localhost,localhost:9449,192.168.1.56,192.168.1.56:9449      - AUTH=tls      - KEYSTORE_PATH=/opt/nifi/nifi-current/certs/jks/nifi-ks.jks      - KEYSTORE_TYPE=JKS      - KEYSTORE_PASSWORD=randomPassword      - TRUSTSTORE_PATH=/opt/nifi/nifi-current/certs/jks/trus

I am currently running Hbase CDH 5.5 on Centos 7. I need to upgrade to RHEL8. Does CDH need to be upgraded as well ? What happens to the existing data ?

Hello ,I'm using NIFI to replicate 250 million messages between Kafka topics.The problem is that NIFI replicates messages in a non-sequential order, resulting in the destination topic storing messages differently than the source topic.for examplesource topic - partition 0offset:5 key:a value:v1offset:6 key:a value:v2offset:7 key:a value:v3destination topic - partition 0offset:5 key:a value:v2offset:6 key:a value:v1offset:7 key:a value:v3The topics are configured with a cleanup policy: compact.I'm using ConsumeKafka and PublishKafka processors to replicate topics.      

I have this JSON:{ "store": { "book": [ { "category": "reference", "author": "Nigel Rees", "title": "Sayings of the Century", "price": 8.95 }, { "category": "fiction", "author": "Herman Melville", "title": "Moby **bleep**", "isbn": "0-553-21311-3", "price": 8.99 }, { "category": "fiction", "author": "J.R.R. Tolkien", "title": "The Lord of the Rings", "isbn": "0-395-19395-8", "price": 22.99 } ]}I want a list/record set of categories. CSV, plain text, doesn't matter, but let's say CSV:category"refereces""fiction""fiction"I have tried many things, too many to repeat them all here. But basically, I have a GenerateFlowFile where the JSON is hard-coded, then a QueryReport where the reader is a JsonPathReader where I have properties for all fields in the JSON:store $.storebook $.store.book[*]category $.store.book[*].categoryetc.Just to see what'

I created a template to convert data from querying Postgres database and then convert it into xml. I want to run it by using command line arguments. Are there any way to do this in NiFi? Thank you

I have set following:nifi.flowfile.repository.implementation: org.apache.nifi.controller.repository.VolatileFlowFileRepositorynifi.provenance.repository.implementation: org.apache.nifi.provenance.VolatileProvenanceRepositoryWith this, I expect disk io operations to go down(checked with nmon). But its just opposite and disk writes have gone up multiple times.Why so?

Hi everyone,I'm trying to figure out what causes this Impala warning that is generated when external tables are created:Impala does not have READ_WRITE access to path 'hdfs://xxxxxxxxxxx'At first, we thought it was a permissions issue with Ranger, but after numerous attempts, we still get the same warning.We also checked the permissions on the HDFS paths, but there are folders with permissions of 775, 750, and 755, so there doesn't seem to be a correlation between the warning and POSIX permissions.Could it be an issue with the user and/or group? In many paths, the owner and group of the folder are hdfs:hdfs. Should the owner be changed to impala?Unfortunately, I haven't found any helpful documentation on this topic that would allow me to eliminate the warning. Many users think that the tables are actually not being created and that there is no way to write to them.

Hi All,we try to fetch the results from database according to the content of cvs file.after converting the csv to json format by SpileRecord  then we try to  get the content of Json record  by  EvaluateJsonPath  then we  try to get value of attribute by  UpdateAttribute  but the  FF_Content attribute into inEvaluateJsonPath not recognized . listFile->FetchFiles->SpileRecord->EvaluateJsonPath ->UpdateAttribute ->ExecuteSQL   

the platform is : CDH 5.14.4after setting federation, the hdfs entrypoint is: viewfs://cluster6/ , the back namenode entrypoint is : hdfs://nameservice/ and hdfs://nameservice2/add new warehouse directory is  /user2/hive/warehouse and managed by nameservice2.but i found the strange things is  if i create database using hdfs://nameservice2, sentry can sync ACL with hdfs, but if i used viewfs://cluster6/ , sentry can not sync ACL.for example:create database test location '/user2/hive/warehouse/test.db'    can not sync ACLbut create database test location 'hdfs://nameservice2/user2/hive/warehouse/test.db' is ok.who knows why ?

Dear all, I have a query regarding the insertion of a flowfile into a MySQL Database. Specifically, I would like to know how to retrieve the auto-incremented ID from the database and store it as a flowfile attribute.Consider the following example flowfile (representing one record, with many more records in reality):My goal is to insert this data into 'table1,' which has an auto-incremented ID field (let's assume that after insertion, the ID will be 3001). Could you please guide me on how to retrieve this auto-incremented ID and assign it to a flowfile attribute?

Hi there, I need some help regarding using a flow.xml.gz from 1.13 to 1.24. First our nifi setup: 2 node cluster running on ec2 instances in aws where we run nifi in docker containers. We have succesfully upgraded from 1.13 to 1.24 after making a backup of the flow.xml.gz file.We then stop the nifi containers and put back the flow.xml.gz from version 1.13. Then startup nifi. Unfortunately we get the following error:Jan 19 07:20:58 nifi-node-1 docker[27934]: 2024-01-19 07:20:58,216 INFO [main] org.eclipse.jetty.server.Server Started @78240msJan 19 07:20:59 nifi-node-1 docker[27934]: 2024-01-19 07:20:58,304 INFO [main] org.apache.nifi.web.server.JettyServer Loading Flow...Jan 19 07:20:59 nifi-node-1 docker[27934]: 2024-01-19 07:20:58,326 INFO [main] o.apache.nifi.io.socket.SocketListener Now listening for connections from nodes on port 8888Jan 19 07:20:59 nifi-node-1 docker[27934]: 2024-01-19 07:20:59,036 WARN [main] o.a.nifi.fingerprint.FingerprintFactory Schema validation error pa