Q&A

We have deployed NIFI application in AKS with 3 pods. The primary pod is working fine without error but pod1 and pod 2 are continuously restarting. I checked the logs for both, getting errors such as 2024-04-05 11:04:52,015 INFO [Thread-1] Initiating shutdown of Jetty web server...2024-04-05 11:04:52,020 INFO [Thread-1] Stopped ServerConnector@fb713e7{SSL,[ http/1.1]}{nifi-1.nifi-headless.iic-nifi-uat.svc.cluster.local:9443}2024-04-05 11:04:52,021 INFO [Thread-1] node0 Stopped scavengingI have checked the resources such as memory leaks and CPU consumption, everything seems to be in ideal state.  Kindly request insights from forum how to overcome this err

I have Nifi 1.25 on Ubuntu EC2 (m6i.2xlarge) = (8CPU/32GB RAM)Bootstrap.conf was set like the following:java.arg.2=-Xms2gjava.arg.3=-Xmx24gjava.arg.13=-XX:+UseG1GC <not sure if needed even>ExecuteSQL is set like the following: The flow is getting about 50-60 tables from mySQL and start creating flow files from them , upload to S3 and then copy into Redshift.Seems like the ExecuteSQL is consuming high CPU & RAM. The java.arg.3=-Xmx24g is the upper limit for the RAM?How can I control the CPU?

I'd like to have a process for automatically exporting the data flow definition and configuration without needing to go into the CDF UI.Right now, if I wanted the configuration, I'd click here in the UI:My thought was to use the CDP CLI, but under the df section, there doesn't appear to be a command for this.Any suggestions?

Hi All community/Support, I would like to ask how to extend this self-signed certificate validity to 6 months. As per default was 60 days. May i know which section of the code that setting this 60 days validity during start-up?  2024-04-02 07:32:08,803 INFO [main] org.apache.nifi.bootstrap.Command Generating Self-Signed Certificate: Expires on 2024-06-012024-04-02 07:32:11,796 INFO [main] org.apache.nifi.bootstrap.Command Generated Self-Signed Certificate SHA-256: 4XXXXXXXXXXXXXXXXXXXXXXX Appreciate if someone could help point out. 

24/04/12 15:15:54 ERROR tools.DistCp: Exception encounteredjava.io.IOException: Failed on local exception: java.io.IOException: Couldn't setup connection for replication_user@xxxxxx.xxx.xxx.xxx.COM to destination ip/destination ip:8020; Host Details : local host is: "xxxxxx.xxxxx.com/192.xxx.x.x"; destination host is: "xx.xxx.x.xxx":8020;

Hey Everyone,We installed CDP Private Cloud Data Services i.e. ECS 1.4.1. After registering environment through management console, when we are trying to create CDE (Cloudera Data Engineering) cluster we are getting aws_key_id error although we have not deployed servers through AWS. Please find the logs below.2023-10-18 08:06:48: Create: Cluster: Data-engineering-1, user:   <admin@cdp.example>, event: Create, properties: [internal-actor-crn=crn:altus:iam:us-west-1:altus:user:__internal__actor__]2023-10-18 08:07:04: Log: Cluster provisioning initiated, cloud platform: STANDARD2023-10-18 08:07:04: Log: Started provisioning a managed cluster, provisionerID: liftie-9bylvhxh2023-10-18 08:07:04: LogDebug: Updating cluster status from ClusterProvisioningInitiated to ClusterProvisioningInProgress2023-10-18 08:07:04: LogDebug: Checking underlying cluster status2023-10-18 08:07:04: Log: Cluster provisioning completed2023-10-18 08:07:04: LogDebug: Updating cluster status from Clus

We are seeing an error / alert and unable to schedule / run jobs in CDE. We are on CDP - PCAny guidance  in resolving this Tks Wert

I have tried this code on jupyter of Cloudera Machine Learning.============================================================import cml.data_v1 as cmldatafrom pyspark import SparkContext#Optional Spark ConfigsSparkContext.setSystemProperty('spark.executor.cores', '4')SparkContext.setSystemProperty('spark.executor.memory', '8g')#Boilerplate Code provided to you by CML Data ConnectionsCONNECTION_NAME = "go01-dl"conn = cmldata.get_connection(CONNECTION_NAME)spark = conn.get_spark_session()# Sample usage to run query through sparkEXAMPLE_SQL_QUERY = "show databases"spark.sql(EXAMPLE_SQL_QUERY).show()=================================================  I cheched enable spark, as well when I tried to create sessionHowever the result was failed with error message 'No data connection named go01-dl found'    While I am trying this, I thought I have to get information of spark.BUT I CANNOT. Where Can I get the connection name of SPARK?What should I do to do ru

 Hi,In the CML, the sessions in the PBJ Workbench show all infos and warnings in red by default (with a little symbol). Is there a way to set the color to black?  Kind regards! 

Dear Team, I am new to Cloudera. We are planning to use cloudera data platform. Is it possible to manage Data Governance activities with cloudera itself? Actually we are using informatica suite for data governance.  Please advice. I like to have a call with Middle East Cloudera Technical Team. Is it possible to connect with me?   Thanks! Regards, S.Ramakrishnan

We recently upgraded to CDE 1.20.3. After the upgrade we encountered issues related to resource allocation. I suspect that the yunikorn is not properly functionining because jobs are waiting time even if there are available resources to distribute to the spark jobs.Is there a yunikorn UI available from the CDE where we can access to monitor so we can easily monitor the yunikorn pods?

I am planning to appear in the Cloudera Administration Certification, CDP Administrator - Private Cloud Base Exam CDP-2001. Can someone please help me with some training links and sample question papers?

Hello!I have a flow that extracts 100 CSV files from a SFTP sever. I need to identify the last CSV file in order to continue the flow after all the files are loaded in GCP bucket.So, let me explain my scenario: I am extracting all 100 files from SFTP and load them in a GCP bucket. However, after the files are loaded, I want to redirect one file (preferably the last one) using the RouteOnAttribute in order to continue the flow for executing another processors.Can someone help me please?

Hi Regarding python updates, how can they affect cloudera?there is a security update RHSA-2023:0833 - Security AdvisoryDescriptionA vulnerability was discovered in Python. A quadratic algorithm exists when processing inputs to the IDNA (RFC 3490) decoder, such that a crafted unreasonably long name being presented to the decoder could lead to a CPU denial of service. Hostnames are often supplied by remote servers that could be controlled by a malicious actor, which could trigger excessive CPU consumption on the client attempting to make use of an attacker-supplied hostname. Do these types of updates affect cloudera?

After deployed Impala Virtual data warehouse of Data service.I faced Error "401 client Error: Unauthorized for url: http://coordinator:28000/cliservice"Sometimes It works well.  To solve this problem what should I check? 

I have a url endpoint that provides a json file with a size of a couple of `GB`. Unfortunately the api does not support pagination which would be the normal approach to my problem.So what i can do in python is to use ijson lib and split the json from the endpoint while receiving it and storing the result to my hard drive. This is very memory efficient and gives me the ability to run this async and start transforming the results while data are still loaded.  import ijsonimport jsonfrom urllib.request import urlopenf = urlopen(url)objects = ijson.items(f, 'item', use_float=True)record = (o for o in objects)for i,r in enumerate(record): print(i) with open(f'/tmp/streamwriter/{i}.json', 'w') as f: f.write(json.dumps(r))  now i want to do this in nifi. Is there a processor that can do the same. The way I understand the InvokeHttp Processor by now is that it has to receive the full payload before it sends the flowfile down stream.----Reference:I asked the same questi

Hi ,I have been trying to bring up stand alone embedded hbase and solr instance of Apache atlas. I followed the steps given in url : Apache Atlas — Installation Guide | by Manjit Singh | Medium . Once I start atlas with command python <atlas home dir>/bin/atlas_start.py, I see following output on my screen,Configured for local HBase.Starting local HBase...Local HBase started!Configured for local Solr.Starting local Solr...Local Solr started!Creating Solr collections for Atlas using config: /apps/apache-atlas/atlas/distro/target/apache-atlas-3.0.0-SNAPSHOT/conf/solrStarting Atlas server on host: localhostStarting Atlas server on port: 21000.................................................................................................Apache Atlas Server started!!!But when I try to access atlas UI from browser  this what I get ,I see few errors in .err and .out log files , please find the errors below,error from .err file ,ERROR StatusLogger Reconfiguration fai

Hi, I would like to execute redis commands in Nifi such as KEYS WITH PATTERN, ZADD, ZGET and so on. Does anyone has a processor or script that is able to do this? Thanks in advance. I already have the redis connection pool service working if that helps.

Hello, I m sending a http requestto an API and facing some difficulties to configure the multipart specificities.The 1st part should be binary (gzip) and named (data) and the second part should be json file and named (parameters). this works corretly with post manbut I m not able to do the same thing with Nifi 1.18this is my workflow   if someone faced this situation, plz share your feedbacks  

Hello everyone,I need help for a curl call with the Execute Stream Command processor. I can't use the invoke http processor because I need to bypass the SSL certificate check. the following request is made. Command Arguments--location;--insecure;--request;POST;--header;'Content-Type: application/json';--data-raw;'{\"username\": \"xxx\",\"password\": \"xxx\"}';https://99.9.99.999:8091/authIf i run the processor, i get the following outputAttribute Valuesexecution.commandcurl execution.command.args--location;--insecure;--request;POST;--header;'Content-Type: application/json';--data-raw;'{\"username\": \"xxx\",\"password\": \"xxx\"}';https://99.9.99.999:8091/auth execution.error% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 101 100 59 100 42 253 180 --:--:-- --:--:-- --:--:-- 433 FlowFile ContentBad Auth Request: Expecting value: line 1 column 1 (char 0) Does anyo

This is the content of my input json flowfile:{"objectIDs":["20607","23679","24365","31444","34130","57462"]} My requirement is to divide it in such a way that each output flowfile has only 5 elements in it.so if an objectIDs array has 17 element then I want 4 flowfile with the same json format with the first 3 having 5 elements and last one with the remaining 2.for my example content provided above, I would want two flow filesFirst One:{"objectIDs":["20607","23679","24365","31444","34130","57462"]}Second one :{"objectIDs":["57462"]}How do I do this in Nifi ?

Hi ,In my profile, I could see company as "London Stock Exchange Group PLC" and it needs to be change to "Refinitiv US LLC".  Thanks & Regards,Vijendar

my cdh version is 6.3.2my hbase version is 2.1.0-cdh6.3.2How can I modify the hbase. online. schema. update. enable parameter of hbase on CDH

Hi all,We have a Nifi community version and we noted that the service is using 46GB of ram on a virtual server that has 47GB.We used these settings to limit the amount of ram to useJava. Ag. 2=-Xms21gJava. Ag. 3=-Xmx42gBut the issue remains.Can you suggest how to address this issue?Regards,Fortunato

I am fetching file(s) from an HDFS path and transferring them to an SFTP server using Nifi. The HDFS file list is created by a Sqoop job, and the HDFS directory may contain one file or more than one file.Here is the list of processors I am using right now:RouteOnAttribute --> GetHDFSFileInfo --> RouteOnAttribute --> UpdateAttribute --> FetchHDFS --> PutSFTP --> UpdateAttributeMy data flow starts from a single FlowFile produced by a Sqoop job, which then becomes many FlowFiles after executing the GetHDFSFileInfo Processor (based on the number of HDFS files). However, I require only a single FlowFile post PutSFTP for downstream processing of job completion.Could you please suggest some solutions to execute the processors after PutSFTP only once? Do we need to create any separate processor group from GetHDFSFileInfo to PutSFTP?My Dataflow looks like below