Q&A

Hi All,I have been using spark 2.2 for long time in CDSW and recently trying to work in spark 3 in CDP. One of my queries is failing in spark 3 with an error of following Py4JJavaError: An error occurred while calling o96.sql.: org.apache.spark.SparkException: Cannot broadcast the table over 512000000 rows: 1235668051 rows              at org.apache.spark.sql.errors.QueryExecutionErrors$.cannotBroadcastTableOverMaxTableRowsError(QueryExecutionErrors.scala:1824)Even though this same query runs fine in spark 2.2 in CDSW. My spark session configuration is following # SET GENERAL SPARK PROPERTIES #print(" Configuring General Spark Properties")spark_session_builder = spark_session_builder.appName(name="Wrangler-Routine")spark_session_builder = spark_session_builder.master(master="yarn")spark_session_builder = spark_session_builder.enableHiveSupport()spark_session_builder = spark_session_builder.config("spark.yarn.que

We are in the process of integrating our services with the Cloudera Platform. The provided link allows us to download Cloudera Manager (Trial) version 7.4.4, which is bundled with CDP 7.1.7. I have tested this version, and everything works as expected. However, to ensure compatibility and avoid potential issues in production, we would like to test our services against newer versions of CDP like 7.1.9 or 7.3.1. From what I can see, access to these versions appears to be restricted and requires a valid subscription and credentials. Could you please advise how we can obtain access to these newer CDP versions for testing purposes? Is there a trial or evaluation license available for them? Thank you.

HiWe are in the process of fine tuning the performance of our NiFi nodes.1. Regarding "Max Timer Driven Thread Count" the default value is set as 10.Is this value okay given that we have 4 core CPU ? 2. For "Concurrent task" how do we figure the optimal value ?Also. does this value has to be same for all the Processors in the flow ? Thanks

Hi I will like to append forward slash / conditionally in EL. But not able to acheive it.${extracted.path:length():gt(0):ifElse(${extracted.path:append('/'):append(${filename})},${filename})}Have tried using ${literal('/')} as well but its not getting the value. Thanks 

Hi, new to Hive and SQL. So please be patient.I've got a dataset with 2 tables:1. organisations - contains names of the organisations and an id2. relationships - contains id of the organisations which organisation is it related to, ie a superiori've got a query to show the immediate relationship next to each other:SELECT sub.organisation_id, sub.organisation_name, sup.organisation_id, sup.organisation_nameFROM organisation supJOIN relationshipON sup.organisation_id = relationship_idJOIN organisation subON organisation_id = relationship_orgidHow do I query this using Hive (or Impala) to display all the organisations in the ancestry? E.g.,baby -> mother -> grandmother -> great-grandmother -> great-great-grandmother -> etc. 

Hello , After importing flows from my Registry in Nifi , while I try to commit Local changes , I get this below error:The logs for Nifi show the following :Caused by: org.apache.nifi.registry.flow.FlowRegistryException: Error creating snapshot: at org.apache.nifi.registry.flow.NifiRegistryFlowRegistryClient.registerFlowSnapshot(NifiRegistryFlowRegistryClient.java:263) at org.apache.nifi.registry.flow.StandardFlowRegistryClientNode.lambda$registerFlowSnapshot$7(StandardFlowRegistryClientNode.java:254) at org.apache.nifi.registry.flow.StandardFlowRegistryClientNode.execute(StandardFlowRegistryClientNode.java:289) at org.apache.nifi.registry.flow.StandardFlowRegistryClientNode.registerFlowSnapshot(StandardFlowRegistryClientNode.java:254) at org.apache.nifi.web.StandardNiFiServiceFacade.registerVersionedFlowSnapshot(StandardNiFiServiceFacade.java:5306) ... 142 common frames omittedCaused by: org.apache.nifi.registry.client.NiFiRegistryException: Error creating snapshot: at org.apache.

Hi,We newly upgraded to CML (previously CDSW). The jupyterlab-runtime sessions go into a timeout according to the value set in the configuration - but the PBJ-runtime sessions not. Right now we assume that the PBJ sessions just don't go into some sort of idle state. Could that be? What could prevent it from going idle? How can we make that the user sessions are timeouted after the time set in the config? Thank you in advance.

Good afternoon,I hope you're doing well. I would like to ask about the potential impacts on Cloudera DWH when upgrading EKS from version 1.29 to 1.31, considering that version 1.29 will no longer be supported.Specifically, I'm interested in understanding which components might be affected and what risks or implications this update could have.I appreciate your support in advance.Best regards!

I have a requirement to gather run duration (time) for the last 3 months, for a particular airflow job.In our CDE environment we use airflow to call spark DBT jobs, of late the run duration of job have increased drastically.I assume there would be a way to gather this jobs runtime duration for further analysis etc. Hoping for some assistance / guidance in getting his done.CDE Version - 1.19.3-b29Thanks Wert

Hello, I’m getting the above error when running this query.select count(*) from table where created_on >= date_sub(current_date(),100));I tried casting the count to int, then I get the same error with “int” in place of “long”.If I remove the where clause, it works just fine.If I execute the same query in hue, I get the results back without error.Any ideas on why this is happening and any workarounds?Thank you.  

HiWe are using DecryptContentPGP(Private key) andEncryptContentPGP(Public key)to encrypt and decrypt the file at rest, which is working fine.This same cert is being used by multiple flows.We are facing an issue when we have to renew this cert. since we need to restart the controller service  StandardPGPPublicKeyService.As this is being used by multiple flows, we need to stop and start all the flows.Is there a easier way to update cert without restarting StandardPGPPublicKeyService? Thanks  

So the use case that I want to implement is some requests will be landing on my NIFI in XML format and then I want to control its rate and then pick the necessary inputs from the XML and then differentiate based on a report number and use rate limit on the the defined report numbers and then further a rate limit on a article number which I pick from the XML and then I call an API and send the response back the initial response now the issue is I am using control rate processor to hold the requests to 5 per second but with hold flowfile config as I cannot fail the request directly but also I want to fail the queueing requests after 5 second and with a specific status code and also I want to log on which of the three control rate failed the request. Any help how this flow can be created? And also monitor how many requests landed in the server per sec how many succeeded and how many failed per second. 

I'm trying to create a new Nifi custom processor.I'm following instructions given on 4 differents tutorials.I only create the structure of the processor, I don't even try to modifiy the code and I'm always having the same mvn clean install [ERROR].Can you please help me to resolve this ?A huge thanks!!!OS : Windows 10 HomeNifi version : 2.0.0-M2Maven : Apache Maven 3.9.6JDK : Java version: 21.0.2, vendor: Oracle Corporation, runtime: C:\Program Files\Java\jdk-21Me version : Java and Maven newbi (0.0.1)Maven is installed and I have added its bin to the path env var.Jdk in installed and I have added its bin to the path env var.JAVA_HOME env var is defined as C:\Program Files\Java\jdk-21, where my jdk is installed.The error:[ERROR] Failed to execute goal on project nifi-helloworld-nar: Could not resolve dependencies for project org.apache.nifi:nifi-helloworld-nar:nar:1.0-SNAPSHOT: The following artifacts could not be resolved: org.apache.nifi:nifi-standard-services-api-nar:nar:1.0-SNAPSHO

Hi community,Our NIFI version: 1.23.0.We are converting JSON message to AVRO format and posting to KAFKA topic using  PublishKafkaRecord_2_0 1.23.0:JSON Tree Reader:AVRO Record Set Writer:Confluent Schema Registry:We are not able to post  the message with the error below.We are able to see the AVRO schema in the KAFKA registry through POSTMAN and using browser URL and we are able to connect to that KAFKA topic with the same parameters in NIFI .Could you please help us to resolve this issue.Error: 

I have a table with one column as JSON file. I am using lateral explode to parse values.The code is working fine for classes except for one where subclasses have similar variable name :Json:{"Customer": {"CustomerId": "19900101","appl": [{***Some values}]"products": [{"ProductCOde": ABC,"Type": C1,"Offer": 16575.0,"Offer": [{"ProductCOde": A1,"Offer": 2000.0"Type": CBA,}{"ProductCOde": A2,"Offer": 2000.0"Type": DBA,}{"ProductCOde": A3,"Offer": 3000.0"Type": BBA,}]}{"ProductCOde": XYZ,"Type": C2,"Offer": 2001,"Offer": [{"ProductCOde": B1,"Offer": 2000.0"Type": BBA,}{"ProductCOde": B2,"Offer": 2000.0"Type": BBA,}]}]}} Code:with product as (selectxml_data,get_json_object(xml_data,'$.customer.customerid')as applicationid,regexp_replace(regexp_replace(get_json_object(xml_data, '$.customer.products'), '\\[|\\]', ''), '\\}\\,\\{', '\\}\\;\\{') AS product_JSOnfrom Custome_info)selectapplicationid,get_json_object(prod_item,'$.ProductCOde') as ProductCOde,get_json_object(prod_item,'$.type')

Hi,I'm trying to use Nifi 2.3.0 and 2.2.0 to list some ftp files using ListFTP and then fetching it using FetchFTP.The problem is that when I try to fetch using 2.3.0 ListFTP, it gets timed out, and using the same parameters in 2.2.0 ListFTP, I can get the files in the ftp. I also tested it with another ftp and got another error:"- Caused by: org.apache.commons.net.MalformedServerReplyException: Could not parse response code."

Hello, we need access to Cloudera archive in order to be able to setup our cdh docker image to test our cdh jar for h2o-3.https://archive.cloudera.com/cdh5/ubuntu/trusty/amd64/cdhHow can we request access to the repo?Thank you.Adam Valenta (adam.valenta@h2o.ai)

Can NiFi handle multiple users hitting a request? Say I have HandleHttpRequest or a ListenHttp processor and my requirement is to handle say 100 concurrent users. Can we do it in NiFi? 

In the context of Cloudera CDP, I've noticed that unlike CDH, YARN does not automatically distribute the hive-site.xml file to the nodes. This seems to be a change in configuration practices, but I'm not sure if this is the correct approach or if there's another recommended method to ensure the hive-site.xml file is properly distributed. Can someone confirm if it is necessary to configure it manually in CDP, or if there is another automated process that YARN handles for distributing the configuration file?

Can we setup the Iceberg table to use ranger authorization on top of spark and how to configure that?

Hello everyone,I'm trying to build Ambari 2.7.9 by following this guide https://cwiki.apache.org/confluence/display/AMBARI/Installation+Guide+for+Ambari+2.7.9But when I run this command mvn -B clean install rpm:rpm -DnewVersion=2.7.9.0.0 -DbuildNumber=da8f1b9b5a799bfa8e2d8aa9ab31d6d5a1cc31a0 -DskipTests -Dpython.ver="python >= 2.6", I'm facing this issue Failed to execute goal org.codehaus.mojo:rpm-maven-plugin:2.0.1:rpm (default-cli) on project ambari-web: RPM build execution returned: '1' executing '/bin/sh -c cd /ambari/apache-ambari-2.7.9-src/ambari-web/target/rpm/ambari-web/SPECS && rpmbuild -bb --buildroot /ambari/apache-ambari-2.7.9-src/ambari-web/target/rpm/ambari-web/buildroot --define '_topdir /ambari/apache-ambari-2.7.9-src/ambari-web/target/rpm/ambari-web' --target noarch-redhat-linux ambari-web.spec' I've already installed rpm-build, g++, and all tools needed to build Ambari, by following https://cwiki.apache.org/confluence/display/AMBAR

Please note that the previous text was translated using a translator, so it might sound a bit awkward. I apologize for any confusion.I’m encountering an issue where everything runs fine initially, but sometimes I experience this error. Could anyone help me understand what might be causing it? [Cloudera][ImpalaJDBCDriver](500051) ERROR processing query/statement. Error Code: ExecuteStatement failed: out of sequence response: expected 6 but got 5, SQL state: HY000,Query: select * from (SELECT tool_path,tag,act_time,value,row_number() over (partition by tool_path,tag order by act_time desc) as rownum FROM `X`.`Y` WHERE (tag like 'A%' or tag like 'B%' or tag like 'C%' or tag like 'D%' or tag like 'E%' or tag like 'F%' or tag like 'G%' or tag like 'H%' or tag like 'I%' or tag like 'J%' or tag like 'K%' or tag like 'L%' or tag like 'M%' or tag like 'N%' or tag like 'O%' or tag like 'P%' or tag like 'Q%' or tag like 'R%' or tag like 'S%' or tag like 'T%' or tag like 'U%') and act_time BE

My goal want to use Nifi to schedule to run python script like daily, monthly.My project are gonna be forecasting Which my python script are gonnaload the ML model then query data from data base and use it as input in my ML model then it will make a prediction then save that data into database.How can I use NiFi to do that? or Should I create API for the Model then use NiFi to call API or using cron Service to schedule it?

Hi Team,I am trying to develop a flow to update the Parameter context on the fly using the InvokeHTTP Processor. Based on the screenshot i have configured the same. However i am getting unAuthorized Error.  My URL format is like : https://<nifi-host>:<port>/nifi-api/parameter-contexts/<context-id>Error Screenshot.  

Hello Cloudera Community,I'm working on a project where I need to provide users with temporary access to call specific Cloudera AI API endpoints, particularly for interacting with deployed models.My goal is to avoid granting users long-lived API V2 keys and instead generate short-lived, scoped access credentials that expire after a set time (e.g., 2 hours).After exploring the CML API v2 and the cmlapi library, I'm finding it challenging to identify a direct method for generating these temporary, scoped access keys.My questions are:Is there a supported mechanism within the CML API v2 or cmlapi to generate temporary access keys with a specified expiration time?If not, are there recommended alternative approaches for providing users with limited, time-bound access to CML API endpoints?Are there plans to introduce temporary access key generation functionality in future CML releases?Is there a best practice to follow for this kind of implementation?I'm aiming for a solution that prioritizes