Solved

Hello all,I could use some help — what’s the current process for obtaining a developer license?We have an older link to a Google request form that we’ve used in the past, but it doesn’t seem to work anymore.Thank you.

HiWe are currently operating a three-node cluster using Apache NiFi version 2.1.During a recent deployment of NiFi flows, we encountered an issue related to replication inconsistencies across the nodes.This problem resulted in one of the nodes being out of the cluster and remained in disconnected state.Furthermore, the affected node did not attempt to reconnect to the cluster on its own.Although the node was disconnected from the cluster, it continued to operate independently, allowing it to consume messages, execute database queries, and perform actions as a standalone instance.I would like to know two things hereIs this expected behavior for nifi node to be active although it is not in clusterCan we alter the behavior through configuration to stop processors temporarily while node is not connected to cluster?Thanks for in advance for your inputs.  Let me know if more details are required.Thanks    

Hi all,we are experiencing a progressive socket leak from the LLAP HiveServer2 daemon to HDFS DataNodes in a cluster running HDP 3.1.5The fd grow constantly until the limit of 6400 and the only resolution is to restart hiveserver2Interactive:[hive@myhost hive]$ netstat -tanp | grep "CLOSE_WAIT" | awk '{print $NF}' | awk -F'/' '{print $1}' | sort | uniq -c(Not all processes could be identified, non-owned process infowill not be shown, you would have to be root to see it all.)1734 -55443 27915 49167[hive@myhost hive]$ ps -ef | grep 2791hive 2791 1 31 Apr11 ? 6-01:44:32 /usr/jdk64/jdk1.8.0_112/bin/java -Dproc_jar -Dhdp.version=3.1.5.0-152 -Djava.net.preferIPv4Stack=true -Xloggc:/var/log/hive/hiveserverinteractive-gc-%t.log -XX:+UseG1GC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCCause -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=100M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/grid/0/tmp/hsi_heapdump.hprof -Dhive.log.dir=/var/log/hive -Dhive.log.f

Hello,I need to enable the HSTS header for Cloudera Management Services - Host Monitor Service and Service Monitor Service.Whilst I managed to do this for all the other services in the cluster using CORE_SETTINGS -> via HTTP Strict Transport Security parameter, this doesn't seem to be applied to Host Monitor Service and Service Monitor Instance from Cloudera Management Services . Is there a way to enable this?I am running a cloudera manager 7.13.1 with a cloudera runtime 7.3.1.Thank you so much!

Hi Cloudera Cummenity,    This documentation lists the versions of components used in Cloudera CDP 7.1.9 SP1:https://docs.cloudera.com/cdp-private-cloud-base/7.1.9/runtime-release-notes/topics/rt-pvc-runtime-component-versions-719sp1.htmlThe Hadoop version is 3.1.1.7.1.9.1000-103.However, when I look at Apache Hadoop Website, I could not find the same version there:The similar thing also happens for HBase:So my question is, why can't I find the hadoop and hbase version that Cloudera CDP used from Apache Hadoop and HBase website?

HiWe are using SiteToSiteReportingRecordSink to generate file transfer report with below settingWe are trying to send the data to the same server where we have added input port processor.Below is the nifi.property setting, do we need to change anything here ?? # Site to Site propertiesnifi.remote.input.host=52564f1f10ffnifi.remote.input.secure=truenifi.remote.input.socket.port=10000nifi.remote.input.http.enabled=truenifi.remote.input.http.transaction.ttl=30 secnifi.remote.contents.cache.expiration=30 secs# web properties ############################################### For security, NiFi will present the UI on 127.0.0.1 and only be accessible through this loopback interface.# Be aware that changing these properties may affect how your instance can be accessed without any restriction.# We recommend configuring HTTPS instead. The administrators guide provides instructions on how to do this.nifi.web.http.host=nifi.web.http.port=nifi.web.http.network.interface.default=we are getti

hi How do we get a report for all the file transfers for a given period ?We are using Nifi to only transfer file from one location to another using SFTP and no transformation on file data.Would like to generate a report with the transferred file name and its transfer status.Thanks  

Hello everyone,I'm seeking technical advice about designing data flows in Apache NiFi. I extract data from my source (Postgres), and my question is whether I should separate as process groups for Extract, Transform, and Load, or one process group covers end-to-end ETL for one table.

Hello, I'm investigating Nifi toolkit (1.16 if it matters)to add key/value pairs to a param-context programmatically( I can't use existing PC providers )Nifi toolkit documents the following action:#> nifi merge-param-context -helpAdds any parameters that exist in the exported context that don't exist in theexisting context. Overwrites any existing inherited parameter contexts with theprovided list....-i,--input <arg> A local file to read as input contents, or apublic URL to fetchI would like to know if the key/value pairs to add to the PC ( -pcid argument )can come from the contents of the -i fileIf yes, what is expected format of this file please ?( random testing and error messages didn't help much)Regards

Hello, I need help to renew validity of web certificate for Nifi.It is for local host and Nifi2.0.0-M4Please refer below screenshot.Thanks & Regards,Gaurav Mahajan 

When I try to install the CM agent 5.16 on SLES12 SP3 I get following error: m4dn136:~ # zypper install cloudera-manager-daemons cloudera-manager-agentLoading repository data...Reading installed packages...'cloudera-manager-daemons' is already installed.No update candidate for 'cloudera-manager-daemons-5.16.1-1.cm5161.p0.1.sles12.x86_64'. The highest available version is already installed.Resolving package dependencies...Problem: nothing provides python-psycopg2 needed by cloudera-manager-agent-5.16.1-1.cm5161.p0.1.sles12.x86_64 Solution 1: do not install cloudera-manager-agent-5.16.1-1.cm5161.p0.1.sles12.x86_64 Solution 2: break cloudera-manager-agent-5.16.1-1.cm5161.p0.1.sles12.x86_64 by ignoring some of its dependenciesThis package is not part of the SLES12 SP3 distribution. Where can I get it?

Hi All,I’ve created a .hql file containing multiple INSERT  statements  for Hive tables and I'm calling the script using Beeline. I need help with capturing error codes in a log file.Also, if any INSERT statement fails, I want the continue the  remaining statements  and need to be write a log into file.I am using below command but it is writing more details log > archive_$(date +\%Y\%m\%d).log 2>&1'Please advise how to handle this? 

Hello All,We have created a custom parcel for Cloudera runtime version 7.1.8 with OS RHEL 7. Our cluster is now operating in mixed OS mode, with some nodes upgraded to RHEL 8 while others are still running on RHEL 7. After the upgrade, Cloudera Manager UI is showing an error for the parcel: "Parcel not available for OS distribution RHEL 7."Is there a way to create parcels for both OS versions so that they can be distributed in mixed OS mode?Thanks,

Hi Team, We are trying to write data from kafka to ozone using nifi but somehow its not working rather its not giving any expection also Flow details generate flow files -> publish kafka -> cosume kafka -> put hdfs (ozone)wherein generate flow files -> put hdfs (ozone) -> works (which confirms there is no authentication issue)changes made in  the core-site.xml config   <property>    <name>fs.defaultFS</name>    <value>ofs://devozone</value>  </property>Are we missing any configs here on the processor level    are we missing anything Need assistance to resolve this issue 

I am trying to create a Ranger plugin audit directory on HBase programmatically using an API. However, I couldn't find any relevant API or documentation that explains how to achieve this.

HiWe are using NiFi API to create ListSFTP processor.When trying to connect to remote host we get below error.Caused by: net.schmizz.sshj.userauth.UserAuthException: Exhausted available authentication methodsWhen we update the "Password" manually in the processor it works fine.We validated that the password value is sent with correct data but for some reason its not getting set using API.Do we need to do anything extra for setting the password using API, Thanks 

账号邮箱。之前的被注销了，现在无法接受验证码，无法登录，请问怎么更改账号绑定邮箱

Hi All,I have been using spark 2.2 for long time in CDSW and recently trying to work in spark 3 in CDP. One of my queries is failing in spark 3 with an error of following Py4JJavaError: An error occurred while calling o96.sql.: org.apache.spark.SparkException: Cannot broadcast the table over 512000000 rows: 1235668051 rows              at org.apache.spark.sql.errors.QueryExecutionErrors$.cannotBroadcastTableOverMaxTableRowsError(QueryExecutionErrors.scala:1824)Even though this same query runs fine in spark 2.2 in CDSW. My spark session configuration is following # SET GENERAL SPARK PROPERTIES #print(" Configuring General Spark Properties")spark_session_builder = spark_session_builder.appName(name="Wrangler-Routine")spark_session_builder = spark_session_builder.master(master="yarn")spark_session_builder = spark_session_builder.enableHiveSupport()spark_session_builder = spark_session_builder.config("spark.yarn.que

We are in the process of integrating our services with the Cloudera Platform. The provided link allows us to download Cloudera Manager (Trial) version 7.4.4, which is bundled with CDP 7.1.7. I have tested this version, and everything works as expected. However, to ensure compatibility and avoid potential issues in production, we would like to test our services against newer versions of CDP like 7.1.9 or 7.3.1. From what I can see, access to these versions appears to be restricted and requires a valid subscription and credentials. Could you please advise how we can obtain access to these newer CDP versions for testing purposes? Is there a trial or evaluation license available for them? Thank you.

HiWe are in the process of fine tuning the performance of our NiFi nodes.1. Regarding "Max Timer Driven Thread Count" the default value is set as 10.Is this value okay given that we have 4 core CPU ? 2. For "Concurrent task" how do we figure the optimal value ?Also. does this value has to be same for all the Processors in the flow ? Thanks

Hello , After importing flows from my Registry in Nifi , while I try to commit Local changes , I get this below error:The logs for Nifi show the following :Caused by: org.apache.nifi.registry.flow.FlowRegistryException: Error creating snapshot: at org.apache.nifi.registry.flow.NifiRegistryFlowRegistryClient.registerFlowSnapshot(NifiRegistryFlowRegistryClient.java:263) at org.apache.nifi.registry.flow.StandardFlowRegistryClientNode.lambda$registerFlowSnapshot$7(StandardFlowRegistryClientNode.java:254) at org.apache.nifi.registry.flow.StandardFlowRegistryClientNode.execute(StandardFlowRegistryClientNode.java:289) at org.apache.nifi.registry.flow.StandardFlowRegistryClientNode.registerFlowSnapshot(StandardFlowRegistryClientNode.java:254) at org.apache.nifi.web.StandardNiFiServiceFacade.registerVersionedFlowSnapshot(StandardNiFiServiceFacade.java:5306) ... 142 common frames omittedCaused by: org.apache.nifi.registry.client.NiFiRegistryException: Error creating snapshot: at org.apache.

Hi,We newly upgraded to CML (previously CDSW). The jupyterlab-runtime sessions go into a timeout according to the value set in the configuration - but the PBJ-runtime sessions not. Right now we assume that the PBJ sessions just don't go into some sort of idle state. Could that be? What could prevent it from going idle? How can we make that the user sessions are timeouted after the time set in the config? Thank you in advance.

Good afternoon,I hope you're doing well. I would like to ask about the potential impacts on Cloudera DWH when upgrading EKS from version 1.29 to 1.31, considering that version 1.29 will no longer be supported.Specifically, I'm interested in understanding which components might be affected and what risks or implications this update could have.I appreciate your support in advance.Best regards!

I have a requirement to gather run duration (time) for the last 3 months, for a particular airflow job.In our CDE environment we use airflow to call spark DBT jobs, of late the run duration of job have increased drastically.I assume there would be a way to gather this jobs runtime duration for further analysis etc. Hoping for some assistance / guidance in getting his done.CDE Version - 1.19.3-b29Thanks Wert

HiWe are using DecryptContentPGP(Private key) andEncryptContentPGP(Public key)to encrypt and decrypt the file at rest, which is working fine.This same cert is being used by multiple flows.We are facing an issue when we have to renew this cert. since we need to restart the controller service  StandardPGPPublicKeyService.As this is being used by multiple flows, we need to stop and start all the flows.Is there a easier way to update cert without restarting StandardPGPPublicKeyService? Thanks