Q&A

I  have received a warning in cloudera manager for 4/5 hosts. It showing "Network Interface Speed", "Concerning : The following network interface(s) appear to be operating at less than full speed: virbr0-nic.2 host network interface(s) appear to be operating at full speed.For 1 host network interface(s), the Cloudera Manager Agent could not determine the duplex mode or interface speed". I already set the value of the 2 configuration parameters "Network Interface Expected Link Speed" as 10000 and "Network Interface Expected Duplex Mode" is Full. I am in need the permanent fix on this issue. Thanks,Doan Gia Hy

Hello,I'm installing CDP Private Cloud Data Service on OCP , but I'm stuck at this part :Please help give soliution.Thank youRegard,Doan Gia Hy

Hello,I'm setting up new cluster for educational purposes and configuring SSL using case 2 "Enabling Auto-TLS with an intermediate CA signed by an existing Root CA ", I just installed cloudera manager and enabled the kerbros using Free IPA & created the certificates using JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk /opt/cloudera/cm-agent/bin/certmanager setup --configure-services --stop-at-csr. But when I sign the csr from the IPA server I got the attached error,, Any help pleaseNote: I'm using the user that automatically created by cloudera manager when setting the kerbros as principle, for more info >>  Step 3: Create the Kerberos Principal for Cloudera Manager Server | CDP Private Cloud

using cdp 7.14 kudu is working and nodes are connected but giving no data ksck error :  Wed Dec 13 15:51:29 +03 2023JAVA_HOME=/usr/java/jdk1.8.0_232-clouderaUsing /var/run/cloudera-scm-agent/process/1546472905-kudu-KUDU_MASTER-kudu_master_ksck_cmd as conf dirUsing scripts/kudu.sh as process scriptCONF_DIR=/var/run/cloudera-scm-agent/process/1546472905-kudu-KUDU_MASTER-kudu_master_ksck_cmdCMF_CONF_DIR=Wed Dec 13 15:51:29 +03 2023: KUDU_HOME: /apps/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kuduWed Dec 13 15:51:29 +03 2023: CONF_DIR: /var/run/cloudera-scm-agent/process/1546472905-kudu-KUDU_MASTER-kudu_master_ksck_cmdWed Dec 13 15:51:29 +03 2023: CMD: ksckWed Dec 13 15:51:29 +03 2023: Found master(s) on m507,m508,m01Master SummaryUUID | Address | Status----------------------------------+---------+---------0a0209b49d6e4f709221336403bae183 | m507 | HEALTHY127407e626be4f4e9290389b1bfc50f3 | m508 | HEALTHY580630447da74d788de916544b487399 | m01 | HEALTHYFlags

Hi:I use HBASE 2.1.6 version, and enable MOB feature on my tablesthen I start to input data to my table,after a while, 4 regionservers are all down, and the error log hs_err_pidxxx.log  generated in the /var/log/hbase/.after to restart the hbase, regionservers still down, it can not start againI have no idea how to check the hs_err_pidxxx.log ## A fatal error has been detected by the Java Runtime Environment:## SIGSEGV (0xb) at pc=0x00007efe5ae4ae40, pid=11890, tid=0x00007efe585d5700## JRE version: Java(TM) SE Runtime Environment (8.0_241-b07) (build 1.8.0_241-b07)# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.241-b07 mixed mode linux-amd64 )# Problematic frame:# V [libjvm.so+0x649e40] void G1ParScanClosure::do_oop_nv<oopDesc*>(oopDesc**)+0x30## Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again## If you would like to submit a bug report, please visit:# http://bugreport.

We have HDP cluster with 152 workers machines - `worker1.duplex.com` .. `worker152.duplex.com` , While all machines are installed on RHEL 7.9 versionWe are trying to delete the last host - `worker152.duplex.com` from Ambari server or actually from PostgreSQL DB as the followingFirst we need to find the `host_id`select host_id from hosts where host_name='worker152.duplex.com';and host_id is:host_id---------51(1 row)Now we are deletion this `host_id` - 51delete from execution_command where task_id in (select task_id from host_role_command where host_id in (51));delete from host_version where host_id in (51);delete from host_role_command where host_id in (51);delete from serviceconfighosts where host_id in (51);delete from hoststate where host_id in (51);delete from kerberos_principal_host WHERE host_id='worker152.duplex.com';delete from hosts where host_name in ('worker152.duplex.com');delete from alert_current where history_id in ( select alert_id from alert_history where host_name in (

I have a Hadoop cluster with these specifications: PC1 --> master: Number of cores: 40 RAM 64G SSD250GPC2 --> slave: Number of cores: 30 RAM 32G SSD128GPC3 --> slave2: Number of cores: 30 RAM 32G SSD128G in each slave, I'll create 2 containers for data nodes, at a total of 5 nodes will setup. Please guide me on what should be the best configuration for memory and vcore on yarn-site.xml and mapred-site.xml for better performance.thanks 

yarn aggregation log on hdfs when I FINISHED a job, the {yarn.nodemanager.remote-app-log-dir}/{user}/{yarn.nodemanager.remote-app-log-dir-sendix} directory permission is 770, I change the directory permission to 775.But when I FINISHED a yarn job, it change the directory permission to 770 again.Why would he happen? How to get it not to modify directory permissions.

Hoping someone can help me with a JOLT spec. I have the following json:{"schemaVersion": "3.2","uniqueIdentifier": "ri.nzRNR5a3ad1AzOkv_2FU9WN6Z_2FluxpEl4eXjgUnYpgkvk_3D","entityType": "CAR"} and I need to use JOLT to transform to this: {"_id": "ri.nzRNR5a3ad1AzOkv_2FU9WN6Z_2FluxpEl4eXjgUnYpgkvk_3D", "schemaVersion": "3.2","uniqueIdentifier": "ri.nzRNR5a3ad1AzOkv_2FU9WN6Z_2FluxpEl4eXjgUnYpgkvk_3D","entityType": "CAR"} I cannot figure out the correct spec. Thanks!

For example, the spring framework of the official nar package of nifi 1-25.0 is 5.3.31. I changed it to 5.3.33 and I can modify these files /home/servers/nifi-1.25.0/lib/properties/spring-core-5.3 .31 .jar/home/servers/nifi-1.25.0/lib/properties/spring-aop-5.3.31.jar/home/servers/nifi-1.25.0/lib/properties/spring-context-5.3.31.jar/home/servers/nifi-1.25.0/lib/properties/spring-jcl-5.3.31.jar/home/servers/nifi-1.25.0/lib/properties/spring-beans-5.3.31.jar/home/servers/nifi-1.25.0/lib/properties/spring-express-5.3.31.jarIs /home/servers/nifi-1.25.0/lib/properties/spring-web-5.3.31.jar implemented? Is it possible to modify the spring framework in these nar packages? I tried to modify these, but the result was that the apache nifi service could not be started./home/servers/nifi-1.25.0/work/nar/framework/nifi-framework-nar-1.25.0.nar-unpacked/NAR-INF/bundled-dependencies/spring-core-5.3.31.jar/home/servers/nifi-1.25.0/work/nar/framework/nifi-framework-nar-1.25.0.nar-unpacked/NAR-INF/b

 is there way can Dashboard/monitor report  which showing statistical report for running NIFI Template/process for example how many success/failure messages are processing and how many message in  queue  and ....etc.  Thank you in advance.

  I have a remote server and servers authenticated Hadoop environment.I want to copy file from Remote server to Hadoop machine to HDFSPlease advise efficient approach/HDFS command to copy files from remote server to HDFS.Any example will be helpful.as ordinary way to copy file from remote server to server itself isscp -rp file remote_server:/tmpbut this approach not support copy directly to hdfs

Hi,I have Nifi 2.0-M2 and Apache Ignite 2.16 database running on my localhost.http://localhost:8443/nifijdbc:ignite:thin://localhost:10800  (I am able to connect to Ignite localhost db from Dbeaver)I have copied ignite-core-2.16.0.jar file to nifi lib directory (before starting nifi) and tried to create DBCPConnectionPool controller service in Nifi to connect to Ignite database.  DB Controller service returns the below error.Failed to invoke @OnEnabled method: java.lang.NoClassDefFoundError: Could not initialize class org.apache.commons.dbcp2.BasicDataSource - Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.NoClassDefFoundError: Could not initialize class org.apache.ignite.IgniteJdbcThinDriver [in thread "Timer-Driven Process Thread-10"]Can you let me know what am I missing here?  Please let me know if more details are required.Screenshot of DB controller service     

We have some execute script processor that supports ECMA engine in NIFI 1.20 now we are migrating to NIFI 2.0 and since ECMA engine is depreciated how do we convert the script to closure or groovy or any other alternatives methods

Hello,We have a situation where we want to connect to both Cloudera Hive and Impala, using a Kerberos authentication. On top of that, we can go through 2 different realms under these two DBs. To proceed with this setup, we are using the MIT tool. This setup works well when it comes to Impala (in both realms), however it does not work with Hive; we essentially get this type of error below (this is the result of the "Test" from the ODBC setup).I found two similar topics on this forum talking about this error, but the resolution was not clear to me (https://community.cloudera.com/t5/Support-Questions/ERROR-28000-Cloudera-ThriftExtension-9-Error-occurred-while/td-p/365047 and https://community.cloudera.com/t5/Support-Questions/Hive-ODBC-kerberos-SASL-1-generic-failure-GSSAPI-Error/m-p/61548)Would anyone have clues on where to go from here to resolve this conflict ?Thank you

The feed json is like below:[ { "orderId": "10000", "orderItems": [ { "orderItemSeqId": "10001", "productIdentifications": [ { "productIdentificationTypeId": "SKU", "idValue": "98706" }, { "productIdentificationTypeId": "UPCA", "idValue": "98788" }, { "productIdentificationTypeId": "HS_CODE", "idValue": "98790" } ] }, { "orderItemSeqId": "10002", "productIdentifications": [ { "productIdentificationTypeId": "SKU", "idValue": "98690" }, { "productIdentificationTypeId": "UPCA", "idValue": "98698" }, { "productIdentificationTypeId": "HS_CODE", "idValue": "98999" } ] } ] }]Requirement- Prepare two new fields for each item of an order.1. Style: idValue of productIdentification wh

we have Hadoop cluster with active/stand by resource manager services the active resource manager is on master1 machine and the stand by resource manager is on master2 machinein our cluster YARN service that include both resource manager services is managing 276 node manager component on workers machinesfrom Ambari WEB UI alerts ( Alerts for Resource Manager ) we notice about the followingResource Manager Web UIConnection failed to http://master2.jupiter.com:8088(timed out)we start to debug the issue by wget with port 8088 , and we found that process is hang on - HTTP request sent, `awaiting response... No data received`.example from resource manager machinewget --debug http://master2.jupiter.com:8088DEBUG output created by Wget 1.14 on Linux-gnu.URI encoding = ‘UTF-8’Converted file name 'index.html' (UTF-8) -> 'index.html' (UTF-8)Converted file name 'index.html' (UTF-8) -> 'index.html' (UTF-8)--2024-02-21 10:13:42-- http://master2` .jupiter.com:8088/Resolving master2.jupiter.com

I read the article But I can't understand a few things1. I havea Hadoop (3.1.3)MIT KDC cluster on a separate serverActive Directory forserver users with hadoop is not included in the domain2. All hadoop NN, DN, JN, etc. services are launched using the keytabs created for them and are working successfully3. any requests from the user from AD fall with an error2024-01-27 15:56:07,949 DEBUG security.SaslRpcClient: Sending sasl message state: NEGOTIATE2024-01-27 15:56:07,950 DEBUG security.SaslRpcClient: Get token info proto:interface org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB info:@org.apache.hadoop.security.token.TokenInfo(value=class org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenSelector)2024-01-27 15:56:07,951 DEBUG security.SaslRpcClient: tokens aren't supported for this protocol or user doesn't have one2024-01-27 15:56:07,951 DEBUG security.SaslRpcClient: Get kerberos info proto:interface org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolP

[root@cdp1~]# kinit hdfsPassword for hdfs@HADOOP.COM:********[root@cdp1~]# klistTicket cache: KCM:0:86966Default principal: hdfs@HADOOP.COM Valid starting Expires Service principal2022-02-11T01:52:21 2022-02-12T01:52:21 krbtgt/HADOOP.COM@HADOOP.COMrenew until 2022-02-18T01:52:21 [root@cdp1 ~]# hdfs dfs -ls /22/02/11 01:53:31 DEBUG util.Shell: setsid exited with exit code 022/02/11 01:53:31 DEBUG conf.Configuration: parsing URL jar:file:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/hadoop/hadoop-common-3.1.1.7.1.7.0-551.jar!/core-default.xml22/02/11 01:53:31 DEBUG conf.Configuration: parsing input stream sun.net.www.protocol.jar.JarURLConnection$JarURLInputStream@4678c73022/02/11 01:53:32 DEBUG conf.Configuration: parsing URL file:/etc/hadoop/conf.cloudera.yarn/core-site.xml22/02/11 01:53:32 DEBUG conf.Configuration: parsing input stream java.io.BufferedInputStream@369f73a222/02/11 01:53:32 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate or

Hello,I recently installed Nifi on a Mac with an M1 chip, Sonoma 14.4.1. I am able to run "nifi.sh run" successfully and navigate to localhost port 8443/nifi. However, when I run "nifi.sh start" it fails to open localhost 8443/nifi. The logs do not throw any errors, I have checked (see screenshot below). I am running openjdk 21.0.2 and nifi 2.0.0-M2. My environment variables for java (/Library/Java/JavaVirtualMachines/jdk-21.0.2.jdk/Contents/Home), nifi (/Users/RZO4145/nifi-2.0.0-M2), and apache maven have been set.I have also checked when doing the nifi.sh start, it looks like there is nothing listening on 8443 (and I checked nifi.properties, it is still set to 8443). I've also tried setting the host to 0.0.0.0 instead of 127.0.0.1 and changing the port. After about 5-10 minutes of running, I will finally be able to see it on localhost 8443 (see attached), but the page will still not load.Is there something I'm missing?

hellomy cluster is cdh5.14   centos7.4   JAVA7We have been receiving alerts for Impala. The health test result for IMPALAD_QUERY_MONITORING_STATUS has become bad: There are 0 error(s) seen monitoring executing queries, and 1 errors(s) seen monitoring completed queries for this role in the previous 5 minute(s). Critical threshold: any.The following picture shows cloudera-scm-agent error logshow to solve this error? 

Hi Everyone,I am trying to perform some data transformations using UpdateRecord processor.The input format is a nested JSON, and I am able to perform shifting of some fields using RecordPath.But facing issues when preparing a field using some conditional logic from a nested list.Here is my scenario:Input JSON[ { "orderId": "1234", "orderName": "Order 1", "orderItems": [ { "orderItemSeqId": "01", "itemStatusId": "ITEM_APPROVED", "orderItemQuantity": 1, "unitPrice": 149, "goodIdentifications": [ { "goodIdentificationTypeId": "SHOPIFY_PROD_ID", "idValue": "43623074824370" }, { "goodIdentificationTypeId": "SKU", "idValue": "2510524-4BB-S" }, { "goodIdentificationTypeId": "UPCA", "idValue": "2510524-4BB-S" } ] }, { "orderItemSeqId": "02", "itemStatusId": "ITEM_APPROVED", "orderItemQuant

I use the PutIcebergProcessor to write data to my data lake. Therefore I need to specify a HiveCatalogService. This Service needs HadoopConfigurationResources. This parameter is a path to an xml file containing the credentials to the S3 where the Iceberg files are stored.My Problem with this, that some content of this file is supposed to be secret to the users interacting with the nifi ui. However, as soon as a UI user knows this path, he can simply use ExecuteProcess Processor to retrieve those information. Is there any way to keep those information safe? Reference:HiveCatalogService PutIceberg 

Hello All,We have installed the Cloudera ODBC driver of version 2.6.11 and configured ODBC driver. Making use of impala ODBC as a data source and custom query in Power BI desktop, developed a report and published it into the power bi service. But when i refreshed the dataset to get the updated data in the Power BI report. We are getting below error, can anyone help me to troubleshoot the refresh issue.Error message 1{"error":{"code":"DM_GWPipeline_Gateway_MashupDataAccessError","pbi.error":{"code":"DM_GWPipeline_Gateway_MashupDataAccessError","parameters":{},"details":[{"code":"DM_ErrorDetailNameCode_UnderlyingErrorCode","detail":{"type":1,"value":"-2147467259"}},{"code":"DM_ErrorDetailNameCode_UnderlyingErrorMessage","detail":{"type":1,"value":"<ccon>ODBC: ERROR [08S01] [Cloudera][ImpalaODBC] (450) Error when calling the Impala Thrift API ExecuteStatement: SSL_read: error code: 0</ccon>"}},{"code":"DM_ErrorDetailNameCode_UnderlyingHResult","detail":{"type":1,"value":"

Can someone tell me how to use GETSFTP processor?For the hostname should we provide value like sftp://host or hostI am not able to fetch the files from SFTP.I have provided hostname, Port, Username, Password, Remote path.For remote path, I am following this pattern : /folder/subfolder/