Q&A

We have newly install CDP-DC 7.1.7 and try to load data using a spark jobs but getting the errors.----Payloadjava -cp /etc/hadoop/conf.cloudera.hdfs/ssl-client.xml:/etc/hbase/conf.cloudera.hbase/hbase-site.xml:/etc/hadoop/conf.cloudera.hdfs/core-site.xml:/etc/hadoop/conf.cloudera.hdfs/hdfs-site.xm:/data/scripts/LeaApp-1.0-SNAPSHOT.jar net.ba.lea.transformation.FileActions "/data/scripts/msc/IN/" "/tmp/nss_processing/" "/data/scripts/msc/reject/" "250" "LEA.DBM_CDR_FILE_HEAD" "NSS" "jdbc:phoenix:gzvlcdpnode01.ba.net:2181:/hbase:phoenix/gzvlcdpnode02@BA.NET:/etc/security/keytab/phoenix.keytab"error :Can't find method newStub in org.apache.phoenix.coprocessor.generated.MetaDataProtos$MetaDataService!--------------------------------But the same code is working fine with CDP-DC 7.1.3.We have changed  the version only.Please guide us .

Hello all,I am using Cloudera Datasservices on CDP Private Cloud. I am using using  the embedded container service (ECS).If i check the kubernetes pods i see that the pods of the namespace shared-services are always crashing (Expect of openshift-idling-controller-manager) kubectl get pods -n shared-servicesNAME READY STATUS RESTARTS AGElog-router-fznw2 0/2 Init:CrashLoopBackOff 17 46mlog-router-jnjrb 0/2 Init:CrashLoopBackOff 17 46mlog-router-tptbg 0/2 Init:CrashLoopBackOff 17 46mopenshift-idling-controller-manager-0 1/1 Running 0 45m If i query the logs of the pod I get the following error:kubectl logs -f -n shared-services log-router-jnjrb sdx-init-container I have already tried to delete the pods but without success.But the behaviour is unchanged:Who can help?Thanks in advance.Best regardsUli  {"timestamp":"2022-10-07 07:35:52.120 +0000","level":"WARN","thread":"Thread-8","logger":"c.cloudera.cdp.config.ConfigTemplate","message":"[2022-10-07 07:35:52.12

Hello Guys! I updated HDP from 3.1.0.0-78 to 3.1.5.6189-1 as well as Ambari from 2.7.3 to 2.7.5.I want to remove the old stack of HDP as the files are still on the servers.Is there any official guide for this move? 

I have cluster 1 (the source cluster in this example) which does not have HA enabled, and I have cluster 2 (destination cluster in this example) which does have HA enabled.  Both clusters are running the same version of HDP.  When I attempt to copy data from cluster 1 to cluster 2 via distcp using the active name node instead of the HA nameservice, it works but ultimately causes problems when the active name node switches.  I have tried using the HA nameservice, but without any configurations, it fails (as one would expect).  When modifying the hdfs-site.xml to add cluster 2's nameservice information (following this guide laid out here but only for one HA cluster: https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/scaling-namespaces/topics/hdfs-distcp-between-ha-clusters.html), the HDFS services on cluster 1 fail to restart, as the name nodes appear to be attempting to join cluster 2.  Is this sort of configuration even possible, or do we need HA on both cl

We want to configure listenhttp processor that listen our web service. question is what are the configuration in http listener processor in nifi such that when we will make curl post request this ListenHttp processor will catch the update. basically i didn't understand what i have to put in Base path (is it i have to put nifi address i.e http://localhost:8080/contentListener or our web app service URL???) what is contentListener. And is it i have to put our web app port number in listening port? in the request command what i have to put in url part curl --request POST 'http://(here i have to used the nifi url)' -H 'Authorization: Key demo-key i know this is very basic question but i'm just using this don't know any thing about nifi and curl command .....thanks in advance

Hi all, this may be an easy question, but just wanted to verify.  When adding new brokers to the cluster, after installing the Kafka role/instance on the new hosts, should you start the new broker instances first before you restart the existing brokers?  Or does it matter?  This is CDP Private Cloud 7.1.7Thanks

Hello everyone,  I am trying to run a python script in a dockerized enviornment using spark and yarn, the cluster is kerberised and I provided the needed realm and keytab for the spark-submit command. I still face this issue where java only sees user null for some reason. What I tried :   spark-submit \--master yarn \--deploy-mode cluster \--conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \--conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=MyImage \--conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS="/etc/passwd:/etc/passwd:ro,/etc/hadoop:/etc/hadoop:ro,/opt/cloudera/parcels/:/opt/cloudera/parcels/:ro,/etc/krb5.conf:/etc/krb5.conf:ro" \--conf spark.executorEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \--conf spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=MyImage \--conf spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS="/etc/passwd:/etc/passwd:ro,/etc/hadoop:/etc/hadoop:ro,/opt/cloudera/parcels/:/o

I have a hive table with three columns, delimited by tabshive (database)> describe formatted my_table;# col_name data_type commentfield1 string field2 string field3 string... ...Storage Desc Params: input.regex (\\S+)\\s+(\\S+)\\s+(.*)hive (database)> select * from my_table limit 2;DataX DataX DataXDataX DataX DataXI want to add two new columns to the hive table schema, so I did:// add two columnshive (database)> alter table my_table add columns(field4 string,field5 string);// add two more (\\S+)\\s+ for the new columnshive (database)> alter table my_table SET SERDEPROPERTIES ("input.regex" = "(\\S+)\\s+(\\S+)\\s+(\\S+)\\s+(\\S+)\\s+(.*)");But seems it doesn't work with the old dataset (the old data file with three columns):hive (database)> select * from my_table limit 2;NULL NULL NULL NULL NULLNULL NULL NULL NULL NULLI expected something like this:DataX DataX DataX NULL NULLDataX DataX DataX NULL NULL&nbsp

Is there any other way to migrate the Hortonworks HDP3  cluster to the Apache Hadoop 3.3.2  other than the distcp?is it possible to upgrade by updating the jar and pointing the older metadata to the newer stack of Hadoop? Thanks

Is CDP 7.1.7 SP1 a stable release? Is it recommended upgrade to this version? We still see some log4j 1X findings in this version so we need to now if someone has faced the same issue.

Hi Team, We have 2 Hive Metastore server services configured on cluster in node1 and node2. In node1 Hive Metastore getting alert with below message. "Hive Metastore Canary" The Hive Metastore canary failed to create a database. I am seeing below errors in servicemonitor log file. 2022-10-12 22:40:06,657 WARN com.cloudera.cmf.cdh6client.hive.MetastoreClientImpl: (2 skipped) Could not drop hive database: cloudera_manager_metastore_canary_test_db_hive_HIVEMETASTORE_0137966f79e5f15b3b5d4dec61b7592ecom.cloudera.cdh6client.hive.shaded.org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed outat com.cloudera.cdh6client.hive.shaded.org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)at com.cloudera.cdh6client.hive.shaded.org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)at com.cloudera.cdh6client.hive.shaded.org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)at c

Hello everyone.i need help. I have a JSON file which contains in an object either an array of other objects or a single object. This is the relevant object "Product": [ or "Product": { .How can I configure with the JOLT Transform processor to always expect an array in the object? The product object must always be treated as an array, even if it contains only a single object. { "costc": 9638, "sum_amount": 543, "sum_invitation": 0, "sum_selfconsumption": 0, "sum_loss": 0, "sum_gross": "2764,000", "sum_net": "2392,480", "sum_margin": "2003,780", "sum_costs_total": "388,700", "sum_costs": "388,700", "sum_costs_invitation": 0, "sum_costs_selfconsumption": 0, "sum_costs_loss": 0, "sum_costs_total_percent": "16,25", "sum_costs_percent": "16,25", "Classes": { "ClassSummary": [ { "class": 1102, "sum_amount": 117, "sum_invitation": 0, "sum_selfconsumption": 0, "sum_loss": 0, "sum_gross": "740,500", "sum_net": "692,060

Running shell actions(Only sleep command) in Oozie workflows for more than 7 days throws JA017 error on exit.We saw this problem in a shell script that submits a spark job.Spark job succeeded but submit shell action failed.We tested by creating a shell action that does a simple sleep and we see the same failure.https://blog.cloudera.com/hadoop-delegation-tokens-explained/I saw this blog and tried to increase max-lifetimedfs.namenode.delegation.token.max-lifetime (hdfs-site.xml)restart NameNode, Oozie, YARN ResourceManagerbut the same fails. If I run the Oozie workflow and look at the logs, I see that I get 3 delegation tokens.RM_DELEGATION_TOKEN, MR_DELEGATION_TOKEN, HDFS_DELEGATION_TOKENBecause the dfs.namenode.delegation.token.max-lifetime setting value is increased, the maxDate of HDFS_DELEGATION_TOKEN is increased.The maxDate of RM_DELEGATION_TOKEN and MR_DELEGATION_TOKEN tokens did not increase.Why does authentication problem occur due to expiration of delegation token when O

Hi, In BDR HDFS replication I want to exclude all files which end with ".tmp" in the directory "/backups/"   Appreciate it if somebody could give the expression for this to add in BDR "Add Exclusion".    Thanks

Hi , We upgraded OS of one of the server in cluster from RHEL 7.5 to 7.9. After that we are unable to start cloudera services from webui, getting below errors in agent log file. can anyone help me to get rid of this issue. AttributeError: 'NoneType' object has no attribute 'get'[30/Sep/2022 19:37:38 +0000] 1048 MainThread agent WARNING Long HB processing time: 7.00362992287[30/Sep/2022 19:38:01 +0000] 1048 DnsResolutionMonitor throttling_logger INFO DnsTest not running. Java not located.[30/Sep/2022 19:38:37 +0000] 1048 MonitorDaemon-Reporter firehoses INFO Creating a connection to the ACTIVITYMONITOR.[30/Sep/2022 19:38:37 +0000] 1048 MonitorDaemon-Reporter firehoses INFO Creating a connection to the SERVICEMONITOR.[30/Sep/2022 19:38:37 +0000] 1048 MonitorDaemon-Reporter firehoses INFO Creating a connection to the HOSTMONITOR.[30/Sep/2022 19:38:37 +0000] 1048 MonitorDaemon-Reporter throttling_logger ERROR Error sending messages to firehose: mgmt-HOSTMONITOR-d85e01b86fca15e352

hello cloudera community we are having a problem with zookeeper on cdp after enabling kerberos on the cluster zookeeper instances are healthy (status green), but the general status of zookeeper shows the message: Canary test of client connection to ZooKeeper and execution of basic operations succeeded though a session could not be established with one or more servers  how can we solve this problem? 

Hi All,I am trying to run a Spark job using a python script in CDE cluster. The script is from a CDE tutorial.The job fails due to the error message in log files (in Subject). Appreciate any inputs.Thanks in advance .... Sekhar   

Hello, if the autoDeletePeriodSeconds is set to 86400, will that trigger the deletion at midnight? We are not seeing the index getting deleted and our cluster is down nightly, so just curious. Also if I donwconfig, edit this setting and upconfig, I am seeing that configuration is getting reverted next day. Not sure why it is getting reverted.Any suggestions? sdl

i got a bunch of files with differert headers for files of one type,their may be two kinds of headers,i.e. the following two type(call it type1 and type 2) of files's headerstype1: a,b,c34,d     or     a,b,c,dtype2: d,e,f,kgg      or     d,e,f,gnotice that the third field names of type1 are c or c34,the forth field names of type2 are kgg or g,the field name c34 and kgg are wrong, and need to be changed to c and gi want to change the headers only(that is, the first line) using  kv pairs, key is the string to be replaced and value is the  replacement value ,such as {"c34":"c""kgg":"g""hqq":"h"....}i want to do this using  replacetext but the processor replacetext could only set one search and  replacement value,that's the problem!!any ideas would be welcome 

In my flow file I have JSON content which contains an array. How can I extract the size of that array into an attribute (e.g. my.array.size = 5)?

I need to use wildcard syntax in a JSON path to return the color of the bike as a scalar value.  However, the problem is that when I use wildcard syntax with the EvaluateJSONPath processor it always returns an array.  For exampleThis path returns a scalar value:     $.store.bicycle.color   ==>  redThis path returns an array:              $..bicycle.color         ==> ['red']         Is it possible to use the EvaluateJSONPath processor with wildcards and get a scalar value in return? Or, is it possible to extract the value from the array without using an additional processor? { "store": { "bicycle": { "color": "red", "price": 19.95 } }, "expensive": 10}    

Hello Supporters, I am trying to use NIFI CLI toolkit so that I can load 100 parameters using parameter context. (doing it using UI is a tedious job and not appropriate for deployments as well) I installed Nifi toolkit on remote server. I generally ssh into this server from local and run my commands.To see the UI, I use ssh tunnel. There is only one user for this Nifi Instance (admin) and it don't have https certificates. (Single User Authentication) When i start CLI prompt it works fine (below) # sh cli.shCLI v1.16.3Type 'help' to see a list of available commands, use tab to auto-complete.Session loaded from /root/.nifi-cli.config#>  But when I use any Nifi commands, it throws below error: #> nifi list-param-contexts -u http://localhost:8443/ERROR: Error executing command 'list-param-contexts' : Unexpected end of file from server if I use https , #> nifi list-param-contexts -u https://localhost:8443/ERROR: Error executing comm

 Can someone explain why the compilation time is so big and also why we don't have other info in the dag timeline?Also if you can explain what exactly happens when the dag is build, submitted and is running.

Hi guys,hope anyone can point me on the right direction.I have the following flow file: [ { "version" : "5", "type" : "REPORT", "dp_code" : "DHA592", "field4" : "83482227299911", "field5" : "83482227299911", "platform_name" : "GHESTEM LA CHAPELLE", "field7" : "01143200", "field8" : "72801371220938", "field9" : "72801371220938", "carrier_name" : "", "shipper_name" : "", "field12" : "", "consignee_name" : "", "consignee_code" : "", "consignee_address" : "", "consignee_city" : "", "field17" : "", "ldc_number" : "", "ldc_date" : "", "shipment_number" : "2022115144", "shipment_code" : "2022115144/SH", "field22" : "", "field23" : "", "field24" : "", "field25" : "liv.cfm", "situation_code" : "LIV", "justification_code" : "CFM", "situation_date" : "20221003", "feedback_date" : "202210061222", "anomaly_packages_number" : "", "comments" : "", "new_appointment_date" : "", "field33" : "", "field34" : "", "field35" : "", "assigned" : "", "field37" : "", "a

Version: NiFi 1.11.x Noticed a behavior where for this version of NiFi, when the remote url is a GSLB, it doesn't recognize when a host is removed from a GSLB until the processor is stopped. Any idea why? Any ideas for a solution?