Solved

Hi , We upgraded OS of one of the server in cluster from RHEL 7.5 to 7.9. After that we are unable to start cloudera services from webui, getting below errors in agent log file. can anyone help me to get rid of this issue. AttributeError: 'NoneType' object has no attribute 'get'[30/Sep/2022 19:37:38 +0000] 1048 MainThread agent WARNING Long HB processing time: 7.00362992287[30/Sep/2022 19:38:01 +0000] 1048 DnsResolutionMonitor throttling_logger INFO DnsTest not running. Java not located.[30/Sep/2022 19:38:37 +0000] 1048 MonitorDaemon-Reporter firehoses INFO Creating a connection to the ACTIVITYMONITOR.[30/Sep/2022 19:38:37 +0000] 1048 MonitorDaemon-Reporter firehoses INFO Creating a connection to the SERVICEMONITOR.[30/Sep/2022 19:38:37 +0000] 1048 MonitorDaemon-Reporter firehoses INFO Creating a connection to the HOSTMONITOR.[30/Sep/2022 19:38:37 +0000] 1048 MonitorDaemon-Reporter throttling_logger ERROR Error sending messages to firehose: mgmt-HOSTMONITOR-d85e01b86fca15e352

hello cloudera community we are having a problem with zookeeper on cdp after enabling kerberos on the cluster zookeeper instances are healthy (status green), but the general status of zookeeper shows the message: Canary test of client connection to ZooKeeper and execution of basic operations succeeded though a session could not be established with one or more servers  how can we solve this problem? 

Hi All,I am trying to run a Spark job using a python script in CDE cluster. The script is from a CDE tutorial.The job fails due to the error message in log files (in Subject). Appreciate any inputs.Thanks in advance .... Sekhar   

Hello, if the autoDeletePeriodSeconds is set to 86400, will that trigger the deletion at midnight? We are not seeing the index getting deleted and our cluster is down nightly, so just curious. Also if I donwconfig, edit this setting and upconfig, I am seeing that configuration is getting reverted next day. Not sure why it is getting reverted.Any suggestions? sdl

i got a bunch of files with differert headers for files of one type,their may be two kinds of headers,i.e. the following two type(call it type1 and type 2) of files's headerstype1: a,b,c34,d     or     a,b,c,dtype2: d,e,f,kgg      or     d,e,f,gnotice that the third field names of type1 are c or c34,the forth field names of type2 are kgg or g,the field name c34 and kgg are wrong, and need to be changed to c and gi want to change the headers only(that is, the first line) using  kv pairs, key is the string to be replaced and value is the  replacement value ,such as {"c34":"c""kgg":"g""hqq":"h"....}i want to do this using  replacetext but the processor replacetext could only set one search and  replacement value,that's the problem!!any ideas would be welcome 

In my flow file I have JSON content which contains an array. How can I extract the size of that array into an attribute (e.g. my.array.size = 5)?

I need to use wildcard syntax in a JSON path to return the color of the bike as a scalar value.  However, the problem is that when I use wildcard syntax with the EvaluateJSONPath processor it always returns an array.  For exampleThis path returns a scalar value:     $.store.bicycle.color   ==>  redThis path returns an array:              $..bicycle.color         ==> ['red']         Is it possible to use the EvaluateJSONPath processor with wildcards and get a scalar value in return? Or, is it possible to extract the value from the array without using an additional processor? { "store": { "bicycle": { "color": "red", "price": 19.95 } }, "expensive": 10}    

Hi guys,hope anyone can point me on the right direction.I have the following flow file: [ { "version" : "5", "type" : "REPORT", "dp_code" : "DHA592", "field4" : "83482227299911", "field5" : "83482227299911", "platform_name" : "GHESTEM LA CHAPELLE", "field7" : "01143200", "field8" : "72801371220938", "field9" : "72801371220938", "carrier_name" : "", "shipper_name" : "", "field12" : "", "consignee_name" : "", "consignee_code" : "", "consignee_address" : "", "consignee_city" : "", "field17" : "", "ldc_number" : "", "ldc_date" : "", "shipment_number" : "2022115144", "shipment_code" : "2022115144/SH", "field22" : "", "field23" : "", "field24" : "", "field25" : "liv.cfm", "situation_code" : "LIV", "justification_code" : "CFM", "situation_date" : "20221003", "feedback_date" : "202210061222", "anomaly_packages_number" : "", "comments" : "", "new_appointment_date" : "", "field33" : "", "field34" : "", "field35" : "", "assigned" : "", "field37" : "", "a

Hi Team, Am using self signed certificates, for API calling but while import the configuration passwords are not stick with the properties, can someone please suggest a method, How we can add truststore and keystore password using NiFi CLi or via API call to SSL ContextService.   

Hbase is failing with below error . error Connection failed: [Errno 111] Connection refused to master1:16000

We have configured both NiFi Settings Maximum Timer Driven to 80 & Maximum Event DrivenThread Count to 120. We have a 4 node cluster with 64 cores. On diagnostics we see only average usage of 5 cores. Additionally, the data is queuing up and processing slowly.Issue is The CPU utilization does not go beyond some percentage. What further configurations can help utilize the CPUs and increase the computation power.

Hi everybody,I need to install CDH at least 5.8 ( Virtual Box) and always face with this error: 403 Forbidden (varnish) the provided credentials were incorrectKnow any how to fix it? Many thanks,Mehdi

I have registered AD in Ranger Usersync and want to sync deleted user in ad. so, I am following the guide on the link (https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/security-ranger-authentication-unix-ldap-ad/topics/security-ranger-authentication-usersync-deleted-users.html) However, even if I delete the account from AD, the account does not change to the hidden state in Ranger. Is there anything else I need to configure besides what's in the document?

Hello there, I have flowfiles with json content, which I transform before using them to send a post request with InvokeHTTP.I have an attribute "url_value" which I can use to determine my "api_value". For now I use EvaluateJsonPath to add those values to the flowfile attributes, then I use UpdateAttribute where I have set some rules to get the correct api_value.Now my question: how do I get the value back into my content? If I use AttributesToJSON, it overrides all previous content, therefore I need to have extracted all necessary attributes with EvaluateJsonPath before. This is not a very elegant solution since there can be many entries and I actually only need to update one of them!Is there no processor similar to UpdateAttribute but for flowfile content? How could I use JoltTransformJSON to change a value depending on another attribute? Edit:I would need something like if url_value:contains("webpage1") set api_value to "wepage1"elseif url_value:contains("webpage2") set api

Most of the same Impala queries respond within 20 seconds. But sometimes it takes more than 60 seconds.I see a hotspot in 'SCAN HDFS' of 'ExecSummary' in slow query profile. (avgTime 6s462ms, maxTime 54s766ms ) low response time Query Info: - Duration : 13.1s - HDFS Scanner Average Read Throughput: 382.2MiB/s - Threads: Network Receive Wait Time: 10.0m high response time Query Info: - Duration : 1m, 4s - HDFS Scanner Average Read Throughput : 988.1 MiB/s - Threads: Network Receive Wait Time: 59.5m If HDFS SCAN is the cause of the slow response time, is the 'HDFS Scanner Average Read Throughput' value higher in the slow query? And one more question.I compared the fast and slow operators of the HDFS_SCAN_NODE stage in the slow query profile.RowBatchQueueGetWaitTime(54.76s) and ScannerThreadsTotalWallClockTime(54.77s) I wonder why the two metric values are high. HDFS_SCAN_NODE (id=0)Hdfs split stats (<volume id>:<# splits>

Good Morning I'm trying to access an address of another Linux server through Nifi that is also installed on a Linux server, to search for files. I'm trying to use the FetchFile Processor for this but I'm not getting the error "Routing to not found". Could you help me how to make this connection between one server and another?

Hi, We're experiencing a peculiar issue these past few days - our python scripts started failing when run with spark-submit. On first glance our logs show that the scripts are encountering syntax errors whenever we are using code related to modules, but further troubleshooting showed that in actuality the modules are the issue. When we comment out pieces of code that throw syntax errors, we instead receive import errors ("No module named..."). We have two versions of python on our cluster but it appears that spark-submit still is using the proper python version with all our modules installed on it. Our scripts run just fine through pyspark, for some reason however, spark-submit does not recognize the imported modules when we run scripts through it. What is more, YARN doesn't seem to recognize these jobs as failed, they are not logged at all, probably because they crash as soon as we start importing modules. So basically we do not have access to YARN logs for these jobs.&

How can I use the output of the Base64EncodeContent (or the Base64Encode function) as input for an XML template, where the flow file content (not an attribute) is the XML with the encoded value included?  There doesn't seem to be a NiFi reader that reads the encoded flow file content, so I'm not sure how best to use this processor aside from using a couple of ReplaceText processors to bracket XML substrings around it.I appreciate any input.  Thanks for reading.example:Initial Flow File: "The quick brown fox jumped over the lazy dog"Encoded Flow File: VGhlIHF1aWNrIGJyb3duIGZveCBqdW1wZWQgb3ZlciB0aGUgbGF6eSBkb2c= <?xml version="1.0" encoding="UTF-8" ?><a xmlns="some/name/space/goes/here"> <b>This is a template</b> <c>ENCODED DATA GOES HERE</c></a>  

Hello,on our system CDP 7.1.7, we have use konfiguration parameterkafka.properties_role_safety_valveadd set attributesasl.kerberos.principal.to.local.rulesto map ActiveDirectory principals to entities created in ranger.In our system, the AD user have a prefix e.g. xjohndoe@SAMPLE.COM maps to a ranger entity "johndoe"During a spark-submit (over yarn), we also need to pass a principal, however as there is no such mapping, we obtain an error saying the unix user "xjohndoe" does not exist. This is true indeed, we eed to map it to "johndoe". Ist there any possibility to map principals during spark-spark-submit possibly similarly to sasl.kerberos.principal.to.local.rules in kafka or any other possibility?Best regardsJaro

Hi peopleI am working with HttpInvoke processor with Post and I try to catch the response body using the propertie "Put Response Body in attribute", the processor execute well and it generate the response on the queue Response but it dosent generate the attribute. Someone has some idea where is my error?

I am trying to set partition retention times on existing Hive manged tables using the following:  ALTER TABLE <table name> SET TBLPROPERTIES ('discover.partitions'='true');ALTER TABLE <table name> SET TBLPROPERTIES ('partition.retention.period'='1d'); as stated on this page below, however I am still able to search partitions older than a day so it appears to not be working? It does mention on the page that this is for 'external' tables, can anyone let me know if this an 'age off' retention period is possible on managed tables? am I missing any commands etc? https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.4/using-hiveql/content/hive-set-partition-retention.html Thanks in advance

Hello,I have setup the Hadoop cluster on single server ( on my laptop). I have uploaded the simple text file data into Hive.I am running the select query :Select * from ratings;but this simple query is not giving me any result even after 10 minutes.All services of Hive are running.Status of Ambari :   my query : What should I do ? Thanks 

Hi everyone,I've trying to monitoring nifi api. In our nifi we have configure oidc login and it works fine.As monitoring, we using zabbix.The problem in zabbix is when try to check api page (https://nificluster.info/nifi-api/flow/cluster/summary) and I receive http 401 error. In the web scenario monitoring, I have configured 2 steps: connect to https://nificluster.info/nifi/  with login and password. This steps it's ok, the required string - "log out" - and required status code 200 works. In this step I've configured viariables username, password for logging and follow redirects flag enable.connect to api https://nificluster.info/nifi-api/flow/cluster/summary without any parameters except requird string with number of cluster nodes and required status code 200.Who could help me resolve this 401 error on the api nifi? 

Is there a size limit to the tar file generated by MergeContent?ERRORMergeContent[id=8a06a034-0183-1000-b8b5-c045959a56e9] Failed to process bundle of 2 files due to entry size '14743306287' is too big ( > 8589934591 ).; rolling back sessions: java.lang.RuntimeException: entry size '14743306287' is too big ( > 8589934591 ).   

hello cloudera community,   we are having problems using livy with job spark to read hive by jupyter notebook   when we run a simple query, for example:   "spark.sql("show show databases").show()"   returns the error below"org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient"   Could you help us with this setup?   ps: we are using cdh 5.16.x