Q&A

Hello,We are observing Impala CatalogException: Error refreshing metadata for table due to lock contention errors on our Cluster.Came across this article - https://community.cloudera.com/t5/Support-Questions/Impala-Error-updating-the-catalog-due-to-lock-contention/td-p/329310 upon checking the suggestions made by Cloudera Tech - load_catalog_in_background is already unchecked nor do we find any error for JVM Pause in Catalogue LogsJava Heap Size of Catalog Server in Bytes = 28GCDH 6.3.3Would appreciate any help in understanding why this happens and ways to fix this.

So I am trying to implement this script  from datetime import datetime, timedelta, date flowfile = session.get()current_date= datetime.today().date()if flowfile != None: src_table_name = flowfile.getAttribute('src_table_name') date_filter = str(current_date.replace(day =11)) if src_table_name in ['Income_project_BP', 'Plan_final_scenario', 'Plan_primary_scenario', 'ProductionCalendar', 'Subcontracting_BP']: flowfile = session.putAttribute(flowfile, 'date_filter', date_filter) session.transfer(flowfile, REL_SUCCESS) else: session.transfer(flowfile, REL_FAILURE)   And I've tried a lot of different ways but when it comes to "replace" method I'm always getting this error. Same thing when I am trying to use "strftime" method What should I do? I really need this method.Thank you beforehand!

Hello folks! I have a cluster with 9 machines, running on CDH 6.2 (OnPremise). I have 3 master, 1 edge and 5 workers. I am not able to up 2 of 5 NodeManager on workers. 3 of them are ok, and 2 of them give me a follow log (attach), without error but a Warning with "NullPointerException":. When I put the NodeManager to run, on Cloudera Manager it doens't fail, but I got two alerts, as follow:- NodeManager can not connect to ResourceManager- ResourceManager could not connect to Web Server of NodeManager Also, I can't access the /jmx of the server. And, when I run NodeManager by Cloudera Manager, my CPU going to use of 100%. On that 2 workers, I have RegionServer and DataNode working fine, the problem is only with NodeManager. Please, any suggest?     

kudu service has 3 kudu master and 5 kudu tablet among these 5 kudu tablet, 3 of them have high memory usage, 60GB has been set for each kudu tablet and it is using more than 85% to 90% of that memory and with that it ends up causing jobs failure below shows the memory usage behavior:TABLET SERVER: hostname04.local - USE MEMORY: 55.28G / %: 92.13%TABLET SERVER: hostname05.local - USE MEMORY: 54.22G / %: 90.38%TABLET SERVER: hostname06.local - USE MEMORY: 29.05G / %: 48.42%TABLET SERVER: hostname07.local - USE MEMORY: 28.47G / %: 47.46%TABLET SERVER: hostname08.local - USE MEMORY: 54.99G / %: 91.66% memory usage graph of all kudu tablet in cloudera manager: would like to understand why kudu is having this behavior and how can we reduce this memory usage? PS: kudu service is installed and being managed by cloudera manager express in version 5.16.2 and kudu service is in version 1.7

Good morning everyone,   I have a question regarding the Failure of the ExecuteSQL processor.   I would like to know if when accusing a failure in this processor I can give a Retry in the Processor with its own Failure.   Thanks

Working on a wait-notify setup, but it is running into issues. It is constantly being populated with an "Address already in use" error although there is just the one set of DistributedMapCacheServer and DistributedMapCacheClient controller services. This oddly doesn't immediately cause an issue when these are first created and the wait-notify actually works fine, but on any subsequent runs with the same input file (same Release Signal Identifier) the files do not wait at all and move through immediately.   

Hi, I am facing an issue with one of our Impala Daemon which continuously has error related to “The health test result for IMPALAD_FRONTEND_CONNECTIONS has become bad: There are 0 (Beeswax pool) 123 (Hive Server 2 pool) active client connections, each pool has a configured maximum of 128” Have checked CM -> Impala -> Charts Library -> check "Active Frontend API connection" this shows too many Hive Server connections                 Also checked server's network side and run netstat -an | grep ESTABLISHED | grep 25000 but I see only 2 active connections.   [root@nucleus ~]$ netstat -an | grep ESTABLISHED | grep 25000tcp 0 0 10.20.20.41:25000 10.20.20.41:49728 ESTABLISHED tcp 0 0 10.20.20.41:25000 10.20.20.41:49732 ESTABLISHED   Any assistance / guidance in resolving this issue is appericated.     CM / CDH - 6.3.3 Thanks  Wert.

I am having 5 region servers in my cluster, I have added an additional 5 region servers,can I directly decommission the older 5 region server with the help of ambari server, will it cause any data loss in HBase? 

Hello, Does Cloudera 6.3.3 (CM/CDH) support Oracle 19c ? 

what version can be used as freeware like cloudera express in present CDP after merger as i see current cloudera express is discontinued If there is any freeware or open source still available to use and list of components please. 

I have been trying to download HDP Sandbox for last 1 month but not able to complete it up-to 100%. Its size is 20 GB and I have utilized more than 150 GB still not able to complete this 20 GB download as if fails after some times. While every other providers provide torrent link for such huge file size I don't know why cloudera has not provided.  And if you complain about my internet speed then FYI I tried downloading it from my Company network too.    I am unable to digest that people at Cloudera can't think of this issue and provide some alternative way to download it. I tried searching for alternate way to download it but could not find it.    Here is what you can do to solve this issue: 1. Provide a link that is resume-able. 2. Provide a Torrent link.     No Thanks for wasting my time and bandwidth.   Sam

Hi everyone!I'm trying to flat array and it view as simple case, but I can't reach the goal.Here is simple json input:{ "MainArray": { "mattr1": "some_value1", "mattr2": "some_value2", "Subarray": { "sattr1": 1, "sattr2": 2, "sattr3": 3 } }}Then I would like to convert it to array of arrays [key, value],Here is Desired Output:{ "ResultArray" : [ [ "mattr1", "some_value1" ], [ "mattr2", "some_value2" ], [ "sattr1", 1 ], [ "sattr2", 2 ], [ "sattr1", 3 ] ]} Here is my spec:[ { "operation": "shift", "spec": { "MainArray": { "*": { "$": "ResultArray[#2][0]", "@": "ResultArray[#2][1]" }, "Subarray": { "*": { "$": "ResultArray[#3][#2][0]", "@": "ResultArray[#3][#2][1]" } } } } }] And I don't understand how to fix problem when Subarrays is inside extra array in my result:{ "ResultArray

I wonder where the query history like the image above is saved to!Help ME! 😞

If we are going to decomm a data node on which a region server is also running : 1. first - is that a viable option? Can a data node be decommissioned when a region server is also running on that host? Doesn't the region server have/use data on that data node - if so, then how can the data node be decommissioned? 2. let us suppose to due some severed hardware error the node has to be decommed. Then is it necessary to shift the region server on to another node? Or (if there is no real issue with the hardware etc) can a region server exist on a machine without a data node on it? 3. Is there anything to be done to HBASE/Region Server (maintenance mode etc) when you decommission a data node? Appreciate the insights.

I am facing an issue while starting resource manager using ambari but able to start it using CLI.when starting using ambari no logs is generated in /var/log/haoop-yarn/ location.Please find logs on ambari. Traceback (most recent call last):File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 995, in restartself.status(env)File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/YARN/package/scripts/resourcemanager.py", line 150, in statuscheck_process_status(status_params.resourcemanager_pid_file)File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/check_process_status.py", line 43, in check_process_statusraise ComponentIsNotRunning()ComponentIsNotRunningThe above exception was the cause of the following exception:Traceback (most recent call last):File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/YARN/package/scripts/resourcemanager.py", line 261, in <module>Resourcemanager().execute()File "/usr/lib/ambari-agent/li

Hello, I am new here, need some help I Have CDP5 with 2 masters and 3 nodes, one of the masters is trying to get HBase DS tables and schemas, when doing from Workers all good but when trying to get from master it goes to localhost:2181 and not creating a job for Zookeeper workers, at least after 3 tries giving error of refused exception, I don't understand why master behave like that and why I cannot get data from Master, looks zookeeper is not working on it? From Cloudera Manager all the Hosts and instances are green and shown great.SSH connect to the master, or workers the terminal master, $ shell:HBase shelllistdescribe and all the commands working great, but when trying to do connection and get from outside to master getting the error. checked hbase-site.xml and all the zookeeper nodes are there like internal ip hosts. what else I may check?

Hello, I'm trying to learn NIFI so this is all new to me, I used to work with Talend and I have hard time translating to NIFI. So the main idea: For example is I have two tables in Postgresql Table CITY :ID (auto generated), city_nameTable PERSON :ID (auto generated), first_name, last_name, city_id and I have a CSV file :first_name, last_name, city_name Can you please explain how I can insert in tow tables from one flowfile and refer in the table PERSON to the ID of the city not the name from the table CITY.Thank you.

Hi, I've been successfully secure EFM to Keycloak server with oidc auth. But Minifi agent wont show up in EFM Dashboard. I've check both EFM and minifi log but not found any clues. Need your help. Here my conf/efm.properties config:# Web Server TLS Propertiesefm.server.ssl.enabled=trueefm.server.ssl.keyStore=/home/efm/certs/keystore.jksefm.server.ssl.keyStoreType=jksefm.server.ssl.keyStorePassword=ksPasswdefm.server.ssl.keyPassword=ksPasswdefm.server.ssl.trustStore=/home/efm/certs/truststore.jksefm.server.ssl.trustStoreType=jksefm.server.ssl.trustStorePassword=changeitefm.server.ssl.clientAuth=WANT# User Authentication Propertiesefm.security.user.auth.enabled=trueefm.security.user.auth.adminIdentities=adminefm.security.user.auth.autoRegisterNewUsers=trueefm.security.user.auth.authTokenExpiration=12hefm.security.user.certificate.enabled=trueefm.security.user.oidc.enabled=trueefm.security.user.oidc.issuerUri=https://keycloak.domain.com:8443/realms/efmefm.security.user.oidc.clientId=

queries in hue using impala are returning the following error: "results have expired, rerun the query if needed" as shown in the screenshot below:what could be causing this error and how to fix this issue?

When I am tring to deploy new cluster, I am getting the below error. Error for parcel CDH-7.1.7-1.cdh7.1.7.p74.21057765-el7.parcel : Hash verification failed.

I am using NiFi v.1.16.1 When I attempt to retrieve the weather data from the below URL, I don't get any response unless I specify "Always Output Response", and then I get an Access Denied response.  However, these are free data feeds that  work just fine using any other client,e.g. curl https://w1.weather.gov/xml/current_obs/KSFO.xml Anyone have any ideas what may be going on here?  

Hi all, I am hoping I can get some help with this. I have a process that's sending a set of JSON records to a NiFi flow. Each JSON record in the array is a set of criteria/conditions for which I run QueryElasticSearchHTTP to get matching records from ElasticSearch. I am using a Merge Content processor at the end to merge all of this data and send back the JSON array as response. It works fine when for example I send 10 JSON criteria records in the array and QueryElasticSearchHTTP is able to find a document in the index for each criteria it receives. It bundles up the 10 resultant records from Elastic and sends the response through. However, if QueryElasticSearchHTTP finds only 8 records matching the criteria, it then waits forever at the Merge Content processor. I realise this is because it has received 10 records initially and split them into 10 flow files. When it looks to merge, it finds only 8 flow files and is waiting for the remaining 2 which never come through. This bl

Hi,is there any documentation to install R and SparkR in a gateway node of a DataHub?I have CM 7.5.2 and a CDP Public subscription.Spark version currently configured:spark 2.4.8spark 3.1.2 Thank youAndrea

Testing this on our non-prod cluster and finding Cloudera manager won't start after switching from Oracle to Open JDK. Followed the Cloudera documentation and it appears to be working, but after removing Oracle RPM things went kablooey. Anyone have any suggestions? From CM log: cResourcePool$ScatteredAcquireTask@1f258f49 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (5). Last acquisition attempt exception:com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failureThe last packet successfully received from the server was 1 milliseconds ago. The last packet sent successfully to the server was 1 milliseconds ago.at sun.reflect.GeneratedConstructorAccessor61.newInstance(Unknown Source)at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)at java.lang.reflect.Constructor.n

Hi, I'm new in big DATA,I'm working with HDP 2.6.5, Ambari 2.6.2, HBase 1.1.2I would like to make a weekly full backups of all tables of HBase (all regionservers and my HBase master)and daily incremental backups of same thingsNo one has made a backup, so that's reason I am trying to make one.HBase is kerberised. so I did a kinit as hbase userI tried those commands on my hbase master (in local on it) :hbase backup create full hdfs://[FQDN server]:[port]/ -set /etc/hbase/hbase_backup_full/test -w 3goal : backup all tables in the file called test (I've created it so I don't understand why it tells me that's empty, and I've made a chown hbase:hbase on it as well as on the directory hbase_backup_full)issue : 634 ERROR [main] util.AbstractHBaseTool: Error running command-line toolIOException: Backup set '/etc/hbase/hbase_backup_full/test' is either empty or does not existcould anyone help me please ?Do I need to modify some config files to do this ? if yes which ?Thanks 🙂