Q&A

2021-06-09 17:00:54,088 WARN ha.HealthMonitor (HealthMonitor.java:doHealthChecks(210)) - Transport-level exception trying to monitor health of NameNode at NAMENODE/NAMENODE:PORTjava.net.SocketTimeoutException: Call From NAMENODE/NAMENODE to NAMENODE:PORT failed on socket timeout exception: java.net.SocketTimeoutException: 45000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/NAMENODE:PORT2 remote=NAMENODE/NAMENODE:PORT]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout2021-06-09 17:00:54,090 INFO ha.HealthMonitor (HealthMonitor.java:enterState(248)) - Entering state SERVICE_NOT_RESPONDING2021-06-09 17:00:54,090 INFO ha.ZKFailoverController (ZKFailoverController.java:setLastHealthState(893)) - Local service NameNode at NAMENODE/NAMENODE:PORT entered state: SERVICE_NOT_RESPONDING2021-06-09 17:00:54,191 WARN tools.DFSZKFailoverController (DFSZKFailoverController.java:getLocalNNThreadDump(249)) - Can

Is it safe to delete older ranger audit logs?/ranger/audit/hdfs/20200901/hdfs_ranger_audit_xyz.logas my nodes running out of space. 

Hi,   I am getting the below error while accessing a Google Cloud Storage bucket for the first time via Cloudera CDH 6.3.3 Hadoop Cluster. I am running the command on the edge node where Google Cloud SDK is installed. Reachability of Google Storage is only possible via HTTP proxy as of now.   Cloudera CDH 6.3.3 cluster is on-prem.   Below is the command that I run:   hadoop --loglevel trace fs -ls gs://distcppoc-2021-08-09/   Error is:   ls: Error accessing: bucket: distcppoc-2021-08-09   Last few lines when the Hadoop command is run: 21/08/10 21:07:42 DEBUG security.UserGroupInformation: hadoop login commit21/08/10 21:07:42 DEBUG security.UserGroupInformation: using local user:UnixPrincipal: <username>21/08/10 21:07:42 DEBUG security.UserGroupInformation: Using user: "UnixPrincipal: <username>" with name <username>21/08/10 21:07:42 DEBUG security.UserGroupInformation: User entry: "<username>"21/08/10 21:07:42 DEBUG security.Us

Hi all,  I have a scenario where:1. I get a list of flowfiles in Json format2. Using Jolt I extract the fields and send it to an API3. The result of the API needs to find the match in the original json that was ingest in the step one. I've not understood yet what kind of processors I need to implement, I was looking at the LookupRecord, but I haven't so far understood what to do to make the lookup happen in the right flowfile?  Could someone guide me what is necessary to perform it?

Hi,I am getting above error from Hive while trying to query a table.This is coincidentally during an insert operation on the table... does it mean I cant access the table while a hive insert operation is ongoing...? table contains lots of rows partitioned by two columns  

Hi Community, I need you support. I have 3 node-managers and after rebooting 1 node-manager does not start.(RedHat 7.6 / Hadoop 2.6.5 / Ambari 2.5.1.0). Added the full log:https://drive.google.com/file/d/1Osz9nV5PSBAeu8eiiYsJOsib0h3E3Gjk/view?usp=sharing ----------------------------------------------------------FATAL nodemanager.NodeManager (NodeManager.java:initAndStartNodeManager(549)) - Error starting NodeManagerorg.apache.hadoop.yarn.exceptions.YarnRuntimeException: NMWebapps failed to start.at org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer.serviceStart(WebServer.java:116)at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:302)at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initA

Hi,We are currently using Python2.7 with HDP3.1.5 (PySpark) and RHEL 7.9.We are planning to upgrade to Python3.x as Python2.7 support has ended.If anyone could please help determining the latest stable Python3.x? version that will best suit or compatible with HDP3.1.5 (Spark2.3) and RHEL 7.9?ORLet me know steps I should follow to determine the best stable Python3.x? version for the HDP stack? Thanks,PB 

Hello,We are observing our Impala Catalogue Server’s process frequently gets exited / killed (This role encountered 1 unexpected exit(s) in the previous 5 minute(s).This included 1 exit(s) due to OutOfMemory errors. Critical threshold: any.)I was going through this article (https://community.cloudera.com/t5/Support-Questions/Cloudera-6-2-1-Impala-GC-Overhead-limit-Exceeded/td-p/293225) it seems to be Heap Memory related, would like to know if there is any way / calculations to find how much Heap should be allocated to avoid these issues. CM / CDH 5.16.2Java Heap Size of Catalog Server in Bytes = 15Gb Appreciate any guidance in this regard.

Hello,I was trying to take report from HDFS Report and get the below messageUpon checking /var/log/cloudera-scm-headlamp and see the following error  021-04-15 18:09:44,934 ERROR com.cloudera.headlamp.HeadlampIndexManager: Index build failed for service hdfsjava.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Unsupported layout version -64 at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.cloudera.headlamp.AbstractIndexBuilder.buildIndex(AbstractIndexBuilder.java:80) at com.cloudera.headlamp.HeadlampIndex.buildIndex(HeadlampIndex.java:257) at com.cloudera.headlamp.HeadlampIndex.reindex(HeadlampIndex.java:325) at com.cloudera.headlamp.HeadlampIndexManager.reindexIndexes(HeadlampIndexManager.java:240) at com.cloudera.headlamp.HeadlampIndexManager.access$100(HeadlampIndexManager.java

I'm encountering the following error while using beeline.  (Note that I'm able to successfully connect using hive cli. [xxxx@ip-11-1-1-111 data_dir_design]$ beeline -u "jdbc:hive2://ip-11-1-1-111.us-west-1.compute.internal:10000/default;principal=hive/ip-11-1-1-111.us-west-1.compute.internal@EXAMPLE.COM"Connecting to jdbc:hive2://ip-11-1-1-111.us-west-1.compute.internal:10000/default;principal=hive/ip-11-1-1-111.us-west-1.compute.internal@EXAMPLE.COMConnected to: Apache Hive (version 1.2.1000.2.6.5.1175-1)Driver: Hive JDBC (version 1.2.1000.2.6.5.1175-1)Transaction isolation: TRANSACTION_REPEATABLE_READBeeline version 1.2.1000.2.6.5.1175-1 by Apache Hive0: jdbc:hive2://ip-11-1-1-111.us-west-1.compu> show databases;Error: Error while compiling statement: FAILED: HiveAccessControlException Permission denied: user [xxxx] does not have [USE] privilege on [Unknown resource!!] (state=42000,code=40000)0: jdbc:hive2://ip-11-1-1-111.us-west-1.compu> When connecting via hive

Hello Team,   I'm raising this issue on behalf of my colleague(Frank van Heukelom).He is not able to change  company's name in his profile. By mistake during account creation, selected incorrect company's name. Request you to kindly fix it.   Thanks, Ananth.

I have a hadoop server which namenode, datanode , YARN component etc are spnego kerberized and when we hit their url's on web we get as below -url - http://host:50070/jmx?qry=Hadoop:*here 50070 is namenode port output :HTTP ERROR 403Problem accessing /index.html. Reason: GSSException: Defective token detected (Mechanism level: GSSHeader did not above response on web indicates that url is kerberized.  I tried to enable spnego kerberos for ambari with using command "ambari-server setup-security" and completed kerberos configuration.but when hit the url of ambari services rest api then getting proper output - url - http://host:8080/api/v1/clusters/cluster_name/services/HDFS I could see the proper output for above url on web . why both url's output is not same , it should be same if enabled kerberos . please help and thanks in advance!!!

Hi Support Team, I have JSON input for Nifi flow with some special characters. Could someone help me with how to remove special characters following payload? we would need only value with array and double-quotes.input json: - {"TOT_NET_AMT" : "[\"55.00\"]","H_OBJECT" : "File","H_GROSS_AMNT" : "[\"55.00\"]","TOT_TAX_AMT" : "[9.55]"}  Expect Result :  {"TOT_NET_AMT" : "55.00","H_OBJECT" : "File","H_GROSS_AMNT" : "55.00","TOT_TAX_AMT" : "9.55"}

Hello , I have this error when executing select count(*) on Hive on Tez , could you guys help me?      

I had a table that it vanished after an issue. I created again the table and then recover the partitions in order to read again the files that were in hdfs hive direcotry. Although all of a sudden now, it throws me the below error: File 'path/data.0.parq' has an incompatible Parquet schema for column 'db.table.parameter_11'. Column type: STRING, Parquet schema: optional int64 amount [i:10 d:1 r:0]  

Hi Team, I have a scenario if an Participant status  Filed value is 'Inactive' then i should update some of the columns data in the file with NULL/empty. Could you please help me in how can we achieve this? 

我在一个启用了kerberos的CDH6.3集群里，提交一个flink1.9版本程序到yarn，需消费集群外一个0.8版本的未启用安全认证的kafka，需要如何设置他不使用安全的方式去连接这个集群外kafka的zookeeper呢？谢谢。 org.apache.flink.shaded.zookeeper.org.apache.zookeeper.client.ZooKeeperSaslClient - Client will use GSSAPI as SASL mechanism.org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Opening socket connection to server ark1.analysys.xyz/10.2.33.189:2181. Will attempt to SASL-authenticate using Login Context section 'Client'org.apache.flink.shaded.zookeeper.org.apache.zookeeper.client.ZooKeeperSaslClient - An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when evaluating Zookeeper Quorum Member's received SASL token. Zookeeper Client will go to AUTH_FAILED state.

Hello Everyonei am running one YARN job on daily basis ,but today its running little bit slow than yesterday, on the same node other peoples also running the same job but they didn't face any issue regarding the slowness job,only my job is running slow and its a daily running jobwhat could be the possible reason behind this? thanks in advance !!

Hello experts!Trying to add the host to an existing SSL-protected demo CDP 7.1.6 cluster (on IBM Power servers, ppc64le).The "Install Agents" wizard step fails with the error "Installation failed. Failed to copy installation files."When looking into the "Details" link, the following is printed there:/tmp/scm_prepare_node.gvqeyBBtFailed to generate certificates for cldr62.ibmcc.ru/opt/cloudera/cm-agent/bin/certmanager is present on both the Cloudera Manager host and on the machine to be added.Can someone please recommend how to resolve this issue?Cloudera Manager logs do not seem to reveal anything useful: 2021-08-12 16:14:24,928 INFO scm-web-499421:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/add-hosts-wizard/install2021-08-12 16:14:24,936 INFO scm-web-499421:com.cloudera.server.cmf.node.NodeConfiguratorService: Creating request with id 162021-08-12 16:14:24,937 INFO scm-web-499421:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing

Issue summary: In cdh6.2.1，with kerberos and sentry enabled, we are getting issues when using statement "insert overwrite" to insert data into new partitions of parttioned table, the sql statement, and detailed metastore and hiveserver2 logs are attached. Note:1. Event though errors are throwed (and show partions xxx will not dispaly the new partition), the underneath hdfs directory and files for the corresponding partition are created successfully.2. after errors throwed for the "insert overwrite" statement, we can use msck repair tablexxx to fix the hive metastore data for the talbe, and after that, we can use "show partitions" to dispaly the new created partition successfully, and use "select xx" to query the new inserted partition data successfully:3. This happens for both static mode partitioning and dynamic mode partitoning；( as long as you are inserting data to new partitions）4. if you are using insert overwrite to insert data into an existing partion (the partition can be eithe

Hi, I am new to using Cloudera Cloudera Data Visualization tool (public)... wanted to know if anyone knows if it is possible to set only a "From" date filter. What I have is; -Data that predicts outcomes from now to a set number times, eg. predict the next 10 events from now.- The predicted events happen over hours- What I want to do is the ability to set only the FROM date filter and then have the vis return all the values from that point At the moment, I have to set a FROM and TO date. Setting the TO date means I only ever gets result TO that point. Thanks

 I have installed Nifi locally and recieved the below error, could some one please let me know what need to be done.   It would be great help if provided in detailed.thanks!

Hello Team, I have the following file to load on Kudu.1829;BN=0;UNIT=VOLUME_ALL;IN=0;TC=0;TCC=0;CT=;FU=1000001;CU=54274;FB=61701;FL=ugw9811_3828500385_360_271530=5126742111750858;U=23059268534;SI=6;SG=1;SR=7;SN=BROWSING;SC=BROWSING;BS=60342256;BR=2581143;TU=2021-04-27 14:02:47;TF=2021-04-27 00:00:00;TA=2021-04-27 14:02:47;TB=2021-04-27 00:00:00;TE=2021-04-27 14:02:47;TS=1619517767;D=16292;R=151;E=0;UDR_cu=0;UDR_fb=BROWSING;DCM=0;UP=Prepaid;ST=BROWSING;MSISDN=23059268534;APN=orange;SGSN=196.192.13.113;GGSN=196.192.13.113;IMSI=617010014925066;BU1=23292;BN=0;UNIT=VOLUME_ALL;IN=0;TC=0;TCC=62923399;CT=;FU=1000000;CU=3586;FB=61701;FL=ugw9811_3828490275_312_87990=5126752111750858;U=23059268534;SI=6;SG=1;SR=7;SN=BROWSING;SC=BROWSING;BS=0;BR=0;TU=2021-04-27 14:02:47;TF=2021-04-27 00:00:00;TA=2021-04-27 14:02:47;TB=2021-04-27 00:00:00;TE=2021-04-27 14:02:47;TS=1619517767;D=16292;R=151;E=0;UDR_cu=0;UDR_fb=BROWSING;DCM=0;UP=Prepaid;ST=BROWSING;MSISDN=23059268534;APN=orange;SGSN=196.192.13.113

Lock on /hadoopfs/fs1/hdfs/namenode/in_use.lock acquired by nodename 8360@xxxx-080060091.xxxx.com 2021-08-11 06:30:25,079 WARN namenode.FSNamesystem (FSNamesystem.java:loadFromDisk(716)) - Encountered exception loading fsimage java.io.IOException: NameNode is not formatted. When i tried to format the namenode by using hadoop namenode -format getting the below error as, 21/08/11 06:45:12 INFO util.ExitUtil: Exiting with status 1: org.apache.hadoop.hdfs.qjournal.client.QuorumException: Could not format one or more JournalNodes. 2 exceptions thrown:10.80.60.88:8485: Directory /hadoopfs/fs1/hdfs/journalnode/dev-30 is in an inconsistent state: Can't format the storage directory because the current directory is not empty. In hdfs-site.xml:<property><name>dfs.data.dir</name><value>/hadoop/hdfs/data</value><final>true</final></property><property><name>dfs.name.dir</name><value>/hadoopfs/fs1/hdfs/namenod

I have to implement one scenario using Nifi flow. 1. There is one attribute from upstream processor like    KeyValue : KGPYT2. I have to find whether KeyValue is unique in the system or not , by validation from Database.   If it is not found in the database, the flowfile should be proceed to success and   if the value is present in database, it should be proceed to failure. Please suggest me how to proceed in this case. thanks!