Solved

Hi team, How to troubleshoot always particular nodemanagers nodes memory reaches >95% when job is running Regards,Hanu

Encountered error with /opt/cloudera/cm/bin/gen_credentials.sh: Cannot access generated keytab file /var/run/cloudera-scm-server/cmf32289673674897789.keytab

Hi Team, I am using Jolt for the input:{"headers": {"query": "NA","param": "false"},"data": [{"SEQ_NUM": [162,162,162,162,162,162,162,162]},{"SEQ_NUM": [162,162,162,162,162,162,162,162]}]}  Jolt Spec:[{"operation": "shift","spec": {"data": {"*": {"SEQ_NUM": {"*": {"@": "[&1].SEQ_NUM"}}}}}}] Output:[ {"SEQ_NUM" : [ 162, 162 ]}, {"SEQ_NUM" : [ 162, 162 ]}, {"SEQ_NUM" : [ 162, 162 ]}, {"SEQ_NUM" : [ 162, 162 ]}, {"SEQ_NUM" : [ 162, 162 ]}, {"SEQ_NUM" : [ 162, 162 ]}, {"SEQ_NUM" : [ 162, 162 ]}, {"SEQ_NUM" : [ 162, 162 ]} ] Expected Output:[ {"SEQ_NUM" : 162}, {"SEQ_NUM" : 162}, {"SEQ_NUM" : 162}, {"SEQ_NUM" : 162}, {"SEQ_NUM" : 162}, {"SEQ_NUM" : 162}, {"SEQ_NUM" : 162}, {"SEQ_NUM" : 162} ] Can you please help me with the same.Thank you

Is there a setting or option that can be used to prevent an appltication  from creating huge temp files?We recently had an application that due to bad syntax, began to cycle and create files in TB temp almost lowering the cluster for space, there is a way to prevent /tmp from overflowing at that level

Hi, After configuration changes in alert publisher for mail alerts the stale configuration restart is required is it impact the cloudera server ?    Regards,Datta Nagargoje

Hi All,I'm not able to access BDF (HIVE/IMPALA) tables from my local machine. I'm using below query. Could you please suggest. from impala.dbapi import connectconn = connect(host='xxxximpala.rxcorp.com', port=21000,user='xxxxxx', password='xxxxx',auth_mechanism='PLAIN')cursor = conn.cursor()cursor.execute('SELECT COUNT(1) FROM my_table LIMIT 100')results = cursor.fetchall() TTransportException Traceback (most recent call last)<ipython-input-199-b13e7182755e> in <module> 1 from impala.dbapi import connect----> 2 conn = connect(host='xxxmpala.rxcorp.com', port=21000,user='xxxx', password='xxxx',auth_mechanism='PLAIN') 3 cursor = conn.cursor() 4 cursor.execute('SELECT COUNT(1) FROM my_table LIMIT 100') 5 #print cursor.description # prints the result set's schema~\AppData\Local\Continuum\anaconda3\lib\site-packages\impala\dbapi.py in connect(host, port, database, timeout, use_ssl, ca_cert, auth_mechanism, user, password, k

Hi I need to extract data from a SQL query into Excel file (or xls)Is it possible to do it by Groovy script ? 

I want to convert csv flowfile content into XLS file usin groovy script + POI APII m using ivy to grab POI dependencies and it seems work well.please find more details below: This is my Nifi flow:   This is my script@Grapes(@Grab(group='org.apache.poi', module='poi-ooxml', version='3.9'))import org.apache.poi.ss.usermodel.*import org.apache.poi.hssf.usermodel.*import org.apache.poi.xssf.usermodel.*import org.apache.poi.ss.util.*import org.apache.poi.ss.usermodel.*import org.apache.poi.hssf.extractor.*import java.nio.charset.*import java.io.*import org.apache.commons.io.IOUtilsdef flowFile = session.get()def date = new Date()if(!flowFile) returnflowFile = session.write(flowFile, {inputStream, outputStream ->try { Workbook wb = WorkbookFactory.create(inputStream,);Sheet mySheet = wb.getSheetAt(0);def record = IOUtils.toString(inputStream, StandardCharsets.UTF_8)outputStream.write(record.getBytes(StandardCharsets.UTF_8))wb.close();} catch(e) { log.error("Error du

I have a three node cluster  Using cores=8 memory=32GB disks=2 hbase=FalseProfile: cores=8 memory=31744MB reserved=1GB usableMem=31GB disks=2Num Container=4Container Ram=7680MBUsed Ram=30GBUnused Ram=1GByarn.scheduler.minimum-allocation-mb=7680yarn.scheduler.maximum-allocation-mb=30720yarn.nodemanager.resource.memory-mb=30720mapreduce.map.memory.mb=3840mapreduce.map.java.opts=-Xmx3072mmapreduce.reduce.memory.mb=7680mapreduce.reduce.java.opts=-Xmx6144myarn.app.mapreduce.am.resource.mb=3840yarn.app.mapreduce.am.command-opts=-Xmx3072mmapreduce.task.io.sort.mb=1024 

What is Kafka Gateway and Kafka MirrorMaker when add Role Instances to Kafka?I created a Kafka service on Cloudera Manager. Now I want to add a new Kafka broker instance inside the Kafka service. I followed the guide on the internet. Choose Instances -> Add Role InstancesA new window comes up. But I notice that I can only add Kafka Brokers Instances (this guide said that I can add Kafka Connect Instance too).Also, there are two other instances called Gateway and Mirrormaker, which I don't know what are they? I search google but only find some info about Kafka MirrorMaker but no luck on finding anything about Kafka Gateway  

Nifi service is not coming up due to the error /var/log/nifi/nifi-app.log 2019-08-16 14:45:57,385 ERROR org.springframework.web.context.ContextLoader: Context initialization failedorg.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.springframework.security.config.annotation.web.configuration.WebSecurityConfiguration': Unsatisfied dependency expressed through method 'setFilterChainProxySecurityConfigurer' parameter 1; nested exception is org.springframework.beans.factory.BeanExpressionException: Expression parsing failed; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied dependency expressed through method 'setJwtAuthenticationProvider' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'jwtAuthenticationProvider' defined in class

I got CM up and running and want to add the HDFS service. Keep getting this error. In fact I went to the linux disk and in desperation made chmod -R 755 to the root namenode folder. Still get this error java.io.IOException: Cannot create directory /media/sanjay/hdd02/dfs/nn/currentat org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.clearDirectory(Storage.java:416)at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:579)at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:601)at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:173)at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1169)at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1610)at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1720)2022-12-30 11:51:27,073 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.java.io.IOException: Cannot create directory

  sudo tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log| grep -i error2022-12-27 19:36:25,061 ERROR CommandPusher-1:com.cloudera.cmf.command.flow.WorkOutputs: CMD id: 31 Failed to complete installation on host hp8300two.2022-12-27 19:36:25,063 ERROR CommandPusher-1:com.cloudera.cmf.command.flow.WorkOutputs: CMD id: 31 Failed to complete installation on host hpenvy01.2022-12-27 19:36:25,063 ERROR CommandPusher-1:com.cloudera.cmf.model.DbCommand: Command 31(GlobalHostInstall) has completed. finalstate:FINISHED, success:false, msg:Failed to complete installation.2022-12-27 19:38:06,200 ERROR ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: (1 skipped) Failed to download manifest. Status code: 404 URI: https://archive.cloudera.com/accumulo6/6.1.0/parcels/manifest.json/2022-12-27 19:38:53,129 ERROR CommandPusher-1:com.cloudera.cmf.command.flow.WorkOutputs: CMD id: 32 Failed to complete installation on host hp8300two.2022-12-27 19:38:53,130 ERROR

Dear NiFi Team,                             I’m trying to Send Bulletin board info to mail using site to site reporting task.Where I’ve used input port and sending to PutEmail Processor for the FlowFile , upon creating  the Site to site Reporting task and our NiFi is on Kubernetes Cluster.While establishing this getting ended up with Warning message.  Suspecting are there any policies need to be added to thisCould you please help in establishing this task. I’ve attached all the evidences/configurations with screenshots.  By default its connecting to primary node. As mentioned in the ReportingConfigure mentioned both Destination URL and instance URL as same  value by following the https:// [Full DNS of cluster]:PortNumber. For your info. Cluster is of 3 nodes and our NiFi is of version 1.15.3      Please do let me know if you need any further info. Thanks!! 

Hello Is there a way to restart my Hbase service in my cluster by command line since I don't have access to CM BR 

Hi, We are unable to insert data into hive table, do not see any error in hive / hive on tez logs. Upon trying to insert data it just sits and does nothing. 0: jdbc:hive2://MY_SERVER.com:218> insert into test_table values(1,'aaa');INFO : Compiling command(queryId=hive_20221223201916_92c0b5ff-fa62-4f28-afb4-0dd88d76c2d8): insert into test_table values(1,'aaa')INFO : Semantic Analysis Completed (retrial = false)INFO : Created Hive schema: Schema(fieldSchemas:[FieldSchema(name:col1, type:int, comment:null), FieldSchema(name:col2, type:string, comment:null)], properties:null)INFO : Completed compiling command(queryId=hive_20221223201916_92c0b5ff-fa62-4f28-afb4-0dd88d76c2d8); Time taken: 0.371 secondsINFO : Executing command(queryId=hive_20221223201916_92c0b5ff-fa62-4f28-afb4-0dd88d76c2d8): insert into test_table values(1,'aaa')INFO : Query ID = hive_20221223201916_92c0b5ff-fa62-4f28-afb4-0dd88d76c2d8INFO : Total jobs = 1INFO : Launching Job 1 out of 1INFO : Starting t

when i tried to running this note%spark2.pysparkimport pysparkfrom pyspark_llap import HiveWarehouseSessionhive = HiveWarehouseSession.session(spark).build()df = hive.execute(" show databases ") the result is  File "/tmp/spark-00e8a412-9248-4a78-803f-3ca9d8ddac2d/userFiles-2f95cdc1-3caa-42d7-8356-5b022b67b3b9/pyspark_hwc-1.0.0.3.1.0.0-78.zip/pyspark_llap/sql/session.py", line 228, in session return HiveWarehouseBuilder.session(session) File "/tmp/spark-00e8a412-9248-4a78-803f-3ca9d8ddac2d/userFiles-2f95cdc1-3caa-42d7-8356-5b022b67b3b9/pyspark_hwc-1.0.0.3.1.0.0-78.zip/pyspark_llap/sql/session.py", line 44, in session jvm.com.hortonworks.spark.sql.hive.llap.HiveWarehouseBuilder.session(jsparkSession)) TypeError: 'JavaPackage' object is not callable how do i fix it ? thanks

Configured new test instance of RangerSettings Binded user has domain admin rights.Now synced only 999 users of 1650 existed in ADWhat problem can be?  

Hello. I am using Cloudera Manager trial version 7.1.4. I need to install accumulo but when I tried to install it using parcel I got the error that I need to use version CDH lower than 7. Is there any way to install accumulo using CDH 7?

we have old hadoop cluster based on HDP from hortonworks version HDP 2.6.4cluster include 2 namenode services when one is the standby namenode and the second is the active namenode , all machines in the cluster are rhel 7.2 version , and we not see any problem on OS level also cluster include 12 workers machines ( worker include the datanode and node manager services ) the story begin when we get alerts from the smoke test script that complain about "`Detected pause in JVM or host machine`" on the standby namenode , so based on that we decided to increase the namenode heap size from 60G to 100G and above setting was based on table that show how much memory to set according to number of files in HDFS and according to the table we decided to set the namenode heapsize to 100G and then we restart the HDFS service after HDFS is completely restarted , we still see the messages about `Detected pause in JVM or host machine` , and this is really strange because we

Hi guys,i am using spark-shell ( spark-shell --master yarn --jars /usr/hdp/current/hive-warehouse-connector/hive-warehouse-connector_2.11-1.0.0.3.1.2.0-4.jar --conf spark.security.credentials.hiveserver2.enabled=false)to read hive tables using spark, i am able to execute commands like create table, create database, show tables, show databases, but i am not able to read data from tables,my code is as belowimport com.hortonworks.hwc.HiveWarehouseSessionval hive = HiveWarehouseSession.session(spark).build()hive.createDatabase("spark_llap01",false)hive.setDatabase("spark_llap01")hive.createTable("hwx_table").column("value", "string").create()hive.executeUpdate("insert into hwx_table values('1')")hive.executeQuery("select * from hwx_table").showi get this error, whenever i try to fetch datajava.lang.AbstractMethodError: Method com/hortonworks/spark/sql/hive/llap/HiveWarehouseDataSourceReader.createBatchDataReaderFactories()Ljava/util/List; is abstracti used beeline to check whether the data

Hello, While browsing through various articles related to Cloudera Manager, I came across an article which talks about exporting Cloudera Manager configs using a script. In this a wget from "${SCM_URL}/cmf/exportCLI" url is doing the trick. Being fairly new with playing APIs, I tried to read across somethings related to CM APIs, however, did not find much. Would be great if someone could help me to understand which exact configurations are exported with the mentioned API. Apologies if seems to be a basic question. However, thought of asking community instead of creating confusions for myself. Thankssnm1523

Hi All,I am using  Postgress for Hive metastoreHow can we configure postgress HA with Hive with true HA on the Hive Metastore without changing the connection string manually?

1:08:30.213 PM ERROR RetryingHMSHandler[pool-9-thread-200]: HMSHandler Fatal error: javax.jdo.JDODataStoreException: Exception thrown flushing changes to datastoreat org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:543)at org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:171)at org.apache.hadoop.hive.metastore.ObjectStore.commitTransaction(ObjectStore.java:728)at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:101)at com.sun.proxy.$Proxy27.commitTransaction(Unknown Source)at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:1821)at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_with_environment_context(HiveMetaStore.java:1993)at sun.re

We are having a CDP 7.1.7 private cluster with phoenix setup.We had created some phoenix tables long back with certain primary keys, however now when we are trying to check those primary keys on a table via phoenix shell, we are getting empty results.I have tried below commands and no information.!primarykeys <tablename>!column <tablename>!describe <tablename>select column_name from system.catalog where table_name='tablename' Phoenix version is 5.0.0 Please help.