Find answers, ask questions, and share your expertise
Recently active
Hi All, Can someone past procedure to decommission disk from Ambari cluster. It will helpful. As we have 3 data node's and we have 3 disks configured each node and we want to remove one disk from it. Thanks, Amit Mayekar
Hi team, How to troubleshoot always particular nodemanagers nodes memory reaches >95% when job is running Regards,Hanu
Encountered error with /opt/cloudera/cm/bin/gen_credentials.sh: Cannot access generated keytab file /var/run/cloudera-scm-server/cmf32289673674897789.keytab
Hi Team, We are trying to enable logging at each processor group level. But we are unable to do it. From nifi-app.log we are getting as a general but we need log for each processor in a processor group level. We cannot use bulletin as well as bulletin will stay only for 5 min interval. Can anyone can help achieving it.
Hi Team, I am using Jolt for the input:{"headers": {"query": "NA","param": "false"},"data": [{"SEQ_NUM": [162,162,162,162,162,162,162,162]},{"SEQ_NUM": [162,162,162,162,162,162,162,162]}]} Jolt Spec:[{"operation": "shift","spec": {"data": {"*": {"SEQ_NUM": {"*": {"@": "[&1].SEQ_NUM"}}}}}}] Output:[ {"SEQ_NUM" : [ 162, 162 ]}, {"SEQ_NUM" : [ 162, 162 ]}, {"SEQ_NUM" : [ 162, 162 ]}, {"SEQ_NUM" : [ 162, 162 ]}, {"SEQ_NUM" : [ 162, 162 ]}, {"SEQ_NUM" : [ 162, 162 ]}, {"SEQ_NUM" : [ 162, 162 ]}, {"SEQ_NUM" : [ 162, 162 ]} ] Expected Output:[ {"SEQ_NUM" : 162}, {"SEQ_NUM" : 162}, {"SEQ_NUM" : 162}, {"SEQ_NUM" : 162}, {"SEQ_NUM" : 162}, {"SEQ_NUM" : 162}, {"SEQ_NUM" : 162}, {"SEQ_NUM" : 162} ] Can you please help me with the same.Thank you
Is there a setting or option that can be used to prevent an appltication from creating huge temp files?We recently had an application that due to bad syntax, began to cycle and create files in TB temp almost lowering the cluster for space, there is a way to prevent /tmp from overflowing at that level
Hi, After configuration changes in alert publisher for mail alerts the stale configuration restart is required is it impact the cloudera server ? Regards,Datta Nagargoje
Hi All,I'm not able to access BDF (HIVE/IMPALA) tables from my local machine. I'm using below query. Could you please suggest. from impala.dbapi import connectconn = connect(host='xxxximpala.rxcorp.com', port=21000,user='xxxxxx', password='xxxxx',auth_mechanism='PLAIN')cursor = conn.cursor()cursor.execute('SELECT COUNT(1) FROM my_table LIMIT 100')results = cursor.fetchall() TTransportException Traceback (most recent call last)<ipython-input-199-b13e7182755e> in <module> 1 from impala.dbapi import connect----> 2 conn = connect(host='xxxmpala.rxcorp.com', port=21000,user='xxxx', password='xxxx',auth_mechanism='PLAIN') 3 cursor = conn.cursor() 4 cursor.execute('SELECT COUNT(1) FROM my_table LIMIT 100') 5 #print cursor.description # prints the result set's schema~\AppData\Local\Continuum\anaconda3\lib\site-packages\impala\dbapi.py in connect(host, port, database, timeout, use_ssl, ca_cert, auth_mechanism, user, password, k
Hi I need to extract data from a SQL query into Excel file (or xls)Is it possible to do it by Groovy script ?
I want to convert csv flowfile content into XLS file usin groovy script + POI APII m using ivy to grab POI dependencies and it seems work well.please find more details below: This is my Nifi flow: This is my script@Grapes(@Grab(group='org.apache.poi', module='poi-ooxml', version='3.9'))import org.apache.poi.ss.usermodel.*import org.apache.poi.hssf.usermodel.*import org.apache.poi.xssf.usermodel.*import org.apache.poi.ss.util.*import org.apache.poi.ss.usermodel.*import org.apache.poi.hssf.extractor.*import java.nio.charset.*import java.io.*import org.apache.commons.io.IOUtilsdef flowFile = session.get()def date = new Date()if(!flowFile) returnflowFile = session.write(flowFile, {inputStream, outputStream ->try { Workbook wb = WorkbookFactory.create(inputStream,);Sheet mySheet = wb.getSheetAt(0);def record = IOUtils.toString(inputStream, StandardCharsets.UTF_8)outputStream.write(record.getBytes(StandardCharsets.UTF_8))wb.close();} catch(e) { log.error("Error du
1:25:46.144 AMINFOTezClient[main]: Failed to retrieve AM Status via proxycom.google.protobuf.ServiceException: java.io.EOFException: End of File Exception between local host is: "b7-40.lab.archivas.com/172.18.105.90"; destination host is: "b7-35.lab.archivas.com":41485; : java.io.EOFException; For more details see: http://wiki.apache.org/hadoop/EOFException at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:247) ~[hadoop-common-3.1.1.7.1.7.1000-141.jar:?] at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) ~[hadoop-common-3.1.1.7.1.7.1000-141.jar:?] at com.sun.proxy.$Proxy44.getAMStatus(Unknown Source) ~[?:?] at org.apache.tez.client.FrameworkClient.getAMStatus(FrameworkClient.java:115) ~[tez-api-0.9.1.7.1.7.1000-141.jar:0.9.1.7.1.7.1000-141] at org.apache.tez.client.TezClient.getAppMasterStatus(TezClient.java:843) [tez-api-0.9.1.7.1.7.1000-141.jar:0.9.1.7.1.7.1000-141] at org.apache.tez.client.TezClient.waitTillReady(Tez
I have a three node cluster Using cores=8 memory=32GB disks=2 hbase=FalseProfile: cores=8 memory=31744MB reserved=1GB usableMem=31GB disks=2Num Container=4Container Ram=7680MBUsed Ram=30GBUnused Ram=1GByarn.scheduler.minimum-allocation-mb=7680yarn.scheduler.maximum-allocation-mb=30720yarn.nodemanager.resource.memory-mb=30720mapreduce.map.memory.mb=3840mapreduce.map.java.opts=-Xmx3072mmapreduce.reduce.memory.mb=7680mapreduce.reduce.java.opts=-Xmx6144myarn.app.mapreduce.am.resource.mb=3840yarn.app.mapreduce.am.command-opts=-Xmx3072mmapreduce.task.io.sort.mb=1024
We have a full ACID hive managed table that we need to access from spark ETL. We used the documentation provided to connect to Hive WareHouse connector ->https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.0.1/integrating-hive/content/hive_hivewarehousecon... In addition to using hive warehouse connector to access the acid tables, what spark execution mode differences are there when using JDBC hwc connector and HiveWareHouseSession vs SparkContext without hwc connector. We don't see any information spark ui/ spark history server and the query takes far too long (x3) than a similar query from SQLContext using a non-acid managed table. from pyspark_llap import HiveWarehouseSessionhive = HiveWarehouseSession.session(spark).build()df= hive.sql("select * from incidents LIMIT 100")...df.show(10)#additional spark transformation code..# NO DAG in spark history server, slower, takes higher memory__________________________The same pattern using SQLContextfrom py
What is Kafka Gateway and Kafka MirrorMaker when add Role Instances to Kafka?I created a Kafka service on Cloudera Manager. Now I want to add a new Kafka broker instance inside the Kafka service. I followed the guide on the internet. Choose Instances -> Add Role InstancesA new window comes up. But I notice that I can only add Kafka Brokers Instances (this guide said that I can add Kafka Connect Instance too).Also, there are two other instances called Gateway and Mirrormaker, which I don't know what are they? I search google but only find some info about Kafka MirrorMaker but no luck on finding anything about Kafka Gateway
Nifi service is not coming up due to the error /var/log/nifi/nifi-app.log 2019-08-16 14:45:57,385 ERROR org.springframework.web.context.ContextLoader: Context initialization failedorg.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.springframework.security.config.annotation.web.configuration.WebSecurityConfiguration': Unsatisfied dependency expressed through method 'setFilterChainProxySecurityConfigurer' parameter 1; nested exception is org.springframework.beans.factory.BeanExpressionException: Expression parsing failed; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied dependency expressed through method 'setJwtAuthenticationProvider' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'jwtAuthenticationProvider' defined in class
I have a Power BI with ODBC Connection to Impala. I have a situation that Power BI will refresh the Dashboard with scheduled everyday. But somehow, when the refresh start, it got an error: ERROR [HY000] [Cloudera][ImpalaODBC] (120) Error while retrieving data from in Impala: [] : I'm so confused, the error log only shown like that in my Power BI log refresh history. When I check on my Impala Log History. It shown such below: User: xDatabase: xQuery Type: QueryPool: root.XDuration: 10.36sAdmission Result: Cancelled (queued)Query State: EXCEPTION The rest of other query were Error, with Query Status : Session closed because it has no active connections.
I'm using Cloudera manager 7.4.4 and running the spark(2.4.4) application. I'm facing below warning and application is going on to infinite loop. WARN cluster.YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources I'm using a simple spark-submit command. spark-submit <filename.py>my python version is 2.7 When I'm running the below command I got many other errors spark-submit --master yarn --deploy-mode cluster --driver-memory 5g --executor-memory 5g --num-executors 3 --executor-cores 2 <filename.py> Below are the errors
Hi,we are using HDP 2.6.5 and are facing an issue for some days now.Yesterday we realized that when we issue a spark-submit command, it is sent to the same yarn queue even if we specify --queue param.Had anybody ever come to that issue?Spark version is 2.3.0.Regards
I got CM up and running and want to add the HDFS service. Keep getting this error. In fact I went to the linux disk and in desperation made chmod -R 755 to the root namenode folder. Still get this error java.io.IOException: Cannot create directory /media/sanjay/hdd02/dfs/nn/currentat org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.clearDirectory(Storage.java:416)at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:579)at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:601)at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:173)at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1169)at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1610)at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1720)2022-12-30 11:51:27,073 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.java.io.IOException: Cannot create directory
sudo tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log| grep -i error2022-12-27 19:36:25,061 ERROR CommandPusher-1:com.cloudera.cmf.command.flow.WorkOutputs: CMD id: 31 Failed to complete installation on host hp8300two.2022-12-27 19:36:25,063 ERROR CommandPusher-1:com.cloudera.cmf.command.flow.WorkOutputs: CMD id: 31 Failed to complete installation on host hpenvy01.2022-12-27 19:36:25,063 ERROR CommandPusher-1:com.cloudera.cmf.model.DbCommand: Command 31(GlobalHostInstall) has completed. finalstate:FINISHED, success:false, msg:Failed to complete installation.2022-12-27 19:38:06,200 ERROR ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: (1 skipped) Failed to download manifest. Status code: 404 URI: https://archive.cloudera.com/accumulo6/6.1.0/parcels/manifest.json/2022-12-27 19:38:53,129 ERROR CommandPusher-1:com.cloudera.cmf.command.flow.WorkOutputs: CMD id: 32 Failed to complete installation on host hp8300two.2022-12-27 19:38:53,130 ERROR
Dear NiFi Team, I’m trying to Send Bulletin board info to mail using site to site reporting task.Where I’ve used input port and sending to PutEmail Processor for the FlowFile , upon creating the Site to site Reporting task and our NiFi is on Kubernetes Cluster.While establishing this getting ended up with Warning message. Suspecting are there any policies need to be added to thisCould you please help in establishing this task. I’ve attached all the evidences/configurations with screenshots. By default its connecting to primary node. As mentioned in the ReportingConfigure mentioned both Destination URL and instance URL as same value by following the https:// [Full DNS of cluster]:PortNumber. For your info. Cluster is of 3 nodes and our NiFi is of version 1.15.3 Please do let me know if you need any further info. Thanks!!
Does anyone know how the cloudera-scm-agent service determines which services were running when it restarts after a system crash and reboot?I would like to be able to prevent CDH services from automatically starting when the cloudera-scm-agent service starts after a system crash.I've been looking at the log file for the cloudera-scm-agent but I didn't see any info level messages that would help me determine where the agent is getting information to determine which services were running before the crash.I've also looked at tables in the database used by the cloudera-scm-server-db service but have not found anything obvious that appears to control which services are automatically started when the cloudera-scm-agent service starts (I thought this might be controlled by the 'configured_state' field in the services table but that does not seem to have any effect on which services are started).Any help is greatly appreciated.
I have a table named 'X' and a column family 'cf. The table contains data of past 5 years. Old data are requested only few times whereas recent data are accessed frequently. I wanted to apply different storage policies for the data based on time. How can i configure ? Also is it possible to specify different compression algorithms for hot and cold data in single column family? I am asking this because in HBase documentation, different algorithms are recommended for hot and cold data.
Hello Is there a way to restart my Hbase service in my cluster by command line since I don't have access to CM BR
Hi, We are unable to insert data into hive table, do not see any error in hive / hive on tez logs. Upon trying to insert data it just sits and does nothing. 0: jdbc:hive2://MY_SERVER.com:218> insert into test_table values(1,'aaa');INFO : Compiling command(queryId=hive_20221223201916_92c0b5ff-fa62-4f28-afb4-0dd88d76c2d8): insert into test_table values(1,'aaa')INFO : Semantic Analysis Completed (retrial = false)INFO : Created Hive schema: Schema(fieldSchemas:[FieldSchema(name:col1, type:int, comment:null), FieldSchema(name:col2, type:string, comment:null)], properties:null)INFO : Completed compiling command(queryId=hive_20221223201916_92c0b5ff-fa62-4f28-afb4-0dd88d76c2d8); Time taken: 0.371 secondsINFO : Executing command(queryId=hive_20221223201916_92c0b5ff-fa62-4f28-afb4-0dd88d76c2d8): insert into test_table values(1,'aaa')INFO : Query ID = hive_20221223201916_92c0b5ff-fa62-4f28-afb4-0dd88d76c2d8INFO : Total jobs = 1INFO : Launching Job 1 out of 1INFO : Starting t
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.