Q&A

Hi I need to extract data from a SQL query into Excel file (or xls)Is it possible to do it by Groovy script ? 

I want to convert csv flowfile content into XLS file usin groovy script + POI APII m using ivy to grab POI dependencies and it seems work well.please find more details below: This is my Nifi flow:   This is my script@Grapes(@Grab(group='org.apache.poi', module='poi-ooxml', version='3.9'))import org.apache.poi.ss.usermodel.*import org.apache.poi.hssf.usermodel.*import org.apache.poi.xssf.usermodel.*import org.apache.poi.ss.util.*import org.apache.poi.ss.usermodel.*import org.apache.poi.hssf.extractor.*import java.nio.charset.*import java.io.*import org.apache.commons.io.IOUtilsdef flowFile = session.get()def date = new Date()if(!flowFile) returnflowFile = session.write(flowFile, {inputStream, outputStream ->try { Workbook wb = WorkbookFactory.create(inputStream,);Sheet mySheet = wb.getSheetAt(0);def record = IOUtils.toString(inputStream, StandardCharsets.UTF_8)outputStream.write(record.getBytes(StandardCharsets.UTF_8))wb.close();} catch(e) { log.error("Error du

1:25:46.144 AMINFOTezClient[main]: Failed to retrieve AM Status via proxycom.google.protobuf.ServiceException: java.io.EOFException: End of File Exception between local host is: "b7-40.lab.archivas.com/172.18.105.90"; destination host is: "b7-35.lab.archivas.com":41485; : java.io.EOFException; For more details see: http://wiki.apache.org/hadoop/EOFException at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:247) ~[hadoop-common-3.1.1.7.1.7.1000-141.jar:?] at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) ~[hadoop-common-3.1.1.7.1.7.1000-141.jar:?] at com.sun.proxy.$Proxy44.getAMStatus(Unknown Source) ~[?:?] at org.apache.tez.client.FrameworkClient.getAMStatus(FrameworkClient.java:115) ~[tez-api-0.9.1.7.1.7.1000-141.jar:0.9.1.7.1.7.1000-141] at org.apache.tez.client.TezClient.getAppMasterStatus(TezClient.java:843) [tez-api-0.9.1.7.1.7.1000-141.jar:0.9.1.7.1.7.1000-141] at org.apache.tez.client.TezClient.waitTillReady(Tez

I have a three node cluster  Using cores=8 memory=32GB disks=2 hbase=FalseProfile: cores=8 memory=31744MB reserved=1GB usableMem=31GB disks=2Num Container=4Container Ram=7680MBUsed Ram=30GBUnused Ram=1GByarn.scheduler.minimum-allocation-mb=7680yarn.scheduler.maximum-allocation-mb=30720yarn.nodemanager.resource.memory-mb=30720mapreduce.map.memory.mb=3840mapreduce.map.java.opts=-Xmx3072mmapreduce.reduce.memory.mb=7680mapreduce.reduce.java.opts=-Xmx6144myarn.app.mapreduce.am.resource.mb=3840yarn.app.mapreduce.am.command-opts=-Xmx3072mmapreduce.task.io.sort.mb=1024 

 We have a full ACID hive managed table that we need to access from spark ETL. We used the documentation provided to connect to Hive WareHouse connector ->https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.0.1/integrating-hive/content/hive_hivewarehousecon...  In addition to using hive warehouse connector to access the acid tables, what spark execution mode differences are there when using JDBC hwc connector  and HiveWareHouseSession vs SparkContext without hwc connector. We don't see any information spark ui/ spark history server and the query takes far too long (x3) than a similar query from SQLContext using a non-acid managed table. from pyspark_llap import HiveWarehouseSessionhive = HiveWarehouseSession.session(spark).build()df= hive.sql("select * from incidents LIMIT 100")...df.show(10)#additional spark transformation code..# NO DAG in spark history server, slower, takes higher memory__________________________The same pattern using SQLContextfrom py

What is Kafka Gateway and Kafka MirrorMaker when add Role Instances to Kafka?I created a Kafka service on Cloudera Manager. Now I want to add a new Kafka broker instance inside the Kafka service. I followed the guide on the internet. Choose Instances -> Add Role InstancesA new window comes up. But I notice that I can only add Kafka Brokers Instances (this guide said that I can add Kafka Connect Instance too).Also, there are two other instances called Gateway and Mirrormaker, which I don't know what are they? I search google but only find some info about Kafka MirrorMaker but no luck on finding anything about Kafka Gateway  

Nifi service is not coming up due to the error /var/log/nifi/nifi-app.log 2019-08-16 14:45:57,385 ERROR org.springframework.web.context.ContextLoader: Context initialization failedorg.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.springframework.security.config.annotation.web.configuration.WebSecurityConfiguration': Unsatisfied dependency expressed through method 'setFilterChainProxySecurityConfigurer' parameter 1; nested exception is org.springframework.beans.factory.BeanExpressionException: Expression parsing failed; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied dependency expressed through method 'setJwtAuthenticationProvider' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'jwtAuthenticationProvider' defined in class

I have a Power BI with ODBC Connection to Impala. I have a situation that Power BI will refresh the Dashboard with scheduled everyday. But somehow, when the refresh start, it got an error: ERROR [HY000] [Cloudera][ImpalaODBC] (120) Error while retrieving data from in Impala: [] : I'm so confused, the error log only shown like that in my Power BI log refresh history. When I check on my Impala Log History. It shown such below: User: xDatabase: xQuery Type: QueryPool: root.XDuration: 10.36sAdmission Result: Cancelled (queued)Query State: EXCEPTION The rest of other query were Error, with Query Status : Session closed because it has no active connections.

I'm using Cloudera manager 7.4.4 and running the spark(2.4.4) application. I'm facing below warning and application is going on to infinite loop.   WARN cluster.YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources     I'm using a simple spark-submit command. spark-submit <filename.py>my python version is 2.7 When I'm running the below command I got many other errors spark-submit --master yarn --deploy-mode cluster --driver-memory 5g --executor-memory 5g --num-executors 3 --executor-cores 2 <filename.py>   Below are the errors      

Hi,we are using HDP 2.6.5 and are facing an issue for some days now.Yesterday we realized that when we issue a spark-submit command, it is sent to the same yarn queue even if we specify --queue param.Had anybody ever come to that issue?Spark version is 2.3.0.Regards

I got CM up and running and want to add the HDFS service. Keep getting this error. In fact I went to the linux disk and in desperation made chmod -R 755 to the root namenode folder. Still get this error java.io.IOException: Cannot create directory /media/sanjay/hdd02/dfs/nn/currentat org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.clearDirectory(Storage.java:416)at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:579)at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:601)at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:173)at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1169)at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1610)at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1720)2022-12-30 11:51:27,073 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.java.io.IOException: Cannot create directory

  sudo tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log| grep -i error2022-12-27 19:36:25,061 ERROR CommandPusher-1:com.cloudera.cmf.command.flow.WorkOutputs: CMD id: 31 Failed to complete installation on host hp8300two.2022-12-27 19:36:25,063 ERROR CommandPusher-1:com.cloudera.cmf.command.flow.WorkOutputs: CMD id: 31 Failed to complete installation on host hpenvy01.2022-12-27 19:36:25,063 ERROR CommandPusher-1:com.cloudera.cmf.model.DbCommand: Command 31(GlobalHostInstall) has completed. finalstate:FINISHED, success:false, msg:Failed to complete installation.2022-12-27 19:38:06,200 ERROR ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: (1 skipped) Failed to download manifest. Status code: 404 URI: https://archive.cloudera.com/accumulo6/6.1.0/parcels/manifest.json/2022-12-27 19:38:53,129 ERROR CommandPusher-1:com.cloudera.cmf.command.flow.WorkOutputs: CMD id: 32 Failed to complete installation on host hp8300two.2022-12-27 19:38:53,130 ERROR

Dear NiFi Team,                             I’m trying to Send Bulletin board info to mail using site to site reporting task.Where I’ve used input port and sending to PutEmail Processor for the FlowFile , upon creating  the Site to site Reporting task and our NiFi is on Kubernetes Cluster.While establishing this getting ended up with Warning message.  Suspecting are there any policies need to be added to thisCould you please help in establishing this task. I’ve attached all the evidences/configurations with screenshots.  By default its connecting to primary node. As mentioned in the ReportingConfigure mentioned both Destination URL and instance URL as same  value by following the https:// [Full DNS of cluster]:PortNumber. For your info. Cluster is of 3 nodes and our NiFi is of version 1.15.3      Please do let me know if you need any further info. Thanks!! 

Does anyone know how the cloudera-scm-agent service determines which services were running when it restarts after a system crash and reboot?I would like to be able to prevent CDH services from automatically starting when the cloudera-scm-agent service starts after a system crash.I've been looking at the log file for the cloudera-scm-agent but I didn't see any info level messages that would help me determine where the agent is getting information to determine which services were running before the crash.I've also looked at tables in the database used by the cloudera-scm-server-db service but have not found anything obvious that appears to control which services are automatically started when the cloudera-scm-agent service starts (I thought this might be controlled by the 'configured_state' field in the services table but that does not seem to have any effect on which services are started).Any help is greatly appreciated.

I have a table named 'X' and a column family 'cf. The table contains data of past 5 years. Old data are requested only few times whereas recent data are accessed frequently. I wanted to apply different storage policies for the data based on time. How can i configure ? Also is it possible to specify different compression algorithms for hot and cold data in single column family? I am asking this because in HBase documentation, different algorithms are recommended for hot and cold data.

Hello Is there a way to restart my Hbase service in my cluster by command line since I don't have access to CM BR 

Hi, We are unable to insert data into hive table, do not see any error in hive / hive on tez logs. Upon trying to insert data it just sits and does nothing. 0: jdbc:hive2://MY_SERVER.com:218> insert into test_table values(1,'aaa');INFO : Compiling command(queryId=hive_20221223201916_92c0b5ff-fa62-4f28-afb4-0dd88d76c2d8): insert into test_table values(1,'aaa')INFO : Semantic Analysis Completed (retrial = false)INFO : Created Hive schema: Schema(fieldSchemas:[FieldSchema(name:col1, type:int, comment:null), FieldSchema(name:col2, type:string, comment:null)], properties:null)INFO : Completed compiling command(queryId=hive_20221223201916_92c0b5ff-fa62-4f28-afb4-0dd88d76c2d8); Time taken: 0.371 secondsINFO : Executing command(queryId=hive_20221223201916_92c0b5ff-fa62-4f28-afb4-0dd88d76c2d8): insert into test_table values(1,'aaa')INFO : Query ID = hive_20221223201916_92c0b5ff-fa62-4f28-afb4-0dd88d76c2d8INFO : Total jobs = 1INFO : Launching Job 1 out of 1INFO : Starting t

when i tried to running this note%spark2.pysparkimport pysparkfrom pyspark_llap import HiveWarehouseSessionhive = HiveWarehouseSession.session(spark).build()df = hive.execute(" show databases ") the result is  File "/tmp/spark-00e8a412-9248-4a78-803f-3ca9d8ddac2d/userFiles-2f95cdc1-3caa-42d7-8356-5b022b67b3b9/pyspark_hwc-1.0.0.3.1.0.0-78.zip/pyspark_llap/sql/session.py", line 228, in session return HiveWarehouseBuilder.session(session) File "/tmp/spark-00e8a412-9248-4a78-803f-3ca9d8ddac2d/userFiles-2f95cdc1-3caa-42d7-8356-5b022b67b3b9/pyspark_hwc-1.0.0.3.1.0.0-78.zip/pyspark_llap/sql/session.py", line 44, in session jvm.com.hortonworks.spark.sql.hive.llap.HiveWarehouseBuilder.session(jsparkSession)) TypeError: 'JavaPackage' object is not callable how do i fix it ? thanks

Hi, I am exploring to learn cloudera, and I am trying to configure cloudera manager.But https://archive.cloudera.com/cm5/ asking me credentials (username and password).Could you please help me to solve this problems please. Thanks,Rahil

Configured new test instance of RangerSettings Binded user has domain admin rights.Now synced only 999 users of 1650 existed in ADWhat problem can be?  

Hello. I am using Cloudera Manager trial version 7.1.4. I need to install accumulo but when I tried to install it using parcel I got the error that I need to use version CDH lower than 7. Is there any way to install accumulo using CDH 7?

Hi. I want to know Where is the Jar folder for Spark in Cloudera? In my previous company, we just put all the needed jars inside $SPARK_HOME/jar folder (on every node), so that we don't have to worry much about the --jars, --packages,... when running the spark-submit job. Also, it saves lots of disk space and time since we don't need to include every package when building a jar.But in my new company, which uses Cloudera, I don't know where is this jar folder. I found 2 places (maybe not the right one):- /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars- /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/spark/jarsWhere should I put the needed jars file on? It seems like all the jars file on the second one are linked to the first one. Also, I found dozens of lib/jar folders everywhere in Cloudera.Or is there any other way to do this with Cloudera? I read some guides about modified Spark configs on the Cloudera manager on the internet. 

we have old hadoop cluster based on HDP from hortonworks version HDP 2.6.4cluster include 2 namenode services when one is the standby namenode and the second is the active namenode , all machines in the cluster are rhel 7.2 version , and we not see any problem on OS level also cluster include 12 workers machines ( worker include the datanode and node manager services ) the story begin when we get alerts from the smoke test script that complain about "`Detected pause in JVM or host machine`" on the standby namenode , so based on that we decided to increase the namenode heap size from 60G to 100G and above setting was based on table that show how much memory to set according to number of files in HDFS and according to the table we decided to set the namenode heapsize to 100G and then we restart the HDFS service after HDFS is completely restarted , we still see the messages about `Detected pause in JVM or host machine` , and this is really strange because we

Hi guys,i am using spark-shell ( spark-shell --master yarn --jars /usr/hdp/current/hive-warehouse-connector/hive-warehouse-connector_2.11-1.0.0.3.1.2.0-4.jar --conf spark.security.credentials.hiveserver2.enabled=false)to read hive tables using spark, i am able to execute commands like create table, create database, show tables, show databases, but i am not able to read data from tables,my code is as belowimport com.hortonworks.hwc.HiveWarehouseSessionval hive = HiveWarehouseSession.session(spark).build()hive.createDatabase("spark_llap01",false)hive.setDatabase("spark_llap01")hive.createTable("hwx_table").column("value", "string").create()hive.executeUpdate("insert into hwx_table values('1')")hive.executeQuery("select * from hwx_table").showi get this error, whenever i try to fetch datajava.lang.AbstractMethodError: Method com/hortonworks/spark/sql/hive/llap/HiveWarehouseDataSourceReader.createBatchDataReaderFactories()Ljava/util/List; is abstracti used beeline to check whether the data

Hello, While browsing through various articles related to Cloudera Manager, I came across an article which talks about exporting Cloudera Manager configs using a script. In this a wget from "${SCM_URL}/cmf/exportCLI" url is doing the trick. Being fairly new with playing APIs, I tried to read across somethings related to CM APIs, however, did not find much. Would be great if someone could help me to understand which exact configurations are exported with the mentioned API. Apologies if seems to be a basic question. However, thought of asking community instead of creating confusions for myself. Thankssnm1523