Q&A

First we created a table based on JSON dataset from hive cli using the below queryCREATE EXTERNAL TABLE json10(fruit string,size string,color string)ROW FORMAT SERDE'org.apache.hive.hcatalog.data.JsonSerDe'STORED AS INPUTFORMAT'org.apache.hadoop.mapred.TextInputFormat'OUTPUTFORMAT'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'LOCATION 's3a://json2/'; we are able to execute select queries form hive cli on top of the table created above, but unable to execute the same from pyspark script. pyspark scriptfrom pyspark.context import SparkContextsc=SparkContext()from pyspark.sql import HiveContexthive_context = HiveContext(sc)json=hive_context.table("default.json")hive_context.sql("select * from json").show()ERROR MESSAGE 22/06/07 15:24:38 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, b7-36.lab.archivas.com, executor 1): java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy to field org.apach

The same operation worked normally in the previous months, but this problem suddenly appeared recently, and the cause could not be found. I hope you can help me find out the cause of the problem, thank you! The error log is as follows :[2022-06-16 13:12:10,571] {ssh.py:141} INFO - Warning: Ignoring non-spark config property: 2022-06-16=21:04:35,281 DEBUG UserGroupInformation:1902 main 281 - PrivilegedAction as:root (auth:SIMPLE) from:org.apache.hadoop.hdfs.tools.GetConf.run(GetConf.java:315) [2022-06-16 13:12:10,572] {ssh.py:141} INFO - Warning: Ignoring non-spark config property: hdfs=//bd.vn0038.jmrh.com:8020/user/spark/applicationHistory [2022-06-16 13:12:12,237] {ssh.py:141} INFO - 22/06/16 21:12:12 INFO client.RMProxy: Connecting to ResourceManager at bd.vn0038.jmrh.com/172.168.100.171:8032 [2022-06-16 13:12:12,557] {ssh.py:141} INFO - 22/06/16 21:12:12 INFO yarn.Client: Requesting a new application from cluster with 7 NodeManagers [2022-06-16 13:12:12,698] {ssh.py:141}

Code deos not have any jar files, I have provided the python folders as zip and using following command to run the code. spark2-submit --queue abc --master yarn --deploy-mode cluster --num-executors 5 --executor-cores 5 --executor-memory 20G --driver-memory 5g --conf spark.yarn.executor.memoryOverhead=4096 --conf spark.sql.shuffle.partitions=400 --conf spark.driver.maxResultSize=0 --conf spark.scheduler.mode=FAIR --conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.kryoserializer.buffer.max=512m --conf spark.executor.heartbeatInterval=100 --conf spark.sql.autoBroadcastJoinThreshold=-1 --conf spark.sql.broadcastTimeout=-1 --py-files /abc/python/dependencies.zip,/abc/python/modules.zip /abc/python/main.py Following is the error:Exit code: 13Shell output: main : command provided 1main : run as user is ***main : requested yarn user is***Getting exit code file...Creating script paths...Writing pid file...Writing to tmp file /Writing to cgroup task files.

I found my error on spark job and i can't solve this error yet until i got stuck, this is the error i got 22/05/19 09:32:40 ERROR util.SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[Executor task launch worker for task 1,5,main] java.lang.OutOfMemoryError: input is too large to fit in a byte array at org.spark_project.guava.io.ByteStreams.toByteArrayInternal(ByteStreams.java:194) at org.spark_project.guava.io.ByteStreams.toByteArray(ByteStreams.java:220) at org.apache.spark.input.WholeTextFileRecordReader.nextKeyValue(WholeTextFileRecordReader.scala:79) at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:39) at org.apache.spark.sql.execution.datasources.HadoopFileWholeTextReader.hasNext(HadoopFileWholeTextReader.scala:52) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409) at org.apache.spark.sql.execution.datasources.FileScanRDD$$an

I have an app where after doing various processes in pyspark I have a smaller dataset which I need to convert to pandas before uploading to elasticsearch. I have  res = result.select("*").toPandas()  On my local when I usespark-submit --master "local[*]" app.pyIt works perfectly fine. I also have a 2 worker cluster, when I run it on my cluster using:spark-submit --master MASTER_IP:7077 app.pyI get the following error:Traceback (most recent call last): File "/home/nitinram/Elisity-Master/esaas/ml_model/landspeed_violation_spark/landspeed_violation_spark.py", line 173, in <module> results = landspeed_calculator(ps_df_compiled,landspeed_params, sc) File "/home/nitinram/Elisity-Master/esaas/ml_model/landspeed_violation_spark/landspeed_violation_spark_utilities.py", line 273, in landspeed_calculator res = result.select("*").toPandas() File "/opt/spark/python/lib/pyspark.zip/pyspark/sql/pandas/conversion.py", line 110, in toPandas split_batches=self_dest

Spark Apache showing error below, when running jobs.+++++++++++++++++++++++++++++++++++++++++++++++++++++++Exception in thread "main" java.lang.IllegalArgumentException: Unable to instantiate SparkSession with Hive support because Hive classes are not found.at org.apache.spark.sql.SparkSession$Builder.enableHiveSupport(SparkSession.scala:878)at br.com.recortv.dadosTratados_FaixaHoraria$.main(dadosTratados_FaixaHoraria.scala:39)+++++++++++++++++++++++++++++++++++++++++++++++++++++++

I have a csv where one of the columns can sometimes come as a pair such as:Column_name1 , 22.5 3.22.9 - 3.2The delimiter can potentially be anything really, but there will be a delimiter. Also the values will only come as pairs. My question is, is there an efficient way to separate this one column into a flowfile where each value has it's own column, such that I have the following result:Column #1Column #2122.5 3.2 Thank you

Dear Friends,I am upgrading CDH to CDP and getting below error while upgrading. Please help.  Show All Steps  Show Only Failed Steps  Show Only Running Steps  Converting configuration parametersFailed due to java.lang.IllegalStateException: HS2 role count in Hive and Hive on Tez services should match.  Failed due to java.lang.IllegalStateException: HS2 role count in Hive and Hive on Tez services should matchjava.lang.IllegalStateException: HS2 role count in Hive and Hive on Tez services should match. 

Hello all,   I cannot read data from hive orc table and load to dataframe. If someone know, could you help me to fix it? Below is my scripts:   from pyspark import SparkContext, SparkConffrom pyspark.conf import SparkConffrom pyspark.sql import SparkSessionfrom pyspark.sql import HiveContext,SQLContext spark = SparkSession.builder.appName("Testing....").enableHiveSupport().getOrCreate() hive_context = HiveContext(spark)sqlContext = SQLContext(spark) df_pgw=hive_context.sql("select * from orc_table")Hive Session ID = 79c9e6c0-1649-41dc-9aea-493c0f62d04622/07/20 11:50:52 WARN SessionState: METASTORE_FILTER_HOOK will be ignored, since hive.security.authorization.manager is set to instance of HiveAuthorizerFactory.22/07/20 11:50:56 WARN HiveMetastoreCatalog: Unable to infer schema for table orc_table from file format ORC (inference mode: INFER_AND_SAVE). Using metastore schema.   df_pgw.show()     => ....Don't have data presents   Thanks,

hello cloudera community, how can we find out which group is tied to a sentry role by command line?

Hello friends, I hope you are very wellQuestion:according tohttps://docs.cloudera.com/cdp-private-cloud-base/7.1.3/managing-clusters/topics/cm-disable-tuned-service.htmlTuned service must be off, but for RHEL 8.x also applies?Regards!

[root@master cloudera-scm-server]# /opt/cloudera/cm/schema/scm_prepare_database.sh -h172.16.4.131 -P3306 -uroot -pZyht_1234 mysql scm scm Zyht_1234caJAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.332.b09-1.el7_9.x86_64Verifying that we can write to /etc/cloudera-scm-serverLoading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.[ main] DbProvisioner ERROR Exception when creating/dropping database with user 'root' and jdbc url 'jdbc:mysql://172.16.4.131:3306/?useUnicode=true&characterEncoding=UTF-8'java.sql.SQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'identified by 'Zyht_1234ca'' at line 1at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:120)[mysql-connector-java.jar:8.0.30]at co

Hi,  I am trying to run spark streaming job on cloudera v5.14.4 which will be reading data from kafka v0.10. I am using following version of spark streaming dependencies  spark-streaming_2.11:2.3.0spark-streaming-kafka-0-10_2.11:2.3.0 The job works fine on local but on cloudera it is not starting up. There is no streaming tab on AM and stages/storage tabs are blank.  There are no errors in the logs. Any suggestion what might be wrong here? 

I have a Spark sql query that works when I execute from inside a Jupyter Notebook that has a a PySpark kernel but fails when I execute it submitting to a Livy session. Usually there's no difference when I execute my queries both ways. I tried to get the spark session parameters with the command below and to guarantee that they are both the same:spark.sparkContext.getConf().getAll()  I'm using spark 2.3. How can I debug this problem? I know that the query works using spark, but I can't make it work submitting with Livy. Here is the query:INSERT INTO sbx_xxxxx.dados_auxiliares_mediana_preco    select        p.ano, p.cod_cfi, p.preco, p.repeticoes, sum(p2.repeticoes) as qtd_antes    from sbx_operacoes_digitais.dados_auxiliares_moda_preco p    left join sbx_operacoes_digitais.dados_auxiliares_moda_preco p2        on p.ano = p2.ano and p.cod_cfi = p2.cod_cfi and p2.preco <= p.preco    group by

Hi all, I have generated a query and the summary of Memory as below.Need your help to elaborate and how to read the summary.Thank you. Query(9c4786bd80e8ed20:1caa788500000000): Limit=3.00 GB Reservation=0 ReservationLimit=2.40 GB OtherMemory=0 Total=0 Peak=520.47 MB

Crossposting from: Stack Overflow(I will respond and give updates on both places)I would like to know if there is also maximum number for items below:Maximum number of databases in a catalog (I assume Hive Metastore only has one catalog, which is "hive")Maximum number of tables per database (as in, can I create 10 million tables in a database or due to limitation must I split them into 10 databases each with 1 million tables)I also would like to know whether the limitations are hard limit (unconfigurable), or configurable by Hive, or dependent on RDBMS it is using.

In this Documentation (https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/installation/topics/cdpdc-configure-mysql-ranger.html) it requires MYSQL connector version 5.7 but i can't find it, anyone help?

Hello Experts, I am new to Flink CEP and trying to test basic things - In below code my expectation is all the input should matched in patter and should print as matched result. But somehow nothing is matching ('matechedStream.print()') any idea about the reason ?Any suggestion/help would be much appreciated. package com.o9.flink;import com.o9.flink.asyncio.DemandSupply;import org.apache.flink.cep.CEP;import java.util.ArrayList;import java.util.List;import java.util.Map;import org.apache.flink.cep.PatternStream;import org.apache.flink.cep.functions.PatternProcessFunction;import org.apache.flink.cep.pattern.Pattern;import org.apache.flink.cep.pattern.conditions.SimpleCondition;import org.apache.flink.streaming.api.datastream.DataStream;import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;import org.apache.flink.streaming.api.windowing.time.Time;import org.apache.flink.util.Collector;public class DemandSupplyPattern { public static void main(String[]

Hi all , is Apache Druid  supported as as a stand alone service in the Cloudera data flow (CDF)

Hello,first of all I apologize for posting this subject here,because I couldn't find an email address to contact the support directly.At this moment I am unable to ask support questions and I have the following messageIf anyone has had this problem please help me.thank you.

Hello,I am working on one of the tasks in Nifi and I am stuck on of the issue, the issue is I am getting the Json data from jolt transforms processor and also I am getting the Json data from input port processor now the issue is I need to combine the data from two processor based on id’s.  How to combined the data from two processors based on common id ? Will you please help me out.Thanks,Sathish

The following code what it does is to transform a string to json and then send them to kafka.But it is sending some blank messages.Does anyone know how I can control what is sent, that is, what comes empty not to send it to kafka. This is code:import sysimport tracebackimport stringimport java.iofrom org.apache.commons.io import IOUtilsfrom java.nio.charset import StandardCharsetsfrom org.apache.nifi.processor.io import StreamCallbackfrom org.apache.nifi.processor import Processorfrom org.apache.nifi.processor import Relationshipfrom org.apache.nifi.components import PropertyDescriptorfrom org.apache.nifi.processor.util import StandardValidatorsfrom org.apache.nifi.logging import ComponentLogimport datetimeimport jsonclass PyStreamCallback(StreamCallback):    def __init__(self, log):        self.log = log    def process(self, inputStream, outputStream):        text = IOUtils.toString(inputStream, StandardCharsets.UTF_8)&n

Hello, I could use some help... I'm using GetHTTP and InvokeHTTP processors for reading certain indices in a elasticsearch database. The URL to this database is a HTTPS, my question is the following: Is it possible to run these processors leaving the SSL Context Service property as "No value" when you insert in the URL property a HTTPs? If I do this I get the following error (InvokeHTTP processor): InvokeHTTP[id=0e07115a-102e-1182-be33-467ed7f3f784] Yielding processor due to exception encountered as a source processor: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target- Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target- Caused by: sun.security.provid

Hi, Could you please help me with the correct URL format to connect to Sybase using the DBCP connection pool of NiFi? The URL format that I have tried is - jdbc:sybase:Tds://host:port I keep getting this error - failed to create Poolable Connection Factory. Incorrect URL format. Thank you in advance.

Dear Experts,We have some files in our HDFS server and want to get these files and put them into Isilon. We use GetHDFS and PutHDFS for this purpose. But, we get the lease and lock errors, as shown in the picture below. How can we fix these problems?Regards