Find answers, ask questions, and share your expertise
Recently active
Hello all, I am a student taking a course in SQL. The final module of this course has us installing Oracle VMWare and Cloudera to utilize Hadoop. I went through a detailed instruction manual for the VMWare installation, including an expansion pack install and guest account install within the VMWare environment. I ran into no problems with this installation process (linked below for reference) Installation_ClouderaQuickstartVirtualMachine Afterwards, we were to follow a Cloudera basic tutorial guide to get started (linked below). Cloudera Quickstart Beginners Tutorial However, when attempting to input the first script into Terminal as provided, I began running into errors. The script and errors received are shown below. [cloudera@quickstart ~]$ sqoop import-all-tables \> m 1 \> connect jdbc:mysql://quickstart:3306/retail_db \> username=retail_dba \> password=cloude
Hi EveryoneWe trying to integrate LDAP to our existing nifi server single node. After configuring ldap details, Nifi fails with timeout connection error. There is no firewall block and we are able to telnet the ldap server. Please see our setup ldap below. LDAP server :CN=Service Account\, nifi_ldap,OU=Service Accounts,OU=Enterprise Dev,DC=dev,DC=abcdeldap://ldap.dev.abcde:389password: 5$qwerty! nifi.properties *Previously our application was running as http but we change it to https*nifi.authorizer.configuration.file=./conf/authorizers.xmlnifi.login.identity.provider.configuration.file=./conf/login-identity-providers.xml# web properties ############################################### For security, NiFi will present the UI on 127.0.0.1 and only be accessible through this loopback interface.# Be aware that changing these properties may affect how your instance can be accessed without any restriction.# We recommend configuring HTTPS instead. The administrators guide provides in
Within my NiFi flow I have a number of datasets, with various schemas where the 'key' of key value pairs are on camelCase but i want any incoming keys to be outputted in snake case, does anyone know what jolt spec i can use to achieve this? Example input:{"aRandomFieldName":"Random Value"} Required Output:{"a_random_field_name":"Random Value"} Been struggling to achieve this using Jolt or replaceText processors, any assistance would be great. Thanks in advanceAndy.
Hello, one of our cluster nodes (a 3 node cluster v1.16.3 on Kubernetes) ran out of disk space. After increasing the disk space, we now get the above mentioned exception in the logs and the node does not start.What can we do to fix this and get the node to start again? Thanks a lot! Related logs:{"@timestamp":"2022-09-02T07:49:50.865Z","log.level": "WARN","message":"Encountered unexpected End-of-File when reading journal file ../flowfile_repository/journals/474192833.journal; assuming that NiFi was shutdown unexpectedly and continuing recovery","ecs.version": "1.2.0","service.name":"nifi-app.log","service.node.name":"nifi-prod-1","event.dataset":"nifi-app.log","process.thread.name":"main","log.logger":"org.apache.nifi.wali.LengthDelimitedJournal"}{"@timestamp":"2022-09-02T07:49:50.866Z","log.level": "INFO","message":"Successfully recovered 0 updates from journal ../flowfile_repository/journals/474192833.journal","ecs.version": "1.2.0","service.name":"nifi-app.log","servi
I have a csv with 50 columns, is there a way to grab a specific column by it's header name and move it in a separate flow file for further processing?
Hi, Basically, I am trying to import a database table from MySQL to hive but it turns out not importing to hive instead it gives me the following error. can anyone please help me with a solution which would be very much appreciated. [cloudera@quickstart ~]$ sqoop import --connect jdbc:mysql://localhost:3306/sqoop --username root --password cloudera --table customer --hive-importWarning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail.Please set $ACCUMULO_HOME to the root of your Accumulo installation.19/11/27 23:11:33 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.13.019/11/27 23:11:33 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.19/11/27 23:11:33 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override19/11/27 23:11:33 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc.19/11/27 23:11:33 INFO manager.MySQLManager: Preparing to use a MySQL s
Hi All, We are planning to upgrade our Nifi application(Linux platform) from 1.9.2 version to 1.16.2.Before proceeding with this major version jump , we wanted to check for any possible issues and backward compatibility . We do not see entry for version jump from 1.9.x to 1.16.x in below : https://cwiki.apache.org/confluence/display/NIFI/Migration+GuidanceWe request your guidance for anything we need to be aware of before we make jump from Nifi 1.9.2 to 1.16.2 Thanks,Alvin
Resources we have in place:7 nodes with each having 250 gb memoryvcpu = 32 per each nodeconfiguration specified in spark-defaults.conf :spark.executor.memory = 100gspark.executor.memoryOverhead = 49gspark.driver.memoryOverhead=200gspark.driver.memory = 500gQuery tried to execute :hive_context.sql("select * from 5mcsv CROSS JOIN 2mcsv").show(8000000) So, we are facing below issue when trying to fetch 8 million rows with the above mentioned query. However , we are not facing any issue with fetching 7 million rows Traceback (most recent call last): File "/root/hivespark.py", line 29, in <module> hive_context.sql("select * from 5mcsv CROSS JOIN 2mcsv").show(8000000) File "/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p1000.24102687/lib/spark/python/lib/pyspark.zip/pyspark/sql/dataframe.py", line 381, in show File "/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p1000.24102687/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__ File "/opt/cl
First we created a table based on JSON dataset from hive cli using the below queryCREATE EXTERNAL TABLE json10(fruit string,size string,color string)ROW FORMAT SERDE'org.apache.hive.hcatalog.data.JsonSerDe'STORED AS INPUTFORMAT'org.apache.hadoop.mapred.TextInputFormat'OUTPUTFORMAT'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'LOCATION 's3a://json2/'; we are able to execute select queries form hive cli on top of the table created above, but unable to execute the same from pyspark script. pyspark scriptfrom pyspark.context import SparkContextsc=SparkContext()from pyspark.sql import HiveContexthive_context = HiveContext(sc)json=hive_context.table("default.json")hive_context.sql("select * from json").show()ERROR MESSAGE 22/06/07 15:24:38 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, b7-36.lab.archivas.com, executor 1): java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy to field org.apach
The same operation worked normally in the previous months, but this problem suddenly appeared recently, and the cause could not be found. I hope you can help me find out the cause of the problem, thank you! The error log is as follows :[2022-06-16 13:12:10,571] {ssh.py:141} INFO - Warning: Ignoring non-spark config property: 2022-06-16=21:04:35,281 DEBUG UserGroupInformation:1902 main 281 - PrivilegedAction as:root (auth:SIMPLE) from:org.apache.hadoop.hdfs.tools.GetConf.run(GetConf.java:315) [2022-06-16 13:12:10,572] {ssh.py:141} INFO - Warning: Ignoring non-spark config property: hdfs=//bd.vn0038.jmrh.com:8020/user/spark/applicationHistory [2022-06-16 13:12:12,237] {ssh.py:141} INFO - 22/06/16 21:12:12 INFO client.RMProxy: Connecting to ResourceManager at bd.vn0038.jmrh.com/172.168.100.171:8032 [2022-06-16 13:12:12,557] {ssh.py:141} INFO - 22/06/16 21:12:12 INFO yarn.Client: Requesting a new application from cluster with 7 NodeManagers [2022-06-16 13:12:12,698] {ssh.py:141}
Code deos not have any jar files, I have provided the python folders as zip and using following command to run the code. spark2-submit --queue abc --master yarn --deploy-mode cluster --num-executors 5 --executor-cores 5 --executor-memory 20G --driver-memory 5g --conf spark.yarn.executor.memoryOverhead=4096 --conf spark.sql.shuffle.partitions=400 --conf spark.driver.maxResultSize=0 --conf spark.scheduler.mode=FAIR --conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.kryoserializer.buffer.max=512m --conf spark.executor.heartbeatInterval=100 --conf spark.sql.autoBroadcastJoinThreshold=-1 --conf spark.sql.broadcastTimeout=-1 --py-files /abc/python/dependencies.zip,/abc/python/modules.zip /abc/python/main.py Following is the error:Exit code: 13Shell output: main : command provided 1main : run as user is ***main : requested yarn user is***Getting exit code file...Creating script paths...Writing pid file...Writing to tmp file /Writing to cgroup task files.
I found my error on spark job and i can't solve this error yet until i got stuck, this is the error i got 22/05/19 09:32:40 ERROR util.SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[Executor task launch worker for task 1,5,main] java.lang.OutOfMemoryError: input is too large to fit in a byte array at org.spark_project.guava.io.ByteStreams.toByteArrayInternal(ByteStreams.java:194) at org.spark_project.guava.io.ByteStreams.toByteArray(ByteStreams.java:220) at org.apache.spark.input.WholeTextFileRecordReader.nextKeyValue(WholeTextFileRecordReader.scala:79) at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:39) at org.apache.spark.sql.execution.datasources.HadoopFileWholeTextReader.hasNext(HadoopFileWholeTextReader.scala:52) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409) at org.apache.spark.sql.execution.datasources.FileScanRDD$$an
I have an app where after doing various processes in pyspark I have a smaller dataset which I need to convert to pandas before uploading to elasticsearch. I have res = result.select("*").toPandas() On my local when I usespark-submit --master "local[*]" app.pyIt works perfectly fine. I also have a 2 worker cluster, when I run it on my cluster using:spark-submit --master MASTER_IP:7077 app.pyI get the following error:Traceback (most recent call last): File "/home/nitinram/Elisity-Master/esaas/ml_model/landspeed_violation_spark/landspeed_violation_spark.py", line 173, in <module> results = landspeed_calculator(ps_df_compiled,landspeed_params, sc) File "/home/nitinram/Elisity-Master/esaas/ml_model/landspeed_violation_spark/landspeed_violation_spark_utilities.py", line 273, in landspeed_calculator res = result.select("*").toPandas() File "/opt/spark/python/lib/pyspark.zip/pyspark/sql/pandas/conversion.py", line 110, in toPandas split_batches=self_dest
Spark Apache showing error below, when running jobs.+++++++++++++++++++++++++++++++++++++++++++++++++++++++Exception in thread "main" java.lang.IllegalArgumentException: Unable to instantiate SparkSession with Hive support because Hive classes are not found.at org.apache.spark.sql.SparkSession$Builder.enableHiveSupport(SparkSession.scala:878)at br.com.recortv.dadosTratados_FaixaHoraria$.main(dadosTratados_FaixaHoraria.scala:39)+++++++++++++++++++++++++++++++++++++++++++++++++++++++
I have a csv where one of the columns can sometimes come as a pair such as:Column_name1 , 22.5 3.22.9 - 3.2The delimiter can potentially be anything really, but there will be a delimiter. Also the values will only come as pairs. My question is, is there an efficient way to separate this one column into a flowfile where each value has it's own column, such that I have the following result:Column #1Column #2122.5 3.2 Thank you
Dear Friends,I am upgrading CDH to CDP and getting below error while upgrading. Please help. Show All Steps Show Only Failed Steps Show Only Running Steps Converting configuration parametersFailed due to java.lang.IllegalStateException: HS2 role count in Hive and Hive on Tez services should match. Failed due to java.lang.IllegalStateException: HS2 role count in Hive and Hive on Tez services should matchjava.lang.IllegalStateException: HS2 role count in Hive and Hive on Tez services should match.
Hello all, I cannot read data from hive orc table and load to dataframe. If someone know, could you help me to fix it? Below is my scripts: from pyspark import SparkContext, SparkConffrom pyspark.conf import SparkConffrom pyspark.sql import SparkSessionfrom pyspark.sql import HiveContext,SQLContext spark = SparkSession.builder.appName("Testing....").enableHiveSupport().getOrCreate() hive_context = HiveContext(spark)sqlContext = SQLContext(spark) df_pgw=hive_context.sql("select * from orc_table")Hive Session ID = 79c9e6c0-1649-41dc-9aea-493c0f62d04622/07/20 11:50:52 WARN SessionState: METASTORE_FILTER_HOOK will be ignored, since hive.security.authorization.manager is set to instance of HiveAuthorizerFactory.22/07/20 11:50:56 WARN HiveMetastoreCatalog: Unable to infer schema for table orc_table from file format ORC (inference mode: INFER_AND_SAVE). Using metastore schema. df_pgw.show() => ....Don't have data presents Thanks,
hello cloudera community, how can we find out which group is tied to a sentry role by command line?
Hello friends, I hope you are very wellQuestion:according tohttps://docs.cloudera.com/cdp-private-cloud-base/7.1.3/managing-clusters/topics/cm-disable-tuned-service.htmlTuned service must be off, but for RHEL 8.x also applies?Regards!
[root@master cloudera-scm-server]# /opt/cloudera/cm/schema/scm_prepare_database.sh -h172.16.4.131 -P3306 -uroot -pZyht_1234 mysql scm scm Zyht_1234caJAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.332.b09-1.el7_9.x86_64Verifying that we can write to /etc/cloudera-scm-serverLoading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.[ main] DbProvisioner ERROR Exception when creating/dropping database with user 'root' and jdbc url 'jdbc:mysql://172.16.4.131:3306/?useUnicode=true&characterEncoding=UTF-8'java.sql.SQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'identified by 'Zyht_1234ca'' at line 1at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:120)[mysql-connector-java.jar:8.0.30]at co
Hi, I am trying to run spark streaming job on cloudera v5.14.4 which will be reading data from kafka v0.10. I am using following version of spark streaming dependencies spark-streaming_2.11:2.3.0spark-streaming-kafka-0-10_2.11:2.3.0 The job works fine on local but on cloudera it is not starting up. There is no streaming tab on AM and stages/storage tabs are blank. There are no errors in the logs. Any suggestion what might be wrong here?
I have a Spark sql query that works when I execute from inside a Jupyter Notebook that has a a PySpark kernel but fails when I execute it submitting to a Livy session. Usually there's no difference when I execute my queries both ways. I tried to get the spark session parameters with the command below and to guarantee that they are both the same:spark.sparkContext.getConf().getAll() I'm using spark 2.3. How can I debug this problem? I know that the query works using spark, but I can't make it work submitting with Livy. Here is the query:INSERT INTO sbx_xxxxx.dados_auxiliares_mediana_preco select p.ano, p.cod_cfi, p.preco, p.repeticoes, sum(p2.repeticoes) as qtd_antes from sbx_operacoes_digitais.dados_auxiliares_moda_preco p left join sbx_operacoes_digitais.dados_auxiliares_moda_preco p2 on p.ano = p2.ano and p.cod_cfi = p2.cod_cfi and p2.preco <= p.preco group by
Hi all, I have generated a query and the summary of Memory as below.Need your help to elaborate and how to read the summary.Thank you. Query(9c4786bd80e8ed20:1caa788500000000): Limit=3.00 GB Reservation=0 ReservationLimit=2.40 GB OtherMemory=0 Total=0 Peak=520.47 MB
Crossposting from: Stack Overflow(I will respond and give updates on both places)I would like to know if there is also maximum number for items below:Maximum number of databases in a catalog (I assume Hive Metastore only has one catalog, which is "hive")Maximum number of tables per database (as in, can I create 10 million tables in a database or due to limitation must I split them into 10 databases each with 1 million tables)I also would like to know whether the limitations are hard limit (unconfigurable), or configurable by Hive, or dependent on RDBMS it is using.
In this Documentation (https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/installation/topics/cdpdc-configure-mysql-ranger.html) it requires MYSQL connector version 5.7 but i can't find it, anyone help?
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.