Solved

Hello all,   I am a student taking a course in SQL.  The final module of this course has us installing Oracle VMWare and Cloudera to utilize Hadoop.     I went through a detailed instruction manual for the VMWare installation, including an expansion pack install and guest account install within the VMWare environment.  I ran into no problems with this installation process (linked below for reference)   Installation_ClouderaQuickstartVirtualMachine    Afterwards, we were to follow a Cloudera basic tutorial guide to get started (linked below).    Cloudera Quickstart Beginners Tutorial   However, when attempting to input the first script into Terminal as provided, I began running into errors.  The script and errors received are shown below.         [cloudera@quickstart ~]$ sqoop import-all-tables \> m 1 \> connect jdbc:mysql://quickstart:3306/retail_db \> username=retail_dba \> password=cloude

Within my NiFi flow I have a number of datasets, with various schemas where the 'key' of key value pairs are on camelCase but i want any incoming keys to be outputted in snake case, does anyone know what jolt spec i can use to achieve this? Example input:{"aRandomFieldName":"Random Value"} Required Output:{"a_random_field_name":"Random Value"} Been struggling to achieve this using Jolt or replaceText processors, any assistance would be great. Thanks in advanceAndy.

Hello, one of our cluster nodes (a 3 node cluster v1.16.3 on Kubernetes) ran out of disk space. After increasing the disk space, we now get the above mentioned exception in the logs and the node does not start.What can we do to fix this and get the node to start again? Thanks a lot! Related logs:{"@timestamp":"2022-09-02T07:49:50.865Z","log.level": "WARN","message":"Encountered unexpected End-of-File when reading journal file ../flowfile_repository/journals/474192833.journal; assuming that NiFi was shutdown unexpectedly and continuing recovery","ecs.version": "1.2.0","service.name":"nifi-app.log","service.node.name":"nifi-prod-1","event.dataset":"nifi-app.log","process.thread.name":"main","log.logger":"org.apache.nifi.wali.LengthDelimitedJournal"}{"@timestamp":"2022-09-02T07:49:50.866Z","log.level": "INFO","message":"Successfully recovered 0 updates from journal ../flowfile_repository/journals/474192833.journal","ecs.version": "1.2.0","service.name":"nifi-app.log","servi

I have a csv with 50 columns, is there a way to grab a specific column by it's header name and move it in a separate flow file for further processing?

Hi,     Basically, I am trying to import a database table from MySQL to hive but it turns out not importing to hive instead it gives me the following error. can anyone please help me with a solution which would be very much appreciated. [cloudera@quickstart ~]$ sqoop import --connect jdbc:mysql://localhost:3306/sqoop --username root --password cloudera --table customer --hive-importWarning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail.Please set $ACCUMULO_HOME to the root of your Accumulo installation.19/11/27 23:11:33 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.13.019/11/27 23:11:33 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.19/11/27 23:11:33 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override19/11/27 23:11:33 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc.19/11/27 23:11:33 INFO manager.MySQLManager: Preparing to use a MySQL s

Hi All, We are planning to upgrade our Nifi application(Linux platform) from 1.9.2 version to 1.16.2.Before proceeding with this major version jump , we wanted to check for any possible issues and backward compatibility . We do not see entry for version jump from 1.9.x to 1.16.x in below :  https://cwiki.apache.org/confluence/display/NIFI/Migration+GuidanceWe request your guidance for anything we need to be aware of before we make jump from Nifi 1.9.2 to 1.16.2 Thanks,Alvin

I have an app where after doing various processes in pyspark I have a smaller dataset which I need to convert to pandas before uploading to elasticsearch. I have  res = result.select("*").toPandas()  On my local when I usespark-submit --master "local[*]" app.pyIt works perfectly fine. I also have a 2 worker cluster, when I run it on my cluster using:spark-submit --master MASTER_IP:7077 app.pyI get the following error:Traceback (most recent call last): File "/home/nitinram/Elisity-Master/esaas/ml_model/landspeed_violation_spark/landspeed_violation_spark.py", line 173, in <module> results = landspeed_calculator(ps_df_compiled,landspeed_params, sc) File "/home/nitinram/Elisity-Master/esaas/ml_model/landspeed_violation_spark/landspeed_violation_spark_utilities.py", line 273, in landspeed_calculator res = result.select("*").toPandas() File "/opt/spark/python/lib/pyspark.zip/pyspark/sql/pandas/conversion.py", line 110, in toPandas split_batches=self_dest

hello cloudera community, how can we find out which group is tied to a sentry role by command line?

Hello friends, I hope you are very wellQuestion:according tohttps://docs.cloudera.com/cdp-private-cloud-base/7.1.3/managing-clusters/topics/cm-disable-tuned-service.htmlTuned service must be off, but for RHEL 8.x also applies?Regards!

[root@master cloudera-scm-server]# /opt/cloudera/cm/schema/scm_prepare_database.sh -h172.16.4.131 -P3306 -uroot -pZyht_1234 mysql scm scm Zyht_1234caJAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.332.b09-1.el7_9.x86_64Verifying that we can write to /etc/cloudera-scm-serverLoading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.[ main] DbProvisioner ERROR Exception when creating/dropping database with user 'root' and jdbc url 'jdbc:mysql://172.16.4.131:3306/?useUnicode=true&characterEncoding=UTF-8'java.sql.SQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'identified by 'Zyht_1234ca'' at line 1at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:120)[mysql-connector-java.jar:8.0.30]at co

I have a Spark sql query that works when I execute from inside a Jupyter Notebook that has a a PySpark kernel but fails when I execute it submitting to a Livy session. Usually there's no difference when I execute my queries both ways. I tried to get the spark session parameters with the command below and to guarantee that they are both the same:spark.sparkContext.getConf().getAll()  I'm using spark 2.3. How can I debug this problem? I know that the query works using spark, but I can't make it work submitting with Livy. Here is the query:INSERT INTO sbx_xxxxx.dados_auxiliares_mediana_preco    select        p.ano, p.cod_cfi, p.preco, p.repeticoes, sum(p2.repeticoes) as qtd_antes    from sbx_operacoes_digitais.dados_auxiliares_moda_preco p    left join sbx_operacoes_digitais.dados_auxiliares_moda_preco p2        on p.ano = p2.ano and p.cod_cfi = p2.cod_cfi and p2.preco <= p.preco    group by

Hi all, I have generated a query and the summary of Memory as below.Need your help to elaborate and how to read the summary.Thank you. Query(9c4786bd80e8ed20:1caa788500000000): Limit=3.00 GB Reservation=0 ReservationLimit=2.40 GB OtherMemory=0 Total=0 Peak=520.47 MB

Crossposting from: Stack Overflow(I will respond and give updates on both places)I would like to know if there is also maximum number for items below:Maximum number of databases in a catalog (I assume Hive Metastore only has one catalog, which is "hive")Maximum number of tables per database (as in, can I create 10 million tables in a database or due to limitation must I split them into 10 databases each with 1 million tables)I also would like to know whether the limitations are hard limit (unconfigurable), or configurable by Hive, or dependent on RDBMS it is using.

Hi all , is Apache Druid  supported as as a stand alone service in the Cloudera data flow (CDF)

Hello,first of all I apologize for posting this subject here,because I couldn't find an email address to contact the support directly.At this moment I am unable to ask support questions and I have the following messageIf anyone has had this problem please help me.thank you.

Hello,I am working on one of the tasks in Nifi and I am stuck on of the issue, the issue is I am getting the Json data from jolt transforms processor and also I am getting the Json data from input port processor now the issue is I need to combine the data from two processor based on id’s.  How to combined the data from two processors based on common id ? Will you please help me out.Thanks,Sathish

Dear Experts,We have some files in our HDFS server and want to get these files and put them into Isilon. We use GetHDFS and PutHDFS for this purpose. But, we get the lease and lock errors, as shown in the picture below. How can we fix these problems?Regards

On CDP Private Cloud 7.1.7, and running without rack awareness (an intentional choice due to some of the older servers running with smaller data disks), I am (as would be expected) seeing this spammed to the namenode logs: 2022-08-24 09:41:50,253 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Not enough replicas was chosen. Reason:{NO_REQUIRED_STORAGE_TYPE=1} Because I'm not logging INFO messages, I can eliminate those entries with something like: hadoop daemonlog -setlevel nnserverhost:9870 org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy WARN What's the best way to persist that across a restart?

Good morning,trying to install Cloudera Manager agents in a RH7.9 (this should be supported as seen in the Cloudera support matrix)I realized that this is not supported ¿!!!?The packages installed in the server are newer as expected (they are from rh7.9) and the cloudera manager is looking for older versions...What crazy is having a support matrix if this is not applied  any idea? downgrade dependencies are not considered. 

Hi , I'm trying to create a host and after enabling auto-tls .browser shows that certificate is invalid and not secure. I got failed to receive heartbeat from agent in the step of installing agent.  

Hi ,We are having four nodes CDP 7.1.7 SP1 cluster. Currently , when a normal hive user creates a database, the database ownership is getting assigned to hive:supergroup  in /external/hive which is default. We want the /external/hive database ownership to be assigned to the user who is creating a database. For example:User "Test " is creates one database, the ownership should be like:drwxr-xr-x - test supergroup 0 2022-07-05 17:35 /warehouse/tablespace/exteranl/hive/test_db Kindly let us know if this is possible.

Hello,i'm using a small 3Master+3workers cluster [Cloudera Manager 7.3.1 (Runtime 7.1.6)]Postgresql HA configured (not embedded)The thing is that I need to restart a tablet server if hte CM host is down, and obviously I can't do it using the CM.is there any way to restart these Kudu tablet servers without CM?NOTE: I've already tried sudo service kudu-tserver stopbutFailed to stop kudu-tserver.service: Unit kudu-tserver.service not loaded.

Im trying to run Apache NiFi on my laptop. I can start the run-nifi.bat But when I try to access http://localhost:8443/.  "This page isn’t working localhost didn’t send any data. ERR_EMPTY_RESPONSE"On the same laptop, I'm accessing different NiFi instances running on AWS. With different ports (Production).System: Windows10Please help to connect

Hi Team,we are trying to integrate kafka and hbase into atlas 2.2.0we are not using embedded kafka and hbase , seperately installed in same machine.kafka version : 2.13-2.5.0atlas version : 2.2.0/home/ranger/apache-atlas-sources-2.2.0/hook-bin/import-kafka.sh --topic Maestrotopic3>>>>> /home/ranger/apache-atlas-sources-2.2.0/hook-bin/import-kafka.sh>>>>> /home/ranger/apache-atlas-sources-2.2.0Using Kafka configuration directory [/data/kafka_2.13-2.5.0/config]Log file for import is /var/log/atlas/import-kafka.logPicked up _JAVA_OPTIONS: -Xms1024m -Xmx1024m -XX:MaxPermSize=512mOpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/data/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/data/kafka_2.13-2.5.0/libs/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/St

Hi all, I am new to Hive, was told below parameter used to improve hive performance, if i were to set and run those code sequence as below, does the sequence matters and correlated to each other? Do we need to put up these code whenever run the query? Or execute once will be sufficient? Thanks.  Set hive.exec.parallel = true;set hive.vectorized.execution.enabled = true;set hive.vectorized.execution.reduce.enabled = true;set hive.cbo.enable=true;set hive.compute.query.using.stats=true;set hive.stats.fetch.column.stats=true;set hive.stats.fetch.partition.stats=true;set mapred.compress.map.output = true;set mapred.output.compress= true;SET hive.exec.dynamic.partition = true;SET hive.exec.dynamic.partition.mode = nonstrict;set hive.auto.convert.join = false;set hive.exec.max.dynamic.partitions=100000;set hive.exec.max.dynamic.partitions.pernode=10000;