Q&A

【Problem】After upgrading Cloudera Manager to 7.11.3, attempting to reboot the cluster causes an error in the Lily HBase Indexer of the Key-Value Store Indexer.【Use】CDP Private Cloud Base 7.1.8 (Will upgrade to 7.1.9 later)OS: RHEL 7.6Java: OpenJDK 11.0.8Python: 3.8.12【Occurrence procedure】Before upgrading, I installed Python 3.8 on all hosts that contained Cloudera Manager Agents.The installation was done according to the following documentation:https://docs.cloudera.com/cdp-private-cloud-base/7.1.9/installation/topics/cdpdc-cm-install-python-3-rhel-custom-location.htmlThen, according to the following documentation, Cloudera Manager was upgraded from 7.7.1 to 7.11.3.https://docs.cloudera.com/cdp-private-cloud-upgrade/latest/upgrade-cdp/topics/ug_cm_upgrade.htmlVerified that the Server and Agent were upgraded successfully.Therefore, I proceeded with the steps in Step 5: After You Upgrade, "Perform Post-Upgrade Steps".When I rebooted the cluster in step 4, I encountered an error with the

i generated a .p12 certificate using tls-toolkit by using following command :- ./tls-toolkit.sh standalone -n 'localhost' -B admin1admin -C 'CN=admin,OU=lending'when i try to upload this certificate in keychain access, it shows invalid password and after trying multiple time with the correct password, an error is thrown "mac verification failed during pkcs12 import (wrong password?)". please help me in solving this issue.

I want to have multiple users in Nifi. I don't want to use LDAP because I don't have a LDAP. I don't want to use Kerberos because I don't have a Kerberos. I don't want to use Apache Knox because I don't have a Apache Knox. I don't want to integrate with the thing because I don't have the thing. I just want to have 3 users, each with their own username and password. I've gotten to the point where I have three users but I have no idea what password to use for the non-admin users.  

HelloI'm running Nifi toolkit v1.12.1 with JDK 8I'm running into a problem when running the following commandcli.sh nifi pg-list -p /nifi.properties --verboseI can do awget --certificate=./my.pem --private-key=./my.key --ca-certificate=./myca.pem https://xxx/nifi-api/flow/process-groups/rootand get a correct outputCan you help me get the corret output with the toolkit ?Is it possible to log the toolkit requests / responses ?The stack trace I get with the toolkit is: org.apache.nifi.toolkit.cli.api.CommandException: Error executing command 'pg-list' : nullat org.apache.nifi.toolkit.cli.impl.command.nifi.AbstractNiFiCommand.doExecute(AbstractNiFiCommand.java:65)at org.apache.nifi.toolkit.cli.impl.command.AbstractPropertyCommand.execute(AbstractPropertyCommand.java:74)at org.apache.nifi.toolkit.cli.impl.command.CommandProcessor.processCommand(CommandProcessor.java:252)at org.apache.nifi.toolkit.cli.impl.command.CommandProcessor.processGroupCommand(CommandProcessor.java:233)at org.apa

Hi all Getting below error, when trying to start hive metastore: Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.fs.FsTracer.get(Lorg/apache/Hadoop/conf/Configuration;)Lorg/Apache/htrace/core/Tracer;  Details are as under Hive version: Apache hive 3.1.3      Remote metastore: postgres version 12 Schematool init created all relevant tables without any error. Only when trying to start metastore services facing error. Checked for htrace core incubating jar versions also. Htrace version details across cluster are attachedapache-hive-3.1.3-bin/lib/htrace-core-3.2.0-incubating.jarhadoop-3.3.4/share/hadoop/common/lib/htrace-core-3.2.0-incubating.jarhadoop-3.3.4/share/hadoop/yarn/timelineservice/lib/htrace-core-3.2.0-incubating.jarhbase-2.4.14/lib/htrace-core4-4.2.0-incubating.jarhbase-2.4.14/lib/client-facing-thirdparty/htrace-core4-4.2.0-incubating.jar Request your help.Thanks

Hi,I am running NIFI in Kubernetes cluster with 3 nodes. After the upgrade of NIFI to 1.23.1 version, most of the time, the JSON messages or CSV files are corrupted something like below, which is causing serious production issues to my workflows. Actual message:{"RequestPeriod":"morning","TRANSMISSION_TYPE":"","ReturnDatetime":"2022-02-07T13:00:00+01:00", "PickupDatetime":"2022-02-07T08:00:00+01:00"}Corrupted message:1:00"}{"RequestPeriod":"morning","TRANSMISSION_TYPE":"","ReturnDatetime":"2022-02-07T13:00:00+01:00", "PickupDatetime":"2022-02-07T08:00:00+0 Additional screenshots:OUTPUT: This is become an URGENT issue now. Any support would be highly appreciated.

I want to configure Apache Spark for collecting metrics using Jolokia. For that I am able to do in standalone but not in Cloudera Data Platform as it requires to edit the conf directory in which bigdata.ini file is present. I want to add the following lines in it:- [Spark-Master]stats: http://127.0.0.1:7777/jolokia/read Do anyone have any idea that how can we configure this in Cloudera? 

Hi, I'm new to apache spark so Im not sure if this is the best set up, my goal is to create an environment where I can test and evaluate before making decision. I set up cluster on Windows using the steps from: https://aamargajbhiye.medium.com/apache-spark-setup-a-multi-node-standalone-cluster-on-windows-63d413296971 The cluster version Im using is the latest: 3.3.1\Hadoop 3 The master node is starting without an issue and Im able to register the workers on each worker node using the following comand:spark-class org.apache.spark.deploy.worker.Worker spark://<Master-IP>:7077 --host <Worker-IP>When I register the worker , its able to connect and register successfully as the message indicates , and Im able to see both workers in the US with the ALIVE status. Then I tried submitting  simple  hello_world py job using:spark-submit --master spark://<Master-IP>:7077 hello_world.py My hello_world.py application is like this:  s

Even after passing the executor memory and driver memory as 1 Gb still it is getting maximum heap size rounded to 512 MB spark-shell --conf spark.executor.memory=1g --conf spark.driver.memory=1gWarning: Maximum heap size rounded up to 512 MB Anything that is missing?

Hello, I have configured & set Hadoop Cluster over 2 nodes and launch it along with Yarn like so :  On the master node :hdfs namenode -regularyarn resourcemanager On the slave node : hdfs datanode -regularyarn nodemanagerAnd it shows through UI that there has been a connection established between the two machines that form the cluster.To note that start-dfs on the master node started both namenode and datanode even after setting slaves and hosts files.Now i submit an application (simple hello world) to Yarn : through this command :Spark-submit --class "main" --master yarn pathToJar But i get the error 15/08/29 12:07:58 INFO Client: ApplicationManager is waiting for the ResourceManager client token: N/A diagnostics: N/AApplicationMaster host: N/AApplicationMaster RPC port: -1queue: root.hdfsstart time: 1440864477580final status: UNDEFINED  user: hdfs 15/08/29 12:07:59 INFO Client: Application report for application_1440861466017_0007 (sta

I ran into an interesting situation using the Impala external table. In short, I used "create external table" statement but ended up with a table like a managed one. Here are details.Step 1: creating an external tablecreated external table testdb1.table1 ( fld1 STRING, fld2 STRING ) PARTITIONED BY ( loaddate INT ) STORED AS PARQUET tblproperties('parquet.compress'='SNAPPY','transactional'='false');Step 2: adding partitions and loading data files.alter table testdb1.table1 add if not exists partition (loaddate=20231018);load data inpath '/mytestdata/dir1' into table testdb1.table1 partition (loaddate=20231018);Step 2 shows that this table1 behaves exactly like a managed table. Files at /mytestdata/dir1 are moved to hdfs warehouse path warehouse/tablespace/external/hive/testdb1.db/table1/loaddate=20231018 path. If I drop this partition 20231018, the directory at warehouse/tablespace/external/hive/testdb1.db/table1/loaddate=20231018 is removed.So what exactly is the difference between a m

Goodmorning,I am having troubles with Ranger Service configuration for HDFS.Once opened the Cloudera Manager panel I go to: Clusters -> HDFS -> configuration, then I look up for "ranger" to enable this setting:The issue is that just after I saved the changes whenever I move to another webpage the setting is lost. Microsoft Edge would pop up a message saying "if you leave now some setting could go lost". This happens ONLY when I try to change this setting (Ranger Service for HDFS). It doesn't happen when I modify other values for HDFS or other services.Basically I can't use Ranger for HDFS. Can someone help me on this ? Why cloudera manager won't properly save this setting ? 

We are working on below configuration platform.Cloudera Version: CDP 7.1.2Kerberos 5 version 1.15.1Hive 3.1.3000.7.1.7.1000-141Phoenix 5.1.5After kerberos enabled on the platform, we are not able to connect to hive and phoenix from DBeaver or sql client.

Hi, I have the following problem - I download data from the database using QueryDatabaseTable, I receive the data in the avro-binary format, so then I use ConvertAvroToJson - everything converts nicely except the UUID field, I am left with the following json:{"Name":"Mary","Surname":"Smith","Mail":"m.smith@google.com","UUID":{"bytes":"_쪨Ը¬I³Sñ])º\u000E•"},"City":"New York"}I tried to extract this UUID as an attribute and replace it with the :toString() method but it doesn't work, does anyone have an idea how to convert it to a string? 

I am new to cloudera rest api.I try to upgrade the cloudera runtime with rest api instead of the web-ui for automation.  I am able to download the CDH with rest api.  curl -X POST -u admin:admin "http://{url}:7180/api/v44/clusters/{cluster-name}/parcels/products/CDH/versions/7.1.8-1.cdh7.1.8.p0.30990532/commands/startDownload" I am stuck on it ask for upgrade the cluster when it finish the download. From the Web-UI. it is a series of process.   I can skip all the checkup step(I understand the risks) and upgrade the cluster. How to do that with the rest api?Thank you very much!! 

Hi ,we have added a user via ldap and using super user we have given Grants to the added user but when we log-in in hue using the same and later try to execute create table statement its giving us:Authorization Exception: User <user> does not have privileges to execute 'CREATE' on: defaultas we don't have sentry admin password with us. I need to reset it so that i can log-in using sentry admin and provide the privilege's.i am thinking it should work. please help if there is another work around for this. urgently we need to give privilege's to the added user.

I have a Spring boot application using HBase Java client, version 2.1.0-cdh6.1.1, connect to a Hbase with kerberos. When running a tomcat instance in our server, a simple query could take 5 minutes to complete and timeout found in log file.  I can't find out what's wrong.Connection configuration:  config.set("hbase.zookeeper.quorum", quorum); config.set("hbase.zookeeper.clientPort", "2181");config.set("hadoop.security.authentication", "kerberos");config.set("hbase.security.authentication", "kerberos");config.set("hbase.cluster.distributed", "true");config.set("hbase.rpc.protection", "privacy");config.set("hbase.regionserver.kerberos.principal", "hbase/_HOST@mycompany.com");config.set("hbase.master.kerberos.principal", "hbase/_HOST@mycompany.com"); config.set("hbase.client.retries.number", "3");config.set("hbase.client.pause", "500");config.set("zookeeper.recovery.retry", "1"); System.setProperty("java.security.krb5.conf",ResourceUtils.getFile("classpath:krb5.conf"

Hello all, How to transfer .csv or txt file from CDSW environment to Unix server using sftp?Or do we have any inbuild service to transfer files from CDSW to Unix?  - Sharing public key to UNIX server- run sftp scripts to trassfer using python or unix script How to enable sftp protocol?   

Hi, I face the following problem.I have a service user say "service_dwh" used for our datawarehouse that queries heavily our data reservoir using hive queries.I have had some cases where due to the query and/or to missing statistics a single hive query could take 100% of the resources available for the "service_dwh" user.I couldn't find a way using capacity scheduler, queues and user limit factor to prevent a single application from taking all the resources for a very long time.Traditional DBMS have mechanism that throttle job resources based of the job duration.That way a long (big) job can't monopolyze ressources for new and potentially shorter jobs for too long.

I added some new nodes to my cluster and it works fine. Then I add Spark Gateway roles to all the new nodes. We're using Yarn to manage and distribute Spark work.Does adding Spark Gateway roles to new nodes enough to make Yarn think like "Hey there are some new nodes here, let's distribute some containers and work to these new nodes"? Or do I have to add Yarn Gateway roles to these new nodes too?How to make sure that Yarn will use these new nodes when executing jobs to reduce the overall workload of my cluster  

In case of imgae, only one single value(number) in second queue(name: matched).  How can I insert data(value in queue) into table use SQL?"insert into table values ( queue )"I want insert 1321 into table in this case.I mean i want make like as "insert into table values (1321)"Which variable or parameter can I use?insert into table values ( ${???????})Thank you for your answer.

Hi all,I downloaded today the Docker Compose configuration file in order to install a local development environment for the Data Stream Community EditionThe prometheus container fails to start exiting with exit code 126 and try forever to restart with the same problem.The image for the container isdocker.repository.cloudera.com/cloudera/csp-ce/prometheus:2.0.0.0Someone else has the same issue?Any idea about the solution?ThanksMassimo 

I found the old thread that is exactly similar to the issue I am having .https://community.cloudera.com/t5/Support-Questions/SQOOP-import-of-quot-image-quot-data-type-into-hive/m-p/376825#M243019column data type in source [sql server] : imagecolumn value in source : 0xFFffDeDJBF.......dDf.      --> ( A hexadecimal value )column data type in hive: string column value in hive: FFffDeDJBF.......dDf (0x is not retained)Also I have tried binary data type in hive. yet the value would not match with what oracle has. Is there a data type hive table can handle and retain the  'Image' datatype orginal value in SQL source while sqoop importing into the hive table ?  

Hi,Every 5 min i have a file generated from the NiFi Processor GenerateFlowFile:filename_0.xml filename_1.xmlfilename_2.xml....I would like to asked how do i reset the sequence no. everyday ?Thanks. Would appreciate help with example and explanation. The first method i found is using nextInt():the second method is using seq:Please advise me with steps and example. thanks 

Hi,I have one flowfile with data I get from an external source (stock of a supplier). The products are already in a database. So now I have to check whether the stock has changed. If the stock of my flowfile is equal to the one in the database, I don't want to write the entry into the database (avoid duplicates).Maybe someone of you has an idea how to implement this best.Many thanks for your help!