Q&A

Dear FriendsThe concerned Cloudera Services are: Spark 3 , Livy for Spark 3, HueAfter I have added Spark SQL interpreter in Hue I am struggling  to get more than 1000 rows results once querying  the DB. Do you know what parameter should be used to change the default quantity. I Already tried to use :spark.livy.sql.max-resultCM --> Livy for Spark3 --> Search for "Livy for Spark 3 Client Advanced Configuration Snippet (Safety Valve) for livy3-conf/livy-client.conf" and add the above parameter with value.spark.livy.sql.max-result=2000However after restarting the concerns services - nothing has changed.  Hereunder you can see what is all about:I would like to change this limitation to e.g. 5000. But not sure on what mentioned services this should be adjusted on.I really appreciate your help in this.Thank you

Hi all, I have a cluster in CDP version 7.1.8 private cloud base. The Cloudera Management service show bad health and indicate that the connection to KDC server is not available. I can regenerate missing credentials under Administration > Security > Kerberos Credentials Moreover, I can do kinit in the CM server host. Therefore I have no idea why the connection shows failure. Please let me know if I should provide further information. Thank you.

Hi All,After starting up a Cluster that has been shutdown for some time I find that one of the Oozie servers will not stay up.  It initially starts then goes down.  There are two errors reported in the log file.  I assume that one of them is responsible but I'm not sure which.  Any help would be appreciated.  The errors are:First Error:Source: PurgeXCommandMessage: <openjpa-2.2.2-r422266:1468616 fatal general error> org.apache.openjpa.persistence.PersistenceException: The column index is out of range: 3, number of columns: 2.FailedObject: select w.id, w.parentId from WorkflowJobBean w where w.endTimestamp < :endTime and w.parentId like '%C@%' [java.lang.String]Second Error:Source: CoordStatusTransitXCommandMessage: SERVER[<redacted>] USER[<redacted>] GROUP[-] TOKEN[] APP[<redacted>] JOB[0000068-201217031325641-oozie-oozi-C] ACTION[-] XException,org.apache.oozie.command.CommandException: E0606: Could not get lock [c

 Hello, I am trying to connect to a remote hive cluster which requires kerberos, using spark mysql connector ```# Importsfrom pyspark.sql import SparkSession# Create SparkSessionspark = SparkSession.builder \.appName('SparkByExamples.com') \.config("spark.jars", "/Users/u530241/spark3/spark-3.3.2-bin-hadoop3/jars/mysql-connector-java-5.1.45-bin.jar").enableHiveSupport().getOrCreate() df = spark.read \.jdbc("jdbc:hive2://p-cdl-knox-prd-svc.hcscint.net:8443/default;transportMode=http;httpPath=gateway/default/hive;ssl=true;sslTrustStore=/Users/u530241/Downloads/gateway_prod_.jks;trustStorePassword=knoxprod","member_connect",properties={"user": "u530241", "password": "..", "driver":"com.mysql.jdbc.Driver"})```I keep getting this error - ```IllegalArgumentException: requirement failed: The driver could not open a JDBC connection. Check the URL: jdbc:hive2://p-cdl-knox-prd-svc.hcscint.net:8443/default;transportMode=http;httpPath=gateway/default/hive;ssl=true;sslTrust

I have a ranger policy defined under hadoop_sql plugin in ranger and I am able to access tables using hive editor but when I use impala editor and access same table am getting authorization error

I tried steps below. 1. Installed harbor. (aaa.bbb.ccc)2. Created new image use the dockerfile and command below--------FROM aaa.bbb.ccc/cloudera/cdsw/ml-runtime-jupyterlab-python3.10-standard:2023.08.2-b8 RUN apt-get update && apt-get upgrade -y && apt-get clean && rm -rf /var/lib/apt/lists/# Cloudera metadata for CMLENV ML_RUNTIME_EDITION="EDITED RUNTIME" \ML_RUNTIME_SHORT_VERSION="0.1" \ML_RUNTIME_MAINTENANCE_VERSION=1 \ML_RUNTIME_DESCRIPTION="EDITED RUNTIME to test"ENV ML_RUNTIME_FULL_VERSION="${ML_RUNTIME_SHORT_VERSION}.${ML_RUNTIME_MAINTENANCE_VERSION}"LABEL com.cloudera.ml.runtime.edition=$ML_RUNTIME_EDITION \com.cloudera.ml.runtime.full-version=$ML_RUNTIME_FULL_VERSION \com.cloudera.ml.runtime.short-version=$ML_RUNTIME_SHORT_VERSION \com.cloudera.ml.runtime.maintenance-version=$ML_RUNTIME_MAINTENANCE_VERSION \com.cloudera.ml.runtime.description=$ML_RUNTIME_DESCRIPTION---------docker build -t aaa.bbb.ccc/cloudera/cdsw/ml-runtime-jupyterlab-

Hello,I'm seeing these logs in the namenode, it seems it can't connect to the datanodes? that we have2024-04-26 14:45:21,470 DEBUG net.NetworkTopology: No node to choose.2024-04-26 14:45:21,470 DEBUG blockmanagement.BlockPlacementPolicy: Failed to choose from the next rack (location = /rack-10.128.44.130), retry choosing randomlyorg.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy$NotEnoughReplicasException:at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:829)at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:717)at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseFromNextRack(BlockPlacementPolicyDefault.java:660)at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseLocalRack(BlockPlacementPolicyDefault.java:636)at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlac

Hi All,I am aware once we upgrade CDP PB from x to y, we need to finalise HDFS upgrade if it is required. However, recently in the team got to know from someone who has done the upgrade that we also need to finalize Kafka upgrade along with HDFS. What needs to be done with finalising Kafka upgrade? Is it a newer thing in latest CDP version or something I have always missed on previous Kafka clusters? Haven't had a need of doing this. Thankssnm1523

Given 6 tables with identical column and all partitioned by dateNeed to get result of a union results from all 6 tablesonly if all tables have data for the given date i.e. partitionelse return nothing.  

Is there a way to do something like:case   when (select count(*) from table1 > 0) then (select * from table1)   when (select count(*) from table2 > 0) and (select count(*) from table3 > 0) then (select * from table3)end

Hi there!I have set up Hive and added Hive Server 2 in Cloudera Manager Trial version in order to connect with beeline.Both are in green status but neither port 10000 nor 10002 are listening.The message I get from beeline is "Failed to connect to <HOST>:10000Could not open connection to the HS2 server. Please check the server URI and if the URI is correct, then ask the administrator to check the server status.Error: Could not open client transport with JDBC Uri: jdbc:hive2://<HOST>:10000/: java.net.ConnectException: Connection refused (Connection refused) (state=08S01,code=0)"Here you have the way I'm hitting beeline:beeline -u jdbc:hive2://<HOST>:10000/;principal=hive/USR@PWDor!connect jdbc:hive2://<HOST>:10000 usr pwd org.apache.hive.jdbc.HiveDriver are there some steps missing to start HS2?  

Configured Knoxsso without Ambari. Knoxsso is configured with Shiro provider and updated the core-site.xml with below configshadoop.http.authentication.typehadoop.http.authentication.authentication.provider.urlhadoop.http.authentication.public.key.pemFollowed: https://knox.apache.org/books/knox-1-6-0/user-guide.html#KnoxSSO+Setup+and+ConfigurationAfter restart, NN UI is redirecting to KnoxSSO and after entering the AD credentials it is throwing below error in the UI. The redirect to originalUrl looks valid from Knoxsso url.ERROR Invalid Redirect: Possible Phishing AttemptAny help is appreciated!

Hi all, I'm looking to create a query to check and validate the math on certain fields from different tables. I've found examples where the SUM function was used- but I struggled putting it together, and am just trying to find the best way to do these calcuations.Sample data:Customer table:Customer IDMoney Spent000150000022500003100Sales table:Order IDAmount50055005006250500750Check sales math work in progress query(the query doesn't seem to work, returns a "delta" column with a {null} result:SELECTCAST(SUM(Customer.Money Spent - Sales.Amount) AS INT) AS DELTAThe desired output that I'd like:Check table Rule NameDeltaSales Check0Sales Check0Sales Check50Thanks in advance,

Hi,I want to change the flowfile content (json format) using UpdateRecord by providing property (to be change) from attribute.Here is the flowfile content      here is the flowfile attribute I want to change the value of "id"  in flowfile content using UpdateRecords like this : ( WHICH IS NOT WORKING ) This works but add a new property to existing flow file Here "id" got change because i provided "/id" (hardcoded property name) , i need same functionality to work with "/${col_name}" Any Pointers ... ?????Thanks,Riyaz

HI Team,I am using clouera manager to install Ranger service.Ranger UI is working and its asking Username and password , I type username & password `admin` but its showing invalid password.Pls let me know the default password for ranger to login into UI.  

Hi Team, I installed atlas in CDP using cloudera manager.Atlas service is up and running but Atlas Web UI showing error.```http://104.198.54.113:31000/HTTP ERROR 503 Service UnavailableURI:STATUS:MESSAGE:SERVLET:/503Service Unavailable-Powered by Jetty:// 9.4.39.v20210325``` 

I'm able to list Knox topologies in Ranger and created Ranger policy for webhdfs. Ran the script ranger-knox-plugin.sh and created ranger-security, audits and policymgr-ssl xmls.Configured one topology for webhdfs in knox and added below<provider><role>authorization</role><name>XASecurePDPKnox</name><enabled>true</enabled></provider>When I run curl to webhdfs I'm getting 403 error and I could see in Ranger audits access is denied by ranger-acl.I could nail it down to policy not getting downloaded from Ranger but don't see any error in Knox gateway.log even after setting the logging to debug.Knox is enabled with self-signed and ranger with no ssl. I imported knox cert to Ranger cacert.Am I missing any steps in Knox configuration that could be preventing the policy download?Any help is appreciated!

Hello my client is a WIN server 2019, setup outbound Kerberos only, I have enabled WIN registry keys for TGT Session sharing, environment var pointing to CDP JARS, CONFIG paths and default Java_Home to the bundled Zulu private JRE which these days is using Java 11 and comes with unlimited strength JCE (per provider).  I could do kinit and receive klist.  SAS is the application.  I have setup the CDP JDBC URI provided by CDP team and trustore w credentials.I have not done krb5.ini, nor jaas config, nor pointed the env variable java.security.auth.login.config to jaas config.I get this error:ERROR: java.sql.SQLException: [Cloudera][HiveJDBCDriver](500168) Error creating login context using ticket cache: Unable to obtain Principal Name for authentication .ERROR: Error trying to establish connection: [Cloudera][HiveJDBCDriver](500168) Error creating login context using ticket cache: Unable to obtain Principal Name for authentication .I researched here and some

Hello.I'm using 1.13.2.RC1 and recently i started to notice something, it suddenly stops to process all the flows and hangs the active threads by not giving any response anymore.There's no error in the log, as the time passes, it's only showing a bunch of Minimal Locking Write Ahead Log, and nothing more than this.As long i do not restart, it keeps as it is infinitely, until i restart and everything starts to work once again.Can somebody please give me a light to what's happening?In this print, there's a process group and in there, a bunch of Query Database Table and some other processors, and it's hang.Example of hang Query Database Table:Once i restart, the infinite active thread quits. Important info:Maximum timer driven thread count: 20Maximum event driven thread count: 5Connection Pool 1.13.2: Max Connection Pool: 15. The database itself is working properly and fine. Thank you in advance.

I was deleted hive services with some tools (livy, hue, kudu, oozie, hive on tez) because I forgot to keep All of the password. Now I want to re-add services that I deleted. but I have no clue for this. I don't know username and password.Please help

Hi everyone. I created 2 queue yarns in cdp 7.1.8. Precisely I created 2 queues: users queue --> configured capacity 50% and maximum capacity 100% hive queue -->configured capacity 50% and maximum capacity 100% I enabled child queues mode for all 2 queues with the following configuration parameters:Dynamic Queue Minimum User Limit100%Dynamic Queue User Limit Factor1Dynamic Queue Maximum Applications1000Dynamic Queue Maximum AM Resource Limit20%Dynamic Queue Ordering PolicyFairThe problem we encounter is that if even just 2 queries are executed in the same queue the second goes pending and is not executed in parallel, I show an example belowAnother example would be an oozie job that only calls a spark job. The oozie job runs infinitely waiting for the pyspark to run. However, in the default queue this problem is not present.The development cluster is made up of 3 worker nodes (144 vcores 180 gb yarn memory).Can you provide us support or optimizations to do at the queue configurati

Hello Team,I am unable to run the import command for Hive, since it is failing with error invalid path.I copied the table content to another folder and it has permission for hive user, but while running import command it says invalid path. i ran import command on the path where the source table location still it fails with same reason.Beeline version 3.1.3000.7.1.8.28-1 by Apache Hive[root@machine1 Log_Files]# sudo -u hive hadoop fs -ls /warehouse/tablespace/managed/hive/demo.db/test1/*Found 1 items-rw-r--r-- 3 hive hive 621 2024-05-02 16:05 /warehouse/tablespace/managed/hive/demo.db/test1/delta_0000004_0000004_0000/bucket_00000_0[root@machine1 Log_Files]# sudo -u hive hadoop fs -ls /jay123/restore_jobid123243/jayesh/*Found 1 itemsdrwxr-xr-x - hive hive 0 2024-05-02 16:05 /jay123/restore_jobid123243/jayesh/test1/delta_0000004_0000004_0000[root@machine1 Log_Files]# sudo -E -u hive beeline -u "jdbc:hive2://machine1.testride.com:2181,machine22.testride.com:2181/;serviceDiscoveryMode=zooKe

I am using Apache nifi invokeHttp processor to call Google API. The url contains a path which i need to encode.So in the URL,only path is encoded e.g. resources%2F8cb90ae6-16cc-4460-99f4-3ae0689b3b66%2Fdata_asset%2FCETIvowels-38.pdfHowever whenver I set encoded url, the processor internally replaces %2F with %252F which results in HTTP 404 response.E.g. Nifi convert URL - https://storage.googleapis.com/storage/v1/b/t-bis-1hg-jk/o/resources%2F8cb90ae6-16cc-4460-99f4-3ae0689b3b66%2Fdata_asset%2FCETIvowels-38.pdf?alt=mediatohttps://storage.googleapis.com/storage/v1/b/t-bis-1hg-jk/o/resources%252F8cb90ae6-16cc-4460-99f4-3ae0689b3b66%252Fdata_asset%252FCETIvowels-38.pdf?alt=media 

I am getting the below error while installing the agent. I am working on Ubuntu 20.04. It's a single machine cluster I want to make. Can anyone help me out with this? Cleaning the package manager cache... View Failed to acquire installation lock. View/tmp/scm_prepare_node.OHxqRMPmusing SSH_CLIENT to get the SCM hostname: 10.4.47.28 49784 22opening logging file descriptorStarting installation script...Acquiring installation lock.../tmp/scm_prepare_node.OHxqRMPm/scm_prepare_node.sh: line 1415: /tmp/.scm_prepare_node.lock: Permission deniedcould not open lock file, giving upwaiting for rollback request detected rollback requestrolling back installationReverting changes...rollback startedRemoving package repositories...Cleaning the package manager cache...unknown packager , exitingclosing logging file descriptor  

How to enable IAM for airflow either as ldap or oauth2. Please suggest