hi cloudera,  I need to use Spark on a host that is not part of the Cloudera cluster to run Spark jobs on the Cloudera cluster. Is it possible to use it this way? If yes, how to configure? what I've already tried: 1. Download "https://www.apache.org/dyn/closer.lua/spark/spark-3.3.4/spark-3.3.4-bin-hadoop3.tgz" 2. Copy the "conf" files from the Cloudera cluster and send them to the new Spark directory 3. exported the variables "HADOOP_CONF_DIR" and "SPARK_CONF_DIR" and "SPARK_HOME" using the new spark directory "spark-3.3.4-bin-hadoop3" with the files 4. When trying to run spark-shell as an example, nothing happens, it hangs as shown below: Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 3.3.4 /_/ Using Scala version 2.13.8 (Java HotSpot(TM) 64-Bit Server VM, Java 11.0.16.1) Type in expressions to have them evaluated. Type :help for more information. note: the cluster has kerberos, so before running spark-shell, kinit was run ... View more

hi community cloudera,  we are verifying a strange behavior of an Impala connection controller on nifi, when the processor starts the connection, it displays the error below: [Cloudera][ImpalaJDBCDriver](500593) Communication link failure. Failed to connect to server. Reason: java.net.SocketException: Connection or outbound has closed communication between hosts is ok, ports communicating without problem. when we start and stop the processor the connection occurs without problem, this shows that the error is intermittent. has anyone experienced this type of problem? ... View more

hello cloudera community,   I have a problem in a datanode only in hdfs   this datanode has 10 disks and in the hdfs settings (dfs.datanode.data.dir) is configured the 10 disks   but for some time this datanode is not recognizing one of the disks configured only   was already done the restart of the datanode and even so did not back to recognize   looking at the datanode logs, it is not really recognizing the disk, only the 9 disks   these disks are also being used by other services in the cluster and these other services are able to recognize the 10 disks without any problem, only hdfs is having this problem   could you help us to solve this problem? ... View more

hello cloudera community,   do I need to insert a new datanode in the hdfs and with that I need to know how much size will be used from this new datanode after performing a rebalance on the hdfs?   for example, I have the following scenario:   datanode 1 has 36tb total to use, 30tb is being used, containing about 2.5 million blocks datanode 2 has 36tb total to use, 30tb is being used, containing about 2.5 million blocks datanode 3 has 36tb total to use, 30tb is being used, containing about 2.5 million blocks datanode 4 has 36tb total to use, 30tb is being used, containing about 2.5 million blocks datanode 5 has 36tb total to use, 30tb is being used, containing about 2.5 million blocks datanode 6 has 36tb total to use, 30tb is being used, containing about 2.5 million blocks   i will insert the "datanode 7" which has 36tb total, after performing the rebalance in hdfs, how much will this new datanode 7 receive data? ... View more