Ask the Community

Hi Community,I am unable to stop or start some Cloudera services.Cloudera version: 7.11.3CDP Version: 7.1.7 SP3Below is the type of error I get while trying to stop (e.g) impala daemon from the Cloudera Manager console.On the UI, it shows the status of the service is unknown as shown below. As you can see, the host 207 has a question mark in its status column, which signifies unknown status.Also, for other host (206, 208), the hdfs datanode has the same issue as that of the impala daemon.Apart from the impala daemon and hdfs datanode instances, I get the same issue on hue load balancer instance.Everything was fine a few days ago until yesterday. I have tried restarting the cloudera-scm-supervisord and the cloudera-scm-agent but no luck.Below is the cloudera-scm-agent.log error I get for all the hosts on which those services are running. It's like nothing else in the cloudera-scm-agent.log apart from the following error. [09/Nov/2024 15:18:43 +0000] 14061 __run_queue process E

Hi all,I am trying to insert a single value into a single column table and yet my job is getting stuck at Starting task [Stage-1:MAPRED] in serial modeYou can see the YARN is occupied completely. I have tried killing all the applications that come up when I do yarn application --list. And yet there will be many others popping up in place of the ones I killed, always occupying the resources. I am at loss of what to do. Could someone please help?

 Hi All,   We have the core configuration and tez not started after cluster restart .Kindly guide us how to troubleshoot and fix it 

I have installed 3 Ubuntu 20.04 live-server os and installed the CM trial version and tried to add a cluster, but I keep getting stuck on the attached image. After getting that error in the last check step, I can't get CM to reopen in any way. I have tried restarting the server and agent on the OS, but there is no response. How can I resolve these issues? Also, please let me know how is the best way to set the parcel url.

HiAm trying to install Impala Pro access through DBvisualizerbut logon fails with Java error messages Java Class ....Jar not foundI have installed the Impala driver on the clients Virtual windows , it seemsto uld place the drivers etc in the download folder.When I go into the DBVis Driver manager for the entry JDBC IMPALAhere is an error message "No driver classes fund check your artifact and jar files"The drivers path is displayed as C;...Downloads/impala/JDBC41.jar (inactive).My knowledge of Impala PRO and Java is very limited appreciate any tips !Could it be that Downloads is not in the Java path ?Thanks !Peter 

Hello Team,I faced an unexpected error during the execution of the SELECT query for Cloudera Apache Hive instance.I see the next message in the logs: errorClass:io.onetouch.exceptions.CrawlerSqlException, msg: java.sql.SQLException: [Cloudera][HiveJDBCDriver](500540) Error caught in BackgroundFetcher. Foreground thread ID: 66051. Background thread ID: 91264. Error caught: null. Driver: com.cloudera.HiveJDBC42 v2.6.25.1033JDK: OpenJDK 17Could you please assist with this issue? 

What am I doing:I am new to minifi and I want to get a bit workout on how to use minifi and generate a dataflow and send it to Apache NiFi.After going through the web I understood that first we need to design the flow using the NiFi and download its JSON definition as json.gz format and place it in the conf folder of minifi. Then start the minifi using run-minifi.bat file.These are things I have done to build the dataflow for MiNiFi:First I have created a pipeline to read from a table present in my Postgres SQL database using a ExecuteSQL processor, Remote Process Group, Input Port and logAttribute Processor.                                                                                                           &n

I’ve a question about my application on CML if someone could help!In the ‘Bob RAG Model’ application below, the status doesn’t change from ‘Starting’ even after the full script has been run successfully and my Gradio app has been created.I’m using the exact same settings (including instance parameters) as I have in the ‘Advanced Analytics LLM’ application, and that status changes to ‘Running’ when it’s gone through its script.The scripts are different, but the structure is pretty much the same – load the LLM/RAG model, define functions to query, open a Gradio app.Has anyone seen this before?Thanks 

Hello,I have exported the json in nifi and renamed it to flow.json.raw, but when I try to start minifi, I get the error like this:This is my flow in nifiAnd this is my json content：{ "externalControllerServices": { }, "flowContents": { "comments": "", "componentType": "PROCESS_GROUP", "connections": [ { "backPressureDataSizeThreshold": "1 GB", "backPressureObjectThreshold": 10000, "bends": [], "componentType": "CONNECTION", "destination": { "groupId": "4d2e3379-5165-32b0-bf2a-cc36a354aa29", "id": "8dd0c880-1b12-3097-90a0-5f20d488f88a", "instanceIdentifier": "846c8039-231d-353b-b87e-aed9f7408030", "name": "fromMinifi", "type": "REMOTE_INPUT_PORT" }, "flowFileExpiration": "0 sec", "groupIdentifier": "cad6891f-5564-3a61-8a99-bb7be344adc8", "identifier": "b0ca1df7-83e8-314b-bb2f-50735ab0ccd2", "instanceIdentifier": "1e076437-0193-1000-f190-07b262d28ba8",

Hi All,I feel this community is teaching me lots of things which In was not aware before.i am working on a project where i need to import the data from oracle and store that data into hdfs.I am using pyspark to load the data into DF and then storing the data into HDFS, however in  Oracle the table size is big and it has 420000000 record.Now i want to read those tables in parallel but but there are multiple tables and i am not able to make partition on the tables. is there any way to read the data in parallel when you don't know the partition column ?   

hdfs, yarn, impala, and kudu are currently in service.If an io error occurs on one disk in one server during operation,Is the service down?Should I just leave it until I replace the disk?

xecuting: /usr/java/jdk1.8.0_202-amd64/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.[ main] DbCommandExecutor ERROR Error when connecting to database.java.sql.SQLNonTransientConnectionException: Public Key Retrieval is not allowedat com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:102)[mysql-connector-j.jar:9.1.0]at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:114)[mysql-connector-j.jar:9.1.0]at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:

Hi Team,I am encountering error while listing topics using kafka-topics command,kafka-topics --list --bootstrap-server localhost:9093 --command-config ssl.properties> cat ssl.propertiessecurity.protocol=SSLssl.truststore.location=truststore_location_jksssl.truststore.password=truststore_password Error:INFO clients.NetworkClient: [kafka-admin-client-thread | adminclient-1]: [AdminClient clientId=adminclient-1] Cancelled in-flight METADATA request with correlation id 13 due to node -1 being disconnected (elapsed time since creation: 302ms, elapsed time since send: 302ms, request timeout: 26959ms)  Please help to solve this problem

Hi      I have hadoop cluster with version 3.4.0,  hive version 3.1.3 and tez 0.9.2When i try to insert data i am facing issue:insert into client_orc_update (client_name) values("hello");Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 95Serialization trace:conf (org.apache.hadoop.hive.ql.exec.TableScanOperator)aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork) another error i am getting on insert command with overwrite when transfering data from one table to another. Caused by: java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError: org.apache.hadoop.fs.FileStatus.compareTo(Lorg/apache/hadoop/fs/FileStatus;)I at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1790) ... 17 moreCaused by: jav

I have been using, on Amazon AWS, ubuntu 20.04 version with Hortonworks (Ambari (port 8080) and file system (port 4200). In ubuntu 20.04, in the Network settings, I created Security Groups to enable access to ports:      22 (Type: ssh, Protocol: TCP, Source Type: AnyWhere)     4200 (Type: Custom TCP, Protocol: TCP, Source Type: My IP)     8080 (Type: Custom TCP, Protocol: TCP, Source Type: My IP)then I installed Hortonworks, and from the browser I can access Ambari by going to http://<<IP address>:8080 ; or going to Hortonworks filesystem by going to http://<<IP address>:4200But now with unbuntu 22.04 and 24.04, I do the same steps above, but when I go to the browser, I cannot access Ambari by going to http://<<IP address>:8080; and I cannot go to Hortonworks filesystem by going to http://<<IP address>:4200I appreciate any insight and help.

Hi     I have hadoop cluster with namenode resourcemanager on on server, datanode o another server and hive, tez on different server.I am getting error on running query on beeline -  below are the yarn logs - it keeps trying to connect2024-10-31 15:57:49,806 [INFO] [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager] |rm.TaskSchedulerManager|: Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1111011112024-10-31 15:57:49,813 [INFO] [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager] |rm.YarnTaskSchedulerService|: YarnTaskScheduler initialized with configuration: maxRMHeartbeatInterval: 1000, containerReuseEnabled: true, reuseRackLocal: true, reuseNonLocal: false, localitySchedulingDelay: 250, preemptionPercentage: 10, preemptionMaxWaitTime: 60000, numHeartbeatsBetweenPreemptions: 3, idleContainerMinTimeout: 5000, idleContainerMaxTimeout: 10000, sessionMinHeldContainers: 02024-10-31 15:57:49,817 [INFO] [ServiceThread:org.apa

I have a python script that takes in text input in the format like this {"order":{"Testorder":{"operation":"create","specification":{"name":"${name}","version":"1.0.0-SNAPSHOT"},"parameters":{"controllerName":"${controllername}","parameters":{"parameter1":"${value1}","parameter2":"${value2}"}}}}}{"name": "TestService", "controllername": "MyController_d","value1": "test","value2": "speed"}  First line is a JSON Template with place holders. Second line another JSON string which holds the value to be replaced in the first line. (whole input (first line and second line together) are not a valid JSON) It creates an output like this  { "order" : { "Testorder" : { "operation" : "create", "specification" : { "name" : "TestService", "version" : "1.0.2-SNAPSHOT" }, "parameters" : { "controllerName" : "MyController_d", "parameters" : { "parameter1" : "test", "parameter2" : "strength" }

We were creating dashboards in Cloudera Cloudera Data Visualization and adding filters.For example, the following is a filter that has 137 records to select from:137 records is quite hard to scroll through, so we entered keywords to search for in the Search Box.But when we did so, we noticed that the keyword search is applied on the entire dataset. In other words, the keyword search is not applied on just the 137 records. This is strange and unexpected. Does anyone have a similar experience when adding filters in dashboards? Thank you.    

Hello everyone,do you happen to know if the GitHub integration of NiFi 2.0.0 M4 also works for GitLab? I think I have entered everything correctly, but I get the error message that my repo does not exist.With "normal" GitHub-API it Works fine!Best regards

Hello , can anyone help with a spec to transform this JSON input to the desired output .JSON INPUT=========={"courseId": "xxx"}== desired output json ==[{"courseId": "xxx","courseSubCategory": 1},{"courseId": "xxx","courseSubCategory": 2},{"courseId": "xxx","courseSubCategory": 3}]

I need to use ORACLE stored procedure with input PARAM and this input PARAM is coming from flow file and get output from that stored procedure. is any processor is available with run stored procedure and return the out put in AVRO schema or any other format? Please help me.

I have created the Cloudera trail account but due to a permission issue,  I tried to activate the default environment I can't activate it. how to fix this issue.  

pig-0.17.0bin/pig  -x localvery basic UDF file:#!/usr/bin/python3from pig_util import outputSchema@outputSchema("as:int")def square(num):if num == None:return Nonereturn ((num) * (num))@outputSchema("word:chararray")def concat(word):return word + wordExceedingly simple pig script:REGISTER '/home/scs/woodcock/SD411/lab_udf/test.py' USING org.apache.pig.scripting.streaming.Python.PythonScriptEngine AS myFuncs;A = LOAD '/home/scs/woodcock/SD411/DATA/accident.csv' USING PigStorage(',') AS (state:int,name:chararray);B = FOREACH A GENERATE myFuncs.square(state) AS state, name; If I do a "DUMP A" I get exactly what I would expect.But, on a "DUMP B", I get a failed job:java.lang.Exception: org.apache.pig.impl.streaming.StreamingUDFException: LINE :at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)Caused by: org.apache.pig.impl.streaming.StreamingUDFException: LINE :at org.apache.p

Hello,I have the following simple topologyi simply create two files in a folder that GetFile processor is configured to `look` , and expected to upload to the FTP serverThe configuration for the PutFTP is the following: If i am trying to FTP from the VM to the FTP server, it works like a charm: Any help, please?

Hi How do I host an API using HandleHttpRequest which can be used to upload files to NiFi server from local using "multipart/form-data".Have tried below approach, however I get read permission issue as the Nifi user can't access local data. Please suggest an approach which can be used to upload file from local.Thanks