Solved

I have executed the command hdfs namenode -reformat on my hadoop cluster and the namenode data files have newly generated VERSION file.This created inconsistency as the datanode and namenode VERSION files have different cluster-ID.I manually changed the cluster-id in the datanode VERSION file to match the namenode one. This resulted in succesful start of the cluster but after the start there is a new block create with new blockpoolID which is empy. All my data is in the old datafile with old blockpoolID but it is currently not mapped to my hadoop filestystem.How can I recover my old data and make it accessible in my hadoop cluster again?Can I somehow just start over with my old datafile from datanode?

using commandALTER TABLE test table DROP PARTITION (START_DATE = DATE_ADD('2023-09-13', -13) )error: SQL Error [500051] [HY000]: [Cloudera][HiveJDBCDriver](500051) ERROR processing query/statement. Error Code: 40000, SQL state: TStatus(statusCode:ERROR_STATUS, info Messages:[*org.apache.hive.service.cli.HiveSQLException:Error while compiling statement: FAILED: ParseException line 1:78 cannot recognize input near 'DATE_ADD' '(' ''2023-09-13'' in constant:17:16Need Result : '2023-08-31'

Hi Team,How can I replace the custom ConsumeASB Processor with the InvokeHttp Processor in NiFi? I am currently using a Topic subscription with my ConsumeASB Processor to read messages from an Azure Service Bus endpoint.How can I configure the InvokeHttp Processor to achieve the same functionality?Could you please provide some inputs.

Hello!Xml file has the following structure:  <root> <nested1> <nested2> <nested1>value</nested1> </nested2> </nested1></root>           EvaluateXPath processor type_rec  property:  name(/*/*[1])And I got the following error:      EvaluateXPath[id=ce4e0c05-169f-387f-520d-77e2e664bacc] Input parsing failed FlowFile[filename=69de5f2c-a887-4042-8734-beca03e3d27b]: org.apache.nifi.xml.processing.ProcessingException: Parsing failed- Caused by: org.xml.sax.SAXParseException; lineNumber: 6; columnNumber: 5; The element type "nested2" must be terminated by the matching end-tag "</nested2>".As I understand, parser except, at 4 level closed tag </nested1> , because it'v met open tag with the same name at level 2 (<nested1>).   But at level 4 file has a new tag with the same name. I tried to use ReplcaseText processor for

2024-11-19 04:52:41,378 WARN [Monitor Processor Lifecycle Thread-1] o.a.n.controller.StandardProcessorNode Timed out while waiting for OnScheduled of ExecuteScript[id=acb441ba-c36b-1fdd-53f2-3a4821d43833] to finish. An attempt is made to cancel the task via Thread.interrupt(). However it does not guarantee that the task will be canceled since the code inside current OnScheduled operation may have been written to ignore interrupts which may result in a runaway thread. This could lead to more issues, eventually requiring NiFi to be restarted. This is usually a bug in the target Processor 'ExecuteScript[id=acb441ba-c36b-1fdd-53f2-3a4821d43833]' that needs to be documented, reported and eventually fixed.

Hi,I wanted to know if it is possible to disable kerberos once it has enabled in cloudera. I couldn't find any option to for kerberos in he configuration in cloudera.

Hello,Yesterday, I purchased the CDP-2001 CDP Administrator - Private Cloud Base Exam, but I haven’t yet received the follow-up email to schedule my exam. I only received the payment confirmation. All of my colleagues received their scheduling email right away. Could you please look into this for me?Thank you!

HiI'm new to NiFi and exploring my options.We are planning to use NiFi as a File Movement platform, where we will be transferring thousands of files everyday of sizes ranging upto 50GB without any extraction, transformation or loading. We just need to pull the file from one location and push to another location using SFTP and/or S3.Is NiFi a right tool to use to do this activity ? Thanks  

Hi,         I am trying to run simple query on Hive with execution engine as Tez,Here is the query     Insert into table test values('a', 'b');there are only two columns in table and both are string columns.If i insert into int column  then it works fine.also if i insert through load data command, this also works fine with string command. Below is the error that i am getting in yarn logs: ERROR : Status: FailedERROR : Vertex failed, vertexName=Map 1, vertexId=vertex_1731065811939_0001_1_00, diagnostics=[Vertex vertex_1731065811939_0001_1_00 [Map 1] killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: _dummy_table initializer failed, vertex=vertex_1731065811939_0001_1_00 [Map 1], java.lang.RuntimeException: Failed to load plan: hdfs://localhost:9000/tmp/hive/hadoopuser/a86dedab-ce0f-4f06-8056-c37aef3cca55/hive_2024-11-11_12-26-32_369_2945486912443049901-2/hadoopuser/_tez_scratch_dir/8d991848-0c76-4ffa-a82e

Hi Community,I am unable to stop or start some Cloudera services.Cloudera version: 7.11.3CDP Version: 7.1.7 SP3Below is the type of error I get while trying to stop (e.g) impala daemon from the Cloudera Manager console.On the UI, it shows the status of the service is unknown as shown below. As you can see, the host 207 has a question mark in its status column, which signifies unknown status.Also, for other host (206, 208), the hdfs datanode has the same issue as that of the impala daemon.Apart from the impala daemon and hdfs datanode instances, I get the same issue on hue load balancer instance.Everything was fine a few days ago until yesterday. I have tried restarting the cloudera-scm-supervisord and the cloudera-scm-agent but no luck.Below is the cloudera-scm-agent.log error I get for all the hosts on which those services are running. It's like nothing else in the cloudera-scm-agent.log apart from the following error. [09/Nov/2024 15:18:43 +0000] 14061 __run_queue process E

 Hi All,   We have the core configuration and tez not started after cluster restart .Kindly guide us how to troubleshoot and fix it 

HiAm trying to install Impala Pro access through DBvisualizerbut logon fails with Java error messages Java Class ....Jar not foundI have installed the Impala driver on the clients Virtual windows , it seemsto uld place the drivers etc in the download folder.When I go into the DBVis Driver manager for the entry JDBC IMPALAhere is an error message "No driver classes fund check your artifact and jar files"The drivers path is displayed as C;...Downloads/impala/JDBC41.jar (inactive).My knowledge of Impala PRO and Java is very limited appreciate any tips !Could it be that Downloads is not in the Java path ?Thanks !Peter 

I’ve a question about my application on CML if someone could help!In the ‘Bob RAG Model’ application below, the status doesn’t change from ‘Starting’ even after the full script has been run successfully and my Gradio app has been created.I’m using the exact same settings (including instance parameters) as I have in the ‘Advanced Analytics LLM’ application, and that status changes to ‘Running’ when it’s gone through its script.The scripts are different, but the structure is pretty much the same – load the LLM/RAG model, define functions to query, open a Gradio app.Has anyone seen this before?Thanks 

Hello,I have exported the json in nifi and renamed it to flow.json.raw, but when I try to start minifi, I get the error like this:This is my flow in nifiAnd this is my json content：{ "externalControllerServices": { }, "flowContents": { "comments": "", "componentType": "PROCESS_GROUP", "connections": [ { "backPressureDataSizeThreshold": "1 GB", "backPressureObjectThreshold": 10000, "bends": [], "componentType": "CONNECTION", "destination": { "groupId": "4d2e3379-5165-32b0-bf2a-cc36a354aa29", "id": "8dd0c880-1b12-3097-90a0-5f20d488f88a", "instanceIdentifier": "846c8039-231d-353b-b87e-aed9f7408030", "name": "fromMinifi", "type": "REMOTE_INPUT_PORT" }, "flowFileExpiration": "0 sec", "groupIdentifier": "cad6891f-5564-3a61-8a99-bb7be344adc8", "identifier": "b0ca1df7-83e8-314b-bb2f-50735ab0ccd2", "instanceIdentifier": "1e076437-0193-1000-f190-07b262d28ba8",

Hi All,I feel this community is teaching me lots of things which In was not aware before.i am working on a project where i need to import the data from oracle and store that data into hdfs.I am using pyspark to load the data into DF and then storing the data into HDFS, however in  Oracle the table size is big and it has 420000000 record.Now i want to read those tables in parallel but but there are multiple tables and i am not able to make partition on the tables. is there any way to read the data in parallel when you don't know the partition column ?   

hdfs, yarn, impala, and kudu are currently in service.If an io error occurs on one disk in one server during operation,Is the service down?Should I just leave it until I replace the disk?

xecuting: /usr/java/jdk1.8.0_202-amd64/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.[ main] DbCommandExecutor ERROR Error when connecting to database.java.sql.SQLNonTransientConnectionException: Public Key Retrieval is not allowedat com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:102)[mysql-connector-j.jar:9.1.0]at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:114)[mysql-connector-j.jar:9.1.0]at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:

Hi      I have hadoop cluster with version 3.4.0,  hive version 3.1.3 and tez 0.9.2When i try to insert data i am facing issue:insert into client_orc_update (client_name) values("hello");Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 95Serialization trace:conf (org.apache.hadoop.hive.ql.exec.TableScanOperator)aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork) another error i am getting on insert command with overwrite when transfering data from one table to another. Caused by: java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError: org.apache.hadoop.fs.FileStatus.compareTo(Lorg/apache/hadoop/fs/FileStatus;)I at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1790) ... 17 moreCaused by: jav

Hi     I have hadoop cluster with namenode resourcemanager on on server, datanode o another server and hive, tez on different server.I am getting error on running query on beeline -  below are the yarn logs - it keeps trying to connect2024-10-31 15:57:49,806 [INFO] [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager] |rm.TaskSchedulerManager|: Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1111011112024-10-31 15:57:49,813 [INFO] [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager] |rm.YarnTaskSchedulerService|: YarnTaskScheduler initialized with configuration: maxRMHeartbeatInterval: 1000, containerReuseEnabled: true, reuseRackLocal: true, reuseNonLocal: false, localitySchedulingDelay: 250, preemptionPercentage: 10, preemptionMaxWaitTime: 60000, numHeartbeatsBetweenPreemptions: 3, idleContainerMinTimeout: 5000, idleContainerMaxTimeout: 10000, sessionMinHeldContainers: 02024-10-31 15:57:49,817 [INFO] [ServiceThread:org.apa

We were creating dashboards in Cloudera Cloudera Data Visualization and adding filters.For example, the following is a filter that has 137 records to select from:137 records is quite hard to scroll through, so we entered keywords to search for in the Search Box.But when we did so, we noticed that the keyword search is applied on the entire dataset. In other words, the keyword search is not applied on just the 137 records. This is strange and unexpected. Does anyone have a similar experience when adding filters in dashboards? Thank you.    

Hello everyone,do you happen to know if the GitHub integration of NiFi 2.0.0 M4 also works for GitLab? I think I have entered everything correctly, but I get the error message that my repo does not exist.With "normal" GitHub-API it Works fine!Best regards

I need to use ORACLE stored procedure with input PARAM and this input PARAM is coming from flow file and get output from that stored procedure. is any processor is available with run stored procedure and return the out put in AVRO schema or any other format? Please help me.

pig-0.17.0bin/pig  -x localvery basic UDF file:#!/usr/bin/python3from pig_util import outputSchema@outputSchema("as:int")def square(num):if num == None:return Nonereturn ((num) * (num))@outputSchema("word:chararray")def concat(word):return word + wordExceedingly simple pig script:REGISTER '/home/scs/woodcock/SD411/lab_udf/test.py' USING org.apache.pig.scripting.streaming.Python.PythonScriptEngine AS myFuncs;A = LOAD '/home/scs/woodcock/SD411/DATA/accident.csv' USING PigStorage(',') AS (state:int,name:chararray);B = FOREACH A GENERATE myFuncs.square(state) AS state, name; If I do a "DUMP A" I get exactly what I would expect.But, on a "DUMP B", I get a failed job:java.lang.Exception: org.apache.pig.impl.streaming.StreamingUDFException: LINE :at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)Caused by: org.apache.pig.impl.streaming.StreamingUDFException: LINE :at org.apache.p

Hello,I have the following simple topologyi simply create two files in a folder that GetFile processor is configured to `look` , and expected to upload to the FTP serverThe configuration for the PutFTP is the following: If i am trying to FTP from the VM to the FTP server, it works like a charm: Any help, please?

Hi How do I host an API using HandleHttpRequest which can be used to upload files to NiFi server from local using "multipart/form-data".Have tried below approach, however I get read permission issue as the Nifi user can't access local data. Please suggest an approach which can be used to upload file from local.Thanks