Q&A

we have ambari cluster , HDP version `2.6.5` cluster include management of two name-node ( one is active and the secondary is standby )and 65 datanode machines we have problem with the standby name-node that not started and from the namenode logs we can see the following 2021-01-01 15:19:43,269 ERROR namenode.NameNode (NameNode.java:main(1783)) - Failed to start namenode.java.io.IOException: There appears to be a gap in the edit log. We expected txid 90247527115, but got txid 90247903412.at org.apache.hadoop.hdfs.server.namenode.MetaRecoveryContext.editLogLoaderPrompt(MetaRecoveryContext.java:94)at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:215)   for now the active namenode is up but the standby name node is down regarding tojava.io.IOException: There appears to be a gap in the edit log. We expected txid 90247527115, but got txid 90247903412. what is the preferred solution to fix this problem?&nbs

Hello,I just installed a new Hadoop server which has hive and impala servers on it.Cloudera Runtime 7.1.8Server version: impalad version 4.0.0.7.1.8.0-801 RELEASE (build a3b56f90d9c31ebfa5ce3c266700284a420db28f)However, I am getting an error when try to import data from .cvs file from Impala shell:The error is:Query: INSERT OVERWRITE TABLE ATABLE SELECT * FROM JMASTER.ATABLEQuery submitted at: 2024-02-14 14:00:53 (Coordinator: http://hiveservername.domainame.com:25000)Query progress can be monitored at: http://hiveservername.domainame.com:25000/query_plan?query_id=0248306127fdcbd1:36d35f5a00000000ERROR: File 'hdfs://hiveservername.domainame.com:8020/user/hivetest/master_tables/hivetesttable/ATABLE.csv' has an invalid Parquet version number: 30 30 30 30 .Please check that it is a valid Parquet file. This error can also occur due to stale metadata. If you believe this is a valid Parquet file, try running "refresh jmaster.hivetesttable".Could not execute command: INSERT OVERWRITE TABLE AT

Hi, Is there any way that Nifi processors stop automatically after no input for example 10 minutes rather than running until they are manually stopped?Kindly let me know if there is any such feature..

We have multiple different sources which we are using same process group. We need to call same process group with different parameters in parallel.

Hi All,I'm having a 6 node on prem nifi cluster which has certain process groups. It had a node disconnect once and got auto connected back. Post this I tried to stop/disable a process group which was in started state in the UI, but it threw me an error stating -Node xxx is unable to fulfill this request due to : [xxxx, xxxxxxxxxxx, xxxxxxxxxxxxx] is not the most up-to-date revision. This component appears to have been modified.Although I didn't modified any component in the past, only a node disconnect and auto connect was the event which happened in between. As per my previous experiences, I had to restart whole nifi from Cloudera Manager UI and then restart the flows which would resolve the problem which I stated above. But I was looking for an alternate solution other than restarting whole NiFi from CM UI as this is my production cluster and I can't get downtime so easily.Any help would be highly appreciated.--Adding the snapshot of the error below which i got while stopping/d

I'm going to take a Cloudera Certificate Exam (CDP Data Engineer Exam CDP-3002). I have already read the FAQ section, but I still have some questions about it. Please understand that I'm just a fresh graduate student from a 3rd world country, so the 333$ cost is very high for me. So I need to make sure I am fully prepared before paying for the exam.1. From payment, how long does it take until I receive an email with detailed information about the test?2. From payment, how long does it take until the test starts?3. The email will guide me in detail on the steps to start the exam, right?4. I read through the FAQ section, is there a person called "proctor" who will instruct me on what to do before the exam?5. I just need to install the Questionmark Secure software, right?6. There are only multiplier choice questions and no exercise questions (I have to code), right? Sorry for all the questions, please understand for me because I'm very confused right now. There isn't much thing

We would like to convert  3   nodes ( these nodes have only zookeeper roles deployed on them ) to  unmanaged nodes by purging the cloudera software . These 3 nodes forming a zookeeper ensemble were created long time ago and are not used by any of our existing CDP  cluster services but some external edge querying applications . we now need to take them off being managed by cloudera manager to avoid being included unnecessarily in the licensing cost .  How should I go about completing this task ?  I don't want to break the external application but there is no reason that we should have these nodes managed by CM . Do I have to rebuild the ensemble using the open source s/w from apache after purging the cm agent s/w ?     

Are there any plans and target dates for CDP Private Cloud to support Java 17 and Spark 3.3? 

Hi, our team has started using Cloudera Data Science Workbench.We want to connect CDSW with VSCODE using SSH and have followed the steps described in the Cloudera documentation.However, I cannot find method to reconnect existing(running) session using cdswctl CLI.I came across  this article stating that reconnecting to an existing session was impossible as of 2020. Is this still the case?"Additionally, if there are any alternative methods available, please let me know." 🙂

Dear all,A bit background about this:I receive fresh data from a data vendor on a weekly basis and aim to either insert or update my existing database. If the lookup identifies matching values (such as company ID, fiscal year, etc.) in a record, we perform an update on specific fields (e.g., announcement date); otherwise, we insert a new record.My chain of lookups:My lookup processor setup:My data:Before lookup:After first lookup (Company ID) (It return me fs_ID):After second lookup (Fiscal_Year) (It return me another fs_ID):Note: fs_ID is retrived value from lookup tablesHowever, I check my lookup table:It does not match the "Company_ID", it only match the "Fiscal_year", It seems like those lookup processor worked independently. I expected this output: myincomingdata.Company_ID = ent_fundamentals.Company_ID AND myincomingdata.Fiscal_year = ent_fundamentals.Fiscal_year..etc., (I can update this record base on those columns if they all match)Can anyone please help me this

My setup looks like above.My question is: what if I have 11111 in total, I think each 1000 flowfiles will be merge, how about the leftover 111 flowfiles? how can I deal with the leftover flowfiles?Thanks!

 2024-02-16 01:50:21,843 ERROR namenode.NameNode: Failed to start namenode.java.io.IOException: Gap in transactions. Expected to be able to read up until at least txid 35923060031 but unable to find any edit logs containing txid 20264370028at org.apache.hadoop.hdfs.server.namenode.FSEditLog.checkForGaps(FSEditLog.java:1751)at org.apache.hadoop.hdfs.server.namenode.FSEditLog.selectInputStreams(FSEditLog.java:1709)at org.apache.hadoop.hdfs.server.namenode.FSEditLog.selectInputStreams(FSEditLog.java:1684)at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:701)at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:323)at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1086)at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:714)at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:632)at org.apache.hadoop.hdfs.server.namenode.NameNode.initi

When I activate an environment with Private Load Balancer, Private Executors I am not able to access Hue from my warehouse.  Any help with what I am missing would be appreciated.

I already have a working connection through ODBC using Cloudera ODBC Driver for Apache Hive, where I have my DSN set and all I need is to call pyodbc.connect(f"DSN={mydsn}", autocommit=True)I'd like to use SQLAlchemy, but I'm struggling how to create a working connection url for multiple hosts.I was trying to base my idea on this guide where I found connection string jdbc:hive2://datanode1:2181,master1:2181,master2:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2However, thisfrom sqlalchemy import create_enginequery = """SELECT TOP 10 * from eb.mobile_sa"""conn_url = f'hive://{UID}@host1:2181,host2:2181,host3:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2'engine = create_engine(conn_url)with engine.connect() as conn: df = pd.read_sql(query, conn)throws an error invalid literal for int() with base 10: '2181,host2:2181,host3:2181 etc. I've also considered cloudera's impyla since Pyhive seems to be currentl

Hello all, I am creating a processor group that read from kafka topic and write it to the clickhouse database. I am using stateless mechanism to ensure that when there is a problem during execution, nifi crash, or nifi restarted or clickhouse database return error, kafka offset will not be committed and process will be retry.Unfortunately clickhouse will create a new row for duplicated message. In order to avoid duplicate message, i would like to check first the database and see if i have duplicated message before processing. Have someone create similiar use case as this one?   

Hello support,I am creating another execute stateless from a processor group. This processor group has one consume jms connected with ibm mq. Below is the processor group and consume jms ssl context service. The processor group running fine without execute stateless. After i push the processor group to be referenced with execute stateless processor. After i got following error:Is there a specific way to define the type of truststore file or password for execute stateless.   

 /* * Licensed to the Apache Software Foundation (ASF) under one or more * contributor license agreements. See the NOTICE file distributed with * this work for additional information regarding copyright ownership. * The ASF licenses this file to You under the Apache License, Version 2.0 * (the "License"); you may not use this file except in compliance with * the License. You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */package com.demo.processors;import org.apache.commons.lang3.StringUtils;import org.apache.nifi.annotation.behavior.InputRequirement;import org.apache.nifi.annotation.behavior.ReadsAttribute;import org.apache.

Hello experts,I have downloaded and trying to run nifi registry 1.24 with meta data db as mysql and Git flow persistence.Getting below error and unable to start the service - log in text : 2024-02-21 16:11:13,716 INFO [main] o.a.n.registry.bootstrap.RunNiFiRegistry Runtime Java version: 11.0.142024-02-21 16:11:13,725 INFO [main] o.apache.nifi.registry.bootstrap.Command Starting Apache NiFi Registry...2024-02-21 16:11:13,725 INFO [main] o.apache.nifi.registry.bootstrap.Command Working Directory: /usr/independent_nifi/nifi-registry-1.24.02024-02-21 16:11:13,726 INFO [main] o.apache.nifi.registry.bootstrap.Command Command: java -classpath /usr/independent_ni.............................2024-02-21 16:11:13,743 INFO [main] o.apache.nifi.registry.bootstrap.Command Launched Apache NiFi Registry with Process ID 190992024-02-21 16:11:14,447 INFO [NiFi Registry Bootstrap Command Listener] o.a.n.registry.bootstrap.RunNiFiRegistry Apache NiFi Registry now running and listening for Bootst

Hi community,From Hue security browser, I have a RBAC role (admin), which was given all permission on all databases.I have this Database A which consists of table1,2,3I want to deny this admin role from dropping Database A -> Table1 and Table2.Can you show me how this can be done on sentry?Thanks!!

Hi! I'm using NiFi 2.0.0 M1, and I'm writing a flow inside a processed group. I had a problem when I wrote the flow.The flow should be run as an individual task. Every time when it is run, it will take in some data from a table on MySQL Server, use many LookupRecord Processors and then process the matched and unmatched data respectively. Finally, it turns out to have 9 branches within the flow.After processing all these, I need to trigger an ExecuteSQLRecord processor which will call some stored procedures in MySQL. Literally, only when all the above branches finish running should this processor be triggered (and it should be triggered only once in one task). To achieve this, I wanted to use Notify and Wait processor to detect whether all the branches finish running and when I should trigger the ExecuteSQLRecord processor. However, I can only set a fixed number of signals to wait for in the Wait processor, while in the real case, sometimes not all of the 9 branches will be run. (Becaus

I am trying to assign a value to a variable using single value result from a query but i keep getting syntax error . I am using HUE - Impala Editor Result of "Select distinct MonthEndSnapshotKey FROM vmge_1h_bmh_lynx_non_pu_sec.rundtcur" = 202005 with data type INT .I want to assign it to variable Testvar  SQL statements : SET $(testvar=Select distinct MonthEndSnapshotKey FROM vmge_1h_bmh_lynx_non_pu_sec.rundtcur};select $(testvar}ParseException: Syntax error in line 1:undefined: SET $(testvar=Select distinct Mon... ^ Encountered: ( Expected: = CAUSED BY: Exception: Syntax error

Hello,Is there a way we can monitor long running queries in Impala on  CDP-PC or a dashboard where this can be viewed and if required an alert to be triggered. Thanks Wert.

I have a job which runs a hive query inside. When it comes the time for the query Oozie throws this error:   Error while compiling statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex re-running, vertexName=Map 1, vertexId=vertex_1668428709182_0049_1_00Vertex re-running, vertexName=Map 1, vertexId=vertex_1668428709182_0049_1_00Vertex re-running, vertexName=Map 1, vertexId=vertex_1668428709182_0049_1_00Vertex failed, vertexName=Map 1, vertexId=vertex_1668428709182_0049_1_00, diagnostics=[Vertex vertex_1668428709182_0049_1_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE, Vertex vertex_1668428709182_0049_1_00 [Map 1] failed as task task_1668428709182_0049_1_00_000000 failed after vertex succeeded.]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0   I can not understand a lot of this error but when I run the job through terminal it ends successfully.

Hi, I have a GetHDFS processor that reads a .TXT and I need to read the content and send a email combined the content with some other default text. I tried a UpdateAttribute to capture flowfile-content and then used in PutEmail sendMessage as Hi ${content} for example, but always get a empty string.Someone can help me

Dear there, While i make the Nifi deployment to build ExecuteGroovyScript like that from integration_pipeline.models.processors.script_processors.script_processor_base import ScriptProcessorBaseclass ExecuteGroovyScriptProcessor(ScriptProcessorBase):    @property    def name(self) -> str:        return 'org.apache.nifi.processors.groovyx.ExecuteGroovyScript'    def insert_custom_properties(self, script_file: str,                                 module_directory: str = None, **kwargs):        self.set_configuration_properties({**{"Script File": script_file,                                              "Module Directory": module_directory,                    &n