Solved

Hello,I am getting the data from GetMongo Processor then i am evaluating the JSON Path using EvaluateJSONPath where i am getting series_id and then I am doing in InvokeHTTP.  I am able to send as json but i am not able to send as form-data. Can you please suggest me what i am doing wrong here.Before invokeHTTP the data flow is     

Hi all, I have a cluster in CDP 7.1.8 private cloud based. Today I tried migrating CM server and following the instruction in https://docs.cloudera.com/cdp-private-cloud-base/7.1.8/managing-clusters/topics/cm-moving-the-cm-server-new-host2.html After I manually restore the cluster following the instruction in https://docs.cloudera.com/cdp-private-cloud-base/7.1.3/configuring-clusters/topics/cm-api-import-configuration.html, the cluster cannot restart. Found that in the Parcel page in CM Console, the parcels are activating but keep on loading forever. Here are the log message related to parcel from cloudera-scm-agent.log: [02/May/2023 03:29:55 -0400] 6258 MainThread parcel ERROR Error while attempting to modify permissions of file '/opt/ cloudera/parcels/CDH-7.1.8-1.cdh7.1.8.p0.30990532/lib/hadoop-0.20-mapreduce/sbin/Linux/task-controller'. File "/opt/clo

Hello Team,I have JSON like following[{"INTEGRATION_ID": "1","ISSUE_STATUS_CAT": "Done","SPRINT_LIST": ["24-w1-2","24-w1-3","24-w1-4"]},{"INTEGRATION_ID": "2","ISSUE_STATUS_CAT": "Done","SPRINT_LIST": "24-w1-2"}]My requirement is to check whether SPRINT_LIST is array or not. so I am using UpdateRecord processor with following formula ${field.value:startsWith(" [ ")}I was getting the correct output earlier, where it returned 'True' when the value was a list and 'False' when it was null or not a list. However, after making some changes and then reverting back to the original settings, I started encountering issues. "Cannot convert value [true] of type class java.lang.String to Object Array for field error"

we have ambari cluster , HDP version `2.6.5` cluster include management of two name-node ( one is active and the secondary is standby )and 65 datanode machines we have problem with the standby name-node that not started and from the namenode logs we can see the following 2021-01-01 15:19:43,269 ERROR namenode.NameNode (NameNode.java:main(1783)) - Failed to start namenode.java.io.IOException: There appears to be a gap in the edit log. We expected txid 90247527115, but got txid 90247903412.at org.apache.hadoop.hdfs.server.namenode.MetaRecoveryContext.editLogLoaderPrompt(MetaRecoveryContext.java:94)at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:215)   for now the active namenode is up but the standby name node is down regarding tojava.io.IOException: There appears to be a gap in the edit log. We expected txid 90247527115, but got txid 90247903412. what is the preferred solution to fix this problem?&nbs

Hi, Is there any way that Nifi processors stop automatically after no input for example 10 minutes rather than running until they are manually stopped?Kindly let me know if there is any such feature..

Are there any plans and target dates for CDP Private Cloud to support Java 17 and Spark 3.3? 

Hi, our team has started using Cloudera Data Science Workbench.We want to connect CDSW with VSCODE using SSH and have followed the steps described in the Cloudera documentation.However, I cannot find method to reconnect existing(running) session using cdswctl CLI.I came across  this article stating that reconnecting to an existing session was impossible as of 2020. Is this still the case?"Additionally, if there are any alternative methods available, please let me know." 🙂

Dear all,A bit background about this:I receive fresh data from a data vendor on a weekly basis and aim to either insert or update my existing database. If the lookup identifies matching values (such as company ID, fiscal year, etc.) in a record, we perform an update on specific fields (e.g., announcement date); otherwise, we insert a new record.My chain of lookups:My lookup processor setup:My data:Before lookup:After first lookup (Company ID) (It return me fs_ID):After second lookup (Fiscal_Year) (It return me another fs_ID):Note: fs_ID is retrived value from lookup tablesHowever, I check my lookup table:It does not match the "Company_ID", it only match the "Fiscal_year", It seems like those lookup processor worked independently. I expected this output: myincomingdata.Company_ID = ent_fundamentals.Company_ID AND myincomingdata.Fiscal_year = ent_fundamentals.Fiscal_year..etc., (I can update this record base on those columns if they all match)Can anyone please help me this

My setup looks like above.My question is: what if I have 11111 in total, I think each 1000 flowfiles will be merge, how about the leftover 111 flowfiles? how can I deal with the leftover flowfiles?Thanks!

Hello support,I am creating another execute stateless from a processor group. This processor group has one consume jms connected with ibm mq. Below is the processor group and consume jms ssl context service. The processor group running fine without execute stateless. After i push the processor group to be referenced with execute stateless processor. After i got following error:Is there a specific way to define the type of truststore file or password for execute stateless.   

 /* * Licensed to the Apache Software Foundation (ASF) under one or more * contributor license agreements. See the NOTICE file distributed with * this work for additional information regarding copyright ownership. * The ASF licenses this file to You under the Apache License, Version 2.0 * (the "License"); you may not use this file except in compliance with * the License. You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */package com.demo.processors;import org.apache.commons.lang3.StringUtils;import org.apache.nifi.annotation.behavior.InputRequirement;import org.apache.nifi.annotation.behavior.ReadsAttribute;import org.apache.

Hi community,From Hue security browser, I have a RBAC role (admin), which was given all permission on all databases.I have this Database A which consists of table1,2,3I want to deny this admin role from dropping Database A -> Table1 and Table2.Can you show me how this can be done on sentry?Thanks!!

Hi! I'm using NiFi 2.0.0 M1, and I'm writing a flow inside a processed group. I had a problem when I wrote the flow.The flow should be run as an individual task. Every time when it is run, it will take in some data from a table on MySQL Server, use many LookupRecord Processors and then process the matched and unmatched data respectively. Finally, it turns out to have 9 branches within the flow.After processing all these, I need to trigger an ExecuteSQLRecord processor which will call some stored procedures in MySQL. Literally, only when all the above branches finish running should this processor be triggered (and it should be triggered only once in one task). To achieve this, I wanted to use Notify and Wait processor to detect whether all the branches finish running and when I should trigger the ExecuteSQLRecord processor. However, I can only set a fixed number of signals to wait for in the Wait processor, while in the real case, sometimes not all of the 9 branches will be run. (Becaus

Hello,Is there a way we can monitor long running queries in Impala on  CDP-PC or a dashboard where this can be viewed and if required an alert to be triggered. Thanks Wert.

Hi, I have a GetHDFS processor that reads a .TXT and I need to read the content and send a email combined the content with some other default text. I tried a UpdateAttribute to capture flowfile-content and then used in PutEmail sendMessage as Hi ${content} for example, but always get a empty string.Someone can help me

Dear there, While i make the Nifi deployment to build ExecuteGroovyScript like that from integration_pipeline.models.processors.script_processors.script_processor_base import ScriptProcessorBaseclass ExecuteGroovyScriptProcessor(ScriptProcessorBase):    @property    def name(self) -> str:        return 'org.apache.nifi.processors.groovyx.ExecuteGroovyScript'    def insert_custom_properties(self, script_file: str,                                 module_directory: str = None, **kwargs):        self.set_configuration_properties({**{"Script File": script_file,                                              "Module Directory": module_directory,                    &n

When I try 'Activate' Environment to create virtual datawarehouse of CDWThe error occurred below image.This is Authentication pageThis is environmentThis is role of cm_admin accoutThe cm_admin account has all permission on ActiveDirectory.And all ports related LDAP of Activedirectory server. Please let me know what shoul I have to do.     

Hello sir,I am using invokedHttp to post request. But it did not get response from invokeHttp. I get request file . But I want to get the response invokeHttp and write to remote server. Here is the diagramhere is invokeHttp properties.Request has been posted to server successfully.How to get response from invokeHttp and write in remote server?Please help me

Dears team, I have been using PySpark 3.4.2 with the following syntax:  sql_query = "INSERT OVERWRITE TABLE table_1 PARTITION(partition_date = {YYYYMMDD})SELECT table_1.a , table_1.b , table_2.cFROM table_2 change_capture_viewFULL OUTER JOIN ( SELECT * FROM table_1 WHERE WHERE partition_date = {YYYYMMDD_D_1}) current_view ON change_capture_view.a <=> current_view.aWHERE change_capture_view.a IS NULL"  and use spark.sql(sql_query ). And encountered the error:  File "/usr/hdp/3.1.0.0-78/spark3/python/lib/pyspark.zip/pyspark/sql/session.py", line 1440, in sqlFile "/usr/hdp/3.1.0.0-78/spark3/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1322, in __call__File "/usr/hdp/3.1.0.0-78/spark3/python/lib/pyspark.zip/pyspark/errors/exceptions/captured.py", line 175, in decopyspark.errors.exceptions.captured.AnalysisException: Cannot overwrite a path that is also being read from. In essence, I am trying to retrieve data from

I am trying run query via HUE api of cloudera data warehouse on azure. However I couldn't get token for authentication.  I tried enter ID and password started with csso_ and machine user.Tried enter token made from knox as well. How could I get token for access Hue API??

Hello,I have nested JSON objects coming in a flowfile, looking like this (example): { "a" : 1, "b" : {"asd" : 156, "bed" : "lalala"}, "c" : "abba", "d" : {"add" : 872, "bla" : "bebebe"} } I'd like to put double quotes around every nested json object and also escape any double quote inside of that nested json. The result would look like this: { "a" : 1, "b" : "{\"asd\" : 156, \"bed\" : \"lalala\"}", "c" : "abba", "d" : "{\"add\" : 872, \"bla\" : \"bebebe\"}" } I tried doing a replacetext with a regex like this: (?<=:.*\{[^}]*)"However this is utilizing a negative lookbehind with varying length and it seems this isn't supported in java/nifi. What would be the best approach here? Thanks!

I have an existing iceberg table and I'm trying to evolve schema by adding a new column:`alter table my_db.my_table add column insert_time timestamp`When I execute the above statement, I get a 'New column(s) have been added to the table.' but the column doesn't appear. I tried running a REFRESH, an INVALIDATE METADATA, and I tried restarting Impala but it had no effect. I tried the same operation on a test table and it worked. I'm using Impala as a query engine and Azure for storage. Impala is running in a Data Hub Cluster.

Hello,I'm new to using oozie and I've been trying to figure out how to use the output of an action as the input of the next action in Oozie. I'm actually coming from using traditional ETLs, where the output of one Step can be used as arguments in the next Step. My question is whether the following points are possible with Oozie:Use the output of one action as the input of the next actionLet a subsequent action be executed "n" times according to the output of the first action. Like sub jobs or something.Actually, in order to know if this is possible, we are trying to run a very basic dump example where we read and insert with two hive actions, one for read a table and another one to write. Something like the following example created with Hue Oozie Editor.<workflow-app name="Workflow" xmlns="uri:oozie:workflow:0.5">  <start to="hive-04f4"/>  <kill name="Kill">  <message>Error al realizar la acción. Mensaje de error [${wf:errorMessage(wf:l

we make database connect to apache hive and it's 'ok , but , the following issues when try to browse tables when we try to create dataset  2024-02-06 09:58:58,682:DEBUG:pyhive.hive:TExecuteStatementReq(sessionHandle=TSessionHandle(sessionId=THandleIdentifier(guid=b'\x9d\xfc\x17\xd3acJ\x81\x87\xf2<\x92mk\x92C', secret=b'\x1b\x8f\x9cX_NIP\x96\xe55\xce\x16\x0e\xf1:')), statement='SHOW VIEWS IN hrdb', confOverlay=None, runAsync=False, queryTimeout=0)12024-02-06 09:58:58,688:DEBUG:pyhive.hive:TExecuteStatementResp(status=TStatus(statusCode=3, infoMessages=["*org.apache.hive.service.cli.HiveSQLException:Error while compiling statement: FAILED: ParseException line 1:5 cannot recognize input near 'SHOW' 'VIEWS' 'IN' in ddl statement:28:27", 'org.apache.hive.service.cli.operation.Operation:toSQLException:Operation.java:374', 'org.apache.hive.service.cli.operation.SQLOperation:prepare:SQLOperation.java:136', 'org.apache.hive.service.cli.operation.SQLOperation:runInternal:SQLOp

Hello.We are running Cloudera CDP 7.1.7 and Cloudera Manager 7.4.4 but when we access Hue UI we can only browse Jobs :Do we have to follow links given in the article ?https://docs.cloudera.com/cdp-private-cloud-base/7.1.8/hue-introduction/topics/hue-about-query-processor.htmlIs it for CDP 7.1.8 and higher ?Because when I look for adding a new service to our cluster, I can add DAS :but Query Processor is not listed :After installing and enabling Query Processor, will we be able to Query information or Timeline like in this article for CDW ?https://docs.cloudera.com/cdw-runtime/1.5.1/using-hue/topics/hue-view-query-information.htmlhttps://docs.cloudera.com/cdw-runtime/1.5.1/using-hue/topics/hue-view-query-timeline.htmlThanks for your help.