Solved

When using the ValidateXML processor with an XSD, how do I capture the reason for the invalid XML so I can troubleshoot instances when the incoming XML is invalid?   Unfortunately I don't have access to a validation tool in the environment where NiFi is running so I need the feedback from the ValidateXML processor. I appreciate the help.    

I have record from kafka like this {"datetime": "2023-08-03 10:46:50", "long": 106.8451, "lat": -6.2146, "city": "Jakarta", "status": "unknown", "nickname": "unknown"} I use Convertrecord JsonTreeReader with Schema Access Strategy Infer Schema and to change Parquet before I stored in Hive But in my table in Hive the datetime change from 2023-08-03 10:46:50 to 2023-08-03 17:46:50 How to Handle that, I want to stay with same kafka record? I live in GMT+7

Hello Everyone Good Day! When I change the old password of "<property name="Manager Password">NewPassword</property>" in authorizer.xml to new password for our maintenance.I stop nifi services and I change password in authorizers.xml and login-identity-providers.xml after I change I start the nifi services but I encountered the error below.   ERROR [NiFi logging handler] org.apache.nifi.StdErr Failed to start web server: Error creating bean with name 'niFiWebApiConfiguration': BeanPostProcessor before instantiation of bean failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'metaDataSourceAdvisor': Cannot resolve reference to bean 'methodSecurityMetadataSource' while setting constructor argument; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.springframework.security.config.annotation.method.configuration.GlobalMethodSe

hi, i have the following problem - i am creating myself quite a few json files via AttributeToJSON processor. Here, as destination, I have set flow-file content. Now I would like to combine them into one csv file. I used MergeContent in my flow first to combine them into one json and then ConvertRecord to turn it into csv. However, it does not create one file for me, but many, I do not know if the problem lies in the MergeContent processor or ConvertRecord. Or should I approach it differently? There are my settings in MergeContent

Hi, We have 3 datanodes but the other 1 is not having expected number of blocks compared to the other 2. Also, the count of under-replicated blocks is high.  Datanode low block logs: 2023-08-01 07:31:30,134 DEBUG datanode.DataNode: isDatanode=false, isClient=true, isTransfer=false2023-08-01 07:31:30,134 DEBUG datanode.DataNode: writeBlock receive buf size 530480 tcp no delay true2023-08-01 07:31:30,134 INFO datanode.DataNode: Receiving BP-1419792351-10.128.20.54-1579223720671:blk_1122742296_49001972 src: /10.128.66.223:58748 dest: /10.128.64.127:98662023-08-01 07:31:30,134 DEBUG datanode.DataNode: BlockReceiver: BP-1419792351-10.128.20.54-1579223720671:blk_1122742296_49001972storageType=DISK, inAddr=/10.128.66.223:58748, myAddr=/10.128.64.127:9866stage=PIPELINE_SETUP_CREATE, newGs=0, minBytesRcvd=0, maxBytesRcvd=0clientname=DFSClient_NONMAPREDUCE_378619176_53, srcDataNode=:0, datanode=hdfs-datanode-0.hdfs-datanode.metering.svc.cluster.local:9866requestedChecksu

hello, everyone;    I am a freshman as a CDH manager user, My CDH version is CDH 6.0.1, my boss let me configure our CDH cluster to support "docker on yarn", but i found that feature not be supported in CDH 6.x manager, that means i have  a only way to support that with configure that by customer configure    so I decide to  edit yarn-site.xml and container-executor.cfg manually at the node of cluster，but I found that will be covered automatically by cm。    how can i modify the yarn-site.xml and container-executor.cfg and will not be covered by cm.    thx

my  CDH Cluster has been online more than half year,  all the machines and service is normal.   but recently , there is one host agent always under bad healthy, and Cloudera Manager will show the red color on this host.   then I check the agent log as below: [16/Nov/2019 02:20:11 +0000] 1142 MainThread agent WARNING Long HB processing time: 9.52771306038[16/Nov/2019 02:20:52 +0000] 1142 MainThread agent WARNING Long HB processing time: 34.8044610023[16/Nov/2019 02:20:52 +0000] 1142 MainThread agent WARNING Delayed HB: 19s since last[16/Nov/2019 02:21:14 +0000] 1142 MainThread agent WARNING Long HB processing time: 22.3401939869[16/Nov/2019 02:21:14 +0000] 1142 MainThread agent WARNING Delayed HB: 7s since last[16/Nov/2019 02:21:51 +0000] 1142 MainThread agent WARNING Long HB processing time: 36.7459609509[16/Nov/2019 02:21:51 +0000] 1142 MainThread agent WARNING Delayed HB: 21s since last[16/Nov/2019 02:22:24 +0000] 1142 MainThread agent WARNING Long HB processing t

I have a json {  "REQUESTID" : "379",  "REQUESTTYPE" : "V",  "REQUESTCONTEXT" : "B2B",  "CARDNUMBER" : [ "5537290000000511", "5537290000000522" ],  "EXPIRYDATE" : [ "08/20", "09/21" ],  "EMAIL" : [ "John.Jones123@abcmail.com", "Jason.456@gmail.com" ],  "PHONENUMBER" : "11234565555"}CARDNUMBER, EXPIRYDATE and EMAIL are varchar in sql table when Json is passed to covertToSQL processor, I am seeing empty string in the attribute for these columns in prepared statement (Insert)

What is the default port used by the Apache NiFi processor ListenFTP for the data channel in active FTP mode? For the control channel the default port is 2221. Is the default port for the data channel in active mode 2220? The default ports for FTP are 21 for control channel and 20 for data channel. Because ports below 1024 require root access, the NiFi default port for control channel is 2221. 

Hello everyone, we are currently working on automaticlly patching the operating system of our CDP Private Cloud Base cluster. We enabled/configured our cluster to be HA, but for two components (YARN Job History Server and Spark History Server) we couldn't find any solution. We designed our automation to be split over the timespan of three days. Starting at 8 PM each day we update one of the master nodes, one of the edge nodes (e.g. know) and one third of the worker nodes in sequential order. Our problem is how to handle an unexpected failure. Is it possible to wait until the next day or is the availabilty in danger if one server (with the mentioned compontens) isn't booting up etc.  How important are the YARN Job History Server and Spark History Server for the successful execution. As far as I understand both are only required for storing/providing logs for executed applications. Do I miss anything here? 

Hello, I´m using QueryDatabaseTable with time_stamp max value column to get in real time +2k records per day. Despite I read all docs that I found I can not understand how it works. It supose to do a select every time a new record is inserted in the table but if I take a look in the postgres log I can see that the query is executed every second in search for new records.I though that this processors works like a trigger but it seems like it doesn´t and I´m afraid that this can affect database performance.Can somebody explain how this processor internally works?  Thank you in advance  

Hello team, I have tried to change the cluster certificates following the indications from the manual, plus internal company documents. I am stucked when starting the cluster, as when I try to start the INFR Solr instance, it returns a "no valid keystore" message. I have tried with the passwords that are configured in the config files for the service, and it works smoothly. I have done manual comparisson to working environments and everything seems to be in order. Any idea where to hit? Thanks in advance 🙂

Hi, I got an error when creating external table with multiple delimiter such as "||".It says "Cannot validate serde: org.apache.hadoop.hive.serde2.MultiDelimitSerDe". I have tried add jar hive-contribute.3.1.3.jar, but still not working and also using regex.My Cloudera Version is CDH 6.1.1.How to solve this problem so i can achieve to set multiple delimiter in hive create external table?Please help. Sincerely, Gideon MaruliBank Mayapada

I am a newbie to Nifi Jolt transform.  I've been looking through the document, but still cannot figure out how to do mutiple headings for one array.input json{"dataPackage": {"datatest": {"field": [{"Name": "test1","content": 45},{"Name": "test2","content": 20},{"Name": "test3","content": 23}]}}}Jolt Spec[{"operation": "shift","spec": {"dataPackage": {"datatest": {"field": {"*": {"Name": {"test1": {"#test1": "field[&3].Name","@(2,content)": "field[&3].content"},"test2": {"#test2": "field[&3].Name","@(2,content)": "field[&3].content"},"test3": {"#test3": "field[&3].Name","@(2,content)": "field[&3].content"}}}}}}}}, {"operation": "shift","spec": {"field": {"*": {"@(0,content)": "firstDetails.@(1,Name)"}}}}]current output{"firstDetails" : {"test1" : 45,"test2" : 20,"test3" : 23}}Expected output{"firstDetails" : {"test1" : 45,"test2" : 20},"secondDetails" : {"test3" : 23}}can you please help

hello. I am trying to CopyTable 5 minutes or 1 hour of data from a 15TB HBase Table to another cluster.But then I get hbase.ipc.CallTimeoutException and the transfer keeps failing. #1688313600000 ('2023-07-03 01:00:00') - 1688313900000 ('2023-07-03 01:05:00') : 5 minuteshbase org.apache.hadoop.hbase.mapreduce.CopyTable -Dhbase.zookeeper.quorum=A01,B01,C02 --peer.adr=dev01dev02,dev03:2181:/hbase-unsecure --starttime=1688313600000 --endtime=1688313900000 'BIG_EMP'  #1688313600000 ('2023-07-03 01:00:00') - 1688317200000 ('2023-07-03 02:00:00')hbase org.apache.hadoop.hbase.mapreduce.CopyTable -Dhbase.zookeeper.quorum=A01,B01,C02 --peer.adr=dev01dev02,dev03:2181:/hbase-unsecure --starttime=1688313600000 --

I've developed a flow that depends on importing an Oracle database dump via data pump (i.e., the impdp command). To run this command, you have to pass in the username and password as part of the parameters of the impdp command to connect to the Oracle database. In early development, I simply put the password in a non-sensitive parameter, and at the appropriate time, I'm using a ReplaceText processor to wipe out the current contents of the flowfile and write in the contents of the shell script used to run the impdp command. Of course, the text I'm writing in to the flowfile includes #{oracle_pw}. I then PutFile to write the script to disk and ExecuteStreamCommand to run the script. It works like a charm. Now that development is over and we're making plans to move this flow into production, I need a way to make this password parameter sensitive and still be able to pass it in to the script. I've given this a lot of thought and done some research, but it seems like I might need to re

command rebalance is not currently available for execution because role balancer must be in state N/A, but it is in state Busy I cant start rebalance on cdh 5.16.2. 

Hi Team,I create a dataflow try to insert data into uppercase table in postgesql.But it does not work.Appreciate any help on this issue, I am new to Nifi.  Here are my setting for QueryDatabaseTable and PutDatabaseRecord   The error message showorg.apache.nifi.processors.standard.db.TableNotFoundException: Table "ACPMA" not found, ensure the Catalog, Schema, and/or Table Names match those in the database exactly  Here is a uppercase table ACPMA in postgesql. 

Redhat 7 OS.I try to do a in-place upgrade Cloudera Manager from 7.4.4 to 7.7.1 following the docs: https://docs.cloudera.com/cdp-private-cloud-upgrade/latest/upgrade-cdp/topics/ug_cm_upgrade_after-cdp.html I do anything, and after the upgrade, I found the cloudera-scm-agent is not stable. the cloudera-scm-agent log is: [26/Jul/2023 15:11:31 +0000] 1956 MainThread filesystem_map ERROR NFS server binding info failed with error :- rpcinfo: RPC: Port mapper failure - Timed outprogram 100003 is not available[26/Jul/2023 15:12:31 +0000] 1956 MainThread filesystem_map ERROR NFS server binding info failed with error :- rpcinfo: RPC: Port mapper failure - Timed outprogram 100003 is not available[26/Jul/2023 15:13:31 +0000] 1956 MainThread filesystem_map ERROR NFS server binding info failed with error :- rpcinfo: RPC: Port mapper failure - Timed outprogram 100003 is not available[26/Jul/2023 15:14:31 +0000] 1956 MainThread filesystem_map ERROR NFS server binding info failed

Hi,I am a newbiee to nifi and its processors.  Want to understand what would be the jolt specification for the following input/output? Or can anyone suggest any other processor . Input JSON:{"cells": {"deviceindicators:1234_32456_789023":"0", "deviceindicators:5678_89213_875943":"110"}} Output JSON:{"cells": {"1234_32456_789023":"0", "5678_89213_875943":"110"}} Want to remove "deviceindicators:" from the key using JoltJSONtransformation.. Please advice

Hello, I have a simple process getting data from sql server and send them by email. Short emails, nothing heavy and not that many during the day. But few days ago, it suddenly started to be frozen and it happened again today. It does not show any error and I do not see any error in logs either. I checked this thread, maybe it's the same, but there is no real solution and in my logs I can't see anything with SMTP or that it's "waiting", so would like to ask you for help or to point me to the right direction.Maybe I'm not checking everything...  ThanksRgdsLudvik 

Hi.I'm using NiFi 1.11.2. I have a set of Json-files which contain a base64-coded field (Jsonpath to this field is $.data.content).  Decoding the field works as expected and I'm able to write the content from this field to a file in a bucket in S3.  But what I would like to do is to be able to replace the coded value for this field with the decoded value in stead of writing the decoded value to file. And after replacing the value write the updated Json-file to a new S3 bucket.I'm thinking taking the output from RouteOnAttribute and ConvertRecord in the figure below and perhaps use UpdateRecord or ReplaceText. But I'm not sure how to do this.What is the best way to "re-connect" the data flow here from these to objects in the process?Any tips? Thanksbjornmy     

Hi all, I have a Hadoop cluster built on Cloudera. Now I am trying to increase concurrent with impala and I have no ideal for change the config files. Do you have experiences to how to resolve this, please help me. Thank you all,

Hi Team , As we have more then 14 million small file as per Cloudera Navigator as below : 14.8M Small files created in the last 30 days  14.8M / 21.4M 69.3% small files We are doing partition of data per day wise can we increase it or any other suggestion is there to overcome the small file problem. Thank You in Advance   

I created a managed table called "questiuon".However, a path called "/warehouse/tablespace/external/hive/question.db" was created on the hdfs path.Conversely, creating an external table also creates a path called "/warehouse/tablespace/managed/hive/question.db".What is the reason?