Solved

Hi everyone, So I'm trying to write a data quality query that is efficient in counting the number of nulls in particular columns. Currently the query that I use is as below(I actually found a related example here on a related topic) :SELECT CustFirstName, CustLastName,CASE WHEN CustFirstName is null then 1 else 0 end CustFirstNameNullCheck,CASE WHEN CustLastName is null then 1 else 0 end CustLastNameNullCheckFROM SchemaName.DbNameWHERE Date = '2023-03-15' The output that I get is:CustFirstNameCustLastNameCustFirstNameNullCheckCustLastNameNullCheck{null}{null}11 And then hundreds of thousands of single rows of {nulls} that follow(I know, it's a long story). Anyhow, the output that I would LIKE to get, please, is a much simpler layout with just the column(or an alias) and a count of the number of total nulls:CustFirstNameCustLastName500,000 (nulls)500,000 (nulls) I've tried different iterations of SELECT COUNT(*) and just could not get the query right. MapR gives

I'm checking whether there are any effects to spark2 by CVE-2022-33891 or not.Is there anyone who can explanin it to me?

Hi, few hours ago zeppelin stopped to run queries and begins to throw errors like WARN org.apache.zeppelin.socket.NotebookServer: Job 20230217-161657_1725299508 is finished, status: ERROR, exception: null, result: %text javax.security.auth.login.LoginException: Unable to obtain password from userIts kerberized AD KDC cluster.ldapsearch and kinit with right principal works fine./etc/hosts are fine too.The rest services on the cluster are green.http://hostname:port/api/credential/ returns {"status":"OK","body":{"userCredentials":{}}}What else can I check?

Hi,I have query on "Delivery guarantee" parameter in publishKafka processor in NIFI. I have query on below two option supported for the "Delivery guarantee"  1. Best effort2. Guarantee single node delivery I know that the "best effort" does not wait for the ack/response from kafka, but in case of publish failure, when either of the above two option is set, will NIFI try to publish among the available nodes? Like in my setup its the cluster of three kafka nodes. If one of the node is down or if we observe connection timeout issue with one of the node, will NIFI tries to publish with other available nodes? Thanks

Hi ,Am using MSSQL version 12+ , I have connected to DB successfully . I could able to fetch the data eg : Select * from dbname.sys.tables But If i try to query fetch the table records using below processor   Getting below error  :   

I have step up 3 node (node1, node2, node3) nifi cluster and I am able to access it with http protocol, When I try to make those 3 node secure(https). I am facing "insufficient permissions untrusted proxy" in web UI.After setup 3 node cluster in node1 i followed the steps to make secure connection. Step:1 By using below step I have Generate Keys and Certificates/opt/nifi-toolkit/bin/tls-toolkit.sh standalone -n 'node1' -B ClientPassword -C 'CN=admin,OU=NIFI' -O -o /opt/nifi-1.19.1/conf/certsafter run this I got below files/opt/nifi-1.19.1/conf/certs/├── CN=admin_OU=NIFI.p12├── CN=admin_OU=NIFI.password├── node1│ ├── keystore.jks│ ├── nifi.properties│ └── truststore.jks├── nifi-cert.pem└── nifi-key.keyStep:2I have over write the /opt/nifi-1.19.1/conf/certs/node1/nifi.properties to conf/nifi.propertiesThen Edit the nifi.propertiesnifi.security.user.authorizer=managed-authorizerStep:3Updated Security Properties Key/opt/nifi-1.19.1/bin/nifi.sh set-sensitive-properties-key xyz12345Step

Hello We have a HDFS cluster configure to work with Kerberos and we want to use nifi processor PutParquet to write file into hadoop cluster.do we need to configure nifi to work with kerberos too in order to connect to Hadoop?what are the steps i need to take in order to do the connection between the two clusters (nifi --> HDFS) Thanks

Hi All, Greetings! Assume that I have a data flow, and the final result is inserting to the SQL database (Microsoft), but the step before! I have the following JSON format for each flow file content as following:  { "sbrActivityType" : { "1" : "ACT+A", "3" : "ACT+A", "2" : "ACT+X", "5" : "ACT+X", "4" : "ACT+X" }, "sbrDateTime" : "Sun Mar 19 13:45:15 AST 2023"} Then, my desired output format look like the following:INSERT INTO sbrTestNifiData (sbrActivityType, sbrDateTime) VALUES (?, ?) So, it does not insert correctly to my database table which is looks like the following: CREATE TABLE [dbo].[sbrTestNifiData]( [sbrId] [int] IDENTITY(1,1) NOT NULL primary key, [sbrActivityType] [varchar](50) NULL, [sbrDateTime] [varchar](50) NULL);  So, it should inserted to my database table as a five records:sbrIdsbrActivityTypesbrDateTime1ACT+ASun Mar 19 11:56:40 AST 20232ACT+ASun Mar 19 11:56:40 AST 20233ACT+XSun Mar 19 11:56:40 AST 2023

Hi everyone, I'm working to a new feature with an existing nifi cluster to provide a new service to add an interface with serveral kerberized HDP Cluster. I would like to know if a single Nifi cluster can use several realms in the same krb5 file.Reading official documentation, nifi can do it (https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#kerberos_properties) : " If necessary the krb5 file can support multiple realms." It seems ok but because at this time I have no way to have several hadoop cluster for testing (to notice : my cluster is working already with one kerberized cluster hadoop ), is anybody can confirm or reject this design: one cluster Nifi with different realms to communicate with multiple kerberized hdp cluster. Thanks for your help and as soon as I have several kerberized cluster hadoop for testing, I will update this article. 

Hi, All!I have a problem with NIFI ListSFTP processor. After started processor errored with Failed to properly initialize Processor. If still scheduled to run, NiFi will attempt to initialize and run the Processor again after the 'Administrative Yield Duration' has elapsed. Failure is due to java.lang.IllegalArgumentException: A HostProvider may not be empty!: java.lang.IllegalArgumentException: A HostProvider may not be empty! ListSFTP processor Version 1.12.0 and the Properties as follows:Error screenshot from UI:

Dear all, I want to run COMPUTE INCREMENTAL STATS with PARTITION option on some partitions of a partitioned table.It is unclear to me from the documentation if DROP INCREMENTAL STATS must be run on those partitions. For my understanding of how INCREMENTAL STATS with the PARTITION option work, it should not be required, but I've found below sentence on this page https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/impala_perf_stats.html#perf_stats "In cases where new files are added to an existing partition, issue a REFRESH statement for the table, followed by a DROP INCREMENTAL STATS and COMPUTE INCREMENTAL STATS sequence for the changed partition.". I'd prefer to avoid running DROP INCREMENTAL STATS if not required. Does anyone know? Many thanks in advance.

Hello,I am trying to solve a strange behavior of NiFi but can't really figure it out.My flow consists of a GenerateTableFetch (which gets executed once per day), a ExecuteSQLRecord (which gets executed once it receives the files from GenerateTableFetch) and finally an PutS3Bucket.Now, both the GenerateTableFetch and the ExecuteSQLRecord are using the same Controller Service, a DBConnectionPool which connects to an Oracle Database.In the previous days I have noticed that I get plenty of sessions from NiFi (which is ok) but these sessions never get killed/terminated and they remain inactive. I am using the following configurations within my DBConnectionPool Service:Max Wait Time: 500 millisMax Total Connections: 51Minimum Idle Connections: 0Max Idle Connections: 5Max Connection Lifetime: -1Time Between Eviction Runs: -1Minimum Evictable Idle Time: 30 minsSoft Minimum Evictable Idle Time: -1Unfortunately I was not able to find any information in the documentation (or maybe I did not fully

Failing at 38%  NAMENODE : AMBARI server  NAMENODE : AMBARI server stderr:   errors-440.txt stdout:   output-440.txt    2021-07-22 09:41:36,191 - Processing identities...2021-07-22 09:41:36,222 - Creating keytab file for hdpcluster-072221@ on host myserver.com2021-07-22 09:41:36,230 - Processing identities completed.      DATANODE : HDP CLUSTER   DATANODE : HDP CLUSTER stdout:   /var/lib/ambari-agent/data/output-438.txt   2021-07-22 09:41:34,474 - Missing keytabs:Keytab: /etc/security/keytabs/kerberos.service_check.072221.keytab Principal: hdpcluster-072221Command completed successfully! This is the krb5.conf file placed in ambari server (name node) and hdpcluster 3.1.5 (datanode) # Configuration snippets may be placed in this directory as wellincludedir /etc/krb5.conf.d/# output settings[logging]default = FILE:/tmp/krb5libs.logkdc =

Dear Members, I have accidently created a partition having characters %01 in the name. %01 is start of header `as per the notepad++`.When i am trying to delete the partition using ALTER statement, there is no error but partition also not getting deleted. can anyone suggest how to delete such partitions with special characters. ALTER TABLE <table_name> DROP IF EXISTS PARTITION (code='YATHAH%0188QW'); 

when we rename partition of a column, would underlying data of the column would also change as per our rename. this is on ORC file format. for ex. I've partition named activeFlag='N', when I rename this partition to activeFlag='Y', would those activeFlag column value change to 'Y' (OR) data would be intact as 'N' only?

HI i am using cdp 6.3i am trying to connect impala shell it is giving me this errorimpala-shell. i am using 3 nodes , and i tried with all nodes . impala service is working fine and even i can run queries from Hue dashboard but self impala shell is not connecting   Running Command : impala-shell Result :Starting Impala Shell without Kerberos authenticationError connecting: TTransportException, Could not connect to mprl508:21000***********************************************************************************Welcome to the Impala shell.(Impala Shell v3.4.0-SNAPSHOT (27b919f) built on Tue Aug 3 21:32:01 UTC 2021)When you set a query option it lasts for the duration of the Impala shell session.***********************************************************************************[Not connected] > 

  日志显示为这样，我不知道如何解决，希望有人能给我提提建议

Hello -I have a 3 node nifi cluster, running version 1.19.1. I have noticed that consistently there is an interruption of communication between zookeeper and the nifi nodes and each of the nifi nodes disconnect and then re-connect. Another office, is running a previous version of NiFi and they are experiencing the same behavior i.e., nodes consistently disconnect and then reconnect and they have zeroed in on garbage collection as a cause. Based on review of previous tickets in this system, I believe this interruption may be caused by garbage collection. Can you provide any guidance on how to deal with this issue?  Can garbage collection be tuned/reduced so as to have less of an impact? Thank you

Hi all, I have installed CDP 7.1.8 runtime and CM 7.7.1. When I would like to download the parcel for CFM, I got the error shown belowOn my VM, I can download the parcel using wget, so I don't know why it shows 404 not found.Please kindly help me out, thanks in advance.

Hi,I am getting the following error on Importing R globalenv module with Python3, other modules are importing fine. Error comes on following modules:from rpy2.robjects import globalenvimport rpy2.robjects as robjectsfrom rpy2.robjects.packages import importrfrom rpy2.robjects import pandas2riimport osfor key in os.environ: if key in ['R_LIBS', 'LD_LIBRARY_PATH', 'R_HOME']: print(key, '=>', os.environ[key])LD_LIBRARY_PATH => /data/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p78.21656418/lib/hadoop/lib/native:/data/cloudera/parcels/PTY_BDP-7.2.0.26_CDP7.p0/jpeplite/lib/*:/data/cloudera/parcels/PTY_BDP-7.2.0.26_CDP7.p0/pepspark/lib/*:/data/cloudera/parcels/PTY_BDP-7.2.0.26_CDP7.p0/pephive/lib/*:/usr/lib/x86_64-linux-gnu/:/usr/local/lib/R/lib/R_LIBS => /usr/lib/x86_64-linux-gnu/:/usr/local/lib/R/lib/:/home/cdsw/R:/usr/local/lib/R/libraryR_HOME => /usr/local/bin/Rimport rpy2from rpy2 import situationfrom rpy2.robjects import globalenvEngine exited with status 2. Co

Hello, I have a 3 node NiFi Cluster (nifi1, nifi2, and nifi3), using v 1.19.1.   The nodes are each disconnecting consistently from the cluster.  They connect, and then disconnect, and then reconnect, and then disconnect over and over... On nifi1, when I log into the GUI, a warning on the right side of the GUI indicates:Event reported for nifi3:9443- Node disconnected from cluster due to Have not received a hearbeat from node in 338 seconds. In the nifi GUI, logging into https://nifi1:9443/nifi, under the cluster section, when i first open this display,  no nodes are indicated as connected.  When I open the cluster window, an error message "Cannot replicate request to Node nifi1:9443 because the node is not connected.  Then, if I wait several minutes and refresh the screen, then all nodes are showing as connected, and then after another refresh nodes disconnect. There is similar behavior when I log into https://nifi2:9443/nifi. I see a

org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down.org.apache.nifi.encrypt.EncryptionException: Decryption Failed with Algorithm [PBEWITHMD5AND256BITAES-CBC-OPENSSL]at org.apache.nifi.encrypt.CipherPropertyEncryptor.decrypt(CipherPropertyEncryptor.java:78)at org.apache.nifi.registry.flow.diff.StandardFlowComparator.decrypt(StandardFlowComparator.java:272)at org.apache.nifi.registry.flow.diff.StandardFlowComparator.lambda$compareProperties$2(StandardFlowComparator.java:282)at java.util.LinkedHashMap.forEach(LinkedHashMap.java:684)at org.apache.nifi.registry.flow.diff.StandardFlowComparator.compareProperties(StandardFlowComparator.java:280)at org.apache.nifi.registry.flow.diff.StandardFlowComparator.compare(StandardFlowComparator.java:258)at org.apache.nifi.registry.flow.diff.StandardFlowComparator.lambda$compareComponents$0(StandardFlowComparator.java:105)at java.util.HashMap.forEach(HashMap.java:1290)at org.apache.nifi.registry.flow.diff.StandardFlowComp

Hello,hope someone can point me in the right direction.I need to delete all files starting with some attributes value, so based on Linux way it should be something like this: rm -f /opt/my_path/myfile_* I decided to use Execute Stream Command in this way: The path seem correct, the command generated, pasted into a Linux shell is working fine, however the processor did not delete the files.If i try with a specific file it works fine.Seems that * is not permitted, any clue? Many Thanks to everybody... 

I'm stating with Jolt Transformation Processor in NiFi and I so far to understand the Jolt logic. I want to apply a simple transformation to insert a provided JSON into a list. provided JSON:{   Label1:Value1,   Label2:Value2} expected result:{GeneralLabel:[                   {                      Label1:Value1,                      Label2:Value2                   }                  ]} Any clue about how to do it? In working with shift operation but without results.

Hi , Currently i am using NIFI to consume and publish data using kafka and i have a requirement to perform retry only for topicAuthorizationException thrown by NIFI processor (publishKafka processor). Like similar to RouteonAttribute, is there any processor or any log attribute through which i can parse or read  this topicAuthorizationException error , so that i can perform retry? when kafka topic does not exist, NIFI throws below exception, the same is seen in nifi-app.log, i need to know how can we read this error in NIFI org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [test_topic]