Q&A

Hi all. I can't figure out why the MergeRecord processor converts fractional values ​​to integers. Reading origin from json file.

I am trying to configure CDP 7.1.8 with Active Directory. Once the Kerberos Configuration is enabled, during the last step where the services are started, the Zookeeper service is failing to start with the following exception. I had checked the encryption types enabled on the Active Directory as well as the encryption types with which the principal are created. Also, I could see that the kinit using the generated keytabs works fine for the zookeeper service and other services as well. Is there anything that is missing in the configuration ?--- 5:01:27.233 AM WARN SaslServerCallbackHandler No password found for user: null5:01:27.235 AM ERROR QuorumPeerMain Unexpected exception, exiting abnormallyjava.io.IOException: Could not configure server because SASL configuration did not allow the ZooKeeper server to authenticate itself properly: javax.security.auth.login.LoginException: No password provided at org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(

If yes, please provide references to the documentation... Can't seem to find any.

Hi,   After Multi Delimiter Issue is solved. I got an error with data insertion. So, after the data successfully inserted to text file format table with rowserde hive.contrib.serde2.MultiDelimitSerDe. Then i need to insert those data to parquet format table. But when i ran the query to insert data, it failed.   Error message: "[FATAL] 10:25:08 vaproject.xxx tHiveRow_1 Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask".   I used this script: "INSERT OVERWRITE TABLE ParquetTable PARTITION (datadt=dateint) SELECT * FROM TextFileTable"   What is wrong or what am i missing? Please need your help. Thank you.     Sincerely,     Gideon Maruli IT Data Management

Cloudera, I am working on a NiFi flow in which incoming dates are treated as follows in my input schema:{ "name": "Launch_Date", "type": ["null",{ "type" : "int", "logicalType" : "date"}] },And my CSVReader uses the following date format:  MM/dd/yy When records are being ingested, I am getting Java errors on the month and day portion of dates that are single digit, such as 6/23/2019 and 11/3/2019, so I used the ReplaceText processor to add leading zeroes (06./23/2019, 11/03/2019).  What I was not able to figure out was if/how I can do three date conversions in one ReplaceText processor ( I also have to add leading zeroes to dates like 6/3/2019 so they become dates like 06/03/2019). So, I have three ReplaceText processors, that do Regex replacements on the following cases:dates w/ single month digitdates w/ single day digitdates w/ single month and single day digitsIn all three cases, zeroes are prepended to the single digits.  The portion of my flow with t

Tried executing following python script body & Module directory as import jsonimport java.iofrom org.apache.commons.io import IOUtilsfrom java.nio.charset import StandardCharsetsfrom org.apache.nifi.processor.io import StreamCallback#from google.cloud import bigquery#from google.oauth2 import service_accountimport googleclass ModJSON(StreamCallback): def __init__(self): pass def process(self, inputStream, outputStream): text = IOUtils.toString(inputStream, StandardCharsets.UTF_8) obj = json.loads(text) print(text) outputStream.write(bytearray(json.dumps(obj, indent=4).encode('utf-8')))flowFile = session.get()if (flowFile != None): flowFile = session.write(flowFile, ModJSON()) flowFile = session.putAttribute(flowFile, "filename", flowFile.getAttribute('filename')+'_translated.json') session.transfer(flowFile, REL_SUCCESS)session.commit() Module Directory /Users/bonthala/Library/Python/2.7/lib/python/site-packages/ &nbsp

I have install nifi-1.16.1 and used it for several ETL perviously. But suddly i'm experiencing it's restarting some times due to ot of memory issues.but when i manually start the NIFI it logs below exceptions for all controller services2023-08-19 21:11:43,480 INFO [main] o.a.n.c.s.StandardProcessScheduler Enabling StandardControllerServiceNode[service=DBCPConnectionPool[id=1654d4a2-0189-1000-7bbe-076de7850b24], name=DBCPConnectionPool, active=false]2023-08-19 21:12:01,993 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@4d7f3adf checkpointed with 317 Records and 0 Swap Files in 19 milliseconds (Stop-the-world time = 13 milliseconds, Clear Edit Logs time = 2 millis), max Transaction ID 192482023-08-19 21:12:03,810 INFO [Cleanup Archive for default] o.a.n.c.repository.FileSystemRepository Successfully deleted 0 files (0 bytes) from archive2023-08-19 21:12:13,481 ERROR [main] o.a.n.c.s.StandardControllerServicePr

In a GenerateTableFetch i have set as Maximum-value Columns the column updated_at. In addition i have define the property "initial.maxvalue.updated_at" with an hard-code initial value. However i do not want to hardcode manual the initial value but instead grab it dynamically, like performing an SQL statement to get the max updated value from the table. How i could feed such information in "initial.maxvalue.updated_at" ?

Hi , I am trying to List the files in an SFTP location. Cannot use ListSFTP because listSFTP does not take incoming connection, but the issue for us is the SFTP location is coming from a response of an API. Currently I am using FetchSFTP. but takes only a single input which is a filename, that is incorrect as per the requirements, i need a list of files in a directory.I came across this ticket [NIFI-4621] Allow inputs to ListSFTP and ListFTP - ASF JIRA (apache.org) which is still in open state. Is there any alternate way to achieve this? Is there a way to use executeScript and run python sftp ?

Hello,I am trying to setup a HTTPS on NiFi Registry and am receiving SSL_ERROR_BAD_CERT_ALERT when attempting to access the URL after loading the certificates.The way I am setting it up is that I create a CSR, have the CA sign it (AD Domain Controller) and import that certificate into the keystore.   I have generated the keystore, the truststore imported the certificates, and when I restart the NiFi Registry service the logs show no errors so I am not sure what I am doing wrong. Below are some sanitized outputs of some commands I've ran to show what I could potentially have done wrong.Curl Output:curl --cert-type P12 --cert ./keystore.p12:Password --cacert /root/ssl/ADcerts/ad1-ca.cer -v https://nifiregistry.our.network.com:8443/nifi-registry* Trying 123.4.56.78:8443...* Connected to nifiregistry.our.network.com (123.4.56.78) port 8443 (#0)* ALPN, offering h2* ALPN, offering http/1.1* CAfile: /root/ssl/ADcerts/ad1-ca.cer* TLSv1.0 (OUT), TLS header, Certifica

Hi Everyone, I have Ranger 2.4 installed. On the ranger ui, when I give a allow policy to my user, the policy is getting applied and my user is able to see the desired table. But when I give the same policy to a whole group instead of a single user, the users in that group are not able to see the desired table.Any idea how can I resolve this?

Profile is displaying as LONDON STOCK EXCHANGE GROUP PLC. But I need to change it REFINITIV US LLC

I tried connecting sqoop with Hue But got error as:Sqoop:could not get connectors in hue.ini # Settings to configure Sqoop2###########################################################################[sqoop]# For autocompletion, fill out the librdbms section.# Sqoop server URLserver_url=http://192.168.2.112:16000/sqoop# Path to configuration directorysqoop_conf_dir=/home/hadoop/sqoop-1.4.7/conf what is dfference between sqoop2 and sqoopI have installed qoop1.4.7

I has NiFi 1.11.4 installed on 3 nodes cluster. I installed the new version of NiFi 1.23.0 in a nearby directoryMoved config files:nifi.properties rolled over Ansibleautorizations.xml copied from the old versionjks files copied from the old versionflow.xml.gz copied from the old versionUpdated sensitive properties: sudo ../bin/nifi.sh set-sensitive-properties-key 'key'sudo ../bin/nifi.sh set-sensitive-properties-algorithm NIFI_PBKDF2_AES_GCM_256 What does this mean when I do update flow with sensitive props? Flow Configuration Property not specified [null]And now both my versions of NiFi could not initialize it flow, no exceptions or critical errors in logs i've found, some was fixed but nothing helpsCluster coordinator is elected successfully, Zookeeper is available.No any logs in nifi-processors.logs, no any updates on directories for provenance, flowfile, content.I enabled TRACE logs and on Initializing I see only this continuously:2023-08-23 14:03:20,513 TRACE [Load-Balan

Dear Team, I am trying to create ADF Linked service to CDP Hive server. CDP kerberized and ssl enabled cluster. Cloud you share the configurations like below: {"name": "HiveLinkedService","properties": {"type": "Hive","typeProperties": {"serverType": "HiveServer2","thriftTransportProtocol": "SASL","serviceDiscoveryMode": true,"authenticationType": "UsernameAndPassword","httpPath": "jdbc:hive2://******;principal=hive/_HOST@******;serviceDiscoveryMode=zooKeeper;ssl=true;zooKeeperNamespace=hiveserver2","host" : "******","port" : "10000","allowSelfSignedServerCert" : true,"allowHostNameCNMismatch": true,"enableSsl": true,"useSystemTrustStore": true,"trustedCertPath": ".cer path file","zooKeeperNameSpace": "hiveserver2","username" : "******","password": {"type": "SecureString","value": "******"}}}} 

Some hdfs operations (hsync, append, ... ) are not supported with the erasure coding (ec) policy. Does hbase work well when the data folders of hbase are set to the erasure coding policy?Is Hbase compatible with HDFS-EC instead of HDFS-replica ?   

I'm running RHEL 7.9. I'm using free trial. While installing agents I got error message like this below  Please help give soliution. Thank you Regard,Doan Gia Hy

I am trying to connect Qlik to hadoop (cloudera ODBC Driver for Apache Hive) , but i am facing the following error. could someone suggest how to resolve the issue please. Error: failed! [cloudera][thriftextension] (14) unexpected response from server during a http connection: connect() failed: errno = 10060. Looking for your support please to find a solution.   Thanks

Hi  TeamI need my company name to be changed to Refinitiv LLC

I have flowfile which needs to be sent to two different processors which does transformation on the Json. Flow will be split into two parts which are independent of each other so they can be performed parallel but the input for these two parts is the same flow file. how can I send a flowfile in a queue to two different processors at same time.

Dears, After installing CDP Base on premise, I'm deploying NIFI services.For this reason as recommanded, I created the Data Context and created compute cluster with two hosts and installed activated the Parcels CDH 7.1.5 by default a CFM 2.1.5, and when arriving to the Service selection (Custom Services) I don't see the NIFI services, all services belongs to the CDH parcelsHow can I get them ? Thank you in advance

Hi, I have 3 data centers dc1,dc2,dc3. There are about 100 million records migrated to dc1 and replicated to both dc2 and dc3. My goal is to find those records which are not replicated to dc2 and dc3 from dc1. Thanks in advance. 

After configuring Hive with Tez, it works properly and the ACID table is created successfully. but when trying to run insert statement to insert record to this table, Tez failed with below log------------------------------------------------------------------------------------------------------------------2023-08-04 09:48:26,265 [INFO] [main] |common.TezClassLoader|: Created TezClassLoader with parent classloader: sun.misc.Launcher$AppClassLoader@677327b6, thread: 1, system classloader: sun.misc.Launcher$AppClassLoader@677327b62023-08-04 09:48:27,355 [INFO] [main] |app.DAGAppMaster|: Creating DAGAppMaster for applicationId=application_1691002829482_0026, attemptNum=1, AMContainerId=container_1691002829482_0026_01_000001, jvmPid=83708, userFromEnv=hadoop, cliSessionOption=true, pwd=/tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/application_1691002829482_0026/container_1691002829482_0026_01_000001, localDirs=/tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/application_

Hi, We are getting an error when initializing the zk Failover Controller here the error below: WARNING: HADOOP_ZKFC_OPTS has been replaced by HDFS_ZKFC_OPTS. Using value of HADOOP_ZKFC_OPTS.log4j:ERROR Could not instantiate class [com.cloudera.cmf.eventcatcher.client.logs.ExceptionForwarderAppender].java.lang.ClassNotFoundException: com.cloudera.cmf.eventcatcher.client.logs.ExceptionForwarderAppenderat java.net.URLClassLoader.findClass(URLClassLoader.java:387)at java.lang.ClassLoader.loadClass(ClassLoader.java:418)at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)at java.lang.ClassLoader.loadClass(ClassLoader.java:351)at java.lang.Class.forName0(Native Method)at java.lang.Class.forName(Class.java:264)at org.apache.log4j.helpers.Loader.loadClass(Loader.java:198)at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:327)at org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:124)at org.apache.log4j.Propert

Hi  Im beginner to apache nifi basics but i've managed to run the processor and execute the script on NIFI locally ,but now i'm using apache nifi on docker so the problem that when I put the path to python environment it said the file not exist , note that apache nifi and python all are on containers and within the same network so i dont why its not working