Q&A

Hello community,   I started using Apache NIFI for my bachelor-thesis. The basics of the data flow are already working. But there are some cases I can not really get the grasp on.  I get my files via HTTP, and they are mostly in TXT, CSV or XML. How my workflow (data flow?) should look like:- Multiple data sources (Question 1)- Splitting the values in multiple lines- Adding a timestamp as a column to each line (Question 2)- Adding the source (name) as a column to each line (Question 2)- Checking if the value was already seen (Question 3)- Adding a new column to each line with the value "already seen" or "first seen" (Question 3)- Merging the content- Changing Filename- PutFile (Question 4) Question 1Do I need to make a new Data flow for each new resource? Because otherwise they have all the same or a totally random file name at the end. Question 2If I add a column with the same value to each line, is it better to add the value before, at or

Hello ,  Amazon RDS is starting the end of life (EOL) process for MySQL major version 5.6 . Hence we are going to migrate Mysql 5.6 to 5.7 . Our CM/CDH Version is 5.11.2 .What would be the impact of Mysql RDS Upgrade from 5.6 to 5.7 on the cloudera cluster ? Do we require any downtime on the cluster ?

Hello Everyone, Sorry if this question seems dumb, but I have query parameter (that I received via ListenHttp) in format a=value_a&b=value_b and so on, I want to convert this to json as the form given below - {"a" : "value_a","b" : "value_b"..} I am not quite sure, but I think I should be using ReplaceText, but I am unable to understand how to write the regex. If using ReplaceText is the one I should not use on Nifi, then please do let me know how to approach the same? Thanks!Ashwin

How do I add a new node to a EMR Kerberised cluster. Once added I need the node to be Kerberised. How do I do this without having to having to do everything from scratch 

Cloudera pricing is per-node: https://www.cloudera.com/products/pricing.htmlIs it only for service nodes or for gateway nodes as well?For example, I have a cluster:HDFS NameNode: 1 nodeHDFS DataNode: 3 nodesHDFS gateway with my custom app using HDFS: 1 nodeWhat is the number of nodes to pay for in this example?

I'm running 3 node docker cluster on a single host lets say `dev.example.com`   I have followed https://nifi.apache.org/docs/nifi-docs/html/walkthroughs.html#creating-and-securing-a-nifi-cluster-with-the-tls-toolkit  but added every node into a seperate docker container   nifi-node1 0.0.0.0:8096 -> 8080/tcpnifi-node2 0.0.0.0:8097 -> 8080/tcpnifi-node3 0.0.0.0:8098 -> 8080/tcp  I'm accessing using https://dev.example.com:8096/nifi . I have imported client cert on my browser, I'm able to access everything from Chrome. I want to use python to start/stop/manipulate the processors using nipyapi      import osimport nipyapiSSL_DIR = '/home/ubuntu/ssl'nipyapi.config.nifi_config.host = 'https://dev.example.com:8096/nifi-api'ca_file = os.path.join(SSL_DIR, 'nifi-cert.pem')client_cert_file = os.path.join(SSL_DIR, 'CN=nifi-node1_OU=NIFI.pem')client_key_file = os.path.join(SSL_DIR, 'CN=nifi-node1_OU=NIFI_key.pem')client_key_password =

HI all,I m using HBase with HDP 3.1.0. Ambari shows that a total of 63 Regions are in transition. This RIT issue is causing problems for other dependent application. Is there a way to fix this without deleting any data/directory?Also, hbck is not working with HBase 2.0.Any help will be deeply appreciated.Thank you!

I've read the question below:https://community.cloudera.com/t5/Support-Questions/MiniFi-to-NiFi-connection-through-load-balancer/td-p/197667but in my case that I've implemented, site to site does not balance the incoming flows from MiNiFi.Here is my scenario:I have a NiFi cluster with 4 nodes. in the MiNiFi setup, I've set the remote process group URL to the nifi.remote.input.host that I've already set in the nifi.properties.Although Port 1026 is opened for receiving data in all nodes, when I use tcpdump command, I can see that only the host for the node which I've mentioned above are getting data and other nodes in the cluster do not receive any data.Nifi version: 1.8.0MiNiFi version: 0.5.0 java

I have that problem and I have followed the steps of copying the flow.xml.gz to the nodes that do not connect, but after copying them, I have problems to decrypt since a proper key is generated for the gz of that moment   this is the log 2019-10-08 16:59:57,735 ERROR org.apache.nifi.web.server.JettyServer: Unable to load flow due to: org.apache.nifi.controller.serialization.FlowSynchronizationException: org.apache.nifi.encrypt.EncryptionException: There was a problem decrypting a sensitive flow configuration value. Check that the nifi.sensitive.props.key value in nifi.properties matches the value used to encrypt the flow.xml.gz fileorg.apache.nifi.controller.serialization.FlowSynchronizationException: org.apache.nifi.encrypt.EncryptionException: There was a problem decrypting a sensitive flow configuration value. Check that the nifi.sensitive.props.key value in nifi.properties matches the value used to encrypt the flow.xml.gz fileat org.apache.nifi.controller.StandardFlowSynchron

Hello Everyone, I'm trying to enable kerberos on my HDP setup. during the wizard, it fails at Check Kerberos step. Before that, it shows that all the Keytabs creation and everything is getting successful.  But the keytabs are not actually created. Please check the below screenshot: I checked in the host in question and the keytab file indeed didn't exist. Can anyone please give ideas where it might be going wrong?In the ambari-server log file I see the below error snippet. 27 Jan 2021 18:34:52,512 INFO [Server Action Executor Worker 6943] KerberosServerAction:353 - Processing identities...27 Jan 2021 18:34:52,730 ERROR [Server Action Executor Worker 6943] FinalizeKerberosServerAction:114 - Failed to update the owner of the keytab file at /etc/security/keytabs/kerberos.service_check.012721.keytab to ambari-qa: chown: cannot access ‘/etc/security/keytabs/kerberos.service_check.012721.keytab’: No such file or directory27 Jan 2021 18:34:52,738 ERROR [Server A

Good afternoon everyoneI am trying to configure 2 containers (Openldap and NIFI) but it is impossible to find what the error is because the nifi screen does not show any change.I have read several incidences about this issue and I have taken those examples as a reference without getting the expected success, so I will need your help to see if I can make it work.It's like all the configuration in the 3 files is not read and executed so that an unknown user can log in.Thanks in advance for any ideas to help me solve this problem. I leave you the data:Apache nifi version 1.12.1OpenLDAP version OpenLDAP 2.4.50Open Ldap configurationou=nifi,dc=example,dc=local* User usernifi* Pass nifiroot@44266e786b13:/# ldapsearch -x -W ldap://localhost -D "uid=usernifi,ou=nifi,dc=example,dc=local"Enter LDAP Password: # extended LDIF## LDAPv3# base <> (default) with scope subtree# filter: (objectclass=*)# requesting: ldap://localhost## search resultsearch: 2result: 32 No such object# numRespons

I am attempting to add a node to my cluster.  My steps are:Unzip a fresh copy of the binary installCopy production authorizer.xml and flow.tar.gz to the new conf directoryUpdate my nifi.properties for the new node this is similar to the a production node with the exception of references to the server itself.When I bring up the new node I see it in 'Connecting' state in cluster manager but it never completes.I receive the following error:2021-02-01 18:52:13,164 ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to java.lang.NoClassDefFoundError: org/apache/nifi/security/xml/XmlUtilsjava.lang.NoClassDefFoundError: org/apache/nifi/security/xml/XmlUtilsat org.apache.nifi.authorization.AbstractPolicyBasedAuthorizer.parsePoliciesUsersAndGroups(AbstractPolicyBasedAuthorizer.java:416)at org.apache.nifi.authorization.AbstractPolicyBasedAuthorizer.checkInheritability(AbstractPolicyBasedAuthorizer.java:348)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect

I have created a process in Nifi to get a file from a first folder, compress it and delete the uncompressed file. I have used:GetHDFS: to get the file, deleting it from the folder (Keep Source File is set to False)PutHDFS: to compress the file and save in a second folder The process seems working, in fact the file is not anymore in the first folder and the compressed file is in the second folder.The problem is that a warning message is displayed:Could not remove <file path> from HDFS. Not ingesting this file... So I have the doubt that the uncompressed file is still somewhere in the HDFS, but I don't know where.What does it mean the warning message? 

Hi, I appeared for CCA 175 exams yesterday and its been more than 24 hours and i have still not received the results. I read some of the other forums that the results are usually sent in few hours.I didn't find results in spam folder either. Any idea how long does it take ? Regards, Sunil Guntuku  

hi , In the develop enviroment , there is a service named CM yarn usage aggregation runs in yarn per hour . It can be found in jobhistoryserver's  web ui . but in the test env , there is no more . The difference between this two is that develop start with root , test env start with a user with previlege sudo . how to find the starting log of CMyarnusageaggregation  to debug the problems ? The log-aggretion properties are both set to enabled. Much Thanks!

-rw-r--r--.  1 hive hive 209715489 Jan  20 23:21 hadoop-cmf-hive-HIVEMETASTORE-masternode3.phenoix.test.com.log.out.2021-01-20-5-rw-r--r--.  1 hive hive 209715610 Jan  21 04:31 hadoop-cmf-hive-HIVEMETASTORE-masternode3.phenoix.test.com.log.out.2021-01-21-1-rw-r--r--.  1 hive hive 209715215 Jan  21 09:38 hadoop-cmf-hive-HIVEMETASTORE-masternode3.phenoix.test.com.log.out.2021-01-21-2-rw-r--r--.  1 hive hive 209715271 Jan  21 14:43 hadoop-cmf-hive-HIVEMETASTORE-masternode3.phenoix.test.com.log.out.2021-01-21-3-rw-r--r--.  1 hive hive 209715387 Jan  21 19:50 hadoop-cmf-hive-HIVEMETASTORE-masternode3.phenoix.test.com.log.out.2021-01-21-4-rw-r--r--.  1 hive hive 209715339 Jan  22 00:54 hadoop-cmf-hive-HIVEMETASTORE-masternode3.phenoix.test.com.log.out.2021-01-22-1-rw-r--r--.  1 hive hive 209715439 Jan  22 05:55 hadoop-cmf-hive-HIVEMETASTORE-masternode3.phenoix.test.com.log.out.2021-01-22-2-rw-r--r--.  1 hive hive 2097

Hello, Does anyone has any clue about below error. Datanode is not coming up with this error in the logs.We are on Cloudera Hadoop 6.3.0. RECIEVED SIGNAL 15: SIGTERM Thankssnm1523

we installed small HDP cluster with one data-node machine HDP version is `2.6.5` and ambari version is `2.6.1` so this is new cluster that contain two name-node with only one data-node ( worker machine ) the interesting behavior that we see is that increasing of `under replica` on ambari dashboard , for now the number is `15000` under replica blocks as we know the most root cause of this problem is network issues between name node to data-node but this isn't the case in our hadoop cluster we can also decrease the under replica by the following proceduresu - <$hdfs_user>bash-4.1$ hdfs fsck / | grep 'Under replicated' | awk -F':' '{print $1}' >> /tmp/under_replicated_files-bash-4.1$ for hdfsfile in `cat /tmp/under_replicated_files`; do echo "Fixing $hdfsfile :" ; hadoop fs -setrep 3 $hdfsfile; done  but we not want to do it because under replica problem should not happens from beginning and maybe need to tune some HDFS paramet

Hi,  I'm not able to start the management service after the reboot of the manager node. I'm not sure what happened, everything was working fine until the reboot.  [21/Jan/2021 11:03:25 +0000] 15063 MainThread agent ERROR Heartbeating to ip-172-31-203-46.ec2.internal:7182 failed.Traceback (most recent call last):File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1399, in _send_heartbeatresponse = self.requestor.request('heartbeat', heartbeat_data)File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 141, in requestreturn self.issue_request(call_request, message_name, request_datum)File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 254, in issue_requestcall_response = self.transceiver.transceive(call_request)File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 483, in transceiveresult = self.read_framed_message()File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/

Hi All, I am trying to build JSON string using Ifelse condition in ReplaText processor but i am getting error as invalid Invalid Expression . Working Ifelse condition : ${status:contains('0'):ifElse('Success','Failed')} This below string is not working and it is throwing invalid expression . Please assist on this . ${status:contains('0'):ifElse('{"filename":"${overall_filename}","status":"${status}","message":"${overall_message}"}','{"filename":"${overall_filename}","status":"${exc_status}","message":"${overall_message}"}')} Thanks

Is Cloudera Manager an Open Source Software

We have a three node NiDi cluster with an F5 load balancer in front of it. The main flow start with a listenHTTP processor to wait for files from AWS. The F5 has a single pool that then sends files round robin to NiFi. How would I configure the flow to use NiFi load balancing instead of the F5 or can they be used together. What confuses me is I think I would send to a single NiFi node and the connection after ListenHTTP would load balance across the cluster. This is a single point of failure so I cannot see the benefit in NiFi load balancing. Can someone explain please Cheers.

Hi @ApacheNifi,i have i extracted the attributes from the filename and used AttributesToJson to create a json file to upload into elasticsearch (to create a catalog).It gives me many problems, what can I do?I also noticed that from updateattribute to attributetojson it adds "\ n" in two fields.This is my flow: UPDATEATTRIBUTE: I have correctly configured updateattribute and this is the result Setting:Result:ATTRIBUTESTOJSON:Configuration:Result: as you can notice in jsonattributes it adds a "\ n" at the end of the strings "giri" and "id-riduttore" fieldsPUTELASTICSEARCHHTTPRECORD:Configure:   Error:How can i do this? thank you so much 

Ranger service is failing to install with below error Error executing: CREATE FUNCTION `getXportalUIdByLoginId`(input_val VARCHAR(100)) RETURNS int(11) BEGIN DECLARE myid INT; SELECT x_portal_user.id into myid FROM x_portal_user WHERE x_portal_user.login_id = input_val; RETURN myid; ENDjava.sql.SQLException: This function has none of DETERMINISTIC, NO SQL, or READS SQL DATA in its declaration and binary logging is enabled (you *might* want to use the less safe log_bin_trust_function_creators variable)SQLException : SQL state: HY000 java.sql.SQLException: This function has none of DETERMINISTIC, NO SQL, or READS SQL DATA in its declaration and binary logging is enabled (you *might* want to use the less safe log_bin_trust_function_creators variable) ErrorCode: 14182021-01-19 10:13:39,544 [E] ranger_core_db_mysql.sql file import failed!

When I try to look at logs from command `yarn logs -applicationId xxx`, some errors raised: Response from the NodeManager:dp006:45454 WebService is not successful, HTTP error code: 500, Server response:Exception reading log file. Application submitted by 'xxx' doesn't own requested log file : syslogCan not get any logs for the log file: stderrResponse from the NodeManager:dp006:45454 WebService is not successful, HTTP error code: 500, Server response:Exception reading log file. Application submitted by 'xxx' doesn't own requested log file : stderrCan not get any logs for the log file: prelaunch.outResponse from the NodeManager:dp006:45454 WebService is not successful, HTTP error code: 500, Server response:Exception reading log file. Application submitted by 'xxx' doesn't own requested log file : prelaunch.outCan not get any logs for the log file: launch_container.sh The question is similar to https://community.cloudera.com/t5/Support-Questions/Cannot-read-log-files-on-ap