Hi @dyadav1   Thanks for pointing out the direction for Kafka topic but I need little more help to determine the strategy, as I mentioned if every topic is putting 100 different types of command and I need to transform each type of command into new type (assuming Jolt transformation and have jolt spec for 100 different command types), how do I store those spec and make generalize flow so that new command types get added I just need to update Spec and flow didn;t require any changes. In more terms need to dynamically figure out which command type coming in the flow file and transform it based on spec, now I feel I need to use lookup and if I store all spec in database but then for each transformation I need to do database trip which I want to avoid. ... View more

Hi Steven,   I used @bingo 's solution to get Nifi to find my JAVA_HOME.   But you mention that Nifi does not need this to run. Do you know what is the impact for running nifi without it knowing where Java is installed? ... View more

thank you for the post   but another question - according to the document - https://docs.cloudera.com/HDPDocuments/Ambari-2.7.0.0/administering-ambari/content/amb_changing_host_names.html   The last stage is talking about – in case NameNode HA enabled , then need to run the following command on one of the name node     hdfs zkfc -formatZK -force     thank you for the post   but since we have active name node and standby name node we assume that our namenode is HA enable   example from our cluster   but we want to understand  what are the risks when doing the following cli on one of the namenode   hdfs zkfc -formatZK -force   is the below command is safety to run without risks ? ... View more

Ambari-metrics-monitor is running on these 2 of 26 hosts also.  Only issue I saw in logs of 1 of the 2 hosts I checked:   Jul 29, 2020 10:31:28 AM java.util.logging.LogManager$RootLogger log SEVERE: Failed to resolve default logging config file: config/java.util.logging.properties     ... View more

Definitively, Nifi is ignoring the core-site.xml and hdfs-site.xml  I put in the configuration. I have filtered the log for my flow and I get:   2020-07-11 16:13:49,123 INFO [NiFi Web Server-298078] o.a.n.c.s.StandardProcessScheduler Starting PutHDFS[id=436939a6-0fd4-1741-bb61-7913fdc19600] 2020-07-11 16:13:49,135 WARN [Timer-Driven Process Thread-5] org.apache.hadoop.conf.Configuration /etc/nifi/confhdi/core-site.xml:an attempt to override final parameter: fs.defaultFS; Ignoring. 2020-07-11 16:13:49,136 WARN [Timer-Driven Process Thread-5] org.apache.hadoop.conf.Configuration /etc/nifi/confhdi/hdfs-site.xml:an attempt to override final parameter: dfs.datanode.data.dir; Ignoring. 2020-07-11 16:13:49,136 WARN [Timer-Driven Process Thread-5] org.apache.hadoop.conf.Configuration /etc/nifi/confhdi/hdfs-site.xml:an attempt to override final parameter: dfs.datanode.failed.volumes.tolerated; Ignoring. 2020-07-11 16:13:49,136 WARN [Timer-Driven Process Thread-5] org.apache.hadoop.conf.Configuration /etc/nifi/confhdi/hdfs-site.xml:an attempt to override final parameter: dfs.namenode.name.dir; Ignoring. 2020-07-11 16:13:49,136 WARN [Timer-Driven Process Thread-5] org.apache.hadoop.conf.Configuration /etc/nifi/confhdi/hdfs-site.xml:an attempt to override final parameter: dfs.webhdfs.enabled; Ignoring. 2020-07-11 16:13:49,145 INFO [Timer-Driven Process Thread-5] o.a.hadoop.security.UserGroupInformation Login successful for user svcqhdfuser using keytab file /etc/nifi-resources/keytabs/svcqhdfuser.keytab 2020-07-11 16:14:04,750 WARN [Timer-Driven Process Thread-5] org.apache.hadoop.hdfs.DFSUtil Namenode for mycluster remains unresolved for ID nn1. Check your hdfs-site.xml file to ensure namenodes are configured properly. 2020-07-11 16:14:21,063 WARN [Timer-Driven Process Thread-5] org.apache.hadoop.hdfs.DFSUtil Namenode for mycluster remains unresolved for ID nn2. Check your hdfs-site.xml file to ensure namenodes are configured properly. 2020-07-11 16:14:21,063 ERROR [Timer-Driven Process Thread-5] o.apache.nifi.processors.hadoop.PutHDFS PutHDFS[id=436939a6-0fd4-1741-bb61-7913fdc19600] HDFS Configuration error - java.io.IOException: Couldn't create proxy provider class org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider: {} 2020-07-11 16:14:21,064 ERROR [Timer-Driven Process Thread-5] o.apache.nifi.processors.hadoop.PutHDFS PutHDFS[id=436939a6-0fd4-1741-bb61-7913fdc19600] Failed to properly initialize Processor. If still scheduled to run, NiFi will attempt to initialize and run the Processor again after the 'Administrative Yield Duration' has elapsed. Failure is due to java.lang.reflect.InvocationTargetException: java.lang.reflect.InvocationTargetException 2020-07-11 16:14:50,810 INFO [NiFi Web Server-298342] o.a.n.c.s.StandardProcessScheduler Stopping PutHDFS[id=436939a6-0fd4-1741-bb61-7913fdc19600] 2020-07-11 16:14:51,076 WARN [Timer-Driven Process Thread-2] org.apache.hadoop.conf.Configuration /etc/nifi/confhdi/core-site.xml:an attempt to override final parameter: fs.defaultFS; Ignoring. 2020-07-11 16:14:51,077 WARN [Timer-Driven Process Thread-2] org.apache.hadoop.conf.Configuration /etc/nifi/confhdi/hdfs-site.xml:an attempt to override final parameter: dfs.datanode.data.dir; Ignoring. 2020-07-11 16:14:51,077 WARN [Timer-Driven Process Thread-2] org.apache.hadoop.conf.Configuration /etc/nifi/confhdi/hdfs-site.xml:an attempt to override final parameter: dfs.datanode.failed.volumes.tolerated; Ignoring. 2020-07-11 16:14:51,077 WARN [Timer-Driven Process Thread-2] org.apache.hadoop.conf.Configuration /etc/nifi/confhdi/hdfs-site.xml:an attempt to override final parameter: dfs.namenode.name.dir; Ignoring. 2020-07-11 16:14:51,077 WARN [Timer-Driven Process Thread-2] org.apache.hadoop.conf.Configuration /etc/nifi/confhdi/hdfs-site.xml:an attempt to override final parameter: dfs.webhdfs.enabled; Ignoring.   Brgds,   Paz ... View more

Hi.   Tis hadoop cluster hacer 1.4PB size, so for this node we have this situation size on the Mount points:   [root@ithbda108 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/md2 459G 50G 385G 12% / tmpfs 126G 36K 126G 1% /dev/shm /dev/md0 453M 77M 349M 19% /boot /dev/sda4 6.6T 6.3T 323G 96% /u01 /dev/sdb4 6.6T 6.3T 321G 96% /u02 /dev/sdc1 7.1T 6.8T 314G 96% /u03 /dev/sdd1 7.1T 6.8T 314G 96% /u04 /dev/sde1 7.1T 6.8T 318G 96% /u05 /dev/sdf1 7.1T 6.8T 323G 96% /u06 /dev/sdg1 7.1T 6.8T 325G 96% /u07 /dev/sdh1 7.1T 6.8T 323G 96% /u08 /dev/sdi1 7.1T 6.8T 324G 96% /u09 /dev/sdj1 7.1T 6.8T 324G 96% /u10 /dev/sdk1 7.1T 6.8T 324G 96% /u11 /dev/sdl1 7.1T 6.8T 322G 96% /u12 cm_processes 126G 200M 126G 1% /var/run/cloudera-scm-agent/process ithbda103.sopbda.telcel.com:/opt/exportdir 459G 338G 98G 78% /opt/shareddir   I suppose that it can be an issue about space disk and  there is no space left on the device at the time of writing into log4j.   Any idea in what action can we do to solve the space left on the mount points?  Some cloudera procedure to optimize that the process can be up?    ... View more

Just to close this thread, we've detected that we miss a configuration inside the new configuration group (nifi.zookeeper.root.node). This parameter should be different in each configuration group in order to set a new path base where to save the new cluster configuration in Zookeeper (we didn't need to create the path, it was created as soon as we change the config). Now we have both groups of nifi instances up and running. Thanks for the help! ... View more

We can configure TailFile processor for single or multiple file and configure the  Rolling Filename Pattern property , as explained in TailFile processor documentation:   https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.6.0/org.apache.nifi.processors.standard.TailFile/   https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.6.0/org.apache.nifi.processors.standard.TailFile/additionalDetails.html ... View more

Great! thanks! ... View more

Once  the rolling/express upgrade stages complete, it gives the option to choose to Finalize the upgrade, to Finalize Later or to Downgrade. Finalizing later gives you a chance to perform more validation on the cluster. Downgrade moves the cluster version back to the previous version (basically: reverses the upgrade process stages). However once you finalized the upgrade, you cannot downgrade back to the previous version.      ... View more

You can start and stop CDH services from command line. Please refer the below document for details:   https://docs.cloudera.com/documentation/enterprise/5-16-x/topics/cdh_ig_cdh_services_start.html https://docs.cloudera.com/documentation/enterprise/5-16-x/topics/cdh_ig_services_stop.html ... View more

Hi, thanks, yes i did that, but in adition, i changed authorizers.xml so that "Initial Admin Identity" is just a username. Now it looks like this: Although "USE_DN" is still in the "Identity Strategy". Any way to get full name instead og username? ... View more

UnpackContent processor do not support .rar file: http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.11.4/org.apache.nifi.processors.standard.UnpackContent/index.html   Hence, yes you can  use an  ExecuteStreamCommand  processor to invoke WinRAR command-line tool. ... View more

It seems y ou have configured your RPG to connect to a NiFi instance and during TLS handshake there is no correct SAN (Subject Alternative Name) found in Nifi certificates , hence you need to add the host FQDN to the server cert as a SAN.  ... View more

To migrate Grafana to another host, first  we need to remove the Grafana service from one node and then need to  re-install it on another node. We can do the below steps:   1. Take backup of Ambari Database 2. Add the component to a new Node using the following API: curl --user username:password -H 'X-Requested-By: ambari' -i -X POST http://AMBARI_SERVER_HOST:8080/api/v1/clusters/CLUSTER_NAME/hosts/NEW_HOST_NAME/host_components/COMPONENTNAME For Example, to add Grafana: curl --user admin:admin -H 'X-Requested-By: ambari' -i -X POST http://<ambari-server-host>:8080/api/v1/clusters/MyLabCluster/hosts/SecondLabNode02.cloudera.com/host_components/METRICS_GRAFANA 3. Go to Ambari Hosts tab and click on the node where the Component has been added using the above API 4. The current status of the Component would be Install Pending 5. Click on Install Pending and select Re-intall to complete the installation 6. Once the above is completed, Start the service using Ambari Service UI. 7. Stop the Service using Ambari Service UI. ( Grafana service on old Host ) 8. Remove the component using the following Ambari API call: curl -u username:password -H 'X-Requested-By: ambari' -X DELETE http://AMBARI_SERVER_HOST:8080/api/v1/clusters/CLUSTER_NAME/hosts/OLD_HOSTNAME/host_components/COMPONENTNAME For example, if Grafana is to be removed, curl --user admin:admin -H 'X-Requested-By: ambari' -X DELETE http://<ambari-server-host>:8080/api/v1/clusters/MyLabCluster/hosts/FirstLabNode01.cloudera.com/host_components/METRICS_GRAFANA   ... View more

You can also try to stop and start [HDP] namenode services from command line by using below command:   If you are running NameNode HA (High Availability), start the JournalNodes by executing these commands on the JournalNode host machines: su -l hdfs -c "/usr/hdp/current/hadoop-hdfs-journalnode/../hadoop/sbin/hadoop-daemon.sh start journalnode" where   $HDFS_USER   is the HDFS user. For example,   hdfs. Execute this command on the NameNode host machine(s): su -l hdfs -c "/usr/hdp/current/hadoop-hdfs-namenode/../hadoop/sbin/hadoop-daemon.sh start namenode" If you are running NameNode HA, start the ZooKeeper Failover Controller (ZKFC) by executing the following command on all NameNode machines. The starting sequence of the ZKFCs determines which NameNode will become Active. su -l hdfs -c "/usr/hdp/current/hadoop-hdfs-namenode/../hadoop/sbin/hadoop-daemon.sh start zkfc" If you are not running NameNode HA, execute the following command on the Secondary NameNode host machine. If you are running NameNode HA, the Standby NameNode takes on the role of the Secondary NameNode. su -l hdfs -c "/usr/hdp/current/hadoop-hdfs-namenode/../hadoop/sbin/hadoop-daemon.sh start secondarynamenode" ... View more