I am running HDF 3.0.1.1 which comes with NiFi 1.2.0.3.0.1.1-5. We are using SiteToSiteBulletinReportingTask to monitor bulletins (for things like Disk Usage and Memory Usage). When we restart NiFi via Ambari (either with a Restart or Stop and then Start), when NiFi comes back up the SiteToSiteBulletinReportingTask no longer works. It throws the following error when it is first trying to start up: SiteToSiteBulletinReportingTask[id=ba6b4499-0162-1000-0000-00003ccd7573] org.apache.nifi.remote.client.PeerSelector@34e976af Unable to refresh Remote Group's peers due to response code 409:Conflict with explanation: null No matter how long we wait, it never works. The ways I have been able to get it to start working again are as follows: Stop and then Start the Remote Input Port the SiteToSiteBulletinReportingTask is using Delete the SiteToSiteBulletinReportingTask and create a new one Wait a while and stop and start the SiteToSiteBulletinReportingTask (however this doesn't work consistently) I have tested the same flow steps using a process that uses a Remote Process Group and a different Remote Input Port, and that RPG throws the same error when first coming up but then starts working after a period of time. So maybe the SiteToSiteBulletinReportingTask isn't trying enough times to connect to the Remote Input Port? ... View more

I need to move SmartSense logs from /var/log to /data/disk1/log. In Ambari for the SmartSense component, I have updated "activity_log_dir" to be /data/disk1/log/smartsense-activity and "hst_log_dir" to be /data/disk1/log/hst. I then restarted all SmartSense components and most of the ".log" files are now going to /data/disk1/log, however some of the ".out" files are still going to /var/log. Does anyone know how I can fix this? For instance, here are the logs going to /data/disk1/log/smartsense-activity: activity-explorer.out activity-explorer.log activity-analyzer.log Here are the logs still going to /var/log/smartsense-activity: activity-analyzer.out Here are the logs going to /data/disk1/log/hst/: hst-server.log Here are the logs still going to /var/log/smartsense-activity: hst-agent.log hst-server.log hst-server.out hst-gateway.log (Also notice how hst-server.log is going to both /data/disk1/log and /var/log) I am running Ambari 2.5.1, HDP 2.6.1, and SmartSense 1.4.0.2.5.1.0-159. ... View more

I have set up a job that runs several hadoop commands including 'hdfs dfsadmin' and 'hdfs dsfs -du' and I was wondering if these could be taxing on my cluster at all if I run them every 5 minutes or would it be harmless? The reason I am running several of these commands every 5 minutes is I have set up a job to run these commands, parse out the output to a structured format for a hive table so we can create historical reports about our system ... View more

I am looking for a way to monitor and track the number of Hive queries that have been run per day/week/month and by who, how long they took, etc. Basically I want to be able to run my own queries to how the Tez View in Ambari works. The data in the Tez view is exactly what I want, however I want to run queries against that data and not just click around in the UI. What is a recommended way to go about this process/has anyone done this before? ... View more