This is a very loaded question. Many things can change and impact performance. You can have suddenly more data that meets the criteria for that job processing, some other process is using more resources, your network band can be impacted by some other process or have some dropped packets. You can also have a setting that due to change in conditions could be insufficient, e.g. JVM heap, file descriptors, etc. You must analyze data provided by your monitoring tools and identify what seems to be the bottleneck: CPU, I/O, network, RAM etc. Check your Ganglia and Nagios monitoring and see what they report. In most recent versions of HDP, you could benefit of SmartSense which you may have based on your subscription level. This is an awesome tool designed by engineers in pain of administrating clusters and dealing with mis-configuration and performance issues. As a short-term approach, you should troubleshoot the current issues, address as much as you can, and then plan for an HDP upgrade to use SmartSense. Read more about SmartSense here: http://hortonworks.com/services/smartsense/
