I'm having the same issue. This is my agent configuration:     [General] server_host=cloudera-1 server_port=7182 max_collection_wait_seconds=10.0 metrics_url_timeout_seconds=30.0 task_metrics_timeout_seconds=5.0 monitored_nodev_filesystem_types=nfs,nfs4,tmpfs local_filesystem_whitelist=ext2,ext3,ext4,xfs impala_profile_bundle_max_bytes=1073741824 stacks_log_bundle_max_bytes=1073741824 stacks_log_max_uncompressed_file_size_bytes=5242880 orphan_process_dir_staleness_threshold=5184000 orphan_process_dir_refresh_interval=3600 scm_debug=INFO dns_resolution_collection_interval_seconds=60 dns_resolution_collection_timeout_seconds=30     This is my Cloudera Server config:     Link to the image: https://imgur.com/SIciDFr   Regards, Silva ... View more

Thanks a lot for the info ... View more

But I need to know which specific queries spills into disk, generating the scratch files. Is possible to have that kind of information?. ... View more

I'm using this one:   /api/v18/clusters/cluster/services/impala/impalaQueries?from=2018-05-31T0%3A0%3A0&filter=(user=userX)"   Also, I solved this issue by restarting the Monitoring Service.   Regards, ... View more

If you want to use spark2-shell and spark2-submit, you don't have to set those ENV variables. I set it because I wanted to point the current spark-shell/submit to spark2.   This should be done in all the nodes that you want to use the shell and/or the submit.   I forgot to add the changes that I made for spark-sumbit. In these files:   /opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/bin/spark-submit /opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/spark/bin/spark-submit Add this ENV var: SPARK_HOME=/opt/cloudera/parcels/SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904/lib/spark2 ... View more

I forgot to mention that Spark 1.6 came with CDH 5.8. I don't know how CDH installed Spark. ... View more