I admin a small Hortonworks Hadoop cluster which runs consistently two custom spark applications, each in separate queue (dev and prod). Default queue is used for services like thrift server / zeppelin etc. Last two weeks some process submits several applications each hour to the root.default queue on behalf of user hadoop, getsss and some others. There are more 12 thousand apps submitted to this date. Node Managers die after several minutes if I restart YARN or the whole cluster. I know nothing about these applications. They are all either in ACCEPTED or FAILED state. What bothers me is: 1. Who (which service or user, from which machine) keeps submitting these apps? The cluster is hosted in cloud in internal network and accessible only via several edge gateway forwarded ports (ssh and web-ui of hue, ambari, yarn-ui and zeppelin) 2. How do I stop this from happening? I see following solutions: - block default queue from any submissions after cluster startup and clear the default queue I didn't find a decent way to clean queue from 12000 apps at a time, and it takes ages to kill them one-by-one. I still have to reopen the queue to restart e.g. zeppelin, and blocking it each time seems a bad idea - delete default queue and reconfigure all services to use other queue as default Also seems like a painful and ugly solution - find out what submits apps and kill it with fire I still have to clear the queue, so the last question is 3. How do I clear the queue from this mess? Thanks for help in advance!
... View more
