1 data node 8 CPU,30 GB RAM Some assumptions : you have 8 container in your cluster . 1. Even if you have 2 Gb of DATA 8 conatiners will be consumed completely by the job. 2. If two parallel job runs together job will slow down significantly if preemption happens. You should tune the Queue , but at least add one more node to achieve some significant advantage of parallelism. ... View more

@Saravanan Ramaraj have you looked into apache knox? The Knox API Gateway is designed as a reverse proxy with consideration for pluggability in the areas of policy enforcement, through providers and the backend services for which it proxies requests. The Apache Knox Gateway is a REST API Gateway for interacting with Apache Hadoop clusters. The Knox Gateway provides a single access point for all REST interactions with Apache Hadoop clusters. In this capacity, the Knox Gateway is able to provide valuable functionality to aid in the control, integration, monitoring and automation of critical administrative and analytical needs of the enterprise. Authentication (LDAP and Active Directory Authentication Provider) Federation/SSO (HTTP Header Based Identity Federation) Authorization (Service Level Authorization) Auditing And then for authorization you can use Apache Ranger which offers a centralized security framework to manage fine-grained access control over Hadoop data access components coupled with kerberos you cluster will be secured and the links shall be authenticed using kerberos and ranger will provide authorization on what services the user has access to. Finally knox will be your perimeter security. ... View more