@Baruch AMOUSSOU DJANGBAN Currently this is not possible. HADOOP-10019 is the community JIRA to add this functionality to HDFS. ... View more

@sagar pavan This will happen when there are not enough resources (memory) to run the AppMaster container needed to control the Tez job. In YARN capacity-scheduler.xml there is a property yarn.scheduler.capacity.maximum-am-resource-percent which controls the percentage of total cluster memory that can be used by AM containers. If you have several jobs running then each AM will consume the memory required for one container. If this exceeds the given % of total cluster memory the next AM to run will wait until there are free resources for it to run. You'll need to increase yarn.scheduler.capacity.maximum-am-resource-percent to get the AM to run. ... View more

@Timo Burmeister I don't believe there are any recommendations written down. Knox itself is more-or-less just a proxy server. It uses Jetty internally and will comfortably run in 8GB of RAM on most production systems. As Knox is multi-threaded, having more CPU threads will allow you to process more simultaneous requests through Knox. I'm not aware of any performance testing done so you'd need to experiment to find out based on your expected load how many CPU threads would work, but in general 8 CPU threads should be a good starting point. The number of NICs needed again depends on the work-load you expect also. If you're pushing large volumes of data through Knox then obviously you'll need to think about 10GbE or multiple bonded 1GbE NICs. You should probably also have a separate NIC for the external network and the internal cluster network; unless you're using VLANs or virtual IPs on a single NIC. For most starting configurations I'd say a 1GbE NIC would be sufficient. ... View more

@Sami Ahmad These messages are the Ranger plugins for HDFS and Hive connecting to Ranger Admin to check that they have the latest policies. If you want to stop these messages then you'll need to turn the Ranger Admin logging down to WARN, but these messages are normal. If you look at the frequency they'll occur every 30 seconds which is the default polling period for the plugins. ... View more

@Arpan Rajani Yes, you can use a wildcard certificate - see https://en.wikipedia.org/wiki/Wildcard_certificate If you're using a CA authority then most will generate wildcard certificates for you. If you're using an internal CA or self-signed certificates then this link shows you how: https://serversforhackers.com/self-signed-ssl-certificates In terms of using it for Hadoop, it is used in the same way as a regular certificate but you only have one certificate for all the services. The main security issue with this is that if someone gets hold of the certificate they can install it on any host in your network that matches the domain in DNS (for example *.example.com) and get a valid certificate on that host. ... View more

@Chris L The Ambari Agent is a python process so uses Python's logging facility. To change the pattern, go to /etc/ambari-agent/conf and copy logging.conf.sample to logging.conf, then edit it and look for the lines [formatter_logfileformatter] format=%(levelname)s %(asctime)s %(filename)s:%(lineno)d - %(message)s Change the 'format' line to suit your needs. There is information on the format for this here. The available attributes that can be written to the file from each log record are given here. ... View more