Is that on all the agents or on ambari server machine? I'm assuming _CLIENT components aren't needed. Thanks. ... View more

Great. I'll try that out here shortly and report back. ... View more

Thanks for the quick response. How does the sandbox actually start all the services? Is there an easier way vs having to keep track of all the services added/removed etc. ? I read somewhere there is some option in ambari.properties (but i can't seem to find anything) ? ... View more

Thanks @Matt Clarke I understand HDF is separate. I'm looking for the best approach to use existing HDP with HDF. I'll look into ambari-nifi-service again - i didn't pursue this service because of the disclaimer about it being for demo purposes. Anyone using this service - is it up to date with 2.4? Assuming i don't care about upgradability at the moment... Prefer to add nifi cluster to existing HDP kerberized cluster. any ideas/posts/links on how to best do that would be helpful. Is the best approach to setup a whole new ambari system? I would like to avoid doing this. Thanks ... View more

Thanks for putting work into this. Can you speak to the difference with your approach vs using: 1. cloudbreak ambari-docker 2. using docker compose and/or ansible container. Also does this support kerberos? If after review this looks promising.... i plan to integrate a few tasks/roles via ansible to automate some of the manual setup ... View more

@Shashang Sheth I'm not sure what you mean. Basically that error from oozie can mean a number of things. e.g. the table actually doesn't exist, if the classpath isn't set correctly it will also give table not found, or if the metastore uris aren't set correct oozie may still fail with table not found. Can you explain what "solution" worked for you? Thanks ... View more

Thanks @bikas so if the spark queue capacity is 40% and max capacity is 50% and user limit is 1 - then in theory regardless of what options are set during spark submit the spark job should never use resources above 50%...correct? Preemption is enabled here is the config: yarn.resourcemanager.monitor.enable=true yarn.resourcemanager.monitor.capacity.preemption.max_wait_before_kill=15000 yarn.resourcemanager.monitor.capacity.preemption.monitor_interval=3000 yarn.resourcemanager.monitor.capacity.preemption.total_preemption_per_round=0.1 yarn.resourcemanager.scheduler.monitor.policies=org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy ordering policy for both queues is 'fair' and Enable Size Based Weight Ordering is not enabled. I will try out a few more tests and make sure it really is utilizing resources it shouldn't. If something else seems off in our config - let me know... ... View more