About sroberts

sroberts · ‎01-12-2016

I've been told that the DV2 series is now recommended over the A series. Shall we update this? https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-size-specs/#standard-tier-dv2-series

sroberts · ‎12-18-2015

@schintalapani - Do you have more details? Such as if there are JIRA issues or dev email threads tracking this?

sroberts · ‎12-10-2015

For configuring/tuning YARN & MR memory settings: the HDP Configuration Utility is referenced in the HDP documentation. and the Ambari "Stack Advisor" provides similar configuration. Which is preferred? Do they overlap? Are they maintained similarly (such that they don't conflict)? And so on ...

sroberts · ‎10-29-2015

@Jonas Straub 2. Yep, that's what I've been doing. It works, but configuration changes shouldn't require such complexity. The API needs to be more human understandable. An alternative is to be really hacky with 'configs.sh' and 'sed': cd /tmp configssh="/var/lib/ambari-server/resources/scripts/configs.sh" ${configssh} get localhost mycluster hive-env \ | sed -e '1,3d' \ -e '/^"content" : / s#",$#\\n\\n WHATEVER-YOU-WANT-TO-ADD \\n",#' \ > /tmp/hive-env.json ${configssh} set localhost mycluster hive-env /tmp/hive-env.json

sroberts · ‎10-29-2015

@Jonas Straub Let me know what you find. I bet you wiped it out. But would love if it worked properly.

sroberts · ‎10-29-2015

@jramakrishnan@hortonworks.com - I recall @jplayer@hortonworks.com @nfakhar@hortonworks.com adapted the code for diverse hardware on AWS.

sroberts · ‎10-29-2015

Ambari Blueprints does not (as of 2.1.2) use the StackAdvisor to provide recommended configurations. It is planned for Ambari 2.1.3 (AMBARI-13487). But you can use the recommendation engine yourself. The following script uses the Recommendation API (/api/v1/stacks/HDP/versions/<ver>/recommendations😞 https://github.com/seanorama/ambari-bootstrap/tree/master/deploy The script generates & deploys a Blueprint, including StackAdvisor recommendations. Alternatively use it to get the recommendations only (set 'export deploy=false' to just generate the configurations) Or borrow the API calls to integrate into your own deployment method. p.s. a few projects have borrowed this code to generate configurations throughout the life of a cluster. p.s.s. these scripts are used by Google's bdutil so have been heavily field tested.

sroberts · ‎10-29-2015

How can we append to '-env' in a clean programatic way? 2 scenarios: Blueprints: How to append to 'hadoop-env' at cluster build? It appears you would have to set the entire block. Which means fetching from an existing cluster and setting the entire thing. This is not reliable, nor is it sustainable since the defaults may change between versions of Ambari/HDP. How to change programmatically after cluster build? Ideally you would be able to append to the existing configuration.

sroberts · ‎10-29-2015

Regarding permanently adding custom JARs for Hive SerDes (Feel free to answer those which you now, and come back later to answer the rest.) What is the difference between these, and do they conflict? hive-env -> hive-env template -> HIVE_AUX_JARS_PATH hive-site -> hive.aux.jars.path Is hive.reloadable.aux.jars.path supported? Is hdfs:// supported as the path? Are wildcards or open paths supported? e.g. /mydir/*.jar or /mydir What other permanent methods are supported? What other temporary methods are supported? Other than `add jar jarfile.jar`

sroberts · ‎10-28-2015

Good answer. Somwa's was more complete with a description of how it works.

Online	Offline
Last Visited	‎05-01-2020 01:20 PM

Member Since	‎09-21-2015 02:23 PM
Last Visited	‎05-01-2020 01:20 PM
Posts	85
Kudos received	71

Cloudera Community

Re: Google Storage and Kerberos integration

Re: What is the Hortonworks recommendation on Swap...

Re: Ambari Blueprint : Sizing Components memory ba...

Re: Ranger KMS won't start

Re: Ambari Kerberos Wizard: How to configure Activ...

Re: Recommendations for Microsoft Azure HDP Deploy...

Re: How to read from a Kafka topic using Spark (st...

YARN&MR Configuration: "HDP Configuration Utils" o...

Re: (ambari) Append to -env templates (e.g. hadoop...

Re: (ambari) Append to -env templates (e.g. hadoop...

Re: Ambari Blueprint : Sizing Components memory ba...

Re: Ambari Blueprint : Sizing Components memory ba...

(ambari) Append to -env templates (e.g. hadoop-env...

(hive) How to permanently add SerDe JARs to Hive?

Re: Falcon mirror recipe: What happens if an insta...