Re: Advance Configuration Snippet in Director Conf...

Vinithra · ‎08-01-2017

Do you use cloud infrastructure for your CDH clusters? You are invited to share your feedback on your usage and management of cloud infrastructure. You will thereby influence the roadmap of Cloudera Director and other cloud orchestration software from Cloudera. Please share your experiences by taking this short survey : http://tiny.cloudera.com/director-survey Key Cloudera Director features recently added include: Automatic repair of clusters Support for long-running clusters: Cloudera Director now syncs with cluster changes made via Cloudera Manager AWS Spot instance support and improved resiliency to spot instance disappearances Support for AWS EBS volumes Support for Azure ADLS, Azure premium and standard storage For more details on which versions support different features, please refer to the release notes. If you have any questions or comments, you can reach me at vinithra@cloudera.com. Thanks for your input, Vinithra Varadharajan Engineering Manager, Cloud

Vinithra · ‎03-20-2017

Hi, If that property does not exist in the CM properties already, then you should find the "safety valve" configuration for the file that it should live in. Example: Safety valve of core-site.xml. You can find the configuration key in the same link, and then use that in the Director config file. Hope this helps.

Vinithra · ‎01-24-2017

Hi, Currently Cloudera customers who have physical hardware / datacenters that they manage do setup clusters using Cloudera Manager, and do not interface with Director (via the BYON plugin) for this purpose. This path works for them as they usually have to setup the cluster once and manage it thereafter - there isn't a need for bringing up/down clusters or growing/shrinking the clusters. Private clouds are a different category: Cloudera has a Director - VMware vSphere plugin integration that is in alpha preview mode. What's the usecase that you were hoping to solve with the BYON plugin?

Vinithra · ‎10-04-2016

Does this list of configuration properties help: http://www.cloudera.com/documentation/enterprise/properties/5-7-x/topics/cm_props.html

Vinithra · ‎10-04-2016

Hi, Your observation is correct. The reason we give the choice of resume or rebuild when submitting the bootstrap command is because you might be trying to bootstrap a second cluster with the same config file. Modifications to an existing cluster should be done using the update command, which requires the config file as input and you can make changes to the config file. However, do note that not all modifications to the config file are supported by update. Specifically, config changes are not supported with update. However, changes to instance count are supported, which allows you to grow and shrink the cluster. Hope this helps.

Vinithra · ‎09-16-2016

Hi, Without Cloudera Director, this is going to be hard. You can try using the CM API to add hosts to the service and grow the service by figuring out what CM does for Solr. http://www.cloudera.com/documentation/enterprise/latest/topics/cm_intro_api.html This will be much easier if you are using Director, as it does the work for you of growing the service by adding the hosts, registering them with CM, etc.

Vinithra · ‎09-15-2016

Hi, I'm assuming you used Cloudera Director to set this cluster up. To grow the cluster based on load, you first need to identify how you can programmatically assess that load has/is going to increased. Examples are: day of month triggered by a cron job, or the output of some monitoring that tells you that the volume of data that needs to be ingested has crossed a threshold. With that indicator that load is increasing as input, write a script that makes a PUT request to the update endpoint on clusters, with a body that describes the cluster with the increased count of instances: PUT /api/v5/environments/{environment}/deployments/{deployment}/clusters/{cluster} You can try out this API by going to the API console in your Director UI. You can find a link to the API console in the upper right ?. Is this something that you can start with? Were you expecting a different way of achieving this? -Vinithra

Vinithra · ‎09-02-2016

Hi, we currently don't have a scheduled date for this, but we will take your request into consideration during prioritization.

Vinithra · ‎07-13-2016

Kudu will be supported in Cloudera Director when Kudu goes GA. Can you give us more details on how you are using Director and your experience so far?

Vinithra · ‎07-12-2016

Amit, I wanted to clarify something based on feedback that I got. If you use Directo to setup a cluster without Kudu, and then add Kudu via CM, then Director will not be able to grow or shrink the cluster thereafter as the topology has changed outside of Director. So, whether to use Director or not depends on your use-case - if you are looking at it as the means of a one-time setup that isn't going to require modifications via Director thereafter, then Director could meet your needs. However, if you want to continue using Director, then adding Kudu out-of-band will not work with Director 2.1.

Vinithra · ‎07-11-2016

Hi Amit, Director 2.1 currently does not support Kudu-beta. However, you should be able to use Director to set up all the services other than Kudu, and then set up Kudu using CM. If you run into any issues, please post your questions here: http://community.cloudera.com/t5/Beta-Releases-Apache-Kudu/bd-p/Beta

Vinithra · ‎06-16-2016

Hi, Regarding elasticity, you can grow and shrink clusters using Director. Auto-scaling is not an out-of-the-box feature; however, there are users who have used Director's API to automate scaling of the cluster based on some parameters. In any case, we have this on the roadmap and we will take your input into prioritization. If you can provide more details on your usecase, that would be welcome too. Thanks, Vinithra

Vinithra · ‎01-30-2016

Damion, Thanks for your question. To clarify, CDH is certified on Azure and clusters deployed in Azure are supported. However, Cloudera Director can currently support creation of Cloudera Manager and CDH clusters only in AWS and GCP, as you've observed. Support for Azure in Cloudera Director itself is on the roadmap. Please stay tuned and let us know how we can help in the meantime.

Vinithra · ‎10-14-2015

Roland, Cloudera Director uses the same spec as CM API. You can find the roletypes that CM (and thereby Director) accepts here: http://cloudera.github.io/cm_api/apidocs/v10/path__clusters_-clusterName-_services_-serviceName-_roles.html Do keep us posted on how things go and if you run into any further issues. -Vinithra

Vinithra · ‎07-10-2015

Kirk, Yes, you can modify the Cloudera Manager and CDH repositories to specify any other version. Make sure that you're using the same minor version of both Cloudera Manager and CDH, i.e. both should be a 5.4.x version. If you're using the config file, refer to aws.reference.conf for examples. There you'll see this in the 'cloudera-manager' block, which you can modify to install CM5.4 instead: # repository: "http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.4/" # repositoryKeyUrl: "http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/RPM-GPG-KEY-cloudera" In the 'cluster' block, you will see this to modify to install CDH5.4 parcels instead: # parcelRepositories: ["http://archive.cloudera.com/cdh5/parcels/5.4/"] -Vinithra

Vinithra · ‎07-10-2015

Hi, You can check the Cloudera Director logs to see what AWS actions are taken. If you're using the Cloudera Director client, the logs can be found at $HOME/.cloudera-director/logs/application.log. If using the Cloudera Director server, the logs can be found at /var/log/cloudera-director-server/application.log. If you let us know what you typically do to setup EC2 instances, we can also tell you whether Director does the same or not. -Vinithra

Vinithra · ‎01-21-2015

You do need to start the server. We will improve the docs to include this. Start the Cloudera Director Server: sudo service cloudera-director-server start Turn off iptables: sudo service iptables stop

Online	Offline
Last Visited	‎07-30-2019 08:36 PM

Date Registered	‎08-08-2013 02:13 PM
Last Visited	‎07-30-2019 08:36 PM
Total Messages Posted	31
Total Kudos Received	7

Re: Advance Configuration Snippet in Director Conf...

Re: re-run Bootstap command does not take use of u...

Re: Error as "API call to Cloudera Manager failed ...

Re: Support for auto scaling

Re: Does Cloudera Director 2.0 allow creation of C...

Survey on Cloud and Cloudera Director

Re: Advance Configuration Snippet in Director Conf...

Re: Cloudera Director BYON plugin blacklisted

Re: re-run Bootstap command does not take use of u...

Re: re-run Bootstap command does not take use of u...

Re: Support for auto scaling

Re: Support for auto scaling

Re: Java 8 on CDH cluster using Cloudera director ...

Re: Error as "API call to Cloudera Manager failed ...

Re: Error as "API call to Cloudera Manager failed ...

Re: Error as "API call to Cloudera Manager failed ...

Re: Support for auto scaling

Re: Does Cloudera Director 2.0 allow creation of C...

Re: Director Config File Role Assignment

Re: Cloudera Directory support for CDH 5.4

Re: Is there a way to see the aws command history ...

Re: Failed to start Cloudera Director UI on 7189 -...