Member since
08-08-2013
31
Posts
7
Kudos Received
7
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1581 | 03-20-2017 04:52 PM | |
1952 | 10-04-2016 01:51 PM | |
2594 | 07-12-2016 06:41 PM | |
5292 | 06-16-2016 05:35 PM | |
1460 | 01-30-2016 04:10 PM |
08-01-2017
06:15 PM
Do you use cloud infrastructure for your CDH clusters? You are invited to share your feedback on your usage and management of cloud infrastructure. You will thereby influence the roadmap of Cloudera Director and other cloud orchestration software from Cloudera. Please share your experiences by taking this short survey : http://tiny.cloudera.com/director-survey Key Cloudera Director features recently added include: Automatic repair of clusters Support for long-running clusters: Cloudera Director now syncs with cluster changes made via Cloudera Manager AWS Spot instance support and improved resiliency to spot instance disappearances Support for AWS EBS volumes Support for Azure ADLS, Azure premium and standard storage For more details on which versions support different features, please refer to the release notes. If you have any questions or comments, you can reach me at vinithra@cloudera.com. Thanks for your input, Vinithra Varadharajan Engineering Manager, Cloud
... View more
03-20-2017
04:52 PM
Hi, If that property does not exist in the CM properties already, then you should find the "safety valve" configuration for the file that it should live in. Example: Safety valve of core-site.xml. You can find the configuration key in the same link, and then use that in the Director config file. Hope this helps.
... View more
01-24-2017
05:30 PM
Hi, Currently Cloudera customers who have physical hardware / datacenters that they manage do setup clusters using Cloudera Manager, and do not interface with Director (via the BYON plugin) for this purpose. This path works for them as they usually have to setup the cluster once and manage it thereafter - there isn't a need for bringing up/down clusters or growing/shrinking the clusters. Private clouds are a different category: Cloudera has a Director - VMware vSphere plugin integration that is in alpha preview mode. What's the usecase that you were hoping to solve with the BYON plugin?
... View more
10-04-2016
01:58 PM
1 Kudo
Does this list of configuration properties help: http://www.cloudera.com/documentation/enterprise/properties/5-7-x/topics/cm_props.html
... View more
10-04-2016
01:51 PM
1 Kudo
Hi, Your observation is correct. The reason we give the choice of resume or rebuild when submitting the bootstrap command is because you might be trying to bootstrap a second cluster with the same config file. Modifications to an existing cluster should be done using the update command, which requires the config file as input and you can make changes to the config file. However, do note that not all modifications to the config file are supported by update. Specifically, config changes are not supported with update. However, changes to instance count are supported, which allows you to grow and shrink the cluster. Hope this helps.
... View more
09-16-2016
05:47 PM
Hi, Without Cloudera Director, this is going to be hard. You can try using the CM API to add hosts to the service and grow the service by figuring out what CM does for Solr. http://www.cloudera.com/documentation/enterprise/latest/topics/cm_intro_api.html This will be much easier if you are using Director, as it does the work for you of growing the service by adding the hosts, registering them with CM, etc.
... View more
09-15-2016
05:14 PM
Hi, I'm assuming you used Cloudera Director to set this cluster up. To grow the cluster based on load, you first need to identify how you can programmatically assess that load has/is going to increased. Examples are: day of month triggered by a cron job, or the output of some monitoring that tells you that the volume of data that needs to be ingested has crossed a threshold. With that indicator that load is increasing as input, write a script that makes a PUT request to the update endpoint on clusters, with a body that describes the cluster with the increased count of instances: PUT /api/v5/environments/{environment}/deployments/{deployment}/clusters/{cluster} You can try out this API by going to the API console in your Director UI. You can find a link to the API console in the upper right ?. Is this something that you can start with? Were you expecting a different way of achieving this? -Vinithra
... View more
09-02-2016
06:17 PM
Hi, we currently don't have a scheduled date for this, but we will take your request into consideration during prioritization.
... View more
08-12-2016
11:08 AM
The suggestion here is not to use S3 to populate HDFS - as you have gathered, that would involve moving data back and forth. It isn't necessary for your use-case, though some folks do indeed do just that if their raw data is in S3 and they need to use HDFS for latency requirements. However, the suggestion here is to have your processing jobs (MR, Impala, etc.) run directly off of S3. See this blogpost as an example: http://blog.cloudera.com/blog/2016/08/analytics-and-bi-on-amazon-s3-with-apache-impala-incubating/ The "off the beaten track" was referring to using the bootstrap script to mount EBS volumes. This is extending Director's existing functionality, which should come out of the box soon. But until then, I'm not aware of anyone else who has written such a bootstrap script. Hence the warning. Hope this helps.
... View more
08-11-2016
10:40 AM
Hi David, My recommendation would be to write a bootstrap script that mounts the EBS volumes. Include this bootstrap script in each instance group, so that the disks are available before the CDH services are initiatlized. If the volumes are mounted after the services are setup, you would have to reconfigure and reinitialize a bunch of CDH services. Do note that you are going off the well-trodden path - and that leveraging S3 for now will probably save you some grief.
... View more
08-10-2016
05:58 PM
Hi David, As of Director 2.1, Director cannot attach EBS volumes to instances and hence cannot setup clusters that use EBS for persistent storage. EBS support is on Director's roadmap, but until then, have you considered storing your data in S3 and having your cluster read/write directly from S3?
... View more
08-10-2016
03:32 PM
Hi, It seems like you are trying to connect to the instance via its public IP address. Have you enabled the "Associate Public IP Address" flag while setting up the environment? This will create the instances with public IP addresses. If you have already enabled that flag, then please check the firewall rules of the security group that you used. Do note that you can connect to the CM instance using the private IP address from the Director instance itself - but if your goal is to be able to access the CM instance from anywhere, then you'd have to associate public IP addresses first.
... View more
07-13-2016
10:51 AM
Kudu will be supported in Cloudera Director when Kudu goes GA. Can you give us more details on how you are using Director and your experience so far?
... View more
07-12-2016
06:41 PM
Amit, I wanted to clarify something based on feedback that I got. If you use Directo to setup a cluster without Kudu, and then add Kudu via CM, then Director will not be able to grow or shrink the cluster thereafter as the topology has changed outside of Director. So, whether to use Director or not depends on your use-case - if you are looking at it as the means of a one-time setup that isn't going to require modifications via Director thereafter, then Director could meet your needs. However, if you want to continue using Director, then adding Kudu out-of-band will not work with Director 2.1.
... View more
07-11-2016
06:26 PM
Hi Amit, Director 2.1 currently does not support Kudu-beta. However, you should be able to use Director to set up all the services other than Kudu, and then set up Kudu using CM. If you run into any issues, please post your questions here: http://community.cloudera.com/t5/Beta-Releases-Apache-Kudu/bd-p/Beta
... View more
06-16-2016
05:35 PM
Hi, Regarding elasticity, you can grow and shrink clusters using Director. Auto-scaling is not an out-of-the-box feature; however, there are users who have used Director's API to automate scaling of the cluster based on some parameters. In any case, we have this on the roadmap and we will take your input into prioritization. If you can provide more details on your usecase, that would be welcome too. Thanks, Vinithra
... View more
05-19-2016
11:49 PM
1 Kudo
Hi Karthik, I'm the Engineering Manager for Cloudera Director, and I'm sorry to see that you've been having difficulty in getting Director to work for your usecase. Regarding your specific concerns, we have customers and community users who have been successful in getting Cloudera Director to deploy clusters in AWS. Some ramp up by following Cloudera Director documentation , and others have used the AWS Quick Start for Cloudera Enterprise to start off their experience with Cloudera Director. In addition, if you would like to work with Cloudera Support, I can put you in touch with our sales department. Hope these resources are helpful to you. Sincerely, Vinithra
... View more
05-17-2016
10:46 AM
Hi, I'm assuming you mean you used the CRaSH shell command. Do you have an idea of what got you into the error condition - it was likely because Director was used in an unexpected way. We'd like to help you identify what you are trying to do and find alternate ways to achieve the same, rather than repeatedly run into errors and use the CRaSH shell command.
... View more
01-30-2016
04:10 PM
1 Kudo
Damion, Thanks for your question. To clarify, CDH is certified on Azure and clusters deployed in Azure are supported. However, Cloudera Director can currently support creation of Cloudera Manager and CDH clusters only in AWS and GCP, as you've observed. Support for Azure in Cloudera Director itself is on the roadmap. Please stay tuned and let us know how we can help in the meantime.
... View more
12-08-2015
12:58 PM
Hi, Yes, it is possible to use custom parcels with Cloudera Director. There can be two things that are preventing proper use of them: 1) The manifest file should have appropriate information about the product name and version, as Director refers to this file in order to do its validation. Please refer a sample manifest file in a repo hosted by Cloudera. 2) You might be running into DNS issues. To diagnose further, can you please provide the part of the log that contains the entire exception, including the stacktrace.
... View more
12-08-2015
12:48 PM
Got it. This thread will be moved to the Cloudera Manager community where you can get further help on this topic.
... View more
12-08-2015
12:18 AM
Hi, Can you please provide more details regarding your setup. Have you used Cloudera Director? It seems like you are referring to an existing thread where you discussed setting up a cluster - did you do this using Director or using Cloudera Manager alone?
... View more
10-14-2015
10:25 AM
Roland, Cloudera Director uses the same spec as CM API. You can find the roletypes that CM (and thereby Director) accepts here: http://cloudera.github.io/cm_api/apidocs/v10/path__clusters_-clusterName-_services_-serviceName-_roles.html Do keep us posted on how things go and if you run into any further issues. -Vinithra
... View more
10-09-2015
02:58 PM
Hi, Your post has been moved to the Cloudera Manager forum. Cloudera Manager, as you've observed, will consume EC2 instances and set up CDH clusters on them. In case you're not already aware of it, Cloudera Director will provision EC2 instances for you and then use Cloudera Manager to deploy CDH clusters on those instances. Cloudera Director provides lifecycle management of cloud instances, including operations such as growing and shrinking of clusters. You can download for free and learn more about Cloudera Director from here: http://www.cloudera.com/content/cloudera/en/products-and-services/director.html.
... View more
07-13-2015
10:40 PM
Hi, Thanks for your feedback! Spot instance support is on the Cloudera Director roadmap. Stay tuned for a future release with this feature and please keep your feedback coming in the meantime. If you have other experiences or requests that you'd like to share, we'd be eager to hear from you.
... View more
07-10-2015
11:52 AM
1 Kudo
Kirk, Yes, you can modify the Cloudera Manager and CDH repositories to specify any other version. Make sure that you're using the same minor version of both Cloudera Manager and CDH, i.e. both should be a 5.4.x version. If you're using the config file, refer to aws.reference.conf for examples. There you'll see this in the 'cloudera-manager' block, which you can modify to install CM5.4 instead: # repository: "http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.4/" # repositoryKeyUrl: "http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/RPM-GPG-KEY-cloudera" In the 'cluster' block, you will see this to modify to install CDH5.4 parcels instead: # parcelRepositories: ["http://archive.cloudera.com/cdh5/parcels/5.4/"] -Vinithra
... View more
07-10-2015
11:30 AM
Hi, You can check the Cloudera Director logs to see what AWS actions are taken. If you're using the Cloudera Director client, the logs can be found at $HOME/.cloudera-director/logs/application.log. If using the Cloudera Director server, the logs can be found at /var/log/cloudera-director-server/application.log. If you let us know what you typically do to setup EC2 instances, we can also tell you whether Director does the same or not. -Vinithra
... View more
01-21-2015
10:25 AM
2 Kudos
You do need to start the server. We will improve the docs to include this. Start the Cloudera Director Server: sudo service cloudera-director-server start Turn off iptables: sudo service iptables stop
... View more