01-16-2017 02:22 AM
We are currently using Cloudera Director on a supported cloud provider, but I am trying to adapt the build scripts to create clusters using the BYON plugin with pre-created VMs for infrastructure build testing.
I can see the BYON plugin is build and distributed with Cloudera Director, however it seems to be blacklisted:
[centos@**** ~]$ /usr/bin/cloudera-director bootstrap-remote "/etc/cloudera-director-server/deployments/**********.conf" --lp.remote.username=**** --lp.remote.password=***** --lp.remote.hostAndPort=*************:7189 Connecting to http://*************:7189 Current user roles: [ROLE_READONLY, ROLE_ADMIN] Found errors in provider configuration: * Unsupported provider type: byon:
[2017-01-16 10:04:59] INFO [main] - c.c.l.p.c.PluggableProviderConfig: Overriding blacklist=[byon, sandbox] (default ) [2017-01-16 10:05:00] INFO [main] - c.c.l.p.c.PluggableCloudProviderFactory: Looking for providers in JAR file /var/lib/cloudera-director-plugins/byon-provider-example-1.0.0/byon-provider-example-1.0.0.jar [2017-01-16 10:05:00] INFO [main] - c.c.l.p.c.PluggableCloudProviderFactory: Loaded launcher com.cloudera.director.byon.BYONLauncher [2017-01-16 10:05:00] WARN [main] - c.c.l.p.c.PluggableCloudProviderFactory: Cannot read configuration file: /var/lib/cloudera-director-plugins/byon-provider-example-1.0.0/etc/labels.conf [2017-01-16 10:05:00] INFO [main] - c.c.l.p.c.PluggableCloudProviderFactory: Not loading blacklisted provider byon. [2017-01-16 10:05:00] WARN [main] - c.c.l.p.c.PluggableCloudProviderFactory: No providers registered from JAR byon-provider-example-1.0.0.jar [2017-01-16 10:05:00] INFO [main] - c.c.l.p.c.PluggableCloudProviderFactory: No plugin JAR found in plugin directory /var/lib/cloudera-director-plugins/byon-provider-example-1.0.0 [2017-01-16 10:05:04] ERROR [main] - c.c.l.b.v.GenericEnvironmentValidator: Unsupported environment instance provider type byon
The plugin jar is found by Director and is loaded, but is concequently blacklisted by a pre-created list of blacklisted plugins:
Overriding blacklist=[byon, sandbox]
Loaded launcher com.cloudera.director.byon.BYONLauncher
Not loading blacklisted provider byon.
Is there any way to unblacklist this plugin? Is there an overriding config parameter for this? At the moment, the only way I can see this working is if I clone the code and build with a different ID.
01-16-2017 02:53 AM - edited 01-16-2017 02:54 AM
I have the answer, it is blacklisted in both the server and client.
[centos@***** ~]$ cat /etc/cloudera-director-server/application.properties | grep blacklist lp.plugin.configuration.blacklist: sandbox
/usr/bin/cloudera-director bootstrap-remote "/etc/cloudera-director-server/deployments/*******.conf" --lp.remote.username=****** --lp.remote.password=****** --lp.remote.hostAndPort=*********:7189 --lp.plugin.configuration.blacklist=sandbox
The important part being: --lp.plugin.configuration.blacklist as a command-line option.
This now lets me use the BYON plugin out of the box.
01-16-2017 09:01 AM
I'm glad you found the answer. The byon and sandbox providers are blacklisted by default since they are not suitable for production use. Overriding the lp.plugin.configuration.blacklist property as you have done is the correct way to enable them for experimentation. If you implement your own production-quality BYON provider you should give it its own ID.
01-23-2017 05:06 PM
We are evaluating Director with BYON plugin too. For the production usage part, since the Cloudera Director Service Provider Interface is hierarchical structured, can you elaborate which part for BYON plugin is not production ready (e.g, compare to AWS plugin)?
01-24-2017 06:45 AM - edited 01-24-2017 06:45 AM
Drake, the BYON plugin frankly hasn't been in development for some time, and is totally unsupported for any sort of production use. Right now it exists solely for demonstration/illustrative purposes. If it does still work, we can't make any guarantees on how well it will work or if it will continue to work in the future.
01-24-2017 04:40 PM
I understand BYON plugin is for showcase purpose. But say if you (Cloudera) have a client who wants to deploy CDH to their private datacenter, is manual deployment the only option then? Or you have separate solution for them?
01-24-2017 05:30 PM
01-24-2017 05:47 PM
Our on-premise software will depend on some services from CDH. As one of the prerequisites, we want to provide some kind of installer (or script) to easiliy provision a CDH cluster. If there is no good solution for auto deploy CDH cluster, we can only refer to the instructions and ask customers install CDH by themselves as you mentioned.
04-11-2017 07:37 PM
I have follwed the instructions , able to see the BYON plugin. Now let me share my Lab configuration here
- VMWare vsphere Platform installed on HP DL380 boxes.
- In a VM Cloudera Director 2.3.0 is running
i want use BYON plugin to provide the hostname and IP address for the CLouder amanger , Master node, Worker node well in advance.
I have populate hostname of the VM in the frist page of BYON plugin and when i clicked next , i have selected new instance template , popup shows me AWS instance template.
I want do customisation here, since my enviornment is vmware; template field listed here is not valid.
Is ther way to do.
Also here is my suggestion that why i cant pickup pre existing VM already spun-up in the VMWare enviornment. so that it make life easier and only deploy the hadoop package role on the existing VM?.
Kindly help me...
04-13-2017 09:09 AM
I just tried to repro your problem and I see a BYON-specific instance template popup.
One thing to note that may be confusing you is that Director requires an Type and Image field even though these concepts are not needed in BYON. Feel free to put anything you want in those fields.
If you click on the "Advanced Options", you can specify the preferred hosts for that template. Director will attempt to pick a host from that list on a best effort basis.