Cloudera Director BYON plugin blacklisted

[#1]

Hi all,

We are currently using Cloudera Director on a supported cloud provider, but I am trying to adapt the build scripts to create clusters using the BYON plugin with pre-created VMs for infrastructure build testing.

I can see the BYON plugin is build and distributed with Cloudera Director, however it seems to be blacklisted:

[centos@**** ~]$ /usr/bin/cloudera-director bootstrap-remote "/etc/cloudera-director-server/deployments/**********.conf" --lp.remote.username=**** --lp.remote.password=***** --lp.remote.hostAndPort=*************:7189
Connecting to http://*************:7189
Current user roles: [ROLE_READONLY, ROLE_ADMIN]
Found errors in provider configuration:
* Unsupported provider type: byon:

[2017-01-16 10:04:59] INFO  [main] - c.c.l.p.c.PluggableProviderConfig: Overriding blacklist=[byon, sandbox] (default [])
[2017-01-16 10:05:00] INFO  [main] - c.c.l.p.c.PluggableCloudProviderFactory: Looking for providers in JAR file /var/lib/cloudera-director-plugins/byon-provider-example-1.0.0/byon-provider-example-1.0.0.jar
[2017-01-16 10:05:00] INFO  [main] - c.c.l.p.c.PluggableCloudProviderFactory: Loaded launcher com.cloudera.director.byon.BYONLauncher
[2017-01-16 10:05:00] WARN  [main] - c.c.l.p.c.PluggableCloudProviderFactory: Cannot read configuration file: /var/lib/cloudera-director-plugins/byon-provider-example-1.0.0/etc/labels.conf
[2017-01-16 10:05:00] INFO  [main] - c.c.l.p.c.PluggableCloudProviderFactory: Not loading blacklisted provider byon.
[2017-01-16 10:05:00] WARN  [main] - c.c.l.p.c.PluggableCloudProviderFactory: No providers registered from JAR byon-provider-example-1.0.0.jar
[2017-01-16 10:05:00] INFO  [main] - c.c.l.p.c.PluggableCloudProviderFactory: No plugin JAR found in plugin directory /var/lib/cloudera-director-plugins/byon-provider-example-1.0.0
[2017-01-16 10:05:04] ERROR [main] - c.c.l.b.v.GenericEnvironmentValidator: Unsupported environment instance provider type byon

The plugin jar is found by Director and is loaded, but is concequently blacklisted by a pre-created list of blacklisted plugins:

Overriding blacklist=[byon, sandbox]

Loaded launcher com.cloudera.director.byon.BYONLauncher

Not loading blacklisted provider byon.

Is there any way to unblacklist this plugin? Is there an overriding config parameter for this? At the moment, the only way I can see this working is if I clone the code and build with a different ID.

Many thanks

[#2 · Accepted solution]

I have the answer, it is blacklisted in both the server and client.

Server:

[centos@***** ~]$ cat /etc/cloudera-director-server/application.properties | grep blacklist
lp.plugin.configuration.blacklist: sandbox

Client:

/usr/bin/cloudera-director bootstrap-remote "/etc/cloudera-director-server/deployments/*******.conf" --lp.remote.username=****** --lp.remote.password=****** --lp.remote.hostAndPort=*********:7189 --lp.plugin.configuration.blacklist=sandbox

The important part being: --lp.plugin.configuration.blacklist as a command-line option.

This now lets me use the BYON plugin out of the box.

Thanks

[#3]

I'm glad you found the answer. The byon and sandbox providers are blacklisted by default since they are not suitable for production use. Overriding the lp.plugin.configuration.blacklist property as you have done is the correct way to enable them for experimentation. If you implement your own production-quality BYON provider you should give it its own ID.

[#4]

Hi Jadair:

We are evaluating Director with BYON plugin too. For the production usage part, since the Cloudera Director Service Provider Interface is hierarchical structured, can you elaborate which part for BYON plugin is not production ready (e.g, compare to AWS plugin)? 

[#5]

Drake, the BYON plugin frankly hasn't been in development for some time, and is totally unsupported for any sort of production use. Right now it exists solely for demonstration/illustrative purposes. If it does still work, we can't make any guarantees on how well it will work or if it will continue to work in the future.

[#6]

Hi Mike,

I understand BYON plugin is for showcase purpose. But say if you (Cloudera) have a client who wants to deploy CDH to their private datacenter, is manual deployment the only option then? Or you have separate solution for them?

[#7]

Hi,

Currently Cloudera customers who have physical hardware / datacenters that
they manage do setup clusters using Cloudera Manager, and do not interface
with Director (via the BYON plugin) for this purpose. This path works for
them as they usually have to setup the cluster once and manage it
thereafter - there isn't a need for bringing up/down clusters or
growing/shrinking the clusters. Private clouds are a different category:
Cloudera has a Director - VMware vSphere plugin integration that is in
alpha preview mode.

What's the usecase that you were hoping to solve with the BYON plugin?

[#8]

Hi Vinithra,

Our on-premise software will depend on some services from CDH. As one of the prerequisites, we want to provide some kind of installer (or script) to easiliy provision a CDH cluster. If there is no good solution for auto deploy CDH cluster, we can only refer to the instructions and ask customers install CDH by themselves as you mentioned.

[#9]

Hello All,

  I have follwed the instructions , able to see the BYON plugin. Now let me share my Lab configuration here

- VMWare vsphere Platform installed on HP DL380 boxes.

- In a VM Cloudera Director 2.3.0 is running

i want use BYON plugin to provide the hostname and IP address for the CLouder amanger , Master node, Worker node well in advance.

I have populate  hostname  of the VM in the frist page of BYON plugin  and when i clicked next , i have selected new instance template ,  popup  shows me AWS instance template.

I want do customisation here, since my  enviornment is vmware;  template field listed here is not valid.

Is ther way to do.

Also here is my suggestion that why i cant pickup pre existing VM already spun-up  in the VMWare enviornment. so that it make life easier and  only deploy the  hadoop package role on the existing VM?.

Kindly help me...

Kind Regards,

Yoga

[#10]

Yoga,

I just tried to repro your problem and I see a BYON-specific instance template popup.

One thing to note that may be confusing you is that Director requires an Type and Image field even though these concepts are not needed in BYON. Feel free to put anything you want in those fields.

If you click on the "Advanced Options", you can specify the preferred hosts for that template. Director will attempt to pick a host from that list on a best effort basis.

David