question Re: Cloudera Director BYON plugin blacklisted in Support Questions

Cloudera Director BYON plugin blacklisted

bushnoh — Fri, 16 Sep 2022 15:19:28 GMT

Hi all,

We are currently using Cloudera Director on a supported cloud provider, but I am trying to adapt the build scripts to create clusters using the BYON plugin with pre-created VMs for infrastructure build testing.

I can see the BYON plugin is build and distributed with Cloudera Director, however it seems to be blacklisted:

[centos@**** ~]$ /usr/bin/cloudera-director bootstrap-remote "/etc/cloudera-director-server/deployments/**********.conf" --lp.remote.username=**** --lp.remote.password=***** --lp.remote.hostAndPort=*************:7189
Connecting to http://*************:7189
Current user roles: [ROLE_READONLY, ROLE_ADMIN]
Found errors in provider configuration:
* Unsupported provider type: byon:

[2017-01-16 10:04:59] INFO  [main] - c.c.l.p.c.PluggableProviderConfig: Overriding blacklist=[byon, sandbox] (default [])
[2017-01-16 10:05:00] INFO  [main] - c.c.l.p.c.PluggableCloudProviderFactory: Looking for providers in JAR file /var/lib/cloudera-director-plugins/byon-provider-example-1.0.0/byon-provider-example-1.0.0.jar
[2017-01-16 10:05:00] INFO  [main] - c.c.l.p.c.PluggableCloudProviderFactory: Loaded launcher com.cloudera.director.byon.BYONLauncher
[2017-01-16 10:05:00] WARN  [main] - c.c.l.p.c.PluggableCloudProviderFactory: Cannot read configuration file: /var/lib/cloudera-director-plugins/byon-provider-example-1.0.0/etc/labels.conf
[2017-01-16 10:05:00] INFO  [main] - c.c.l.p.c.PluggableCloudProviderFactory: Not loading blacklisted provider byon.
[2017-01-16 10:05:00] WARN  [main] - c.c.l.p.c.PluggableCloudProviderFactory: No providers registered from JAR byon-provider-example-1.0.0.jar
[2017-01-16 10:05:00] INFO  [main] - c.c.l.p.c.PluggableCloudProviderFactory: No plugin JAR found in plugin directory /var/lib/cloudera-director-plugins/byon-provider-example-1.0.0
[2017-01-16 10:05:04] ERROR [main] - c.c.l.b.v.GenericEnvironmentValidator: Unsupported environment instance provider type byon

The plugin jar is found by Director and is loaded, but is concequently blacklisted by a pre-created list of blacklisted plugins:

Overriding blacklist=[byon, sandbox]

Loaded launcher com.cloudera.director.byon.BYONLauncher

Not loading blacklisted provider byon.

Is there any way to unblacklist this plugin? Is there an overriding config parameter for this? At the moment, the only way I can see this working is if I clone the code and build with a different ID.

Many thanks

Re: Cloudera Director BYON plugin blacklisted

bushnoh — Mon, 16 Jan 2017 10:54:18 GMT

I have the answer, it is blacklisted in both the server and client.

Server:

[centos@***** ~]$ cat /etc/cloudera-director-server/application.properties | grep blacklist
lp.plugin.configuration.blacklist: sandbox

Client:

/usr/bin/cloudera-director bootstrap-remote "/etc/cloudera-director-server/deployments/*******.conf" --lp.remote.username=****** --lp.remote.password=****** --lp.remote.hostAndPort=*********:7189 --lp.plugin.configuration.blacklist=sandbox

The important part being: --lp.plugin.configuration.blacklist as a command-line option.

This now lets me use the BYON plugin out of the box.

Thanks

Re: Cloudera Director BYON plugin blacklisted

jadair — Mon, 16 Jan 2017 17:01:21 GMT

I'm glad you found the answer. The byon and sandbox providers are blacklisted by default since they are not suitable for production use. Overriding the lp.plugin.configuration.blacklist property as you have done is the correct way to enable them for experimentation. If you implement your own production-quality BYON provider you should give it its own ID.

Re: Cloudera Director BYON plugin blacklisted

drake4 — Tue, 24 Jan 2017 01:06:46 GMT

Hi Jadair:

We are evaluating Director with BYON plugin too. For the production usage part, since the Cloudera Director Service Provider Interface is hierarchical structured, can you elaborate which part for BYON plugin is not production ready (e.g, compare to AWS plugin)?

Re: Cloudera Director BYON plugin blacklisted

Mike Wilson — Tue, 24 Jan 2017 14:45:27 GMT

Drake, the BYON plugin frankly hasn't been in development for some time, and is totally unsupported for any sort of production use. Right now it exists solely for demonstration/illustrative purposes. If it does still work, we can't make any guarantees on how well it will work or if it will continue to work in the future.

Re: Cloudera Director BYON plugin blacklisted

drake4 — Wed, 25 Jan 2017 00:40:30 GMT

Hi Mike,

I understand BYON plugin is for showcase purpose. But say if you (Cloudera) have a client who wants to deploy CDH to their private datacenter, is manual deployment the only option then? Or you have separate solution for them?

Re: Cloudera Director BYON plugin blacklisted

Vinithra — Wed, 25 Jan 2017 01:30:13 GMT

Hi,

Currently Cloudera customers who have physical hardware / datacenters that
they manage do setup clusters using Cloudera Manager, and do not interface
with Director (via the BYON plugin) for this purpose. This path works for
them as they usually have to setup the cluster once and manage it
thereafter - there isn't a need for bringing up/down clusters or
growing/shrinking the clusters. Private clouds are a different category:
Cloudera has a Director - VMware vSphere plugin integration that is in
alpha preview mode.

What's the usecase that you were hoping to solve with the BYON plugin?

Re: Cloudera Director BYON plugin blacklisted

drake4 — Wed, 25 Jan 2017 01:47:48 GMT

Hi Vinithra,

Our on-premise software will depend on some services from CDH. As one of the prerequisites, we want to provide some kind of installer (or script) to easiliy provision a CDH cluster. If there is no good solution for auto deploy CDH cluster, we can only refer to the instructions and ask customers install CDH by themselves as you mentioned.

Re: Cloudera Director BYON plugin blacklisted

YogaT — Wed, 12 Apr 2017 02:37:03 GMT

Hello All,

I have follwed the instructions , able to see the BYON plugin. Now let me share my Lab configuration here

- VMWare vsphere Platform installed on HP DL380 boxes.

- In a VM Cloudera Director 2.3.0 is running

i want use BYON plugin to provide the hostname and IP address for the CLouder amanger , Master node, Worker node well in advance.

I have populate hostname of the VM in the frist page of BYON plugin and when i clicked next , i have selected new instance template , popup shows me AWS instance template.

I want do customisation here, since my enviornment is vmware; template field listed here is not valid.

Is ther way to do.

Also here is my suggestion that why i cant pickup pre existing VM already spun-up in the VMWare enviornment. so that it make life easier and only deploy the hadoop package role on the existing VM?.

Kindly help me...

Kind Regards,

Yoga

Re: Cloudera Director BYON plugin blacklisted

dhan — Thu, 13 Apr 2017 16:09:31 GMT

Yoga,

I just tried to repro your problem and I see a BYON-specific instance template popup.

One thing to note that may be confusing you is that Director requires an Type and Image field even though these concepts are not needed in BYON. Feel free to put anything you want in those fields.

If you click on the "Advanced Options", you can specify the preferred hosts for that template. Director will attempt to pick a host from that list on a best effort basis.

David