Created 10-07-2015 11:42 PM
Hi All,
I was going through the namenode and datanode configuration and found that there is a way to add plugin to both of them.
I did some searching but could not find any relevant information. Does anyone know, what is a namenode plugin and datanode plugin?
Thanks
Sebin
Created on 10-08-2015 06:47 AM - edited 10-08-2015 07:13 AM
I'm not aware of any examples of such plugins, but found some details looking through the code.
Plugins should implement the org.apache.hadoop.util.ServicePlugin Java interface, and you can find the code for that here: https://github.com/cloudera/hadoop-common/blob/cdh5-2.6.0_5.4.7/hadoop-common-project/hadoop-common/... It says "Service plug-ins may be used to expose functionality of datanodes or namenodes using arbitrary RPC protocols". Basically the service will read in the class name(s), instantiate them, and then call the start() method and pass it a reference to the service. You can then do whatever you want, and it will later call stop() on the plugin when things are shutting down.
The are other ways to write plugins for Hadoop. Sentry's HDFS support is implemented as a plugin, but it's a more specific type of authorization plugin, rather than a class that's just started and stopped along with the service.
Hope that helps!
Created 10-07-2015 11:44 PM
Created 10-08-2015 02:37 AM
Created on 10-08-2015 06:47 AM - edited 10-08-2015 07:13 AM
I'm not aware of any examples of such plugins, but found some details looking through the code.
Plugins should implement the org.apache.hadoop.util.ServicePlugin Java interface, and you can find the code for that here: https://github.com/cloudera/hadoop-common/blob/cdh5-2.6.0_5.4.7/hadoop-common-project/hadoop-common/... It says "Service plug-ins may be used to expose functionality of datanodes or namenodes using arbitrary RPC protocols". Basically the service will read in the class name(s), instantiate them, and then call the start() method and pass it a reference to the service. You can then do whatever you want, and it will later call stop() on the plugin when things are shutting down.
The are other ways to write plugins for Hadoop. Sentry's HDFS support is implemented as a plugin, but it's a more specific type of authorization plugin, rather than a class that's just started and stopped along with the service.
Hope that helps!
Created 10-09-2015 11:44 PM
Created 10-14-2015 11:55 AM
I had a quick search around on Github for anything that implements this, and couldn't find anything I'm afraid. Pretty sure Cloudera's product line doesn't include any such plugins...