Reply
PNS
Explorer
Posts: 38
Registered: ‎05-20-2014

Caching command classes (e.g., for multithreaded loading)

Hi...

 

Is there any way of somehow caching (or accelerating) the loading of morphline command classes?

 

The standard approach is apparently to search the JAR and instantiate each constructor after the corresponding class is loaded. This takes quite some time, so in a scenario where hundreds of different morphlines (.conf files) need to be compiled, each such morphline requires even over 1 sec to be loaded.

 

Attempting to do the loading in parallel results in multiple threads locked which contending with eache other for the lock on the JAR file iterator. So, using 8 threads offers a marginal improvement of 10% - 20% at the best.

 

If the classes were cached, then the instantiation of each constructor would just need nanoseconds, instead of several hndrud milliseconds (or more).

 

Thanks,

 

PNS

Announcements
The Kite SDK is a collection of docs, sample code, APIs, and tools to make Hadoop application development faster. Learn more at http://kitesdk.org.