Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Morphlines and multithreading

Solved Go to solution
Highlighted

Morphlines and multithreading

Explorer

Hi...

 

Does the single-threaded design of morphilnes mean that there can be concurrency issues by creating multiple morphlines from the same config file and running each of them in a separate thread?

 

For example, if one compiles 10 morphlines from the same config file and then run each of those in a separate thread, can that cause any thread safety issues?

 

Thanks.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Morphlines and multithreading

Expert Contributor
bq. For example, if one compiles 10 morphlines from the same config file and then run each of those in a separate thread, can that cause any thread safety issues?

There?s no thread safety issue when doing that. (And there?s minimal contention as well, which improves throughput). In fact, that?s exactly what the hbase-indexer does in order to take advantage of multiple CPU cores, and that?s also what users of the Flume Morphline Solr Sink sometimes do.

Wolfgang.

2 REPLIES 2

Re: Morphlines and multithreading

Expert Contributor
bq. For example, if one compiles 10 morphlines from the same config file and then run each of those in a separate thread, can that cause any thread safety issues?

There?s no thread safety issue when doing that. (And there?s minimal contention as well, which improves throughput). In fact, that?s exactly what the hbase-indexer does in order to take advantage of multiple CPU cores, and that?s also what users of the Flume Morphline Solr Sink sometimes do.

Wolfgang.

Re: Morphlines and multithreading

Explorer

I assumed so, but wanted to confirm that there is no state sharing "behind the scenes", especially in the Command building process.

 

Thanks for the confirmation.