Is it fine to use SolR + Banana UI for Big Data Reporting?

[#1]

Can i use SolR + Banana Ui as Big Data reporting? Im looking for realtime update reporting for our project and i saw this Solr. Now im trying it and its fine. But the problem is, when im running my nifi job going to Solr, it will fucntion well but after 20 secs the solr and banana started to crashed and and the service stops automatically. I saw that the JVM Memory is full.

By the way im using sandbox.

Any tips how can i fixed this?

Thank you!

Regie

[#2 · Accepted solution]

@regie canada

You can certainly use Solr and Banana to do Big Data reporting. Some things you should to keep in mind:

1. You need to ensure you are giving enough resources to Solr in terms of memory and CPU for indexing and querying your data. In test environments this isn't a huge issue, but it will certainly be something you take into consideration for production environments.

2. You need to be careful with the dashboards and queries you use with Banana. It's running in a web browser and all of the data that you are manipulating within Banana is loaded into memory. It's relatively easy to create very taxing queries and dashboards that consume alot of memory and put a strain on Solr. Additionally, this can also cause alot of memory usage within the end-user web browser making it unresponsive.

The above two points leads me to answering your problem. How much memory have you allocated to the Sandbox? The minimum memory requirement is 8GB, however 10-12GB will work much better. Are most of the components of HDP turned on in the Sandbox? All of these things take up memory and can cause the Solr JVM to run out of memory.

My recommendation would be:

1. Stop any unused components in the HDP stack. This will free up some system memory.

2. Allocate more memory to the Sandbox. As I said, 10GB works much better than 8. I prefer to give it 12GB.