How to fix Ambari-Infra (Solr) throwing 'OutOfMemoryError: unable to create new native thread'

Symptom:

After adding additional configurations to Solr, the service fails with both 'no servers hosting shard' and 'OutOfMemoryError' errors. A restart will resolve the issue for some time and it will eventually hit the error again.

Root Cause:

In some versions, the default setting for max user processes for the infra-solr user is 1024, which may be set too low for certain use cases.

Resolution:

Increase the max user process user limit.

This can be done by creating the file /etc/security/limits.d/infra-solr.conf on the solr node and setting the nproc property.

For example if the new value chosen is 6000, the contents would be:

infra-solr - nproc 6000