Support Questions
Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Innovation Accelerator group hub.

Nifi invokehttp processor to make web API calls to kerberized Solr

Explorer

I'm trying to create a standard ingest flow for nifi that will process java application logs in json format and index them to Solr. I want this flow to create the necessary Solr collections if they don't already exist. I can do this with a simple API call to the /admin/collection endpoint; however, I don't know how kerberos authentication will work from the invokehttp processor. Does anyone have any knowledge on the subject to help get me going in the right direction on this?

Another option would be to invoke the "solrctl" command on the OS; I'm running CDH 5.11. The main issue there is that I don't know of a way to execute a script or command while still using the nifi expression language to pass parameters from flow file attributes.

1 ACCEPTED SOLUTION

Ok, how about ExecuteStreamCommand which accepts incoming flow files?

View solution in original post

3 REPLIES 3

As far as I know, none of the HTTP processors in NiFi support Kerberos authentication, so I don't think you'll be able to do the first idea. For the second idea, you should be able to use the ExecuteProcess processor, the command arguments property supports expression language.

Explorer

Thank you for your suggestion. However, this will not work because ExecuteProcessor does not accept an incoming relationship. I have a preceding processor that has an initialize relationship as an output. Upon initialization of my dataflow a flowfile is is sent to this relationship and contains the necessary information to create the Solr Collection.

The more I look at this issue the more appropriate a CreateSolrCollection processor sounds. For now I'm going to continue with my scripted processor to run the solrctl command. This just requires the nifi user to have authority to invoke the necessary api calls. Unless there's something I'm missing, I think that's the best solution for my question.

Ok, how about ExecuteStreamCommand which accepts incoming flow files?