I was recently tinkering with the walmart rest-api. This is publicly available interface and can be used for a quick price look up for products. The overall goal of the project is to keep track of the cost of specific shopping carts day to day but this intermediate step provides an interesting example-case.
The requirements of this stage:
- Use UPC codes to provide a lookup
- Avoid having the pass the walmart API key to the internal client making the call
- Extract features such as price in preparation for a JDBC database entry.
Core Concepts in Nifi
- Nifi has the ability to serve as a custom restful api which is managed with the "HandleHttpRequest" and "HandleHttpResponce" processors. This can be a Get/Post or any of the other common types
- Nifi can make calls to an external rest API via the "InvokeHTTP" Processor
- XML Data can be extracted with the "EvaluateXPath" Processor
The HandleHttpRequest Processor
This processor receives the incoming rest call and makes a flow file with information pertaining to the headers. As you can see in the image below it is listening on port 9091 and only responding to the path '/lookup'. Additionally the flow file it created has an attribute value for all of the headers it received, particularly "upc_code"
And the flow file
The InvokeHTTP Processor
This processor takes the header and makes a call to the walmart API directly. As you can see i am using the attribute for upc_code received from the request handler. This then sends an XML file in the body of the flow file to the next stage
The EvaluateXPath Processor
In this article I covered how the xpath processor works in more detail. I am extracting key attributes for later analysis.
https://community.hortonworks.com/articles/25720/parsing-xml-logs-with-nifi-part-1-of-3.html
HandleHTTPResponse (Code 200 or Dead Letter)
After successfully extracting the attributes I send a response code of 200(success) back to the rest client along with the xml that walmart provided. In my above example if i do not successfully extract the values the message goes to a dead letter que. This is not ideal and in a production setting I would send the appropriate HTTP error code.
Closing Thoughts
This process group provides a solid basis for my pricing engine. I still need to write in the error handling but this start provides a feature-rich flow file to the next stage of my project.
