Community Articles
Find and share helpful community-sourced technical articles
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.
Labels (1)
Contributor

I was recently tinkering with the walmart rest-api. This is publicly available interface and can be used for a quick price look up for products. The overall goal of the project is to keep track of the cost of specific shopping carts day to day but this intermediate step provides an interesting example-case.

The requirements of this stage:

  • Use UPC codes to provide a lookup
  • Avoid having the pass the walmart API key to the internal client making the call
  • Extract features such as price in preparation for a JDBC database entry.

Core Concepts in Nifi

  • Nifi has the ability to serve as a custom restful api which is managed with the "HandleHttpRequest" and "HandleHttpResponce" processors. This can be a Get/Post or any of the other common types
  • Nifi can make calls to an external rest API via the "InvokeHTTP" Processor
  • XML Data can be extracted with the "EvaluateXPath" Processor

7360-walmart-rest-api.png

The HandleHttpRequest Processor

This processor receives the incoming rest call and makes a flow file with information pertaining to the headers. As you can see in the image below it is listening on port 9091 and only responding to the path '/lookup'. Additionally the flow file it created has an attribute value for all of the headers it received, particularly "upc_code"

7381-request-processor.png

And the flow file

7383-stage-1-flowfile.png

The InvokeHTTP Processor

This processor takes the header and makes a call to the walmart API directly. As you can see i am using the attribute for upc_code received from the request handler. This then sends an XML file in the body of the flow file to the next stage

7384-invokehttp.png

The EvaluateXPath Processor

In this article I covered how the xpath processor works in more detail. I am extracting key attributes for later analysis.

https://community.hortonworks.com/articles/25720/parsing-xml-logs-with-nifi-part-1-of-3.html

7385-nifi-xml.png

HandleHTTPResponse (Code 200 or Dead Letter)

After successfully extracting the attributes I send a response code of 200(success) back to the rest client along with the xml that walmart provided. In my above example if i do not successfully extract the values the message goes to a dead letter que. This is not ideal and in a production setting I would send the appropriate HTTP error code.

Closing Thoughts

This process group provides a solid basis for my pricing engine. I still need to write in the error handling but this start provides a feature-rich flow file to the next stage of my project.


stage-1-flowfile.png
10,805 Views
Comments
New Contributor

Thanks Chris, it is also my use case for NiFi. Can you provide a download link for the template?

Thanks.

Don't have an account?
Coming from Hortonworks? Activate your account here
Version history
Revision #:
2 of 2
Last update:
‎08-17-2019 10:21 AM
Updated by:
 
Contributors
Top Kudoed Authors