Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Storing dynamic data in Apache NiFi

Highlighted

Storing dynamic data in Apache NiFi

New Contributor

Hi all,

 

I am a really beginer in Apache NiFi and I am trying to find the best way to do the following process.

 

I have an API that exposes an endpoint that provides me a token for next requests (available for few hours). I would like to store this token for general purpose, because I need other flows accessing to this token.

 

I would preffer not using Files or external Databases, so  I was wondering about using variables... but honestly... no idea if this is the best way.

 

So far, I know how to connect with the API, (InvokeHTTP), and getting the token (EvaluateJsonPath), but I don't know how to store and where (Variable?) and how to use it in another flow.

 

Current flow so far:

Store Token: GenerateFlowFile -> InvokeHTTP ->EvaluateJsonPath -> ??

New request using token: GenerateFlowFile -> InvokeHTTP (Here I need to access the token)

 

Can anyone help me?

 

Thank u so much!!

 

Javier

1 REPLY 1

Re: Storing dynamic data in Apache NiFi

Cloudera Employee

Hello  @jdelarosa91,

 

First off welcome to Apache NIFI and its ecosystem.

 

You can accomplish this by using PutDistributedMapCache and FetchDistributedMapCache processors.

First, you must configure theDistributedMapCacheServer and DistributedMapCacheClientService controller services.

Then your dataflow should be like so:

 

Get token:

InvokeHttp > UpdateAttribute ( Assign an attribute for instance MyToken and give it value TOKEN ) > EvaluateJsonPath ( get the value you want ) > PutDistributeMapCache ( For cache entry identifier give it the attribute ${MyToken}  )

 

Now in the MapCache we have a key, value like so:

TOKEN / <the string you got from EvaluateJsonPath>

 

And when you need to grab the token:

 

GenerateFlowFile ( Configure an attribute MyToken give it value of TOKEN ) > FetchDistributeMapCache ( cache identifier ${MyToken} and "out identifier in attribute" give it a name to call your token with) > InvokeHttp ( Here when you need to use the token you can call it using nifi attribute ${the attribute with the token retrieved from you MapCache )

 

 

Don't have an account?
Coming from Hortonworks? Activate your account here