Member since
09-20-2018
12
Posts
0
Kudos Received
0
Solutions
01-07-2019
02:04 PM
Okay. Thanks a lot @Shu much appreciated.
... View more
01-07-2019
11:34 AM
@Shu
Thank you very much. Ill check with Hortonworks on how to get these processors into the Nifi version that I am using. Meanwhile, I wanted to know is there any other tool that I can use other than Spark or NiFi ? Thanks In Advance
... View more
01-06-2019
03:21 PM
Thank you for your reply. But I don't see lookuprecord processor present in nifi under add processor menu. The NiFi version that I am using is 1.2.0.3.0.1.1-5. Is there anything that I am missing ?
... View more
01-06-2019
12:25 PM
Hi Guys, I need a suggestion/advice on a use case that i have. I have a bunch of data being streamed from NiFi on 3 different mongo db collections let's say A,B,C. I need to perform a lookup based on a particular field say emp ID which is present in A. If the value of Emp ID present in A is matching with that of the payloads present in collection B & C. Only then i need to attach the document present in B & C to document present in A(i am creating new key/value pair, key being a random UID and value being the payload from B & C in A for performing update), If the value doesn't exists then I need to insert the document as it is in A from B or C. Currently i am using Spark to do this. I believe its an overkill for this job and doesn't make sense to use spark. Any other big data tech suggestions ? P.S I need to perform update on real time basis. -Thanks in Advance
... View more
Labels:
- Labels:
-
Apache NiFi
-
Apache Spark
11-09-2018
02:59 PM
Hey Guys, I have a detect duplicate processor in my nifi flow. Currently i have left the age of duration blank. So is there a default age of duration value if no value is entered ?. @Shu @Matt Burgess Thanks Krishna
... View more
Labels:
- Labels:
-
Apache NiFi
09-24-2018
03:48 PM
Hi All, I am new to NiFi. I am hoping you guys will be able to help me out. I have the below mentioned python script to extract fields from a JSON payload. But the problem i am facing is suppose lets say there's no key value pair present in the payload, so how do i set it to null if the value is not present in the JSON. I tried with using ternary operator but the script failed. The below code is just a sample of what I am trying to achieve. In actuality I need to extract 100 fields from payload so using EvaluateJson processor is not feasible. Script: import json import java.io
from org.apache.commons.io import IOUtils
from java.nio.charset import StandardCharsets
from org.apache.nifi.processor.io import StreamCallback
class PyStreamCallback(StreamCallback):
def __init__(self):
pass
def process(self, inputStream, outputStream):
text = IOUtils.toString(inputStream, StandardCharsets.UTF_8)
obj = json.loads(text)
newObj = {
"Name": obj['name'],
"Age": obj['age'],
"Gender" : obj['gender'],
"test": obj['test'] if obj['test'] else "NULL" //trying to extract test value from Json payload if not present then set it to NULL }
outputStream.write(bytearray(json.dumps(newObj).encode('utf-8')))
flowFile = session.get()
if (flowFile != None):
flowFile = session.write(flowFile,PyStreamCallback())
flowFile = session.putAttribute(flowFile, "filename", flowFile.getAttribute('filename').split('.')[0]+'_translated.json')
session.transfer(flowFile, REL_SUCCESS) -Thanks In Advance
... View more
Labels:
- Labels:
-
Apache NiFi
09-21-2018
07:07 PM
The 100 fields are to be used by another down stream application. I actually referred to your blog on writing groovy script to extract the fields. But facing issues with that as well.
... View more
09-21-2018
03:54 PM
@ShuThank you very much for pointing me in the right direction. I wrote a script for it.
... View more
09-20-2018
10:46 PM
Hi All, I have a big payload coming in from Kafka topic to EvaluateJsonPath Processor in Nifi. I need to extract about 100 fields from the JSON payload that's coming in. I was wondering, if there was a way to specify these fields as property in EvaluateJsonProperty at once instead of specifying them one by one, which is time consuming. Thanks
... View more
Labels:
- Labels:
-
Apache NiFi