Member since
09-20-2018
12
Posts
0
Kudos Received
0
Solutions
02-12-2019
02:07 PM
Hi Guys, I am currently using detect duplicate processor in my NiFi flow. I was just wondering if its possible to clear the state of detect duplicate processor since i have set the age off duration to 24 hrs and i am performing QA on my flow with the same data. Thanks in advance, Krishna
... View more
Labels:
01-07-2019
02:04 PM
Okay. Thanks a lot @Shu much appreciated.
... View more
01-07-2019
11:34 AM
@Shu
Thank you very much. Ill check with Hortonworks on how to get these processors into the Nifi version that I am using. Meanwhile, I wanted to know is there any other tool that I can use other than Spark or NiFi ? Thanks In Advance
... View more
01-06-2019
03:21 PM
Thank you for your reply. But I don't see lookuprecord processor present in nifi under add processor menu. The NiFi version that I am using is 1.2.0.3.0.1.1-5. Is there anything that I am missing ?
... View more
01-06-2019
12:25 PM
Hi Guys, I need a suggestion/advice on a use case that i have. I have a bunch of data being streamed from NiFi on 3 different mongo db collections let's say A,B,C. I need to perform a lookup based on a particular field say emp ID which is present in A. If the value of Emp ID present in A is matching with that of the payloads present in collection B & C. Only then i need to attach the document present in B & C to document present in A(i am creating new key/value pair, key being a random UID and value being the payload from B & C in A for performing update), If the value doesn't exists then I need to insert the document as it is in A from B or C. Currently i am using Spark to do this. I believe its an overkill for this job and doesn't make sense to use spark. Any other big data tech suggestions ? P.S I need to perform update on real time basis. -Thanks in Advance
... View more
Labels:
11-09-2018
02:59 PM
Hey Guys, I have a detect duplicate processor in my nifi flow. Currently i have left the age of duration blank. So is there a default age of duration value if no value is entered ?. @Shu @Matt Burgess Thanks Krishna
... View more
Labels:
10-15-2018
04:23 PM
Hi Guys, I am new to spark and i am trying to update fields of a dataframe derived from mongodb. Is there any example that i can refer to update the fileds of that dataframe using values from variables ? Can you assist ? -Thanks
... View more
09-24-2018
03:48 PM
Hi All, I am new to NiFi. I am hoping you guys will be able to help me out. I have the below mentioned python script to extract fields from a JSON payload. But the problem i am facing is suppose lets say there's no key value pair present in the payload, so how do i set it to null if the value is not present in the JSON. I tried with using ternary operator but the script failed. The below code is just a sample of what I am trying to achieve. In actuality I need to extract 100 fields from payload so using EvaluateJson processor is not feasible. Script: import json import java.io
from org.apache.commons.io import IOUtils
from java.nio.charset import StandardCharsets
from org.apache.nifi.processor.io import StreamCallback
class PyStreamCallback(StreamCallback):
def __init__(self):
pass
def process(self, inputStream, outputStream):
text = IOUtils.toString(inputStream, StandardCharsets.UTF_8)
obj = json.loads(text)
newObj = {
"Name": obj['name'],
"Age": obj['age'],
"Gender" : obj['gender'],
"test": obj['test'] if obj['test'] else "NULL" //trying to extract test value from Json payload if not present then set it to NULL }
outputStream.write(bytearray(json.dumps(newObj).encode('utf-8')))
flowFile = session.get()
if (flowFile != None):
flowFile = session.write(flowFile,PyStreamCallback())
flowFile = session.putAttribute(flowFile, "filename", flowFile.getAttribute('filename').split('.')[0]+'_translated.json')
session.transfer(flowFile, REL_SUCCESS) -Thanks In Advance
... View more
Labels:
09-21-2018
07:07 PM
The 100 fields are to be used by another down stream application. I actually referred to your blog on writing groovy script to extract the fields. But facing issues with that as well.
... View more
09-21-2018
04:56 PM
Hi All, I have written a groovy script to fetch fields from a incoming json payload. The json fileds contain hyphens in them, due to this the script fails to execute and throws an missing property exception saying 'no such property for class'. Is there a way to specify hyphens in groovy script? Thanks
... View more
Labels:
09-21-2018
03:54 PM
@ShuThank you very much for pointing me in the right direction. I wrote a script for it.
... View more
09-20-2018
10:46 PM
Hi All, I have a big payload coming in from Kafka topic to EvaluateJsonPath Processor in Nifi. I need to extract about 100 fields from the JSON payload that's coming in. I was wondering, if there was a way to specify these fields as property in EvaluateJsonProperty at once instead of specifying them one by one, which is time consuming. Thanks
... View more
Labels: