Member since
05-16-2019
25
Posts
1
Kudos Received
0
Solutions
09-18-2018
09:08 AM
Hi Akhil, i do before yum repolist and this was the result: status
HDF-3.2-repo-1 HDF-3.2-repo-1 0
HDP-UTILS-1.1.0.22-repo-1 HDP-UTILS-1.1.0.22-repo-1 0
ambari-2.7.0.0 ambari Version - ambari-2.7.0.0 0
base/7/x86_64 CentOS-7 - Base 0
epel/x86_64 Extra Packages for Enterprise Linux 7 - x86_64 0
extras/7/x86_64 CentOS-7 - Extras 0
updates/7/x86_64 CentOS-7 - Updates 0 This two repositories: HDF-3.2-repo-1
HDP-UTILS-1.1.0.22-repo-1 Dont was added by me, i think that was installed by the mpack. Are you sure it's a good idea to remove them? Test also the yum clean and the update before and the result was the same.
... View more
02-19-2018
11:38 AM
Just to be more specific 1. Driver talks to namenode to find the location of the HDFS blocks. 2. The info is available to the AM. 3. Driver request for AM, Am requests for the required resources based on the blocks info. 4. YARN has no business to talk to namenode directly.
... View more
12-20-2017
08:23 AM
Thank you very much for your help, I didn't consider the expression support.
... View more
12-11-2017
03:00 PM
Thank you so much for the help, I appreciate it.
... View more
03-27-2019
05:08 PM
@Lanic - With release of Apache NiFi 1.7 and HDF 3.2 in mid 2018, the ability in terminate threads still executing on a processor that is in a state of "stopping" is now possible. After changing state of processor from start to stop, you will see processor display red square. You should give the running threads an opportunity to complete their execution. If it appears the processor is just not going to stop (hung threads) you can right click on the processor and select "Terminate" from the context menu displayed as follows:
... View more
11-08-2017
10:19 AM
Thanks Matt, I'll be running tests today, thank you very much for your answer.
... View more
10-24-2017
05:41 PM
@xav webmaster Straight answer: flowFile = session.get()
if (flowFile != None):
flowFile = session.putAttribute(flowFile, 'myAttr', 'myValue')
# implicit return at the end More info on executeScript processor: https://community.hortonworks.com/articles/75032/executescript-cookbook-part-1.html In your particular case, in callback function where you read from input stream, you can scan from org.apache.commons.io import IOUtils
from java.nio.charset import StandardCharsets
from org.apache.nifi.processor.io import StreamCallback
class PyStreamCallback(StreamCallback):
def __init__(self):
self.topic_name=''
pass
def get_topic_name(self):
return self.topic_name
def process(self, inputStream, outputStream):
Log = IOUtils.toString(inputStream, StandardCharsets.UTF_8)
Log2 = str(Log).split(',')
Brand = Log2[0]
Color = Log2[5]
Model = Log2[1]
if Brand == 'ford' and Color == 'gray':
NewLog = str(Log2)
self.topic_name = 'ford'
outputStream.write(bytearray((NewLog).encode('utf-8')))
if Brand == 'audi' and Color == 'black':
NewLog = str(Log2)
self.topic_name = 'audi'
outputStream.write(bytearray((NewLog).encode('utf-8')))
if Brand == 'bmw' and Color == 'white':
NewLog = str(Log2)
self.topic_name = 'bmw'
outputStream.write(bytearray((NewLog).encode('utf-8')))
# add exception handling if needed for empty flowfile content, etc
if(flowFile != None):
caller = PyStreamCallback()
flowFile = session.write(flowFile, caller)
topic_name = caller.get_topic_name()
flowFile = session.putAttribute(flowFile, 'kafka_topic', topic_name)
Hope that will help.
... View more
10-17-2017
01:51 AM
Hi @xav webmaster, i think this answer will help you for sure, Sample Flow :- GenerateFlowFile--> ExtractText-->UpdateAttribute-->PublishKafka GenerateFlowFile:- As for testing purpose i'm using this but in your case you are having some other processors ExtractText Processor:- in this processor i'm extracting the contents of flowfile as attribute. Ex:- adult,dog,bulldog,9,23,male,brown,4,etc The above content of flowfile by adding new property to ExtractText i'm going to extract the content and keeping that as attribute of the flowfile cnt_attr as (.*) //capture everything and add to ff as cnt_attr Configs:- Output of this processor:- Every flowfile will associated with the attribute called cnt_attr to it, we can use this attribute in UpdateAttribute Processor:- To dynamically change the topic names based on the cnt_attr attribute, for this case we need to use Advanced Usage of Update Attribute processor. Right Click on UpdateAttribute processor and click on Advanced Button in the lower right corner. Steps:- open above screenshot in new tab to see 1,2,3,4 steps and refer them with below steps 1. As mentioned in the above screenshot click on FlowFile Policy change to UseOriginal 2. Click on + sign at Rules and give name as adult_dog 3. Click on + sign at Conditions and give our check condition in it ${cnt_attrt:matches('.*adult.*dog.*')} 4. Click on + sign at Actions and give the attribute name as kafka_topic Value for the kafka_topic attribute as adult_dog New Rule:- for cat_dog conditions check is ${cnt_attr:matches('.*cat.*dog.*')} and Actions add attribute name as kafka_topic and value is cat_dog same as 2,3,4 steps above. summarize all the steps:- step 1 we are using original flowfile and
step2 we are creating a rule and
step3 adding conditions to check if the cnt_attr attribute satisfies or not
step4 if it satisfies then adding kafka_topic attribute with desired name to it. like this way we can add as many rules as we want in same UpdateAttribute Processor as you can see in my screenshot i have added 2 Rules(adult_dog,cat_dog). This processor checks which Rule has satisfied and updates kafka_topic attribute with the mentioned name in it. PublishKafka:- use the kafka_topic attribute in Topic Name property of processor ${kafka_topic} Flow Screenshot:- In this way we can use only one UpdateAttribute to dynamically change the value of kafka_topic based on update attribute processor and use same kafka_topic attribute to publish messages to respective topics.
... View more
10-15-2017
07:45 PM
That's a great answer. Thank you so much for the help. It'll really come in handy. I can see that he has taken time to respond, and I thank him for that.
... View more
10-13-2017
12:45 PM
It worked perfectly. Thanks!
... View more