Member since
02-04-2016
132
Posts
52
Kudos Received
7
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
6195 | 07-25-2018 10:53 AM | |
1756 | 07-25-2018 05:15 AM | |
1817 | 10-03-2017 12:08 PM | |
3072 | 04-04-2017 05:36 AM | |
3327 | 11-29-2016 05:40 PM |
08-23-2016
04:46 PM
@Simon Elliston Ball
Thanks for the thorough answer.
I've tried to use putemail processor for failed relationships - but i didn't get any email even though the step completed with an error.
To be more exact - I do receive email for successful steps without a problem, however, i have a flow in which the last step finishes with error but i get no email unless i set the relationship to retry.
If i set the relationship to the putemail processor from the last step as success - no email,
If i set it to failed - no email
if i set to retry - i do receive email. Isn't "error" means the step failed ?
Any idea ? Adi
... View more
08-22-2016
08:22 AM
1 Kudo
Hello y'all :)
When i execute a flow - how can i tell if it is running and what exactly it is doing ?
I couldn't find any indication or log so how can i know how the flow progresses and on which step it is at a particular moment.
In the UI itself i noticed that only if a step fails - it shows a yellow tool-tip that displays information, but i'd rather have an indication on the current doing.
Is there a way to get notified (even in the UI) that a flow is finished ? Thanks Adi
... View more
Labels:
- Labels:
-
Apache NiFi
08-21-2016
03:50 PM
@Constantin Stanca
Hi and thank you for replying.
Just to be clear - i have two tables: staging and permanent.
I want to select from the staging table and insert the results to the permanent table.
So i created a process group with SelectHiveQL processor that queries a specific column (event_time) In that process group i created a sub process group for the insert process. I started the process group with an Input port and connected it to a ReplaceText processor that has the following syntax for replacement value:
INSERT INTO nifitest (event_time) VALUES ('${event_time}')
and of course connected it to a "PutHiveQL" processor. The flow ends successfully - but the column is blank. Not NULL. But blank. No values what so ever.
I tried all kinds of syntax and even using "UpdateAttribute" processor - but nothing seems to work.
I am able to insert hard coded values into a the column using: INSERT INTO nifitest (event_time) VALUES ('2222222') It seems as if the select output isn't transferred to the sub-process group or that my syntax is incorrect. Could you advise ? Thanks in advance!
Adi
... View more
08-18-2016
09:45 AM
3 Kudos
Hello
NiFi 0.70
I have 2 questions regarding Hive processors in NiFi. 1. I created a basic flow and using "SelectHiveQL" processor i'm able to extract data from hive. However, i have no idea where the output resides. When i choose csv as output format the execution fails. When i choose Avro it succeeds but i don't know what Avro is and where to find it on my NiFi machine.
So where exactly can i find the output of my query ? 2. After executing the "SelectHiveQL" processor i wanted to insert the data to a different table in Hive.
I tried using "PutHiveQL" processor but it doesn't have any field for writing my insert command. So i used the "SelectHiveQL" processor for inserting the data by using "insert select" statement (see attached screenshot)
This works - but it seems kind of a workaround. There must be a proper way to insert data into hive.
Any ideas ?
Adi
... View more
Labels:
- Labels:
-
Apache NiFi
08-17-2016
03:34 PM
Hi @Pierre Villard
Unfortunately there is no stack trace. Just this error as is.
Also, it only happens when output format is "CSV". Using "Avro" it succeeds without errors - but i have no idea where the output results are (not familiar with Avro).
My goal is to select data from hive and insert it to another table. If i don't get a file with all the output of the select - how can i insert the output ?
Btw, we are using 0.70 NiFi
... View more
08-17-2016
08:03 AM
Hello
I've created HiveConnectionPool in controller services so i could execute SelectHiveQL processor.
The connection parameters for hive are good (i know this because if i enter a wrong user or password in the connection settings it fails due to login error) but when i execute the Select HiveQL processor it fails with the following error:
2016-08-17 10:56:46,159 ERROR [Timer-Driven Process Thread-6] o.a.nifi.processors.hive.SelectHive
QL SelectHiveQL[id=dc11ed58-3734-4100-bc92-0202d88d9480] SelectHiveQL[id=dc11ed58-3734-4100-bc92-
0202d88d9480] failed to process session due to java.lang.NullPointerException: java.lang.NullPoin
terException
2016-08-17 10:56:46,159 ERROR [Timer-Driven Process Thread-6] o.a.nifi.processors.hive.SelectHive
QL
java.lang.NullPointerException: null Any ideas what i'm doing wrong ? Thank you, Adi J
... View more
Labels:
- Labels:
-
Apache NiFi
04-14-2016
08:19 AM
Thank you all for your responses !
Eventually the balancer stopped by itself so no intervention needed.
... View more
04-13-2016
01:55 PM
1 Kudo
Hello We've added new DataNodes to the cluster. These new DataNodes have bigger disks than the old ones.
The balancer is running flawlessly for days now but my concern is that because the balancing is using percentage for spreading blocks - the end result would be that most of the data would be located in the new DNs and a way smaller amount in the old DNs.
(For example - old DNs are 20TB, new DNs are 50 TB - so end result would be 2 TB in old DNs and 5 TB in new if threshold is 10%). 1. Doesn't this division of data misses the parallel computing advantage ? (most of the data is centralized in few DNs)
2. In case a new DN is down for any reason the recovery process (fixing under replicated blocks) would take longer, no ? Wouldn't it be smart to stop the balancer when data is spread evenly in size and not in percentage ? Thank you Adi J.
... View more
Labels:
- Labels:
-
Apache Hadoop