Member since
10-13-2016
17
Posts
0
Kudos Received
0
Solutions
01-04-2018
10:33 PM
@Srikaran Jangidi When you say hive:hive is the owner I am assuming that is the user (Kerberos Principal) you are providing in your GetHDFS Processor. Also please check the permissions on the hdfs folders. The user (Kerberos Principal) has to have write permissions on the folders you are trying to delete files in or move files from and to. Can you try the same operation using a hdfs command on the console to confirm it works outside of nifi. Do a kerberos authentication using the same user and keytab and try a move comman kinit -k -t hive.keytab hive hdfs dfs -mv /user/hadoop/file1 /user/hadoop/file2
... View more
10-05-2017
10:10 PM
Thanks for the response. Here are the challenges/questions we have on this solution. LookupAttribute processor does not support with ScriptedLookupService. So we have to use LookupRecord processor to do the lookup. What this means is I take my event content, parse it as a record, do lookup for the encryption key and add the returned key to the record/content since lookuprecord only allows to enhance the record and not set attribute, store the key as attribute, remove the key from the content and then encrypt. It sounds like a hack to add the key to the content. Is there any way we can add the result of the lookup as an attribute rather than messing up the content? Also the lookup is not working for me. It matches but I get null. The difference is I am trying to lookup using a string value. I even tried adding the property key with the quotes but it did not work.
... View more
10-04-2017
10:41 PM
We have a use case
that we need a solution for urgently. We have just recently upgraded to HDF
Version 3.0.1.1 (Powered by Apache NiFi - Version 1.2.0.3.0.1.1-5) We have a streaming
flow where we consume events from Kafka topic and store the raw data in Hadoop.
So our flow starts with ConsumeKafka. We have a need to encrypt part of the
data before we store it. Our encryption key is stored in a separate Cassandra database.
There are some internal restrictions due to which we cannot store the key in a
property file. We need a mechanism to retrieve the encryption key from our
Cassandra store periodically and "cache" it or make it available in
our realtime flow to use for encrypting data. We do not want to keep querying
cassandra for every event. We are fine with making the call for the key once
for every node if required. We had looked at
distributed cache in HDF Version 2.1.0.1 (Powered by Apache NiFi - Version
1.1.0.2.1.0.1-1) back but we realized it is not distributed and actually tied
to a node. Also the client has to specify only one node to retreive from. So if
that node goes down that cache cannot be accessed from any node and we have to
manually switch the node. We have not checked the behavior in HDF 3.0.1 but if
this has changed and this cache is usable please let us know. Basically we need
a way to get the key and load it in memory one time and as part of the flow
retrieve it from memory and use it for encrypting data on every event.
... View more
Labels:
- Labels:
-
Apache NiFi
12-08-2016
07:31 AM
When you say declarations above, you mean actually defining the columns and datatypes? What is the syntax for that? I am writing hdfs files from NiFi so it stores the schema header in the Avro file. I want to leverage that to create the hive table but do not have any good example to do that. If you could elaboratethe declarations part with some examples of a few columns it would really help.
... View more
11-09-2016
03:46 PM
Firefox ESR v38.2.0
... View more
11-09-2016
03:38 PM
Thanks a lot for helping me resolve this and especially for the quick replies.
... View more
11-09-2016
03:37 PM
I do not believe this. I did two things. Deleted all ReplaceText processors. Created a new one. Then I just came out of the process group and went back in and now it connected. I did not even have to try what you suggested. I have seen this happen while connecting other processors also. The drag feature is very unstable.
... View more
11-09-2016
02:48 PM
Tried it. No luck.I even tried with creating a GenerateFlowFile processor and connect to ReplaceText. That also does not work. Is there some processor that needs to be used in conjunction with ReplaceText like ExtractText or something?
... View more
11-09-2016
02:22 PM
No. When I try to connect the relationship from ConsumeKafka to ReplaceText, it does not highlight it for me to be able to connect. But If I try to connect from ConsumeKafka to PutCassandraQL it highlights and lets me establish relationship. Is there something that prevents from the output of ConsumeKafka to go to ReplaceText. I thought ReplaceText could take any FlowFile with content and that is what ConsumeKafka produces.
... View more
11-09-2016
07:43 AM
I am trying to create a flow to take the entire content of my kafka message and insert in one column in Cassandra. I have configured ConsumeKafka so that I get one message per FlowFile. I am trying to connect the Processors as shown below: ConsumeKafka -> ReplaceText -> PutCassandraQL I am unable to connect ConsumeKafka to ReplaceText. Why is that?
... View more
Labels:
- Labels:
-
Apache Kafka
-
Apache NiFi