Member since
03-26-2018
110
Posts
10
Kudos Received
7
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
9739 | 06-24-2020 09:21 PM | |
6718 | 05-20-2020 06:12 AM | |
1768 | 02-25-2019 12:39 AM | |
4549 | 02-21-2019 11:33 AM | |
1487 | 02-21-2019 11:16 AM |
04-30-2018
06:27 AM
Hi All , Need some help here with the "PutHiveStreaming" Processor. I have followed all the prerequisites to set up the data flow onto "PutHiveStreaming" processor however I continue to get the following error :- Hive Streaming connect/write error, flow file will be penalized and routed to retry. org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint I have seen this is a fairly common error across the Nifi Forums and I guess @Matt Burgess had posted a NAR file to fix this issue however the NAR file was of version 1.0 and in my case wont work as I am using version 1.4 which is a stand alone open source version of Apache Nifi ( not HDP ). Our requirement is to extract data from database and stream it into a partitioned table on Hive. We have used the streaming option to write onto HDFS using QueryDatabaseTable -> ConvertAvrotoOrc --> putHDFS however this would just land files onto HDFS , we need to directly insert data onto the partitioned table and make data available to the end users in real time. @Matt Burgess @Shu Do you have any idea on how to get this processor working by any chance
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache NiFi
04-29-2018
11:26 AM
1 Kudo
Thanks @Shu .. I will definitely get this done and then share the template and all the details so that others can benefit just as others are benefiting with your detailed answers ..
... View more
04-29-2018
12:34 AM
1 Kudo
Hey Thanks alot @Shu . As always the answer is detailed and still to the point .One quick question before I start using an exchange server or for that case gmail. I will have to open the firewalls isnt it so that my Nifi Linux server can listen to the Gmail or any other exchange server isnt it ?
... View more
04-27-2018
08:40 AM
@Ok , I managed to get this fixed. In oracle I can see it works a little differently. I used the following and this worked.I passed the table name followed by a "." and that seemed to do the trick. @Shu GIM.GIDB_GC_SKILL.*,'${now():toNumber():format('yyyy-MM-dd HH:mm:ss')}' AS LOAD_TMS
... View more
04-27-2018
08:28 AM
Done @Shu . I have accepted 🙂
... View more
04-27-2018
03:28 AM
1 Kudo
Hi , I have a requirement where by I need to read a specific email in my inbox , extract the attachment and push the attachment onto HDFS. At a high level , i think i need to use the comsume imap processor and then the extract email header and extract email attachment processors to achieve this however I need some idea on how to implement this. @Shu have you worked on this kind of a request before ? If so , then I would love to hear from you . Thanks
... View more
Labels:
- Labels:
-
Apache NiFi
04-27-2018
03:06 AM
1 Kudo
Ok got this fixed . In oracle , this works well if we add the table name and then a .*. So this is what I put in and my problem got fixed GIM.GIDB_GC_SKILL.*,'${now():toNumber():format('yyyy-MM-dd HH:mm:ss')}' AS LOAD_TMS Thanks
... View more
04-27-2018
12:00 AM
Thanks @Shu for trying this out . I used the double quotes but it isnt working with Oracle. For now , I will use your JSON approach as a stop gap arrangement till I find a way out to fix this. In case you have any ideas then do let me know. Thansk again
... View more
04-26-2018
07:25 AM
Hey thanks . I am using 1.4 version but still I am not able to get this working. Please find the screen shot below columns-from-rdbms.png So , I can see that the regular expression is working as in the error section , the timestamp is coming up however the issue perhaps seems to be with the syntax which is *, ............... . Isnt this supported ? Waiting for your brilliant feedback as always @Shu
... View more
04-26-2018
05:09 AM
@Shu Meanwhile , I tried using your other option and that worked which is convert Avro to JSON and then using a replace text. That was a cool approach 🙂 . However , we are doing a streaming ingestion and want to limit the number of processors , and hence prefer option 1 which is putting that extra column at the "Query Database Table" processor layer itself. cheers Abhi
... View more
- « Previous
- Next »