Member since
04-22-2016
931
Posts
46
Kudos Received
26
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 1855 | 10-11-2018 01:38 AM | |
| 2217 | 09-26-2018 02:24 AM | |
| 2246 | 06-29-2018 02:35 PM | |
| 2915 | 06-29-2018 02:34 PM | |
| 6096 | 06-20-2018 04:30 PM |
03-22-2018
03:06 PM
1 Kudo
hive-nifi-load.xmlif I load the hive table via sqoop the loading is extremely fast compared to if I load it via Nifi. with Nifi its loading only 100 records / 30 mins how can I make NIFI table loading faster ? I am attaching the nifi flow file. the table ddl is as follows : CREATE TABLE default.purchase_acct_orc (
acct_num STRING,
pur_id STRING,
pur_det_id STRING,
product_pur_product_code STRING,
prod_amt STRING,
accttype_acct_type_code STRING,
accttype_acct_status_code STRING,
emp_emp_code STRING,
plaza_plaza_id STRING,
purstat_pur_status_code STRING)
PARTITIONED BY (pur_trans_date TIMESTAMP)
CLUSTERED BY(acct_num) INTO 5 BUCKETS
STORED AS ORC
TBLPROPERTIES
("transactional"="true");
thanks
... View more
Labels:
03-21-2018
01:50 PM
why do I have "LogAttribute" with QueryDatabasetable processor? its showing a red block on the corner which normally indicates issues I think. from QueryDatabaseTable there are two relations one going to PutHiveStreaming and one going to LogAttribute . why do I need two relationships for success ? (please see attached) capture.jpg
... View more
03-21-2018
01:40 PM
hi Shu yes the table structure you gave fixed the issue and its loading now but something strange is happening , it got stuck on 104 for a while and I stopped the QueryDatabaseTable processor , then after a while I noticed it had dumped more records . I guess its just slow ? how can I tune the loading app as its extremely slow. thanks 18555604 747495984 657020052 31 25 01 NULL 9985 009500 10 2015-01-03 10:02:41
Time taken: 0.089 seconds, Fetched: 104 row(s)
hive>
<br>
... View more
03-20-2018
06:38 PM
db-record.zipi think its the data issue , I am uploaded a queue entry please take a look. the table ddl is as follows : CREATE TABLE purchase_acct_orc ( acct_num BIGINT, pur_id BIGINT, pur_det_id BIGINT, product_pur_product_code STRING, prod_amt FLOAT, accttype_acct_type_code STRING, acctstat_acct_status_code STRING, emp_emp_code STRING, plaza_plaza_id STRING, purstat_pur_status_code STRING ) PARTITIONED BY (pur_trans_date TIMESTAMP) CLUSTERED BY(acct_num) INTO 5 BUCKETS STORED AS ORC TBLPROPERTIES ("transactional"="true")
... View more
03-15-2018
08:32 PM
hi Shu can you please put the properties of the HiveConnectionPool ? hiveconnectionpool-properties.jpg
... View more
03-12-2018
09:03 PM
Shu this flow you mentioned 1) QueryDatabasetable 2)PutHiveStreaming 3)LogAttribute is not working for me , can you please take a look at the errors / the attached flow file ?
... View more
03-12-2018
09:02 PM
Shu this flow you mentioned 1) QueryDatabasetable 2)PutHiveStreaming 3)LogAttribute is not working for me , can you please take a look at the errors / the attached flow file ?
... View more
03-12-2018
09:01 PM
my hive table is bucketed and in ORC format. so why am I getting error ? and what does this error means?
... View more
03-08-2018
09:22 PM
puhive-after-querydatabase.jpg I was told by the HW engineer who setup this initial setup that putHiveStreaming processor is not able to write into hive table cause the data is not in JSON format and I need to convert it to JSON first .. ? please see the attached error when I put HiveStreaming after the QueryDatabase processor.
... View more
03-07-2018
04:41 PM
db-hive-temp.xml nifi-db-hive.jpg please see the attached error , i am feeding it JSON data but its complaining about AVRO format.
... View more
Labels: