@ANKIT PATEL You need to use LookUpRecord processor in NiFi for this case. For more details regards to usage/configurations of LookUpRecord processors refer to this link. ... View more

@Mahendiran Palani Samy Could you mention which of hive are u using because i tried to execute same exact command in Hive-1 and Hive-2 versions(HDP) but both are failed. Hive-2 error: hive> create table u(id int) row format delimited fields terminated by '\;'; Error: Error while compiling statement: FAILED: ParseException line 1:67 mismatched input '<EOF>' expecting StringLiteral near 'by' in table row format's field separator (state=42000,code=40000) Hive-1 Error: hive> create table u(id int) row format delimited fields terminated by '\;'; Error: Error while compiling statement: FAILED: ParseException line 1:67 mismatched input '<EOF>' expecting StringLiteral near 'by' in table row format's field separator (state=42000,code=40000) ... View more

@Mahendiran Palani Samy Create table with unicode character of ; i.e \u003B create table ... ... row format delimited fields terminated by '\u003B'; ... View more

@n c Make sure you are having same number of columns and datatypes of both sides of union are same. Union: Eliminates duplicates from the result. Ex: hive> select "1" id,"2" age from (select 1)t union select "1" id,"2" age from(select 1) t; +---------+----------+--+ | _u2.id | _u2.age | +---------+----------+--+ | 1 | 2 | +---------+----------+--+ as i'm having 1,2 as result but in the result we are having only one row as hive eliminated duplicates. UnionAll: Shows all duplicates also. hive> select "1" id,"2" age from (select 1)t union all select "1" id,"2" age from(select 1) t; +---------+----------+--+ | _u1.id | _u1.age | +---------+----------+--+ | 1 | 2 | | 1 | 2 | +---------+----------+--+ In both cases we need to have same number of columns/datatypes while performing union or union all operations. If you don't have same number of columns/datatypes from different tables then use null value to match number of columns on both sides. Ex: hive> select "1" id,"2" age from (select 1)t union select "1",null from(select 1) t; +---------+----------+--+ | _u2.id | _u2.age | +---------+----------+--+ | 1 | NULL | | 1 | 2 | +---------+----------+--+ in the above example i have used null for age column and result we going to have 2 columns. Refer to this link for more details regards to union and union all operators. ... View more

@nifi_is_awesome, Make sure you are having same field names in target table with your mentioned in avro schema. Even you can also make use of Unmatched Field/Column Behaviour to ignore incase you are having any unmatched fields in your data. Refer to this thread regards to similar kind of issue.. ... View more

@nifi_is_awesome In your UpdateAttribute processor change schema.name property value to STUDENT instead of BIGDATA_DL.STUDENT. ... View more

@Lenu K 1.Using Spark-Hbase Connector: You can use Spark-Hbase connector to get data from Hbase table using Spark and store until what time you have pulled of records from the HBase table. For the next run get the state and use it as lower bound and current time as upper bound pull the data from Hbase table and insert into Hive table. By using this way we are not creating full snapshot of HBase table as Hive orc table instead we are incrementally loading the data into hive table and use hive table data for analytics. 2.Using Hive Merge strategy: You can use Hive Merge strategy introduced in HDP-2.6 but for this case your hive table needs to be Transactional enabled. merge into transactional_table using <hbase_hive_table>... etc for more details refer to this link. another way using hive would be using CTAS as mentioned above in comments for the first run it will take more time but from the 2 run you can only pull the incremental records from HBase table and load into Hive orc table(if you are following this approach then using spark-hbase connector will give more performence.) 3.Using Apache-Phoenix: Using Apache phoenix to get the data from HBase table as Phoenix table will be pointed to HBase table and allows to run sql queries on top of HBase stored data. Difference between Hive-Hbase integration vs Phoenix-Hbase integration ... View more