What is an empty bag, empty tuple, and Null? How to represent empty bag, empty tuple, and Null? which operations will generate empty tuple, empty bag? I have this confusion because I do not have clarity when to use datafu.pig.bags.NullToEmptyBag() and EmptyBagToNullFields functions? Can someone give some example for my understanding ... View more

This is from Hive Manual:- Recover Partitions (MSCK REPAIR TABLE) Hive stores a list of partitions for each table in its metastore. If, however, new partitions are directly added to HDFS (say by using hadoop fs -put command) or removed from HDFS, the metastore (and hence Hive) will not be aware of these changes to partition information unless the user runs ALTER TABLE table_name ADD/DROP PARTITION commands on each of the newly added or removed partitions, respectively. Doubt:- How to add new partitions directly using hadoop fs -put? can someone give one example for this.To my knowledge, I know about alter table add partition or dynamic partition. ... View more

How to apply rows between unbounded preceding and unbounded following in Pig? Currently, I am using below code calculate the cumulative sum: A = load 'T' AS (si:chararray, i:int, d:long, f:float, s:chararray); C = foreach (group A by si) { Aord = order A by d; generate flatten(Stitch(Aord, Over(Aord.f, 'sum(float)'))); } D = foreach C generate s, $5;This is equivalent to the SQL statement select s, sum(f) over (partition by si order by d) from T; I know I need to modify the Over(Aord.f, 'sum(float)' clause but not sure what exactly I need to do? ... View more

I have set the No of reducers to 2 but still Hive is executing with 1.Any body help on this set hive.exec.reducers.max=2 Hive (default)> insert overwrite directory '/input123456' > select count(*) from partitioned_user; Total jobs = 1 Launching Job 1 out of 1 Number of reduce tasks determined at compile time: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number> In order to set a constant number of reducers: set mapred.reduce.tasks=<number> Starting Job = job_201810122125_0003, Tracking URL = http://ubuntu:50030/jobdetails.jsp?jobid=job_201810122125_0003 Kill Command = /home/naresh/Work1/hadoop-1.2.1/libexec/../bin/hadoop job -kill job_201810122125_0003 Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1 2018-10-12 21:36:24,774 Stage-1 map = 0%, reduce = 0% 2018-10-12 21:36:32,825 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 4.12 sec 2018-10-12 21:36:41,919 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 4.12 sec 2018-10-12 21:36:42,926 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 6.38 sec MapReduce Total cumulative CPU time: 6 seconds 380 msec Ended Job = job_201810122125_0003 Moving data to: /input123456 MapReduce Jobs Launched: Job 0: Map: 1 Reduce: 1 Cumulative CPU: 6.38 sec HDFS Read: 354134 HDFS Write: 5 SUCCESS Total MapReduce CPU Time Spent: 6 seconds 380 msec OK _c0 Time taken: 37.199 seconds ... View more

I am new to Avro and any input on below clarifications is appreciated. I got clarifications While reading Article on Avro with Hive. Article Link 1)we are not mentioning .avsc schema file here.I read in some other article where they will be mentioning .avsc file in Table properties.Is it mandatory or optional? hive> CREATE EXTERNAL TABLE user_profile (id BIGINT, name STRING, bday STRING) STORED AS avro; 2)For the below change do I need to recreate the .avsc file each and every time?Is it one-time activity or for every step like a,b,c i need to recreate the .avsc file? You can see that a number of operations can be allowed as a simple requirement change: a)Adding a new column to a table (the “country” column in the 2nd file) b)Dropping a column from a table (the “id” column in the 3nd file) b)Renaming a column (the “birthday” column in the 4th file) ... View more

I am new to pig any input is really appreciated.plays relation does not contain field ninetieth.How can we use out_max.ninetieth in step 5? ... View more

The process involved in the implementation of SCD Type 1 are Identifying the new record and inserting it into the dimension table. Identifying the changed record and updating the dimension table. How to implement SCD1 in Hive? Found below things and not sure how to get the complete solution: a)we can Surrogate key using datafu.pig.hash.SHA(); in pig or http://www.remay.com.br/blog/hdp-2-2-how-to-create-a-surrogate-key-on-hive/ in hive using row_number? b)For change capture I can use full outer to identify new record and update record c)To use update statement in hive we have to use transaction property+ORC Format I want to do either in hive or pig source and target is as below ... View more

This is very superb article and any I have following clarifications? https://community.hortonworks.com/articles/29410/four-step-strategy-for-incremental-updates-in-apac.html Clarification 1:-base_table is not a external table.How we are loading the data into base_table is not clear during first run is not clear?could you please provide input on this?we are not using any load statement or insert into statement? I think after using below statement.Manually we have to load the data from files present in path:- /user/hive/incremental_table/incremental_table into table base_table using load data statement sqoop import --connect jdbc:teradata://{host name or ip address}/Database=retail --connection-manager org.apache.sqoop.teradata.TeradataConnManager --username dbc --password dbc --table SOURCE_TBL --target-dir /user/hive/incremental_table -m 1 Clarification 2:-During first run only base_table will be loaded and there is no need to implement the Reconcile,compact and Purge process since we do not have incremental data.Please correct me if i am wrong ... View more