I have a requirement to calculate greatest value in multiple columns. Input: name sub1 sub2 sub3 st1 10 12 15 st2 50 30 35 st3 15 18 13 expected output: name max_sub st1 15 st2 50 st3 18 When I try to use greatest function as below, I'm getting Failed with exception java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: Error evaluating greatest(sub1,sub2,sub3) select name,greatest(sub1,sub2,sub3) from students; any other functions or simple udf available? Note: I cant' use CASE statement as I have to consider around 15 columns. ... View more

Ideally the number of files should be equal to number of buckets declared in a table DDL. It is working fine whenever an initial insert or every insert overwrite is performed. But, insert into hive bucketed table is creating extra files from next insert on wards. ex: # of Buckets = 4 No. of files after Initial insert --> 4 No. of files after 2nd insert --> 8 No. of files after 3rd insert --> 12 No. of files after n insert --> n* # of Buckets. First insert : -rwxrwxrwx 3 hvallur hdfs 49 2016-08-25 12:42 hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=vr/000000_0 -rwxrwxrwx 3 hvallur hdfs 49 2016-08-25 12:42 hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=vr/000001_0 -rwxrwxrwx 3 hvallur hdfs 308 2016-08-25 12:42 hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=vr/000002_0 -rwxrwxrwx 3 hvallur hdfs 49 2016-08-25 12:42 hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=vr/000003_0 2nd Insert: -rwxrwxrwx 3 hvallur hdfs 49 2016-08-25 12:42 hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=vr/000000_0 -rwxrwxrwx 3 hvallur hdfs 49 2016-08-25 12:47 hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=vr/000000_0_copy_1 -rwxrwxrwx 3 hvallur hdfs 49 2016-08-25 12:42 hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=vr/000001_0 -rwxrwxrwx 3 hvallur hdfs 49 2016-08-25 12:47 hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=vr/000001_0_copy_1 -rwxrwxrwx 3 hvallur hdfs 308 2016-08-25 12:42 hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=vr/000002_0 -rwxrwxrwx 3 hvallur hdfs 302 2016-08-25 12:47 hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=vr/000002_0_copy_1 -rwxrwxrwx 3 hvallur hdfs 49 2016-08-25 12:42 hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=vr/000003_0 -rwxrwxrwx 3 hvallur hdfs 49 2016-08-25 12:47 hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=vr/000003_0_copy_1 Does any one know how to resolve this issue ? I thought of writing a program to consolidate all the copy files to related bucket file (i.e before _copy). But, I doubt it might effect Hive meta store mapping because Hive stores the file name related to each bucket#. below query gives the details. SELECT input__file__name,block__offset__inside__file,id from test2 where id%4==3 order by block__offset__inside__file; (Trying to display all the files related to 3rd bucket) hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=v/000003_0_copy_1 163 3 hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=v/000003_0 166 3 Thank you! ... View more

Please help me with below scenarios. I have a flow file with a single record as data. Is there a way to apply expression language on content of flow file if I want to add a new flow file attribute using update attribute? in my current approach I'm using extract text to assign the entire row as an attribute and then using update attribute to add new attribute with EL. I feel like it is not a correct approach. Please suggest a better way to do this. Also, Can we add one flow file's attribute to another flow file? ex: I want to extract some part of filename attribute of flow file A and use that to add an attribute for flow file B. ... View more