I am just wondering has anybody come across the scenario where you need to import or read the data from excel to Hadoop? Is there such thing like Flume Excel source around? btw, I know I can convert the excel file to csv then deal with it. Really just trying to explore flume source a bit further here. ... View more

What are the prerequisites dor starting HWI on HDP 2.2 ... View more

I installed Hive ODBC Driver for HDP 2.2 on my windows 7 machine and trying to connect to hive through ODBC(hadoop istalled on CENTOS).I encoutered with following error. configs are all default. For example authentication for hiveserver2 is "none"(default).Is anything i missed out.I followed the document of hortoworks.I gave the server ip and port is 10000.I assumed hiveserver2 is running because beeline command line is working for following command beeline -u jdbc:hive2://ip:10000 ... View more

Hi All, According to my requirement i need script like following A = load '/bsuresh/sample' USING PigStorage(',') as (id,name,sal,deptid); B = GROUP A by deptid; C = foreach B { D = A.name,A.sal;--two fields E = DISTINCT D; generate group,COUNT(E); }; In relation 'D', i am extracting two fields.Where exactly i am facing error. If i chnaged the script like,which is working fine. C = foreach B { D = A.name; --one filed E = DISTINCT D; generate group,COUNT(E); }; But i need count based on distinct of two columns .Can any one help me?? ... View more

I have some number of files having format like 1)filename+date.filefomat 2)filename.fileformat now i need to copy only files which have some number before .(dot). ... View more

I would like to know what are the different ways to enable hcatalog and tez in writing pig scripts. ... View more

I would like to print or read pig_home from terminal.Is there any way??? ... View more

Apache Pig version 0.12.1.2.1.7.0-784 I have data where one of the field doesn't have data like 2015,,08 2015,,09 2015,,11 2015,,04 2015,,05 Now i run the pig command like grunt> given_input = load '/pigtest/flightdelays/' using PigStorage(',') as (year,month,day); grunt> ori = foreach given_input generate month; grunt> illustrate ori; generating error like : Caused by: java.lang.RuntimeException: No (valid) input data found! when i replace the loader with CSVExcelStorage like grunt> given_input = load '/pigtest/flightdelays/' using org.apache.pig.piggybank.storage.CSVExcelStorage(',') as (year,month,day); grunt> ori = foreach given_input generate month; grunt> illustrate ori; getting output like ------------------------------------------------------------------------------- | given_input | year:bytearray | month:bytearray | day:bytearray | ------------------------------------------------------------------------------- | | 2015 | | 05 | ------------------------------------------------------------------------------- -------------------------------- | ori | month:bytearray | -------------------------------- | | | -------------------------------- So,I would like to know 1)What is the problem with Pigstorage. 2)Is it loader problem or pig version problem. 3)If i want to use PigStoarage in this,How is should??? Not only illustrate even dump behaves the same. ... View more

Sqoop version: 1.4.4.2.1.7.0-784 The following is the sqoop command that i used to export simple(comma separated) records . sqoop-export --connect jdbc:oracle:thin:@ipaddress:1521:orcl --username user --password password --table EMP --columns EMPNO,ENAME,JOB,MGR --export-dir /sqooptest/export -m 1 --batch The above command stucks at 95% and doesn't get completed. ... View more

select t1.student_id, t2.course_name from table_one t1 join table_two t2 on (t1.course_id=t2.course_id); The above one is the basic query i am dealing with.Where t1 and t2 are less in size 160 bytes and 50 bytes. When i run the above query ,Map only join is running as usual as "hive.auto.convert.join=true" by default in 0.14. Now, I made "set hive.auto.convert.join=false;".I run the query .Then query chooses "Default or map reduce join " as i taught. Now i changed query like select /*+ MAPJOIN(t2) */ t1.student_id,t2.course_name from table_one t1 join table_two t2 on (t1.course_id=t2.course_id); i.e I am giving hint through query.But In this case also,query chooses default join only.So i have 2 queries(still auto conversion set false) 1) How to give hint through query,so that query should select map only join.(when auto conversion set false). 2)What is the configuration parameter to know size where tables can fit it.So that map only join run. or How to know the size where tables can fit with in to run map only join. ... View more

I know some of the admin commands to check cluster detail.I am planning to write HDP practical as well as real exam.Other than admin commands ,Is there any way to check AWS ??? ... View more

Upto now, I am thinking like Normal join is simple map/reduce job where two tables are two inputs .Based on columns given in equality expression ,Key and values will be decided in map side and they are grouped when comes to reducer side. But when i read following lines, i confused a little bit about "Difference between Map side join and Normal join". In every map/reduce stage of the join, the last table in the sequence is streamed through the reducers where as the others are buffered. Therefore, it helps to reduce the memory needed in the reducer for buffering the rows for a particular value of the join key by organizing the tables such that the largest tables appear last in the sequence. e.g. in SELECT a.val, b.val, c.val FROM a JOIN b ON (a.key = b.key1) JOIN c ON (c.key = b.key1) Can any one explain above lines in normal join in details??? Once explain stream table and buffering in normal join?? ... View more