Hi experts, There exists any way to make a query to sys.tables like we do on T-SQL, like: SELECT *FROM sys.tables Is this possible in Impala or Hive? Thanks! ... View more

Hi, Do you know any good tutorial/use case using Hadoop that shows a good approach to clean our data (specially the outliers detection phase)? Thanks! ... View more

Hi experts, Afther some scala programming, I'm getting this output: [40146844020121125,WrappedArray(1726)] [40148356620121118,WrappedArray(7205)] [40148813920120703,WrappedArray(3504, 1703)] [40148991920121112,WrappedArray(5616)] [40150340320130324,WrappedArray(9909)] [40150796920120926,WrappedArray(3509)] [40151143320130423,WrappedArray(9909)] [40153957220120426,WrappedArray(9909)] [40154761720120504,WrappedArray(9909)] [40154969620130124,WrappedArray(9909, 9909)] But I want to extract this: 40146844020121125,1726 40148356620121118,7205 40148813920120703,3504, 1703 40148991920121112,5616 40150340320130324,9909 40150796920120926,3509 40151143320130423,9909 40153957220120426,9909 40154761720120504,9909 40154969620130124,9909,9909 I'm trying to analyze the frequent products purchase together and my Scala code is: val data = sc.textFile("FILE"); case class Transactions(Transaction_ID:String,Dept:String,Category:String,Company:String,Brand:String,Product_Size:String,Product_Measure:String,Purchase_Quantity:String,Purchase_Amount:String); def csvToMyClass(line: String) = { val split = line.split(','); Transactions(split(0),split(1),split(2),split(3),split(4),split(5),split(6),split(7),split(8)) } val df = data.map(csvToMyClass).toDF("Transaction_ID","Dept","Category","Company","Brand","Product_Size","Product_Measure","Purchase_Quantity","Purchase_Amount"); df.show; val df2 = df.groupBy("Transaction_ID").agg(collect_list($"Category")) df.groupBy("Transaction_ID").agg(collect_list($"Category")).show How can map the DataFrame to a normal list? Many thanks!!! ... View more

Hi experts, Imagine that I've this example stored on HDFS in a .CSV file: Stock_ID Sales_ID 1 A 2 A 3 C 3 D 4 A I want to map the row using CombineByKey to list the elements and after that I wanto to reduce it to I can get the RDD that I expect. I only have this line: val textFile = sc.textFile("/input/transactions.csv") How can map and reduce it using Scala in Spark? Many thanks!!! ... View more

Hi guys, I'm doing a Big Data Analytics project and I am in the phase of detection of outliers. I only have experience using SAS, and in there I ussually use Histogram charts to detect the outliers. In Hadoop which component is used to identify this records? Normally you use Pig or Hive? Or just use any tool outside Hadoop like Python or Java? Many thanks! ... View more

Hi, I've the follwoing code: from datetime import datetime loc_offers = "offers.csv" loc_transactions = "transactions.csv" loc_reduced = "reduced2.csv" # will be createddef reduce_data(loc_offers, loc_transactions, loc_reduced): start = datetime.now() #get all categories on offer in a dictoffers = {} for e, line in enumerate( open(loc_offers) ): offers[ line.split(",")[1] ] = 1#open output filewith open(loc_reduced, "wb") as outfile: #go through transactions file and reducereduced = 0for e, line in enumerate( open(loc_transactions) ): if e == 0: outfile.write( line ) #print headerelse: #only write when category in offers dictif line.split(",")[3] in offers: outfile.write( line ) reduced += 1#progressif e % 5000000 == 0: print e, reduced, datetime.now() - start print e, reduced, datetime.now() - start reduce_data(loc_offers, loc_transactions, loc_reduced) I've the two .csv files in my HDFS. How can I submit this Python code over the two files in HDFS? Thanks! ... View more

Hi,   I'm doing a small Big Data project using Hadoop in cloudera-quickstart-vm-5.7.0-0-virtualbox. I've a file in HDFS that have 22GB of size. When I try to do some job in Pig, like: A = LOAD "/user/cloudera/file.csv"; DUMP A; It stays with staus:Running like a long of time. There exists any configuration that I need to do to proces all this data? Thanks ... View more

Hi experts, I'm doing a small project in Hadoop using cloudera-quickstart-vm-5.7.0-0 at virtualbox. I'm trying to Java in Apache PIG, basically use pig java UDF 's. I already do: In eclipse: 1) Create the Project 2) Convert to Marven project 3) Add the depenencies -> pig : 0.15.0 and hadoop-core : 0.20.2 4) Generate the Jar file in the following direcotry /home/cloudera/workspace Now I want to apply my UDF in my PIG Script: REGISTER '/home/cloudera/workspace/UDFs.jar'; emp_data = LOAD '/user/cloudera/teste.txt' USING PigStorage(' ') as (name:chararray, idade:chararray, func:chararray); Upper_case = FOREACH emp_data GENERATE myUDFS.isNumeric(name); DUMP Upper_case; I also try to put the Jar file in HDFS and add at the properties in Pig Editor but when I submit the Pig it gives me error...  Anyone knows If I'm failling any step? Many thanks!     ... View more

src-data.txt I've the following code: Data = LOAD '.../teste1.txt' using PigStorage(''); Fields = FOREACH Data GENERATE (chararray)$0 AS ID, (chararray)$1 AS Time, (chararray)$2 AS Code, (chararray)$3 AS B_In_Activity, (chararray)$4 AS B_Out_Activity, (chararray)$5 AS In_Activity, (chararray)$6 AS Out_Activity, (chararray)$7 AS Activity); Transf = FOREACH Data_Fields GENERATE ID, ToUnixTime(Time,'dd/MM/yyyyHH:mm:ss', 'GMT') AS Time, Code, B_Activity, B_Activity, In_Activity, Out_Activity, Activity; SPLIT Transf INTO Src31 IF ToDate(Time) == ToDate('2014-12-31', 'yyyy-mm-dd'); STORE Src31 INTO '.../TESTE2' using PigStorage(''); I want the following activities: 1) Transform the field Time into a Unix TimeStamo 2) Split the datasets based on date When execute my code it gives me an error... I've upload my source data to show my data. Anyone can help me? Many thanks!!!! ... View more