Hi all, Is possible to create an workflow on Oozie that automatically execute some Hive, Pig and Spark scripts in order to automate my analytics process? Many thanks! ... View more

Hi experts, How can I overwrite an existing file by a new one (data update). Imagine that I've this: result.map(pair => pair.swap).sortByKey(true).saveAsTextFile("FILE/results") And Imagine that I want to do this: test.map(pair => pair.swap).sortByKey(false).saveAsTextFile("FILE/results") How can I overwrite the results of the var result to the results of the val test in same directory? ... View more

What is the bigger advantage of using Hadoop instead SQL Server or ODI when we aren't in a Big Data Scenario? Many thanks! ... View more

Hi experts, I've this statment in Apache PIG: ... Count = FOREACH data GENERATE SUM(Field); ... How can do a IF Statement like this: IF(SUM(Field) > 10) Store into X; ELSE STORE into Y; Is possible to do this? Many thanks! ... View more

Hi, I have four tables in .csv. All of them can be conected through a fact table (that are in .csv too). I wanna to do some data cleansing to this files and next put them into a Big Table in Have. But in Apache PIG should I've to create a script by table individually, or is better to join in PIG and then aplly some data cleansing in this normalized table? Thanks! ... View more

I doing a small project in Hadoop which the main goal is create some KPI in Hive. However I needed to do some ETL jobs using Pig to clean my data and I put the transformed files into a new directory in HDFS. To ensure that all the files are in correct form, I want to create some data quality activities in Java or Python. I tried to to use PIG UDFs to achieve this but I couldn't connect the Jar file with Pig. Since I can't use PIG UDFs, I'm planning a new approach to do the data quality phase: 1) Run the PIG scripts to clean the data and extract the new files into a new directory in HDFS 2) Put Java/Python independentely read the new files and perform the data quality activities 3) If the Data Quality tests return sucessfully load the files into Hive In your opinion this a good approach for a Big Data project? I'm new in this topic... If not, what a good alternative for perform data quality jobs in this project? Many thanks for your help! ... View more

Hi experts, I've the following code: SPLIT A INTO Src01 IF (Date=='2016-07-01'), Src02 IF (Date=='2016-07-02'), Src03 IF (Date=='2016-07-03'), Src04 IF (Date=='2016-07-04'), Src05 IF (Date=='2016-07-05'), Src06 IF (Date=='2016-07-06'), Src07 IF (Date=='2016-07-07'), Src08 IF (Date=='2016-07-08'), Src09 IF (Date=='2016-07-09'), Src10 IF (Date=='2016-07-10'), Src11 IF (Date=='2016-07-11'), Src07 IF (Date=='2016-07-12'), Src13 IF (Date=='2016-07-13'), Src14 IF (Date=='2016-07-14'), Src15 IF (Date=='2016-07-15'), Src16 IF (Date=='2016-07-16'), Src17 IF (Date=='2016-07-17'), Src18 IF (Date=='2016-07-18'), Src19 IF (Date=='2016-07-19'), Src20 IF (Date=='2016-07-20'), Src21 IF (Date=='2016-07-21'), Src22 IF (Date=='2016-07-22'), Src23 IF (Date=='2016-07-23'), Src24 IF (Date=='2016-07-24'), Src25 IF (Date=='2016-07-25'), Src26 IF (Date=='2016-07-26'), Src27 IF (Date=='2016-07-27'), Src28 IF (Date=='2016-07-28'), Src29 IF (Date=='2016-07-29'), Src30 IF (Date=='2016-07-30'), Src31 IF (Date=='2016-07-31'), Src011 IF (Date=='2016-06-01'); STORE Src01 INTO '/path/2016-07-01' using PigStorage('\t'); STORE Src02 INTO '/path/2016-07-02' using PigStorage('\t'); STORE Src03 INTO '/path/2016-07-03' using PigStorage('\t'); STORE Src04 INTO '/path/2016-07-04' using PigStorage('\t'); STORE Src05 INTO '/path/2016-07-05' using PigStorage('\t'); STORE Src06 INTO '/path/2016-07-06' using PigStorage('\t'); STORE Src07 INTO '/path/2016-07-07' using PigStorage('\t'); STORE Src08 INTO '/path/2016-07-08' using PigStorage('\t'); STORE Src09 INTO '/path/2016-07-09' using PigStorage('\t'); STORE Src10 INTO '/path/2016-07-10' using PigStorage('\t'); STORE Src11 INTO '/path/2016-07-11' using PigStorage('\t'); STORE Src07 INTO '/path/2016-07-12' using PigStorage('\t'); STORE Src13 INTO '/path/2016-07-13' using PigStorage('\t'); STORE Src14 INTO '/path/2016-07-14' using PigStorage('\t'); STORE Src15 INTO '/path/2016-07-15' using PigStorage('\t'); STORE Src16 INTO '/path/2016-07-16' using PigStorage('\t'); STORE Src17 INTO '/path/2016-07-17' using PigStorage('\t'); STORE Src18 INTO '/path/2016-07-18' using PigStorage('\t'); STORE Src19 INTO '/path/2016-07-19' using PigStorage('\t'); STORE Src20 INTO '/path/2016-07-20' using PigStorage('\t'); STORE Src21 INTO '/path/2016-07-21' using PigStorage('\t'); STORE Src22 INTO '/path/2016-07-22' using PigStorage('\t'); STORE Src23 INTO '/path/2016-07-23' using PigStorage('\t'); STORE Src24 INTO '/path/2016-07-24' using PigStorage('\t'); STORE Src25 INTO '/path/2016-07-25' using PigStorage('\t'); STORE Src26 INTO '/path/2016-07-26' using PigStorage('\t'); STORE Src27 INTO '/path/2016-07-27' using PigStorage('\t'); STORE Src28 INTO '/path/2016-07-28' using PigStorage('\t'); STORE Src29 INTO '/path/2016-07-29' using PigStorage('\t'); STORE Src30 INTO '/path/2016-07-30' using PigStorage('\t'); STORE Src31 INTO '/path/2016-07-31' using PigStorage('\t'); STORE Src011 INTO '/path/2016-06-01' using PigStorage('\t'); There's a way that I can make this more automatically? Like using a loop or other iterative way? Many thanks! ... View more

Hi, Anyone already do (or read) any social network analysis using Spark Mllib with Python? I need to do a research with Spark to see the relationships between the people in a organization. Many thanks! ... View more

Hi experts, I used Apache Pig to add a new column to my 3 text files inserted on HDFS. The three texts files was: 2016-06-25.txt 2016-06-24.txt 2016-06-23.txt However after I execute my Pig code I've in my HDFS 7 files (because the Map Reduce): part-m-0000 part-m-0001 part-m-0002 part-m-0003 ... part-m-0006 How can I obtain only 3 files with it orignally name? Basically I want to add the new column but still have the same files with the same name... My code is: Src = LOAD '/data/Src/' using PigStorage(' ','-tagFile'); STORE Src INTO '/data/Src/Src2' USING PigStorage(' '); ... View more

Hi experts, I'm trying to do some data transformations (simple) in my text files using Apache Pig. I've 80 text files in my HDFS and I want to add a new column based on filnename. I test the code for to only one text file and works fine. But when I put the code reading all the files it don't do the job (it stays 0% at long time). Here is my code: A = LOAD '/user/data' using PigStorage(' ','-tagFile') STORE A INTO '/user/data/Data_Transformation/SourceFiles' USING PigStorage(' '); In your opinion, Pig are the best way to this? Thanks!! ... View more