Hi all, Is possible to create an workflow on Oozie that automatically execute some Hive, Pig and Spark scripts in order to automate my analytics process? Many thanks! ... View more

Hi experts, How can I overwrite an existing file by a new one (data update). Imagine that I've this: result.map(pair => pair.swap).sortByKey(true).saveAsTextFile("FILE/results") And Imagine that I want to do this: test.map(pair => pair.swap).sortByKey(false).saveAsTextFile("FILE/results") How can I overwrite the results of the var result to the results of the val test in same directory? ... View more

What is the bigger advantage of using Hadoop instead SQL Server or ODI when we aren't in a Big Data Scenario? Many thanks! ... View more

Hi experts, I've this statment in Apache PIG: ... Count = FOREACH data GENERATE SUM(Field); ... How can do a IF Statement like this: IF(SUM(Field) > 10) Store into X; ELSE STORE into Y; Is possible to do this? Many thanks! ... View more

Hi, I have four tables in .csv. All of them can be conected through a fact table (that are in .csv too). I wanna to do some data cleansing to this files and next put them into a Big Table in Have. But in Apache PIG should I've to create a script by table individually, or is better to join in PIG and then aplly some data cleansing in this normalized table? Thanks! ... View more

I doing a small project in Hadoop which the main goal is create some KPI in Hive. However I needed to do some ETL jobs using Pig to clean my data and I put the transformed files into a new directory in HDFS. To ensure that all the files are in correct form, I want to create some data quality activities in Java or Python. I tried to to use PIG UDFs to achieve this but I couldn't connect the Jar file with Pig. Since I can't use PIG UDFs, I'm planning a new approach to do the data quality phase: 1) Run the PIG scripts to clean the data and extract the new files into a new directory in HDFS 2) Put Java/Python independentely read the new files and perform the data quality activities 3) If the Data Quality tests return sucessfully load the files into Hive In your opinion this a good approach for a Big Data project? I'm new in this topic... If not, what a good alternative for perform data quality jobs in this project? Many thanks for your help! ... View more

Hi experts, I've the following code: SPLIT A INTO Src01 IF (Date=='2016-07-01'), Src02 IF (Date=='2016-07-02'), Src03 IF (Date=='2016-07-03'), Src04 IF (Date=='2016-07-04'), Src05 IF (Date=='2016-07-05'), Src06 IF (Date=='2016-07-06'), Src07 IF (Date=='2016-07-07'), Src08 IF (Date=='2016-07-08'), Src09 IF (Date=='2016-07-09'), Src10 IF (Date=='2016-07-10'), Src11 IF (Date=='2016-07-11'), Src07 IF (Date=='2016-07-12'), Src13 IF (Date=='2016-07-13'), Src14 IF (Date=='2016-07-14'), Src15 IF (Date=='2016-07-15'), Src16 IF (Date=='2016-07-16'), Src17 IF (Date=='2016-07-17'), Src18 IF (Date=='2016-07-18'), Src19 IF (Date=='2016-07-19'), Src20 IF (Date=='2016-07-20'), Src21 IF (Date=='2016-07-21'), Src22 IF (Date=='2016-07-22'), Src23 IF (Date=='2016-07-23'), Src24 IF (Date=='2016-07-24'), Src25 IF (Date=='2016-07-25'), Src26 IF (Date=='2016-07-26'), Src27 IF (Date=='2016-07-27'), Src28 IF (Date=='2016-07-28'), Src29 IF (Date=='2016-07-29'), Src30 IF (Date=='2016-07-30'), Src31 IF (Date=='2016-07-31'), Src011 IF (Date=='2016-06-01'); STORE Src01 INTO '/path/2016-07-01' using PigStorage('\t'); STORE Src02 INTO '/path/2016-07-02' using PigStorage('\t'); STORE Src03 INTO '/path/2016-07-03' using PigStorage('\t'); STORE Src04 INTO '/path/2016-07-04' using PigStorage('\t'); STORE Src05 INTO '/path/2016-07-05' using PigStorage('\t'); STORE Src06 INTO '/path/2016-07-06' using PigStorage('\t'); STORE Src07 INTO '/path/2016-07-07' using PigStorage('\t'); STORE Src08 INTO '/path/2016-07-08' using PigStorage('\t'); STORE Src09 INTO '/path/2016-07-09' using PigStorage('\t'); STORE Src10 INTO '/path/2016-07-10' using PigStorage('\t'); STORE Src11 INTO '/path/2016-07-11' using PigStorage('\t'); STORE Src07 INTO '/path/2016-07-12' using PigStorage('\t'); STORE Src13 INTO '/path/2016-07-13' using PigStorage('\t'); STORE Src14 INTO '/path/2016-07-14' using PigStorage('\t'); STORE Src15 INTO '/path/2016-07-15' using PigStorage('\t'); STORE Src16 INTO '/path/2016-07-16' using PigStorage('\t'); STORE Src17 INTO '/path/2016-07-17' using PigStorage('\t'); STORE Src18 INTO '/path/2016-07-18' using PigStorage('\t'); STORE Src19 INTO '/path/2016-07-19' using PigStorage('\t'); STORE Src20 INTO '/path/2016-07-20' using PigStorage('\t'); STORE Src21 INTO '/path/2016-07-21' using PigStorage('\t'); STORE Src22 INTO '/path/2016-07-22' using PigStorage('\t'); STORE Src23 INTO '/path/2016-07-23' using PigStorage('\t'); STORE Src24 INTO '/path/2016-07-24' using PigStorage('\t'); STORE Src25 INTO '/path/2016-07-25' using PigStorage('\t'); STORE Src26 INTO '/path/2016-07-26' using PigStorage('\t'); STORE Src27 INTO '/path/2016-07-27' using PigStorage('\t'); STORE Src28 INTO '/path/2016-07-28' using PigStorage('\t'); STORE Src29 INTO '/path/2016-07-29' using PigStorage('\t'); STORE Src30 INTO '/path/2016-07-30' using PigStorage('\t'); STORE Src31 INTO '/path/2016-07-31' using PigStorage('\t'); STORE Src011 INTO '/path/2016-06-01' using PigStorage('\t'); There's a way that I can make this more automatically? Like using a loop or other iterative way? Many thanks! ... View more

Hi, Anyone already do (or read) any social network analysis using Spark Mllib with Python? I need to do a research with Spark to see the relationships between the people in a organization. Many thanks! ... View more

Hi experts, I used Apache Pig to add a new column to my 3 text files inserted on HDFS. The three texts files was: 2016-06-25.txt 2016-06-24.txt 2016-06-23.txt However after I execute my Pig code I've in my HDFS 7 files (because the Map Reduce): part-m-0000 part-m-0001 part-m-0002 part-m-0003 ... part-m-0006 How can I obtain only 3 files with it orignally name? Basically I want to add the new column but still have the same files with the same name... My code is: Src = LOAD '/data/Src/' using PigStorage(' ','-tagFile'); STORE Src INTO '/data/Src/Src2' USING PigStorage(' '); ... View more

Hi experts, I'm trying to do some data transformations (simple) in my text files using Apache Pig. I've 80 text files in my HDFS and I want to add a new column based on filnename. I test the code for to only one text file and works fine. But when I put the code reading all the files it don't do the job (it stays 0% at long time). Here is my code: A = LOAD '/user/data' using PigStorage(' ','-tagFile') STORE A INTO '/user/data/Data_Transformation/SourceFiles' USING PigStorage(' '); In your opinion, Pig are the best way to this? Thanks!! ... View more

Hi, I've multiple files (in hDFS) with the same schema and I will aggregate all of them into Hive at only one table. Each files represents a date but I only have this info on file title. Which is the best way to insert the file title (the date) as a new column on this files. Java? NiFi? Thanks! ... View more

I'm planning analyse some data using Hadoop. I've 200 text files to analyze. I'm thinking: Using Spark to load data into HDFS (are PIG or Sqoop better?) Create the structure in Hive, creating the tables (basically this first data model will have 200 tables, each table will be a text file) Load data into Hive (all the files) Do some data cleansing with Spark (I will need to put Spark reading from the Hive) and try to reduce the amount of data Create the new data model in Hive (now with a smaller amount of data after the cleansing in previous step) Use a Analytical Tool (like SAS, Tableau, etc.) to do some analytical operations (in this tool I will put the all the data returned in previous step) I believe that this will not be the best way to analyze big data . My goal is in the end of the process in Hadoop have a smaller data set in order to successfully integrate in SAS , for example. What is your opinion ? Many thanks! ... View more

There exists some use case that shows how Hadoop and Spark work together? I already read the theory but I want to see something pratical to have a better understand. Thanks!!! ... View more

I've download the cloudera-quickstart-vm-5.7.0-0-virtualbox Virtual Machine to do my Big Data Project. In my PC i've two Zips files (2GB each on) that contains my source data (there are a lot of txt files). I need to upload this files to HDFS in Virtual Machine, however I getting some troubles When I try to copy/drag the txt files to Virtual Machine.   I was thinking loading the files directly on HDFS (no use Sqoop, for example), my question is: Is there a way that can I load the Source Data from my Local PC to HDFS? Java? Sqoop? There a lot of Txt Files... Thanks! ... View more

Hello experts, I've two simple questions: In your opinion which is the best way to load data to HDFS (My source data are txt files)? Pig, Sqoop, directly in HDFS, etc. Second question is: Is a good option use Spark to do some data transformation, segmentation? Thanks! ... View more