Hi Paul, thnaks for your attention. My goal is do some Social Analysis (find patterns, etc.) that's why I want SAS too. The subject is the relationships between a company. I've the emails, telephones, etc. What I've: 5 months data collection (Aug, Set, Oct, Nov and Dec) Each text file correspond to a day Each type of communication have an specific ID (imagine, email I've ID 1, Phone ID 2, etc.) Each line corresponds to an aggregation of multiple communications (separated by the department and every 30 minutes) The attributes are: Communication ID Time Department Email Code Phone Code Phone Duration One possible line of the text file would be: 1 10:30:87 3 12 1 10:30:22 1 10:45:21 3 12 2 10:30:22 2 12 2 10:30:22 1 12 10:30:22 So as you can see, I can have multiple Communication ID by line (that's one of my doubts to create the Hive tables). The size of the text files are 6GB. Many thanks for your help Paul 🙂 Hope you can understand the problem. Thanks! ... View more

Yes when I think about Hadoop I'm saying to storage the data into HDFS. I don't know what type of advantage that can I take with Spark. Data cleansing? ... View more

Hi Lester, many thanks for your attention 🙂 I was thinking use Sqoop to get the correct format of my data but I think it will be better in terms of simplicity and speed put the files directly on HDFS. When I talk about segmentation, I was thiking in clusters analysis, basically divide the date into more smaller data sets. However, I think I can do that in Hive. Many thanks!!! ... View more

Sean, just many thanks for your response. This machine have pySpark ??? ... View more