Hello everyone, I have a situation and I would like to count on the community advice and perspective. I'm working with pyspark 2.0 and python 3.6 in an AWS environment with Glue. I need to catch some historical information for many years and then I need to apply a join for a bunch of previous queries. So decide to create a DF for every query so easily I would be able to iterate in the years and months I want to go back and create on the flight the DF's. The problem comes up when I need to apply a join among the DF's created in a loop because I use the same DF name within the loop and if I tried to create a DF name in a loop the name is read as a string not really as a DF then I can not join them later, So far my code looks like: query = 'SELECT * FROM TABLE WHERE MONTH = {}' months = [1,2] frame_list = [] for item in months: df = 'cohort_2013_{}'.format(item) query = query_text.format(item) frame_list.append(df) # I pretend to retain in a list the name of DF to recall it later df = spark.sql(query) df = DynamicFrame.fromDF( df , glueContext, "df") applyformat = ApplyMapping.apply(frame = df, mappings = [("field1","string","field1","string"), ("field2","string","field2","string")], transformation_ctx = "applyformat") for df in frame_list: create a join query for all created DF. Please if someone knows how could I achieve this requirement let me know your ideas. thanks so much ... View more

Hi guys, I would like to know if in Sqoop is possible load too many files at the same time as we load whole included files in a folder into a pig relation just indicating in the load something like: test = LOAD '/user/me/datasets/*' USING PigStorage(','); I have tried to apply the same logic in a SQOOP stement to load or update in one shoot many files with the same structure into a mysql table with the code: sqoop export --connect jdbc:mysql://nn01.itversity.com/retail_export --username retail_dba --password itversity \ --table roles --update-key id_emp --update-mode allowinsert --export-dir /user/ingenieroandresangel/datasets/export/* \ -m 1 The weird thing in my scenario is that even if at the end of my statement the systems shows up a message: 17/08/14 15:05:02 INFO mapreduce.Job: Counters: 8 Job Counters Failed map tasks=1 Launched map tasks=1 Rack-local map tasks=1 Total time spent by all maps in occupied slots (ms)=5548 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=2774 Total vcore-milliseconds taken by all map tasks=2774 Total megabyte-milliseconds taken by all map tasks=4260864 17/08/14 15:05:02 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead 17/08/14 15:05:02 INFO mapreduce.ExportJobBase: Transferred 0 bytes in 19.3098 seconds (0 bytes/sec) 17/08/14 15:05:02 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead 17/08/14 15:05:02 INFO mapreduce.ExportJobBase: Exported 0 records. 17/08/14 15:05:02 ERROR mapreduce.ExportJobBase: Export job failed! 17/08/14 15:05:02 ERROR tool.ExportTool: Error during export: Export job failed! the table in my sql has load all records in my two proposed txt files so, in theory that worked even with this error but i'm not sure so please if someone can explain that behavior in technical sens i appreciate. thanks. ... View more