Hello community, I've planning take the HDPCD certification for Spark developers but then according to the official HW site the exam is under maintenance so they post: The HDPCD : Apache Spark Exam NOTICE: This exam is currently undergoing maintenance. It is available for purchase, however, exams cannot be scheduled until an estimated date of August 22, 2018. Thank you for your patience. I was wondering if that means that I can now on Aug 27th purchase and schedule my exam and if they will post new features or any new updates they include for the exam new version. thanks so much for the support. ... View more

Hello everyone, I have a situation and I would like to count on the community advice and perspective. I'm working with pyspark 2.0 and python 3.6 in an AWS environment with Glue. I need to catch some historical information for many years and then I need to apply a join for a bunch of previous queries. So decide to create a DF for every query so easily I would be able to iterate in the years and months I want to go back and create on the flight the DF's. The problem comes up when I need to apply a join among the DF's created in a loop because I use the same DF name within the loop and if I tried to create a DF name in a loop the name is read as a string not really as a DF then I can not join them later, So far my code looks like: query = 'SELECT * FROM TABLE WHERE MONTH = {}' months = [1,2] frame_list = [] for item in months: df = 'cohort_2013_{}'.format(item) query = query_text.format(item) frame_list.append(df) # I pretend to retain in a list the name of DF to recall it later df = spark.sql(query) df = DynamicFrame.fromDF( df , glueContext, "df") applyformat = ApplyMapping.apply(frame = df, mappings = [("field1","string","field1","string"), ("field2","string","field2","string")], transformation_ctx = "applyformat") for df in frame_list: create a join query for all created DF. Please if someone knows how could I achieve this requirement let me know your ideas. thanks so much ... View more

Bonjour everyone, Listen guys, I have previously grouped a relation and the output was declared as a new scheme for a new relation. Then I create a second grouping in the output schema to apply an AVG function bu, guys seems like this new scheme is very different and I'm not able to reach out the required column to apply the AVG. Details: First grouping on a relation called trips_dp to get a new output relation with independent schema trips_gr_dp = GROUP trips_dp BY (departure_station,day,hour); trips_dp_qty = FOREACH trips_gr_dp { qty_trips = COUNT(trips_dp); GENERATE FLATTEN(group) as (departure_station,day,hour), qty_trips as (qty_dp_trip:int); }; Now the output and schema status for the new relation is (image 1): Then, finally I need to get the AVG on hour for whole days within this relation so I grouped and apply the required transformation: trips_dp_qty_gr = GROUP trips_dp_qty by (departure_station,hour); trips_dp_myn = FOREACH trips_dp_qty_gr { myn_trips = AVG(trips_dp_qty.qty_dp_trip); GENERATE FLATTEN(group) as (departure_station,hour), myn_trips as (myn_trips:float); }; My Issue: As you can see in the second group I try to get an AVG from the previous relation trips_dp_qty in the column qty_dp_trip. Unfortunately, I got an error when I run the script because seems like pig can't reach out the desired schema. ... View more

Hi guys, I would like to know if in Sqoop is possible load too many files at the same time as we load whole included files in a folder into a pig relation just indicating in the load something like: test = LOAD '/user/me/datasets/*' USING PigStorage(','); I have tried to apply the same logic in a SQOOP stement to load or update in one shoot many files with the same structure into a mysql table with the code: sqoop export --connect jdbc:mysql://nn01.itversity.com/retail_export --username retail_dba --password itversity \ --table roles --update-key id_emp --update-mode allowinsert --export-dir /user/ingenieroandresangel/datasets/export/* \ -m 1 The weird thing in my scenario is that even if at the end of my statement the systems shows up a message: 17/08/14 15:05:02 INFO mapreduce.Job: Counters: 8 Job Counters Failed map tasks=1 Launched map tasks=1 Rack-local map tasks=1 Total time spent by all maps in occupied slots (ms)=5548 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=2774 Total vcore-milliseconds taken by all map tasks=2774 Total megabyte-milliseconds taken by all map tasks=4260864 17/08/14 15:05:02 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead 17/08/14 15:05:02 INFO mapreduce.ExportJobBase: Transferred 0 bytes in 19.3098 seconds (0 bytes/sec) 17/08/14 15:05:02 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead 17/08/14 15:05:02 INFO mapreduce.ExportJobBase: Exported 0 records. 17/08/14 15:05:02 ERROR mapreduce.ExportJobBase: Export job failed! 17/08/14 15:05:02 ERROR tool.ExportTool: Error during export: Export job failed! the table in my sql has load all records in my two proposed txt files so, in theory that worked even with this error but i'm not sure so please if someone can explain that behavior in technical sens i appreciate. thanks. ... View more