@Mats Johansson Understood. However core of services like hive/pig use map reduce. Does that have the same constraints for node labeling? it seems node labeling is only applicable to storm/spark/kafka/hbase/etc. Services which do not use map reduce as its engine. ... View more

@Vadim thanks for sharing. This article is about R on hadoop. I am intersted in R on Spark. ... View more

@Vadim Are suggesting R can not run in distributed mode? ... View more

Also verify CBO is turned on and you are utilizing vectorization (helps with memory). Run a explain plan on the query to determine what types of joins are being used. ... View more

@azeltov @Paul Hargis R is used for model training. Can this be done distributed fashion across your HDP cluster? Right now I am seeing a few data scientist run strictly on edge node and not utilizing their spark data nodes. What am i missing? it is difficult to get all the R libraries on each spark data node? If if R libraries are pushed to data nodes will the model training run distributed mode? will it run parallel execution? ... View more

both sites I shared have files over 1gb. Definitly the flight data by year is more then 1gb. ... View more

I agree with @Benjamin Leonhardi. Provide the log file because vectorization occurs during opertaions like scans, filters, aggregates, and joins. ... View more