In Spark a transformation works directly on the RDD. Transforms are implemented lazely and closely coupled to the RDDs. You can not use them separately. What you are looking for is a tool that can generate Saprk code for you based on the transformation rule. I don't think that something like that exists.   Wilfred ... View more

The fact that the job runs as the hive user is correct. You have impersonation turned off when you turned on Sentry, at least that is what you should have done. The Hive user is thus the user that executes the job. However the end user should be used to retrieve which queue the application is submitted in (if you use the FairScheduler). This does require some configuration on your side to make this work. There is a Knowledge Base article in our support portal on how to set that up for CM and non CM clusters. Search for "Hive FairScheduler".     I can remember already providing the steps using CM before on the forum:   Login to Cloudera Manager Navigate to Cluster > Yarn > Instances > ResourceManager > Processes Click on the link fair-scheduler.xml, this will open a new tab or window Copy the contents into the a new file called: fair-scheduler.xml On the HiveServer2 host create a new directory to store the xml file (for example, /etc/hive/fsxml) Note: This file should not be placed in the standard Hive configuration directory since that directory is managed by Cloudera Manager and the file could be removed when changing other configuration settings. Upload the fair-scheduler.xml file to the above created directory In Cloudera Manager navigate to Cluster > Hive > Service-Wide > Advanced > Hive Service Advanced Configuration Snippet (Safety Valve) for hive-site.xml and add the following property: <property> <name>yarn.scheduler.fair.allocation.file</name> <value>/etc/hive/fsxml/fair-scheduler.xml</value> </property> Save changes Restart the Hive Service  NOTE: you must have the follwoing rule as the first rule in the placement policy: <rule name="specified" />   Wiflred ... View more

We do not support upgrading Spark without upgrading the rest of CDH. Spark is compiled against a version of Hadoop and the versions of Hadoop can change between releases of CDH. You also need to take into account the dependencies of Spark (like Hive) which might change between versions.   Even if you would be able to upgrade the package you might get weird failures due to the dependency breakage.   Wilfred ... View more

No there is nothing that you can run to check if the log aggregation has finished. It is a distributed state only known inside the NM's   The only thing you can do is retry the log retrieval. Log aggregation is performed by the NodeManager(s) when the containers finish. There is no possibility to tell how long that will take since one node could be running more than one container that finishes at almost the same time. The load on HDFS is also a factor: copying to HDFS will only be as fast as HDFS can handle it at that point.Wilfred     ... View more