That it was working on CDH 5.16.1 but not on CDH 6.3.3 tells me that there's some kind of classpath conflict that didn't exist before. I'd suggest modifying Envelope's top-level pom.xml file to point to the Cloudera Maven repository and to use the Spark version "2.4.0-cdh6.3.3". ... View more

You don't need to declare them in the conf file, but for environment variables you can't have them inside the SQL string because of the way the file format handles variable substitution. Concatenation with variables is reasonably easy though, for example: "SELECT * FROM tableA A INNER JOIN tableB B ON A."${primaryKey}" = B."${primaryKey}   ... View more

You just need to use local environment variables since you are running in client mode.   For example, export tableA=dbA.tableA export tableB=dbB.tableB spark2-submit \ --master yarn \ --deploy-mode client \ envelope-0.7.2.jar comparison.conf   For sudo you would need to use -E to pass the variables through, but it is not good practice to run jobs as the HDFS superuser instead of your own user. ... View more