It works, thanks a mill Felix. I had to make the change manually in the hive-site.xml file as per your suggestion and bypass Ambari. When I rebooted I still had column names. spark-sql> use sparkpluralsight; Response code Time taken: 2.14 seconds spark-sql> select * from customers; ID NAME ADDRESS 2222 Emily WA 1111 John WA 3333 Ricky WA 4444 Jane CA 5555 Amit NJ 6666 Nina NY Time taken: 2.815 seconds, Fetched 6 row(s) spark-sql> ... View more

It works inline in hive at the command prompt, but it doesn't work in spark-sql at the command prompt. Hive works fine: [root@sandbox-hdp ~]# hive log4j:WARN No such property [maxFileSize] in org.apache.log4j.DailyRollingFileAppender. Logging initialized using configuration in file:/etc/hive/2.6.4.0-91/0/hive-log4j.properties hive> USE SparkPluralsight; OK Time taken: 2.988 seconds hive> SET hive.cli.print.header=true; hive> SELECT * FROM Customers; OK customers.id customers.name customers.address 2222 Emily WA 1111 John WA 3333 Ricky WA 4444 Jane CA 5555 Amit NJ 6666 Nina NY Time taken: 1.008 seconds, Fetched: 6 row(s) hive> But spark-sql doesn't work: spark-sql> USE SparkPluralsight; Time taken: 0.028 seconds spark-sql> SET spark.hadoop.hive.cli.print.header=true; spark.hadoop.hive.cli.print.header true Time taken: 0.016 seconds, Fetched 1 row(s) spark-sql> SELECT * FROM Customers; 2222 Emily WA 1111 John WA 3333 Ricky WA 4444 Jane CA 5555 Amit NJ 6666 Nina NY Time taken: 0.212 seconds, Fetched 6 row(s) spark-sql> ... View more

Hi Felix, thanks for responding. I ran spark-sql with the above command prompt as you suggested but it's not working. I then added the line "spark.hadoop.hive.cli.print.header=true" to my custom Ambari settings under spark, and I also changed the default "false" value in advanced hive Ambari settings in "hive.cli.print.header". I have restarted both spark-sql and hive and I'm still not getting any column headings showing. [root@sandbox-hdp ~]# SPARK_MAJOR_VERSION=2 spark-sql --conf "spark.hadoop.hive.cli.print.header=true" SPARK_MAJOR_VERSION is set to 2, using Spark2 18/06/02 00:13:45 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0 18/06/02 00:13:45 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException 18/06/02 00:13:48 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041. spark-sql> use sparkpluralsight; Time taken: 2.701 seconds spark-sql> select * from customers; 2222 Emily WA 1111 John WA 3333 Ricky WA 4444 Jane CA 5555 Amit NJ 6666 Nina NY Time taken: 3.106 seconds, Fetched 6 row(s) spark-sql> ... View more

Thanks for the help Jay, I've gone and logged into Ambari as "admin" instead of "maria_dev" (I had to root into the host VM and reset ambari password first) and made the changes from there. The default "maria_dev" profile doesn't have admin rights so I couldn't make any changes in the sandbox initially, hence the reason I had gone into /etc folder. You're right, accessing Ambari as admin makes it unnecessary to go into file structure, and it stores a log of changes made to all settings, which is an added bonus. ... View more