Spark and Hive use separate catalogs to access SparkSQL or Hive tables in HDP 3.0 and later. The Spark catalog contains a table created by Spark. The Hive catalog contains a table created by Hive. By default, standard Spark APIs access tables in the Spark catalog. To access tables in the hive catalog, we have to edit the metastore.catalog.default property in hive-site.xml (Set that property value to 'hive' instead of 'spark').   Config File Path: $SPARK_HOME/conf/hive-site.xml   Before change the config   <property> <name>metastore.catalog.default</name> <value>spark</value> </property>   After change the config   <property> <name>metastore.catalog.default</name> <value>hive</value> </property>   ... View more

@jtaras, thank you for this wonderful article! I have a follow-up question (just curious) : do Spark workers use the same custom Python path as Zeppelin driver? If yes, how do they know it? (the Zeppelin UI setting seems to apply to the Zeppelin driver application only and its SparkContext) If no, why is there no version conflict between the driver's python and the workers' python? (I tested with the default of 2.X and the custom Anaconda python of 3.7) NOTE: I had to additionally set the PYSPARK_PYTHON environment variable in Spark > Config to the same path in Ambari, in order to be able to use all the Python libraries in the scripts submitted via "spark-submit", but this does not affect the Zeppelin functionality in any way. ... View more

Hello All , Newbie and learning. sandbox seems to stuck at the following . 00:00:09.966414 GIM: KVM: VCPU 0: Enabled system-time struct. at 0x0000000002722000 - u32TscScale=0xcd20cfc1 i8TscShift=-1 uVersion=2 fFlags=0x1 uTsc=0x2a3bff811 uVirtNanoTS=0x10eb9c43d 00:00:09.966475 TM: Host/VM is not suitable for using TSC mode 'RealTscOffset', request to change TSC mode ignored 00:00:12.234594 GIM: KVM: Enabled wall-clock struct. at 0x0000000002721008 - u32Sec=1561631780 u32Nano=946977500 uVersion=2 00:00:12.238143 GUI: UIFrameBufferPrivate::performResize: Size=720x400, Using fallback buffer since no source bitmap is provided 00:00:12.363769 PIT: mode=2 count=0x4a9 (1193) - 1000.15 Hz (ch=0) 00:31:50.924499 NAT: Old socket recv size: 64KB 00:31:50.924620 NAT: Old socket send size: 64KB Also, there seems to be a following error 00:00:04.874822 NAT: Failed to redirect TCP 127.0.0.1:8005 -> 0.0.0.0:8005 (Unknown error) tried to open 127.0.0.1 and no luck please advise. have 16gm on a windows 10 machine . ... View more

I should have mentioned I was using VirtualBox v5.1.14 ... View more