It depends on if you would prefer your timestamps to be converted to UTC (knowing the HiveServer would need to reside in the time zone the data was generated) or left alone.  I personally prefer not modifying data automatically and control any time zone corrections within my ETL process.  I also prefer to include a time zone column to be able to specify what time zone the data originated to be able to do any corrections later.   Just a note, Hive also has a feature to disable time zone conversion now as well, hive.parquet.timestamp.skip.conversion, that was added in 1.2 as part of https://issues.apache.org/jira/browse/HIVE-9482.   Finally, I'll note that both Hive and Impala try to be smart about when to apply the conversion when reading because multiple engines handle it differently so Hive will detect if it was written by a hive process or Impala and Impala will do the same.  It makes sense to play around with the settings some inserting and reading from different settings to ensure you fully understand how it works. ... View more

When spark determines it needs to use yarn's localizer, it will always load the jar to HDFS, it does not attempt to check if the file changed before loading.   When using the Spark distributed included with CDH, the spark jar is already loaded to all nodes and specifies the jar is local.  When specifying it is local, spark will not upload the jar and yarn's localizer is not used. ... View more

Hi Ranan,   Because this is an older thread and already marked as solved, lets keep this conversation on the other thread you opened: http://community.cloudera.com/t5/Advanced-Analytics-Apache-Spark/Debug-Spark-program-in-Eclipse-Data-in-AWS/m-p/48472#U48472 ... View more

Hi zhuangmz,   Looks like you found the Hive service option withing the Spark configuration and that solved your problem based one your other post.  Feel free to mark this as solved with the link provided or another brief description here. ... View more

I've seen issues with some hardware where using local[*] doesn't use number of cores like expected.  The java method to get the number of cores available for the process isn't always consistent.  Instead, try specifying the number explicitly like local[6] and try again. ... View more

There are two tasks for this spark job, both should be able to run in parrallel.  It looks like there is an issue while running this locally.  How are you launching the job?  If you are setting master on command line, could there be somewhere in the code that is overriding to a single thread?   When launching this within YARN and multiple executors, you should see the two tasks run in parallel, but it should be possible locally as well. ... View more