About sandyy006

sandyy006 · ‎06-09-2018

@Robert Cornell, you'd need to shade the protobuf dependency which is being used in the spark application to avoid conflicts between application and spark dependencies. On a side note you can try if setting below properties helps with the current state. spark.driver.userClassPathFirst true spark.executor.userClassPathFirst true

sandyy006 · ‎06-04-2018

@irfan aziz Glad that it helped! Please click on "Accept" to mark this thread as closed.

sandyy006 · ‎06-04-2018

@irfan aziz Importtsv is not a hbase shell command, Please exit hbase shell and execute the same command.

sandyy006 · ‎05-30-2018

@Developer Developer As @Felix Albani suggested above i'd go with spawning multiple threads to process the dataframes in parallel. This article has a good example : https://hadoopist.wordpress.com/2017/02/03/how-to-use-threads-in-spark-job-to-achieve-parallel-read-and-writes/

sandyy006 · ‎05-30-2018

@Victor If this helped, Please consider "Accepting" the answer and close this thread.

sandyy006 · ‎05-30-2018

@Victor This article should help you : https://community.hortonworks.com/content/supportkb/146508/how-to-use-alternate-python-version-for-spark-in-z.html

sandyy006 · ‎05-30-2018

@Victor Spark 2.2.0 supports Python 2.7+/3.4+. But you'd need to install these python versions separately and point spark accordingly.

sandyy006 · ‎05-29-2018

@Rajesh Reddy If the above answer helped you, Please consider clicking "Accept" button and close this thread.

sandyy006 · ‎05-29-2018

@RAUI If the above answer helped you, Please consider clicking "Accept" button and close this thread.

sandyy006 · ‎05-28-2018

@Michael Bronson, It is highly unlikely that you will get the same version number when using epoch timestamp, The number changes every millisecond. So each time you execute "version"+str(int(time.time() * 1000000)) you will get a new unique number.

Online	Offline
Last Visited	‎01-23-2020 02:33 AM

Member Since	‎02-01-2019 10:51 AM
Last Visited	‎01-23-2020 02:33 AM
Posts	650
Kudos received	142

Cloudera Community

Re: Distributed I/O Benchmark of HDFS

Re: How to reset Ambari admin password in Ambari 2...

Re: Discovering existing Hive tables in Atlas

Re: Does Distcp use Tez now in HDP 3.0 instead of ...

Re: hive server2 interactive logs

Re: Spark-submit is forcing an earlier protobuf ve...

Re: Can we load csv from local directory to Hbase ...

Re: Can we load csv from local directory to Hbase ...

Re: Process multiple dataframes in parallel using ...

Re: what is the latest version I can use with spar...

Re: what is the latest version I can use with spar...

Re: what is the latest version I can use with spar...

Re: Kafka topic rename

Re: Create Bucketed hive table from spark.

Re: how to generate tag number by API