Hi, Getting below error when trying to run a spark streaming job to read from Kafka to HBase.Not getting any error when I am creating a jar file but getting error when I am running spark submit command.  Any help is much appreciated.   Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/client/Put ... View more

Hi, when I run it in spark-shell, it will execute fine, but when I run the same statement in scala program it error out as "overloaded method value".   Spark-Shell: spark.read.json(dataframe.select("col_name").as[String]).schema   After research I found that in scala we need to pass rdd as an argument instead of dataframe.   is there a way to make Spark-Shell statment to run in scala without converting it as an rdd.   Note: because my dataframe has both straight column values and json columns. above statement takes only json value as input.   Help is much appreciated.   Thanks, Waseem   ... View more

Hi,   How do I compute dataframe record count without re-running dataframe. I mean can we pull this information from any spark stats table?   Few options I am aware of are: 1. dataframe.cache() -- Don't want to store result in memory. 2. dataframe.describe("col").show -- again it will re-run the dataframe to get count. 3. dataframe.count().show() -- again it will re-run the dataframe to get count.   Thanks!   Thanks, Waseem   ... View more