Spark Streaming Error: java.lang.NoClassDefFoundEr...

S_Waseem · ‎05-05-2020

Hi, Getting below error when trying to run a spark streaming job to read from Kafka to HBase.Not getting any error when I am creating a jar file but getting error when I am running spark submit command. Any help is much appreciated. Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/client/Put

S_Waseem · ‎02-25-2020

Hi, Thanks for reply. but it will spend time in writing dataframe into disk. I am looking for an option like is there a way dataframe counts can be displayed into logs, with spark logger option. intention is to avoid re-running dataframe.

S_Waseem · ‎02-19-2020

Hi, How do I compute dataframe record count without re-running dataframe. I mean can we pull this information from any spark stats table? Few options I am aware of are: 1. dataframe.cache() -- Don't want to store result in memory. 2. dataframe.describe("col").show -- again it will re-run the dataframe to get count. 3. dataframe.count().show() -- again it will re-run the dataframe to get count. Thanks! Thanks, Waseem

S_Waseem · ‎02-06-2020

Hello Sir, It worked... thank you so much!!!

S_Waseem · ‎02-05-2020

sudo su - ${user} -c "beeline -u 'jdbc:hive2://server:port,connectionstring' -e 'insert statement'; " above line works fine if I take out sudo, with sudo it doesnt work, any help or advice is much appreciated. Thanks!

Online	Offline
Last Visited	‎05-20-2020 04:27 PM

Date Registered	‎02-05-2020 06:10 AM
Last Visited	‎05-20-2020 04:27 PM
Total Messages Posted	6

Spark Streaming Error: java.lang.NoClassDefFoundEr...

Re: dataframe count - without re-running it.

dataframe count - without re-running it.

Re: UNIX sudo with beeline is not working.

UNIX sudo with beeline is not working.