About DURAISAM

DURAISAM · ‎11-11-2021

while running the spark job , we are facing "com.esotericsoftware.kryo.KryoException: Buffer underflow." . this is happening intermittently . Logic :Read few tables via spark-sql and write into hdfs using DS.insertinto(..). here insertInto only makes an action in the flow of the code. Analysis : As checked in spark-history UI ,while execution moves from one stage to another stage it happens. Env: CDH 7.x., Spark 2.x with Java

DURAISAM · ‎08-16-2021

Hi, I am going with workaround as of now for this issue. Workaround: Drop partition explicit via code from metadata

DURAISAM · ‎07-28-2021

Hi , Are you manually removing the partitions? Yes . we cant use "set hive.msck.path.validation=ignore" because if we run msck repair .. automatically to sync HDFS folders and Table partitions right? this is not happening and no err. our aim: Make HDFS path and partitions in table should sync in any condition

DURAISAM · ‎07-28-2021

Hi, Mostly the issue occured due to partitions path in HDFS based on defined partitions in the table if partition columns and HDFS path matches, this issue never occur

Online	Offline
Last Visited	‎11-11-2021 08:04 PM

Member Since	‎07-26-2021 02:07 AM
Last Visited	‎11-11-2021 08:04 PM
Posts	8

Cloudera Community

com.esotericsoftware.kryo.KryoException: Buffer un...

Re: In Spark 3.1.1 , Failed while InsertInto with...

Re: CDH 7.1 : MSCK Repair is not working properly...

Re: MSCK repair issue