Issue with handle kafka data into hive table in hdfs

Striver — Fri, 16 Sep 2022 13:53:15 GMT

I just used streaming to handle kafka data, and write it into hive table in hdfs.The hive table was partitioned by month/day which the time in the kafka data and I compute the month/day with it as below:

val sdf=new SimpleDateFormat("yyyy_MM")
val sdf1=new SimpleDateFormat("dd")
val adjustTime = data.getLong(12)
val month = sdf.format(new Date(adjustTime))
val day = sdf1.format(new Date(adjustTime))

and I used repartition when I parse the data

repartition($"month",$"day").write.mode(SaveMode.Append).partitionBy("month","day")

when I checked the data in hdfs I found the problems:

1.The partitioned day appeared like month="2018_09" day ="31",the problem is that Sep could not have 31th.

2.in the someday partition, the data is not belog to this day, just like the adjusttime is 2018-09-30,but the data in partition 2018_09_30 have more data with other time like 2018_03_08 and ... The data in it is not correct.

So I will be appreciated for it if any suggestions or ideas to solve these problems. Thanks~

Re: Issue with handle kafka data into hive table in hdfs

Striver — Thu, 06 Dec 2018 09:23:06 GMT

The Problem was solved because of the thread unsafe SimpleDateFormat method，replaced the method with

val fdf = FastDateFormat.getInstance("yyyy_MM", tz)

checked the data is fine now .

question Re: Issue with handle kafka data into hive table in hdfs in Archives of Support Questions (Read Only)

Issue with handle kafka data into hive table in hdfs

Re: Issue with handle kafka data into hive table in hdfs