Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Inserting 30GB(txt file) text data into ORC stuck at 0%, please help

Inserting 30GB(txt file) text data into ORC stuck at 0%, please help

Expert Contributor

Hi, I had a table with over 100 million records and delimited by | and also separated by quotes, so i used csvserde to ignore quotes to write into hive table. After succesfully done, count(*) and inserting into an orc table is taking forveer and stuck at 0%

I though serde is creating performance issues and deleted quotes fromt he text file and loaded into a regular hive text table. I can do a count(*) on it but writing into an ORC table is stuck at 0% ... i am not sure why. Please help

data size is 30GB and 100million rows.

Thanks so much.

1 REPLY 1
Highlighted

Re: Inserting 30GB(txt file) text data into ORC stuck at 0%, please help

Expert Contributor

i used this property on table tblproperties("skip.header.line.count"="1"); after i took it out, it works fine ... but still not sure why was it an issue.

Don't have an account?
Coming from Hortonworks? Activate your account here