Skip to main content
Atif_Tariq_PK
Atif_Tariq_PK
Silver ⭐️⭐️⭐️
January 24, 2020
Solved

how to read csv have comma in cell and new line character as well

how to read csv have comma in cell and new line character as well . like i have somme columns with description (hi,how are you).

 i am trying to read with spark in scala .

please guide

mridley
mridleyAnswer
Gold ⭐️
January 27, 2020

Can you use sqoop to retrieve the data directly from the database and dump it into Hive? That will solve your delimiter problem.

View in thread

9 replies

mridley
Gold ⭐️
mridley
Gold ⭐️
January 24, 2020

I would recommend not using CSV in your case. If you have commas in the fields then you can't really delimit them with commas because, as you have noticed, you will have field breaks in the middle of a field.

 

Can you get the source data exported some other way?

Translate
    Atif_Tariq_PK
    Atif_Tariq_PKAuthor
    Silver ⭐️⭐️⭐️
    January 25, 2020

    Right and Thank You. What format you suggest?

    Translate
    Atif_Tariq_PK
    Atif_Tariq_PKAuthor
    Silver ⭐️⭐️⭐️
    January 25, 2020

    do you think ? does it work if i copy file direct into mysql then replace commas and after read with spark?

    Translate
    Manoj_hadoop
    Manoj_hadoop
    Silver ⭐️⭐️
    January 24, 2020

    Have you tried using openSerde In DDL?

    Create Table tablename(columns)

    ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde';

     

    Thanks,

    Manoj

    Translate
    Atif_Tariq_PK
    Atif_Tariq_PKAuthor
    Silver ⭐️⭐️⭐️
    January 25, 2020

    actually, i am getting file from ftp and reading using spark. i didn't try to push directly to hive because some enrichment i have to made. this is a big problem for me.

    Thanks

    Translate
    Terms & ConditionsAccessibility statement