- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
rename columns of the dataframe
- Labels:
-
Apache Spark
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi I have a dataframe (loaded CSV) where the inferredSchema filled the column names from the file. I am trying to get rid of white spaces from column names - because otherwise the DF cannot be saved as parquet file - and did not find any usefull method for renaming.
The method withColumnRenamed("Company ID","Company_ID") works, but I need to repeat it for every column in the dataframe. I tried to to use toDF method,
such as:
val dfnew = df.toDF( df.columns.map( a => a.replace(" ","_") ) );
but it failed.,
Any ideas?
Created 01-15-2016 07:00 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I have found a solution to this:
df.registerTempTable("tmp");
val newdf = sqlContext.sql(""" select 'Company ID' as Company_ID, 'Product ID' as Product_ID, .. from tmp""");
newdf.saveAsParquetFile(...);
T.
Created 01-15-2016 07:00 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I have found a solution to this:
df.registerTempTable("tmp");
val newdf = sqlContext.sql(""" select 'Company ID' as Company_ID, 'Product ID' as Product_ID, .. from tmp""");
newdf.saveAsParquetFile(...);
T.
Created 01-15-2016 07:19 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
"""select `Company ID` as Company_ID, .... """
