https://community.cloudera.com/t5/Support-Questions/Rowwise-manipulation-of-a-DataFrame-in-PySpark/m-p/226337#M188197 <P>I have a PySpark dataframe with 87 columns. I want to pass each row of the dataframe to a function and get a list for each row so that I can create a column separately.</P><P>`</P><H2>PySpark code</H2><H2>UDF:</H2><P><CODE>def make_range_vector(row,categories,ledger): print(type(row),type(categories),type(ledger))<BR /> category_vector=[] for category in categories: if(row[category]!=0): category_percentage=func.round(row[category]*100/row[ledger]) category_vector.append(category_percentage) else: category_vector.append(0) category_vector=sqlCtx.createDataFrame(category_vector,IntegerType())<BR /> return category_vector </CODE></P><H2>Main function</H2><P><CODE> pivot_card.withColumn('category_debit_vector',(make_range_vector(struct([pivot_card[x] for x in pivot_card.columns] ),pivot_card.columns[3:],'debit'))) </CODE></P><P>I am beginner in PySpark, and I can't find answers to below questions.</P><OL><LI><P>Print statement outputs <CODE>&lt;class 'pyspark.sql.column.Column'&gt; &lt;class 'list'&gt; &lt;class #'str'&gt;</CODE>. Shouldn't it be StructType?</P></LI><LI><P>Can I pass a Row object and do something similar, like we do in Pandas ?</P></LI></OL> Mon, 12 Aug 2019 18:33:59 GMT preethyvarma 2019-08-12T18:33:59Z