question Re: Iterate a dataframe in Archives of Support Questions (Read Only)

Iterate a dataframe

nanyim_alain — Tue, 14 Jun 2016 15:43:54 GMT

Hello, Please I will like to iterate and perform calculations accumulated in a column of my dataframe but I can not. Can you help me? Thank you

Here the creation of my dataframe. I would like to calculate an accumulated blglast the column and stored in a new column

from pyspark.sql import HiveContext from pyspark import SparkContext from pandas import DataFrame as df sc =SparkContext() hive_context = HiveContext(sc) tab = hive_context.table("table") tab.registerTempTable("tab_temp") df=hive_context.sql("SELECT blglast FROM tab_temp AS b limit 50")

df.show()

Re: Iterate a dataframe

rajkumar_singh — Tue, 14 Jun 2016 16:10:06 GMT

after df=hive_context.sql("SELECT blglast FROM tab_temp AS b limit 50"), you can get the row object in which you can perform your custom logic.

>>> for row in df.rdd.collect():

... dosomething ...

Re: Iterate a dataframe

sandyy006 — Tue, 14 Jun 2016 18:11:27 GMT

iterate = df.map(lambda p: "Name: " + p.blglast )
for iteration in iterate.collect():
  print(iteration)

@alain TSAFACK

Please refer : http://spark.apache.org/docs/latest/sql-programming-guide.html for more information

Re: Iterate a dataframe

nanyim_alain — Tue, 21 Jun 2016 15:30:05 GMT

Hello, Thank you for the directive. But I 'm new to the dataframe and what I try to do is be able to make it to retrieve the values of the indices i and i + 1 for example. Best regards