question Do the Spark REPLs have a way to list current variables? in Archives of Support Questions (Read Only)

Do the Spark REPLs have a way to list current variables?

LesterMartin — Wed, 11 Nov 2015 21:21:46 GMT

I'm specifically using pyspark and I'm wondering if there is something similar to Pig's "aliases" command that shows all currently available variables. If there is something like that in pyspark, I'm just missing it and I hope someone straightens me out! 😉 I'm not using spark-shell much, but knowing how to do this in that REPL would be useful, too.

Re: Do the Spark REPLs have a way to list current variables?

ofermend — Thu, 12 Nov 2015 00:05:45 GMT

The pyspark shell is just Python too. So using dir() should show all existing python variables (although it also shows all imports and a bunch of things you may not be looking for).

Re: Do the Spark REPLs have a way to list current variables?

jwiden — Wed, 18 Nov 2015 12:18:46 GMT

$intp.definedTerms.foreach(println)

That will print all the variables.

Re: Do the Spark REPLs have a way to list current variables?

afomenko — Tue, 08 Dec 2015 09:42:06 GMT

pyspark:

dir(): the list of in scope variables
globals(): the dictionary of global variables
locals(): the dictionary of local variables

spark-shell:

$intp.allDefinedNames: the list of all defined variables

Re: Do the Spark REPLs have a way to list current variables?

clay_stevens — Wed, 15 Nov 2017 05:06:33 GMT

If you keep using `globals()` you will eventually get an error, as it keeps adding itself to itself and you eventually get one of the following errors:

RuntimeError: maximum recursion depth exceeded while getting the repr of a list

RuntimeError: dictionary changed size during iteration