Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Do the Spark REPLs have a way to list current variables?

Solved Go to solution

Do the Spark REPLs have a way to list current variables?

I'm specifically using pyspark and I'm wondering if there is something similar to Pig's "aliases" command that shows all currently available variables. If there is something like that in pyspark, I'm just missing it and I hope someone straightens me out! ;-) I'm not using spark-shell much, but knowing how to do this in that REPL would be useful, too.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Do the Spark REPLs have a way to list current variables?

New Contributor

pyspark:

  • dir(): the list of in scope variables
  • globals(): the dictionary of global variables
  • locals(): the dictionary of local variables

spark-shell:

  • $intp.allDefinedNames: the list of all defined variables
4 REPLIES 4

Re: Do the Spark REPLs have a way to list current variables?

New Contributor

The pyspark shell is just Python too. So using dir() should show all existing python variables (although it also shows all imports and a bunch of things you may not be looking for).

Re: Do the Spark REPLs have a way to list current variables?

Expert Contributor

$intp.definedTerms.foreach(println)

That will print all the variables.

Re: Do the Spark REPLs have a way to list current variables?

New Contributor

pyspark:

  • dir(): the list of in scope variables
  • globals(): the dictionary of global variables
  • locals(): the dictionary of local variables

spark-shell:

  • $intp.allDefinedNames: the list of all defined variables

Re: Do the Spark REPLs have a way to list current variables?

New Contributor

If you keep using `globals()` you will eventually get an error, as it keeps adding itself to itself and you eventually get one of the following errors:

RuntimeError: maximum recursion depth exceeded while getting the repr of a list

or

RuntimeError: dictionary changed size during iteration
Don't have an account?
Coming from Hortonworks? Activate your account here