About kevin_jacobs

kevin_jacobs · ‎10-17-2018

I have some questions regarding the profiler. How can the output of the PROFILER_GET function be used by other distributed processing applications? We would like to visualize features computed by the profiler and aggregated features computed by our models. Therefore, we need the output of the profiler as input for Pyspark in which we implemented our models. An other option would be to implement our pipeline completely in Pyspark, but then we don't have access to the already implemented windowing features of the profiler and since the profiler is a build-in functionality for Metron, we are not sure about the downsides about not using the profiler at all. What would be possible downsides if we implemented our pipeline completely in Pyspark? And even without Pyspark, how is it possible to visualize the profiler data and baselines in Zeppelin? We have more questions about the profiler, these are our critical ones.

kevin_jacobs · ‎10-17-2018

I have some questions regarding the profiler. How can the output of the PROFILER_GET function be used by other distributed processing applications? We would like to visualize features computed by the profiler and aggregated features computed by our models. Therefore, we need the output of the profiler as input for Pyspark in which we implemented our models. An other option would be to implement our pipeline completely in Pyspark, but then we don't have access to the already implemented windowing features of the profiler and since the profiler is a build-in functionality for Metron, we are not sure about the downsides about not using the profiler at all. What would be possible downsides if we implemented our pipeline completely in Pyspark? And even without Pyspark, how is it possible to visualize the profiler data and baselines in Zeppelin? We have more questions about the profiler, these are our critical ones regarding the profiler.

kevin_jacobs · ‎08-02-2018

The Python Anaconda interpreter is a perfect interpreter for our situation. However, we failed to get it working. We installed Anaconda in /opt/anaconda3/. So far, so good. The next thing we did, is trying to print information about Anaconda. This also succeeds (see s1.png). Then, we tried to activate the environment and list the information a second time. In the information, no environment is activated (see s2.png). Then, if we try to execute some Python code, nothing is shown. Only "Took 0s" (see s3.png). What could be the cause of these issues? How can we fix Python? I also tried it with Anaconda2 instead of Anaconda3; same result.

kevin_jacobs · ‎08-01-2018

Thank you for your answer. We tried it. It worked using spark-submit on the command line, but it did not work in Zeppelin (in which we configured the config flags for Spark). What we then tried, was setting the Spark submit options in Ambari (with no luck): export SPARK_SUBMIT_OPTIONS="--conf spark.pyspark.virtualenv.enabled=true --conf spark.pyspark.virtualenv.bin.path=/usr/local/bin/virtualenv --conf spark.pyspark.virtualenv.python_version=3.6 --conf spark.pyspark.virtualenv.requirements=/home/zeppelin/requirements.txt"

kevin_jacobs · ‎08-01-2018

Currently, we have one big Data Science Python environment and we would like to be able to create multiple environments for different tasks. We build the environment on one machine on the cluster and then distribute it manually to all other nodes. Is there an easy way to ship one environment to all the nodes in a cluster? How can we activate a different environment per notebook without having to create a different interpreter (Spark / Livy) for each environment?

Online	Offline
Last Visited	‎10-17-2018 12:33 PM

Date Registered	‎08-01-2018 10:11 AM
Last Visited	‎10-17-2018 12:33 PM
Total Messages Posted	5

Questions regarding the profiler

Re: What are the machine learning (ML) capabilitie...

Python / Anaconda interpreter in Zeppelin

Re: Different Python environments

Different Python environments