About YannBarraud

YannBarraud · ‎03-23-2016

Hi, Parcels are part of it. But then how do I create and activate virtual envs for a specifics jobs ?

YannBarraud · ‎03-23-2016

Hi, As many of you I guess, I mix pyspark jobs and regular pandas dataframes and scikit-learn for data science. But I'm sharing the platform with many other data scientists and we might end with a libs mess. I'd love to be able to have conda envs for projects I could activate separately and in parallel for each job context. Manageable ? Thanks. Yann

YannBarraud · ‎02-27-2016

Hi guys, Thanks for anwsers. Wiuld you recommend using Oozie and refer spark jars like in this post or use SharedLibs (which would resukt in a mess if I have like 3 Spark versions in this folder) ? Cheers, Yann

YannBarraud · ‎02-19-2016

Hi, Is there a way to have various Spark version running on the cluster, and specifying which version to use at job startup ? Thanks. Cheers, Yann

Online	Offline
Last Visited	‎03-23-2016 10:21 AM

Date Registered	‎02-19-2016 09:01 AM
Last Visited	‎03-23-2016 10:21 AM
Total Messages Posted	4

Re: Conda env

Conda env && Spark Jobs

Re: Have various Spark version running on the clus...

Have various Spark version running on the cluster