Reply
Explorer
Posts: 12
Registered: ‎01-11-2017

Hive-site.xml in spark conf dir removed after each redeploy

[ Edited ]

Hello,

 

We are using Zeppelin to run spark job that require hive databases access. In the first time it didn't worked because of the missing hive-site.xml in spark2 conf directory.

 

Now it works fine except that each time we redeploy the config via cloudera manager the spark2 conf folder is crushed with the new one and we lost the hive-site.xml.

 

Any suggestion on how avoid this ?

 

Many thanks,

Regards,

Guillaume,

Posts: 453
Topics: 13
Kudos: 75
Solutions: 41
Registered: ‎09-02-2016

Re: Hive-site.xml in spark conf dir removed after each redeploy

@gvd07

 

Have you tried soft link option instead of copy paste hive-site.xml ?

Explorer
Posts: 12
Registered: ‎01-11-2017

Re: Hive-site.xml in spark conf dir removed after each redeploy

@saranvisa Yes, I did a symlink.

 

Note that this could also be a Zeppelin's issue since I can list Hive DB for spark2-shell without the hive-site symlink in spark2/conf dir.

 

whitout hive-site :

 

2017-03-10_16h36_20.png

 

With :

 

2017-03-10_16h06_05.png 

Cheers,

 

 

Explorer
Posts: 12
Registered: ‎01-11-2017

Re: Hive-site.xml in spark conf dir removed after each redeploy

Hello,

 

I have some news from Zeppelin side, the hive-site.xml is required in the spark folder to support multi-version as dsicribed here : https://issues.apache.org/jira/browse/ZEPPELIN-2244

 

For now I have to create the symlink to hive-site.xml again and again after each config deploy. Any workaround to solve that ?

 

Regards,

Explorer
Posts: 12
Registered: ‎01-11-2017

Re: Hive-site.xml in spark conf dir removed after each redeploy

Hello,

 

Nobody have an advice ? Is there a way I can create the symlik while the config deploy from a .sh or ?

 

Regards,

Highlighted
Explorer
Posts: 24
Registered: ‎01-07-2016

Re: Hive-site.xml in spark conf dir removed after each redeploy

We are feeling the same pain here.

 

In cloudera manager there is usually a "safety valve" for relevant configuration files where you get the opportunity to tweak the configuration for each role. In the Spark2 section in cloudera manager there is no safety valve for hive-site.xml.

 

BR,

Petter

Announcements