Expert Contributor
Posts: 71
Registered: ‎03-04-2015
Re: Cannot pass value from Hive query output directly in workflow

Thanks ... here's what happened:

 

  • LinuxContainerExecutor was a dead-end.  Set up all the config as recommended - Yarn switched to user 'nobody', but not the WF owner ('myao').  This put MR out of order cluster-wide - even Hive CLI stopped working.  yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users = false has no effect ... I was kinda expecting that given that YARN-2424 was for Hadoop 2.6.0, and our CDH 5.2.0 has 2.5.0.
  • Your workaround #2, set HADOOP_USER_NAME=myao in shell action, succeeded.  I was able to run "hive -e <query>" and capture the result.

 

To summarize, the minimal manual config to get shell action working normally as workflow owner for us:

  • HADOOP_CONF_DIR=/etc/hive/conf
  • HIVE_CONF_DIR=/etc/hive/conf
  • HADOOP_USER_NAME=${wf:user()}

 

Appreciate your advices.  Please feel free to comment and share your experiences!

 

 

View solution in original post

Who Me Too'd this solution