question Re: How to execute shell commands using ExecuteProcess processor in a kerberized environment ? in Support Questions

How to execute shell commands using ExecuteProcess processor in a kerberized environment ?

shishir_saxena4 — Fri, 19 Feb 2016 08:57:53 GMT

I have a Kerberized cluster and a shell script containing some sqoop commands. Can we use NiFi ExecuteProcess processor to execute this shell script ?

I don't want to use ExecuteSQL processor because sqoop script has some processing logic that can't be easily replicated in NiFi.

Re: How to execute shell commands using ExecuteProcess processor in a kerberized environment ?

aervits — Fri, 19 Feb 2016 19:29:49 GMT

@Shishir Saxena execute process is intended for Linux commands, nifi is not a scheduling tool, you still need to use Oozie. Especially if you want to maintain order of your transactions. Oozie is still the solution for that with sqoop job maintaining the last record processed in the sqoop metastore

Re: How to execute shell commands using ExecuteProcess processor in a kerberized environment ?

nsabharwal — Fri, 19 Feb 2016 19:39:06 GMT

@Shishir Saxena

You are confusing the NiFi functionality.

Solution: See this example https://prodlife.wordpress.com/2013/11/22/using-oozie-in-kerberized-cluster/

Re: How to execute shell commands using ExecuteProcess processor in a kerberized environment ?

shishir_saxena4 — Fri, 19 Feb 2016 21:23:56 GMT

Thank You for clarifying.

Re: How to execute shell commands using ExecuteProcess processor in a kerberized environment ?

aervits — Fri, 19 Feb 2016 21:24:28 GMT

@Shishir Saxena you can wrap the sqoop command in shell script and use ExecuteProcess if you'd like, would love to see it as an article on HCC when you get it done.

Re: How to execute shell commands using ExecuteProcess processor in a kerberized environment ?

shishir_saxena4 — Fri, 19 Feb 2016 21:24:57 GMT

Thank you for above link.

Re: How to execute shell commands using ExecuteProcess processor in a kerberized environment ?

aervits — Fri, 19 Feb 2016 21:25:08 GMT

questions is re: Nifi not Oozie

Re: How to execute shell commands using ExecuteProcess processor in a kerberized environment ?

nsabharwal — Fri, 19 Feb 2016 21:30:01 GMT

@Shishir Saxena Never hurt to look into alternate solutions @Artem Ervits I know it's NiFi question but there is context to Oozie and kerberos use too.

Re: How to execute shell commands using ExecuteProcess processor in a kerberized environment ?

shishir_saxena4 — Fri, 19 Feb 2016 22:02:50 GMT

Thanks @Artem Ervits. This is what I originally intended, but ExecuteProcess processor doesn't have any option to specify kerberos credentials.

Re: How to execute shell commands using ExecuteProcess processor in a kerberized environment ?

aervits — Fri, 19 Feb 2016 22:05:20 GMT

ok good, but in general my original answer still applies, Nifi is not an Oozie and Sqoop replacement. @Shishir Saxena you need either Change Data Capture (CDC) like Golden Gate or Attunity Replicate or Sqoop job with metastore keeping track of all rows processed. Good luck!

Re: How to execute shell commands using ExecuteProcess processor in a kerberized environment ?

RaviPapisetti — Sat, 22 Oct 2016 05:05:38 GMT

Hi, I got into very similar issue. Did you find a way to run shell with kerberos principal. We are trying to do the same, wrap the sqoop script inside shell and use ExecuteProcess processor. It fails to execute with GSS credentials exception.

We tried to start the nifi service with 'nifi' user that has nifi service principal. We want nifi to use user kerberos principal while running this sqoop script.

Any thoughts?