You will need to install the hadoop binaries and configs on the edge node. That will give you the command line clients that come out of the box with HDP, like pig and beekeeper. See the following on edge nodes: http://www.dummies.com/programming/big-data/hadoop/edge-nodes-in-hadoop-clusters/ https://dwbi.org/etl/bigdata/187-set-up-client-node-gateway-node-in-hadoop-cluster See this post making the good point of installing hadoop binaries via Ambari so they are always up to date with the rest of the cluster. https://community.hortonworks.com/questions/39568/how-to-create-edge-node-for-kerberized-cluster.html ... View more

@Magesh Kumar I believe this question is identical to another one you asked: https://community.hortonworks.com/questions/63947/incremental-flat-file-data-loading-into-hadoop.html#answer-63978 If there are differences, please elaborate. ... View more

You must have a table with a column updating the modifed date (date of insert or update). If you have this column, then use: --check-column {modified_date_col} --incremental {lastmodified} --last-value {modified_date}. If you do not have this column, you cannot avoid your issue. ... View more

The first and most straightforward approach is simply to run tail -f | grep <ProcessorName> against log/nifi-app.log and append it to a log. Ex tail -f log/nifi-app.log | grep ExecuteScript >> ExecuteScript.log You could even use NiFi itself. Create a flow to tail the log file, extract text that you want (e.g. ExecuteScript) and route this to a file. GetFile (set for tail) -> ExtractText -> PutFile. You could make this as complex as you like (e.g one flow but conditionally route extracted text to each target file based on the processor name). Finally, you could build scripts around the NiFi Rest API to monitor your flows and processors: https://nifi.apache.org/docs/nifi-docs/rest-api/ E.g. API for Reporting Tasks, Flow, System Diagnostics. ... View more

For Hive, make sure the HiveServer2 service is running (use Ambari) and then use the beeline command line shell: https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients (Note: beeline replaces an older Hive CLI. Beeline is the preferred way moving forward). For Pig, write your script to a file and then use pig <pigscriptFileName> See the following for more (including passing parameters, running locally or against hadoop, etc) http://pig.apache.org/docs/r0.16.0/ https://www.tutorialspoint.com/apache_pig/apache_pig_execution.htm http://pig.apache.org/docs/r0.16.0/start.html#run https://community.hortonworks.com/articles/51884/pig-doing-yoga-how-to-build-superflexible-pig-scri.html (For the Pig tutorials http://hortonworks.com/hadoop-tutorial/how-to-use-basic-pig-commands/ you will see pig scripts written in Pig View in Ambari. You can run these on the command line simply by placing them in their own file and run as shown above). Let me know this is what you are looking for by accepting the answer; else let me know what gaps remain. ... View more

Use TransformXML processor to do it in one shot: https://community.hortonworks.com/articles/29474/nifi-converting-xml-to-json.html https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.TransformXml/ ... View more