Setting this property creates a tmp directory in BOTH local and HDFS. It does so in HDFS because other properties use hadoop.tmp.dir as a base path to store data in HDFS. Example: dfs.name.dir=${hadoop.tmp.dir}/dfs/name creates this path in hdfs. There is no way to have this property NOT create a path locally. See these links for a good discussion: https://stackoverflow.com/questions/2354525/what-should-be-hadoop-tmp-dir https://stackoverflow.com/questions/40169610/where-exactly-should-hadoop-tmp-dir-be-set-core-site-xml-or-hdfs-site-xml ... View more

Using your sed approach, this should replace all NULL with empty character sed 's/[\t]/,/g; s/NULL//g' > myfile.csv If there is a chance that NULL is a substring of a value you will need to do the following where ^ is beginning of line and $ is end of line and , is your field delimiter sed 's/[\t]/,/g; s/^NULL,/,/g; s/,NULL,/,,/g; s/,NULL$/,/g;' > myfile.csv Note that if your resultset is large, it is probably best to use Pig on HDFS and not sed (to leverage the parallel processing of hadoop and save yourself a lot of time. Note also: To use empty character as nulls in the actual hive table, use the following in the DDL TBLPROPERTIES('serialization.null.format'=''); ... View more

Both are similar in their awesome drag-and-drop UI to process data in motion, However, they differ fundamentally in purpose and underlying technology. Differences Purpose NiFi is meant for data flow management while Streaming Analytics Manager (SAM) is meant for advanced (complex) real-time analytics. In general, for NiFi think acquiring, transforming and routing data to target destinations and for SAM think complex analytics on data as it is flowing across the wire. Here is a more detailed comparison between flow management (NiFI) and stream analytics (SAM) Flow Management (NiFi) Stream Analytics (SAM) data velocity batch, microbatch or streaming (from diverse sources) streaming (from diverse sources) data size (per content) small (kb) to large (GB) small (KB, MB) per message in stream data manipulation rich: parse, filter, join, transform, enrich, reformat minimal changes to data data flow management powerful: queue prioritization, back pressure, route/merge, persist to target minimal: mostly route/merge and persist to target real-time analytics basic powerful So NiFi is great to manage the movement of data from diverse sources (from small sensors, ftp locations, relational databases, rest apis in the cloud, and so on) to similar targets while modifying and making decisions on the data in between. SAM is great at watching real-time streams of data and doing advanced analytics (dashboarding/visualizations, alerting, predictions, etc) as it flows by. Technology NiFi is built around processors and connections with repositories underneath. SAM is built on top of Storm and Kafka (and Druid). Shared What do they have in common? Both have easy UI development that hides complexity underneath. Both are components of Hortonworks Data Flow (HDF) distribution. Both share Kafka (see below). Both are managed by the Ambari (admin and monitoring) and Ranger (authorization and security). Both can use the same Schema Registry to work with data structure of content. Do they connect? A very common pattern is this: stream data using NiFi (and possibly filter, transform, enrich) and pass it to a Kafka queue to make it durable (persistent until consumed). SAM pulls from the queue (subscribes to a topic) and does advanced analytics from there (dashboarding/visualizations, alerting, predictions, etc). SAM pushes to hadoop (HBase or Hive) to persist for further historical analysis and exploration (data science, business intelligence, etc) Tutorial mentioned by @Wynner is an excellent example of this pattern and the separate strengths of NiFi and SAM. ... View more

@Aditya Jadhav Small mistake: you need uppercase PigStorage('|'). lp = load '/employee.txt ' using PigStorage('|') as (aa,bb,cc,dd,ee); Error shows that it is looking for a java function called pigStorage and cannot find it. In addition to Pig's native functions (which PigStorage belongs) functions can be found in referenced libraries (e.g. 3rd party or ones you build yourself as User Defined Functions). ... View more