Currently any NiFi templates that are created are not stored anywhere outside of flow.xml.gz unless they are explicitly downloaded using the NiFi UI. I can see them within my flow.xml.gz file and can still download them and import them using the UI, so I am not experiencing any issues in terms of the template function. However it was my understanding that active templates would automatically be persisted in directory conf/templates, or otherwise in a custom location using the nifi.templates.directory property. Is this the correct behaviour? Or is the templates directory more a convenience for downloaded templates to be manually stored into? I have NiFi version 1.1.2. Thanks! ... View more

When specifying fully-qualified paths to copy data between two HA clusters with DistCp, e.g: hdfs://nn1:8020/foo/bar Is the address of nn1 really referring to the where the active HDFS NameNode is, or is it looking for the active ResourceManager? Thanks! ... View more

We currently have a Pig script that just loads in image (blob) data using AvroStorage with a predefined Avro schema, then stores into HBase with HBaseStorage specifying which columns to use. Each row from the original DB consists of a row ID and 5 image columns, although any number of the image columns could be empty/NULL. e.g. KEY IMG1 IMG2 IMG3 1 null blob blob 2 blob null blob In the HBase table, the column family for the images is i, with column names i1, i2, etc. It was my understanding that any cells containing a NULL value would automatically not be stored in HBase, however those cells are being stored as a key-value pair with a full key and an empty value '', rather than not existing. Is this the normal/expected behaviour? If not, what is the best way to get around it? Do I have to project the key with each of the five columns, filter out nulls and store them individually (i.e. store 5 times)? Or could it be related to a difference in the way Avro, Pig and HBase all represent null values? Is there a simple type conversion I could do on all the image columns so that they would automatically not be stored if they are empty? Versions: we are running plain HDP 2.1, soon to be upgraded to 2.4 ... View more

http://hbase.apache.org/0.94/book/important_configurations.html suggests manually managing HBase region splits. Do others in the community do this? If so: Do you have a use case or example regarding the steps required (including setting hbase.hregion.max.filesize) and how/what you use to implement them? and Have you found it worthwhile in terms of effort vs benefits? Thanks ... View more

When copying files from HDFS to a local file system: hdfs dfs -copyToLocal <source> <dest> you have options -crc and -ignoreCrc to turn the checksum files on/off. I am merging/copying out to local using hdfs dfs -getmerge <sourceDir> <destFile> and end up with a hidden .destFile.crc file for each destFile. Is there an equivalent way to turn this function off, or otherwise automatically remove the .destFile.crc if the corresponding destFile is deleted (from the local file system)? Thank you! ... View more

AIM: To grab a daily extract of data stored in HDFS/Hive, process it using Pig, then make results available externally as a single CSV file (automated using bash script). OPTIONS: 1. Force output from Pig script to be stored as one file using 'PARALLEL 1' and then copy out using '-copyToLocal' extractAlias = ORDER stuff BY something ASC; STORE extractAlias INTO '/hdfs/output/path' USING CSVExcelStorage() PARALLEL 1; 2. Allow default parallelism during Pig STORE and use '-getmerge' when copying out extract results hdfs dfs -getmerge '/hdfs/output/path' '/local/dest/path' QUESTION: Which way is more efficient/practical and why? Are there any other ways? ... View more

We have experienced an issue where (re)processing data in Hive overwrites timestamp data. This occurs with HDP 2.1, but not 2.3. We are using Hive to run an ad hoc 'reorg' or 'reprocess' on existing Hive tables to reduce the number of files stored - improving query performance and reducing pressure on the cluster (found a nice explanation from @david.streever here https://community.hortonworks.com/questions/4024/how-many-files-is-too-many-on-a-modern-hdp-cluster.html). The active Hive table is added to daily, creating at least one ORC file per day. The schema contains several timestamp columns (e.g. created_timestamp for when each record was origingally created on the source system). We then create a reorgTable with an identical schema to activeTable, copy the data from activeTable to the reorgTable which combines many of the smaller daily files reducing the overall number. However, this process edits/overwrites timestamp data (and does not touch other columns): 1. Contents of activeTable ID created_timestamp 01 2000-01-01 13:08:21.110 02 1970-01-01 01:02:03.450 03 1990-10-08 03:09:02.780 2. Copy data from activeTable to reorgTable INSERT INTO TABLE reorgTable SELECT * FROM activeTable; 3. Contents of the reorgTable ID created_timestamp 01 1990-10-08 03:09:02.780 02 1990-10-08 03:09:02.780 03 1990-10-08 03:09:02.780 Has anyone else experienced this? Is there a solution other than upgrading? Or an alternative way to reprocess the data that might not have the same effect? Thank you! ... View more