Hi @Srini Nalluri. I am not 100% sure if i have understood your question correctly, but are you mostly interested in manipulating the attributes of the data you are passing through as flowfiles, rather than the data itself? If so, Have you seen the NiFi Expression Language guide here https://nifi.apache.org/docs.html under General -> Expression Language Guide? It describes the ways in which we can manipulate attributes. For 1, you could use an UpdateAttribute processor and use the jsonPath function to pull out various individual values from your JSON attribute and assign them to new attributes. There are some good examples in the language guide: For 2, you could set the Put Response Body In Attribute property in your InvokeHTTP processor to store it as an attribute, and then use the jsonPath expression language function in an UpdateAttribute processor to evaluate it. For 3, in an UpdateAttribute processor you could use the append/prepend functions to merge or concatenate two attribute values together, e.g. (where attribute1 and attribute 2 have previously been set): Or, there is an AttributesToJSON processor https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.AttributesToJSON/index.html if you need to get your new attributes back into JSON format. Hope it helps ... View more

You can use Pig with HCatalog to load your data from the table and SPLIT it into your 142 aliases if you know exactly how you want the data separated. You would then have to manually force each alias to be stored into a single file using PARALLEL 1 (one STORE command per alias/file). These multiple STORE commands should execute concurrently. Note that you can't choose the name of the files that will be output, so it is a good idea to use a meaningful output directory names instead. If however, your use case has one condition that determines which records go into which file, e.g. you want all the records with the same name to go into the same file, you can use the Multistorage function. A quick example in the answer here on stack overflow. Hope this helps! ... View more

Hi @Kaliyug Antagonist, could you please share a full Sqoop command? Are you using sqoop to import directly to your Hive table? Have you tried using the drop only (and not a replace)? ... View more

Hi @Kaliyug Antagonist A similar question on stack overflow (http://stackoverflow.com/questions/28076200/hive-drop-import-delims-not-removing-newline-while-using-hcatalog-in-sqoop) suggests explicitly specifying the column as the Java String type to get --hive-delims-replacement and --hive-drop-import-delims working correctly: --map-column-java administrationnote=String You can use column type mapping for either Hive types or Java types, both might be worth a try. Thanks! ... View more

Hi @Ashish Vishnoi. In the sqoop-export doco, it says: "The --input-null-string and --input-null-non-string arguments are optional. If --input-null-string is not specified, then the string "null" will be interpreted as null for string-type columns. If --input-null-non-string is not specified, then both the string "null" and the empty string will be interpreted as null for non-string columns. Note that, the empty string will be always interpreted as null for non-string columns, in addition to other string if specified by --input-null-non-string " There is another discussion here around a similar issue. If it doesn't work even for string columns, it may be that a workaround of some kind is needed. E.g. conversion of blanks to another character (or set of characters that wouldn't normally be a part of your data set) prior to export which can be converted back to blanks once in Teradata. Hope this helps. ... View more

Hi @Kashif Khan My understanding is that Hive does not support materialised views at this stage (i.e. saving/running on a point in time snapshot). Some links on Hive locking here, but it sounds as though you have this set up already: https://cwiki.apache.org/confluence/display/Hive/Locking#Locking-LockinginHiveTransactions https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-Locking There is also a blog (from roughly two years ago) that may help if your use case has room to point user queries at views/alternative tables as described here http://hortonworks.com/blog/four-step-strategy-incremental-updates-hive Following, in case someone has another clever work-around. ... View more

Hi @Sridharan Govindaraj. Did running your command from Unix shell instead of HBase shell solve this issue? You might also want to fully qualify your hdfs file path: $ hbase org.apache.hadoop.hbase.mapreduce.ImportTsv -Dimporttsv.separator="," -Dimporttsv.columns="HBASE_ROW_KEY,id,temp:in,temp:out,vibration,pressure:in,pressure:out" sensor hdfs:///user/hbase.csv ... View more

Also I found a similar question that may be helpful re Journal Nodes: https://community.hortonworks.com/questions/17483/is-it-recommended-to-use-same-set-of-journal-nodes.html ... View more

Hi @DumindaJ - while this is not supported, I am still interested in any theoretical answers/suggestions to your question. Did you find any information? ... View more

Thanks @mkataria ... View more

Hi @mkataria. Just wondering what your use case is (don't have to say anything business specific)? E.g., is this for a production and disaster recovery type scenario? Or to decommission an old cluster? Or something else? Just curious, thanks! ... View more

Excellent reference, thanks! ... View more