In this article we will be creating a flow to
read files from hdfs and insert the same into hive using the putHiveStreaming
processor. Before going to NiFi we need update some configurations in Hive. To enable Hive streaming we need to update the following properties
hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager hive.compactor.initiator.on = true hive.compactor.worker.threads > 0 Coming to NiFi we will be making use of the following processors :
1.ListHdfs + FetchHdfs processor – While configuring the List and Fetch HDFS processors we need to make sure that both these processors run on the primary node only so that the flow files are not duplicated across nodes 2.Convert Json to Avro processor – PutHiveStreaming processor supports input in the Avro format only. So any Json input needs to be converted to avro format 3.PutHiveStreaming processor Lets construct the Nifi flow as below : ListHDFS--> FetchHDFS--> ConvertJsonToAvro-->PutHiveStreaming Configuring the PutHiveStreaming processor Set the values for the above as follows The Hive meta store Uri --- Should be of
the format thrift://<Hive Metastore host>:9083. Note that hive meta store
host is not the same as the hive server host. Hive Configuration Resources – Paths to Hadoop
and hive configuration files. We need to copy the Hadoop and hive configuration
files i.e. Hadoop-site.xml, core-site.xml and hive-site.xml to all the NiFi
hosts. Database Name – the database to which you
want to connect Table name – Table name in which you
want to insert the data. Again note that the
a.ORC is the only format supported
currently. So your table must have "stored as orc" b.transactional = "true" should
be set in the table create statement c.Bucketed but not sorted. So your table
must have "clustered by (colName) into (n) buckets" Auto-create partitions – If set to true hive
partitions will be auto created Kerberos Principal – The Kerberos principal
name Kerberos keytab – the path to the Kerberos
keytab This completes the configuration part. Now we can start the processors to insert data into hive from hdfs.
... View more
