About winnie_philip

winnie_philip · ‎05-09-2018

I'm working on a NIFI flow to update incoming JSON file data using UpdateRecord processor. Incoming JSON sample data is provided below: [ { "created_at" : "Tue May 08 16:05:42 CDT 2018", "id_store" : 1, "event_type" : "generated", "id_transaction" : "7276464206868934013", "id_product" : 1, "value_product" : 1100152002, "prod_desc" : "MapRecord[{id_product=appleA}]" } ] I used HBaseLookup Service to lookup "id_product" value from incoming data in HBase database to derive "prod_desc" value. HBase database lookup is working perfectly. The lookup process returned the value "MapRecord[{id_product=appleA}]" under "prod_desc" field for id_product value of "1". Now I'm using UpdateRecord processor to remove everything else except the value "appleA". Attached is the screenshot for UpdateRecord processor setting. It seems not replacing the value as I expected, instead the value "null" is returned under "prod_desc" field. I'm not sure what went wrong in my setting.

winnie_philip · ‎03-05-2018

Thanks Tim. NiFi Flow worked with HDF version of NiFi.

winnie_philip · ‎03-05-2018

The PutHiveStreaming worked w/ HDF build of NiFi version without any problem. I downloaded nifi-1.2.0.3.0.0.0-453-bin.tar.gz from HDF, and installed it on a node as standalone application, and file streamed to hive table without any issues. Thanks for your help. .

winnie_philip · ‎02-27-2018

Thanks Timothy. The cluster is not Kerberized. I can access Hive from command line of that machine. I will probably have to try installing HDF version of NIFI. Attached is the NIFI app log. nifi-applog.txt

winnie_philip · ‎02-27-2018

Thanks Timothy. This could be a reason, I don't have a NIFI user that I've created. How do I create a NIFI user? The table(OLYMPICS) is in default schema. Here is the DDL for the Hive table. --created hive table CREATE TABLE OLYMPICS(CITY STRING,EDITION INT,SPORT STRING,SUB_SPORT STRING,ATHLETE STRING,COUNTRY STRING,GENDER STRING,EVENT STRING,EVENT_GENDER STRING,MEDAL STRING) CLUSTERED BY (EDITION)INTO 3 BUCKETS ROW FORMAT DELIMITED STORED AS ORC LOCATION '/tmp/olympics' TBLPROPERTIES('transactional'='true');

winnie_philip · ‎02-26-2018

I'm working on NIFI 1.5.0. I downloaded Apache NIFI and installed on one of my HDP cluster nodes. I'm running the test against a 3 nodes HDP 2.6 cluster. I'm following a tutorial from following link: https://community.hortonworks.com/articles/52856/stream-data-into-hive-like-a-king-using-nifi.html Everything works fine, but I get error at PutHiveStreaming step. It's basically complaining about an error connecting against Hive endpoint and the table. The table in Hive is created as managed table, and not as an external table. I can telnet to Hive metastore server and port from my NIFI node. What would be root-cause of this issue?nifi-error2.png Error attached: nifi-error.png

winnie_philip · ‎10-06-2017

Thank you Sridhar!

winnie_philip · ‎10-06-2017

Thank you Geoffery

winnie_philip · ‎10-06-2017

Also, I went ahead and ran commands for all other nodemanager yarn directory chain on all nodes: sudo chown -R yarn:hadoop -R /appl/hadoop/yarn sudo chown -R yarn:hadoop /hadoop/yarn/ sudo chown -R yarn:hadoop /usr/db/hadoop/yarn/ sudo chown -R yarn:hadoop /usr/local/opt/hadoop/yarn/ Thank you very much for all help from everyone! I really appreciate!!

winnie_philip · ‎10-06-2017

I ran the command with -R, looks like fixed it, I will watch for errors little bit, I restarted YARN also, this directory tree had mixed owner:grp. chown yarn:hadoop -R /appl/hadoop/yarn

Online	Offline
Last Visited	‎07-19-2018 04:36 PM

Member Since	‎08-11-2017 12:58 AM
Last Visited	‎07-19-2018 04:36 PM
Posts	51
Kudos received	2

Cloudera Community

NiFi UpdateRecord processor is not updating JSON p...

Re: NIFI HiveStreaming error

Re: NIFI HiveStreaming error

Re: NIFI HiveStreaming error

Re: NIFI HiveStreaming error

NIFI HiveStreaming error

Re: Hive error on HDP6

Re: Hive error on HDP6

Re: Hive error on HDP6

Re: Hive error on HDP6