1973
Posts
1225
Kudos Received
124
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 1850 | 04-03-2024 06:39 AM | |
| 2886 | 01-12-2024 08:19 AM | |
| 1594 | 12-07-2023 01:49 PM | |
| 2354 | 08-02-2023 07:30 AM | |
| 3246 | 03-29-2023 01:22 PM |
05-10-2017
07:21 PM
@Bryan Bende, placed the phoenix jar file in NiFi's work directory path and restarted NiFi instance; now seeing a different error in the log when I enable the HBase client service - "org.apache.nifi.StdErr java.lang.NoSuchMethodError: org.apache.hadoop.security.authentication.util.KerberosUtil.hasKerberosKeyTab(Ljavax/security/auth/Subject;)Z" Failed to invoke @OnEnabled method due to java.lang.NoSuchMethodError: org.apache.hadoop.security.authentication.util.KerberosUtil.hasKerberosKeyTab(Ljavax/security/auth/Subject;)Z
2017-05-10 14:13:50,016 ERROR [NiFi logging handler] org.apache.nifi.StdErr [StandardProcessScheduler Thread-6] ERROR org.apache.nifi.controller.service.StandardControllerServiceNode -
2017-05-10 14:13:50,016 ERROR [NiFi logging handler] org.apache.nifi.StdErr java.lang.NoSuchMethodError: org.apache.hadoop.security.authentication.util.KerberosUtil.hasKerberosKeyTab(Ljavax/security/auth/Subject;)Z
2017-05-10 14:13:50,016 ERROR [NiFi logging handler] org.apache.nifi.StdErr at org.apache.hadoop.security.UserGroupInformation.<init>(UserGroupInformation.java:623)
2017-05-10 14:13:50,017 ERROR [NiFi logging handler] org.apache.nifi.StdErr at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1200)
2017-05-10 14:13:50,017 ERROR [NiFi logging handler] org.apache.nifi.StdErr at org.apache.nifi.hadoop.SecurityUtil.loginKerberos(SecurityUtil.java:52)
2017-05-10 14:13:50,017 ERROR [NiFi logging handler] org.apache.nifi.StdErr at org.apache.nifi.hbase.HBase_1_1_2_ClientService.createConnection(HBase_1_1_2_ClientService.java:226)
2017-05-10 14:13:50,017 ERROR [NiFi logging handler] org.apache.nifi.StdErr at org.apache.nifi.hbase.HBase_1_1_2_ClientService.onEnabled(HBase_1_1_2_ClientService.java:178)
2017-05-10 14:13:50,017 ERROR [NiFi logging handler] org.apache.nifi.StdErr at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2017-05-10 14:13:50,017 ERROR [NiFi logging handler] org.apache.nifi.StdErr at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2017-05-10 14:13:50,017 ERROR [NiFi logging handler] org.apache.nifi.StdErr at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2017-05-10 14:13:50,017 ERROR [NiFi logging handler] org.apache.nifi.StdErr at java.lang.reflect.Method.invoke(Method.java:498)
2017-05-10 14:13:50,018 ERROR [NiFi logging handler] org.apache.nifi.StdErr at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:137)
2017-05-10 14:13:50,018 ERROR [NiFi logging handler] org.apache.nifi.StdErr at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:125)
2017-05-10 14:13:50,018 ERROR [NiFi logging handler] org.apache.nifi.StdErr at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:70)
2017-05-10 14:13:50,018 ERROR [NiFi logging handler] org.apache.nifi.StdErr at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:47)
2017-05-10 14:13:50,018 ERROR [NiFi logging handler] org.apache.nifi.StdErr at org.apache.nifi.controller.service.StandardControllerServiceNode$2.run(StandardControllerServiceNode.java:348)
2017-05-10 14:13:50,018 ERROR [NiFi logging handler] org.apache.nifi.StdErr at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
2017-05-10 14:13:50,018 ERROR [NiFi logging handler] org.apache.nifi.StdErr at java.util.concurrent.FutureTask.run(FutureTask.java:266)
2017-05-10 14:13:50,018 ERROR [NiFi logging handler] org.apache.nifi.StdErr at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
2017-05-10 14:13:50,019 ERROR [NiFi logging handler] org.apache.nifi.StdErr at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
2017-05-10 14:13:50,020 ERROR [NiFi logging handler] org.apache.nifi.StdErr at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
2017-05-10 14:13:50,020 ERROR [NiFi logging handler] org.apache.nifi.StdErr at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
2017-05-10 14:13:50,020 ERROR [NiFi logging handler] org.apache.nifi.StdErr at java.lang.Thread.run(Thread.java:745)
2017-05-10 14:13:50,020 ERROR [NiFi logging handler] org.apache.nifi.StdErr [StandardProcessScheduler Thread-6] ERROR org.apache.nifi.controller.service.StandardControllerServiceNode - Failed to invoke @OnEnabled method of HBase_1_1_2_ClientService[id=102e119a-19a2-1409-671f-dddd93a063de] due to java.lang.NoSuchMethodError: org.apache.hadoop.security.authentication.util.KerberosUtil.hasKerberosKeyTab(Ljavax/security/auth/Subject;)Z
... View more
04-28-2017
09:39 PM
3 Kudos
People were asking me when you could throw some more acronyms into an article. So I did that. iBeacons are tiny devices that are broadcasting over Bluetooth Low Energy (BLE) that are very good for advertising and for broadcasting proximity. I have been thinking of the retail space so I purchased a beacon, got a beacon simulator on my phone and found a really cool gateway device. It reads the BLE transmissions from these devices, packages them as JSON MQTT messages and sends them over WIFI to a MQTT Broker of your choice. I am using a cloud hosted MQTT broker to hold these and then Apache NiFi subscribes to these messages. This can happen amazingly fast as MQTT is very light weight, NIFI is fast and BLE is fast. These tiny messages are constantly advertising their identity and other information. With multiple beacons in a building, store, room, facility; you can see who is close to where. As you might expect there's multiple standards for this, iBeacon is Apple's and Google Eddystone. Fortunately this gateway works with both and so can you. Walk into the Physical Web. Use Cases Happy Bubbles Technology In Apache NiFi, it doesn't matter the data or location, it's easy as pie to ingest Beacon data via MQTT and store it in ORC files for Hive table queries. Easy to visualize and analyze the data in Apache Zeppelin against these Hive tables. What I like about this gateway from Happy Bubble technology besides the awesome Hippo logo is that this gateway is dead simple to setup. Plug it in, wait a bit, connect to it's WIFI, set WIFI and MQTT settings. Then you are running! They also provide an open source server you can use to see the data. If BLE messages are ibeacon format, they will get one JSON format. If they are Google Eddystone format it's a second one. And if it's something else, it will get a generic format. I have three listeners in NiFi to grab them all.
Example Messages happy-bubbles-ble
{"rssi":-63, "heap_free":27912}
{"hostname": "happy-bubbles-ble",
"mac": "fa909c522836",
"rssi": -51,
"is_scan_response": "1",
"type": "0",
"data": "020a000816f0ff640000000011094d696e69426561636f6e5f3331303033"}
{"hostname": "happy-bubbles-ble",
"beacon_type": "ibeacon",
"mac": "fa909c522836",
"rssi": -51,
"uuid": "e2c56db5dffb48d2b060d0f5a71096e0",
"major": "0000",
"minor": "0000",
"tx_power": "c5"}
Phones There's a great free application for IPhone called Locate Beacon that let's you simulate beacons. This is great for testing. For the inexpensive beacon from Happy Bubbles I bought it is a minew MS49_nrf51822 and there's an app for that called BeaconSET. This will let you set properties and see it over your phone's BlueTooth. DDL CREATE EXTERNAL TABLE IF NOT EXISTS beaconstatus (rssi INT, heap_free INT)
STORED AS ORC LOCATION '/beacons';
CREATE EXTERNAL TABLE IF NOT EXISTS ibeacon (hostname STRING, beacon_type STRING, mac STRING, rssi INT, uuid STRING, major STRING, minor STRING, tx_power STRING)
STORED AS ORC LOCATION '/beacons/ibeacon';
CREATE EXTERNAL TABLE IF NOT EXISTS beacongateway (hostname STRING, mac STRING, rssi INT, is_scan_response STRING, type STRING, data STRING)
STORED AS ORC LOCATION '/beacons/gateway';
Reference https://en.wikipedia.org/wiki/IBeacon https://en.wikipedia.org/wiki/Bluetooth_low_energy http://developer.estimote.com/ibeacon/tutorial/part-3-ranging-beacons/ https://en.wikipedia.org/wiki/MQTT https://developers.google.com/beacons/proximity/guides https://www.beaconzone.co.uk/BeaconTriggerDataAndServers https://developer.apple.com/ibeacon/ https://developers.google.com/beacons/ https://www.happybubbles.tech/presence/detector https://www.happybubbles.tech/presence/docs/setup/ https://github.com/happy-bubbles/ https://itunes.apple.com/us/app/locate-beacon/id738709014?mt=8
... View more
Labels:
01-11-2018
05:52 PM
In case that someone will face the same problem we solved this by making the table internal, keeping the TextFile format and storing data under default Hive directory. The definition of table look like this at the moment: create table test1(c1 int, c2 int) CLUSTERED BY(c1) SORTED BY(c1) INTO 4 BUCKETS ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS TEXTFILE;
... View more
01-04-2018
05:18 PM
make sure that this error haven't caused by tables that Hive Create in your MySql database: check out if there is something looks like this error: Error: Index column size too large. The maximum column size is 767 bytes. (state=HY000,code=1709)
or just:
The maximum column size is 767 bytes
... View more
04-10-2017
08:51 PM
@Neeraj Sabharwal Please see if you can help me.
... View more
04-02-2017
09:16 PM
2 Kudos
The QueryDatabaseTable processor can easily ingest data from a table based on a incrementing key. A sequence id or primary key that is autogeneratored like Postgresql and MariaDB do is ideal. You can also do an incrementing data or Oracle Sequence ID. As long as it increments when you get a new one you can set. If your tables don't this, you could write a trigger or procedure in your database that sends it to a transaction table with such an autogenerated id and NiFi will grab that. Clearly real CDC involves reading Write Ahead Logs or Transaction logs at a deep level and grabbing all changes. That is coming and can now be done by tools like Atunity + NiFi. For use cases that I have, I just need to grab new rows when they are added to a table and I control the ID. I convert from AVRO to JSON so I can extract attributes since I want to do some routing based on column values. Based on one field in the table, I want to determine where I land the data. It can be sent to HBase (and Phoenix), HDFS or Hive. I split my records for easy processing. One thing you I highly recommend you do for SQL safety and to prevent errors. Example SQL for CDC: upsert into trials (trialid, trialdescription, fileName) values (1,'FENTANYL','5ab2d068-dd53-4674-bcf8-17f7d80d0553')
CREATE EXTERNAL TABLE IF NOT EXISTS trials2 (trialid INT, trialdescription STRING, trialtype STRING) STORED AS ORC
location '/hiveorc'
CREATE TABLE trials (trialid integer not null primary key, trialdescription varchar, filename varchar);
Set your SQL Attributes for SQL Safety. The types are the numeric values for JDBC Types. 12 is String. -5 is BIG INT. Then your SQL is standard JDBC syntax with ?'s for place markers. Here is some cool data: I used Google Location API called via NiFi REST CALL to enhance some data and get lat and long from a vague location. This kind of thing happens in Twitter all the time.
Reference: https://www.mockaroo.com/ https://community.hortonworks.com/articles/51902/incremental-fetch-in-nifi-with-querydatabasetable.html
... View more
Labels:
04-28-2017
02:58 AM
i have not tried, but there's no reason you couldn't. and you could send it to a remote cluster via site 2 site
... View more
03-23-2018
02:36 PM
You have to run Apache Spark on HDP 2.6. You should start with HDP cluster then add HDF to it. see https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.1.1/bk_installing-hdf-and-hdp/content/ch_install-ambari.html
... View more