Member since
07-17-2019
738
Posts
433
Kudos Received
111
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 3473 | 08-06-2019 07:09 PM | |
| 3671 | 07-19-2019 01:57 PM | |
| 5195 | 02-25-2019 04:47 PM | |
| 4666 | 10-11-2018 02:47 PM | |
| 1768 | 09-26-2018 02:49 PM |
10-14-2016
04:00 PM
With sufficient backpressure, I would imagine that NiFi will drop data eventually. Based on your screenshot, you are writing no data via this processor (Out is 0bytes). I would recommend you verify that Phoenix and HBase are healthy before trying to introduce NiFi into the picture.
... View more
10-14-2016
03:54 PM
Oh, that's different than what you said earlier: Both values have "false" (not the values being inverted). Maybe it is a bug with 2.4.0.0. I'll have to see if I can use that exact version and see if I can reproduce the issue.
... View more
10-13-2016
05:30 PM
"In the csv file that value is represented as 1/0 and when it goes into
phoenix it should go as true/false. but it is going as false/true" > create table booltest(pk varchar not null primary key, truth boolean);
> upsert into booltest values('true', 1);
Error: ERROR 203 (22005): Type mismatch. INTEGER cannot be coerced to BOOLEAN (state=22005,code=203) It seems like UPSERTS will not coerce integers into a boolean. I'm curious how the CSV tool is doing this. psql.py seems to do this fine: $ echo "true,1" > ~/booleans.csv
$ echo "false,0" >> ~/booleans.csv
$ /usr/local/lib/phoenix/bin/psql.py -t BOOLTEST localhost:2181:/hbase-1.2 ~/booleans.csv
$ sqlline.py ...
> 0: jdbc:phoenix:localhost:2181:/hbase-1.2> select * from booltest;
+--------+--------+
| PK | TRUTH |
+--------+--------+
| false | false |
| true | true |
+--------+--------+
Similarly, using the CsvBulkLoadTool shows the same for me: yarn jar /usr/local/lib/phoenix/phoenix-4.9.0-HBase-1.2-SNAPSHOT-client.jar org.apache.phoenix.mapreduce.CsvBulkLoadTool -Dmapred.map.child.java.opts="-Xmx1G" --table BOOLTEST --input booleans.csv -z localhost:2181:/hbase-1.2 Maybe do a sanity check on your processing?
... View more
10-13-2016
05:08 PM
1 Kudo
"phoenix putsql is unabel toinsert it seems" Are you referring to the INSERT SQL command? Phoenix does not expose INSERT, it exposes UPSERT which matches the actual semantics of how data is written. "Nifi - putsql for phoenix upsert very slow , getting records 1000/sec" The PutSQL processor seems to be written in a way that would allow for optimal performance with Phoenix. Care to share your PutSQL processor configuration? Have you tried to write a simple JDBC application to verify the "theoretical" performance of your system? One thing I can see is that if PutSQL is getting triggered very frequently, you will be making a large number of RPCs and not batching updates into HBase. How many FlowFiles does PutSQL process per invocation?
... View more
10-12-2016
04:22 PM
Check for: 1. JVM GC pauses. If the JVM is doing a stop-the-world garbage collection, it will cause the server to become disconnected from ZK, and likely lose its session. Read the lines in the HBase service log prior to this error. 2. Errors in the ZooKeeper log about maxClientCnxns (https://community.hortonworks.com/articles/51191/understanding-apache-zookeeper-connection-rate-lim.html) 3. Ensure operation system swappiness is reduced from the default (often 30 or 60), to a value of 0. You can inspect this via `cat /proc/sys/vm/swappiness`.
... View more
10-05-2016
02:57 PM
1 Kudo
Please refer to the documentation on ports for HBase: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_reference/content/hbase-ports.html
... View more
10-04-2016
05:34 PM
1 Kudo
Some extra thoughts on top of Rajeshbabu's reply: 1. Increase the heapsize of the Phoenix Query Server via the PHOENIX_QUERYSERVER_OPTS variable hbase-env.sh 2. For writing data, make sure the addBatch() and executeBatch() API calls are used for the best performance
... View more
09-28-2016
11:03 PM
I would not recommend anything other than Phoenix 🙂
... View more
09-28-2016
10:10 PM
1 Kudo
First off: you should not use HBase APIs to write data into Phoenix tables. Use the Phoenix API. The reason you should not do this is the reason that you are not seeing the data you've written. Using Phoenix API's (the UPSERT command) is what triggers the update to the secondary index table. When you add data via the HBase APIs, Phoenix has no idea that you did this and thus cannot ensure referential integrity with your secondary indices. If you want to use Phoenix, use the Phoenix APIs to read and write data.
... View more