Member since
07-17-2017
143
Posts
16
Kudos Received
17
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1465 | 07-03-2019 02:49 AM | |
1657 | 04-22-2019 03:13 PM | |
1390 | 01-30-2019 10:21 AM | |
8047 | 07-25-2018 09:45 AM | |
7146 | 05-31-2018 10:21 AM |
08-22-2017
08:49 AM
Hi, When I use sqoop command to import and load data from postgresql DB, even if I set the to params to: --null-string '\\N' \
--null-non-string '\\N' \ but I find that the null values in the HDFS file are replaced by 'N' instead of '\N' : #sudo -u hdfs hdfs dfs -cat /user/hive/warehouse/db/categories/part-m-00000 | head -3 371,2017-01-10 18:43:25,Cars,N,CA,N
372,2017-01-10 18:43:26,Motos,N,CA,N
373,2017-01-16 11:01:12,Tracks,N,CA,1 Which results to unrecognized NULL values mostly for the STRING type (because for the INT type the 'N' is considered like NULL..). # select * from categories limit 3; +-----+----------------------+--------+--------+------+----------+
| id | date | name | status | cc | valid |
+-----+----------------------+--------+--------+------+----------+
| 371 | 2017-01-10 18:43:25 | Cars | N | CA | NULL |
| 372 | 2017-01-10 18:43:26 | Motos | N | CA | NULL |
| 373 | 2017-01-16 11:01:12 | Tracks | N | CA | 1 |
+-----+----------------------+--------+--------+------+----------+ #select count(*) from categories where status is null; Error converting column: 5 to INT Error parsing row: file: hdfs://XXX:8020/user/hive/warehouse/...., before offset: 18698 Error parsing row: file: hdfs://XXX:8020/user/hive/warehouse/...., before offset: 18698 +----------+ | count(*) | +----------+ | 0 | +----------+ Fetched 1 row(s) in 0.19s But when I tested to import and load manually with a direct hdfs put a categories.csv file that have a '\N' instead of NULL value I don't face any problem. #sudo -u hdfs hdfs dfs -put categories.csv /user/hive/warehouse/db #sudo -u hdfs hdfs dfs -cat /user/hive/warehouse/db/categories/categories.csv | head -3 371,2017-01-10 18:43:25,Cars,\N,CA,\N 372,2017-01-10 18:43:26,Motos,\N,CA,\N 373,2017-01-16 11:01:12,Tracks,\N,CA,1 # select * from categories limit 3; +-----+----------------------+--------+--------+------+----------+ | id | date | name | status | cc | valid | +-----+----------------------+--------+--------+------+----------+ | 371 | 2017-01-10 18:43:25 | Cars | NULL | CA | NULL | | 372 | 2017-01-10 18:43:26 | Motos | NULL | CA | NULL | | 373 | 2017-01-16 11:01:12 | Tracks | NULL | CA | 1 | +-----+----------------------+--------+--------+------+----------+ #select count(*) from categories where status is null; +----------+ | count(*) | +----------+ | 3 | +----------+ Fetched 1 row(s) in 0.20s Thanks for your help.
... View more
Labels:
- Labels:
-
Apache Impala
-
Apache Sqoop
-
HDFS
07-25-2017
08:05 AM
1 Kudo
After a discussion in Kudu slack channel, we have found that it's concern a bug, and @JD Cryans published it in Apache's JIRA to fix it in the next version. https://issues.apache.org/jira/browse/KUDU-2076 We hope to find a solution as the soon as possible.
... View more
07-21-2017
10:31 AM
Hi Here it is the logs file : kudu-tserver.INFO: link1: http://www41.zippyshare.com/v/xRs7t60g/file.html link2: https://ufile.io/mwia8 kudu-tserver.WARNING: link1: http://www41.zippyshare.com/v/iFSgcXO0/file.html link2: https://ufile.io/z4bqf & Thanks for your help J-D.
... View more
07-21-2017
09:30 AM
Yes I understand that dropping table will be perfect in this case, but the problem is I'm facing this error message also in a kind of delete like: delete from test2p2 where id < 300000000; Okay brother, I'm collecting the 871c8123905c4e529a233c18751f8154 logs, and I will post them in a few minutes.
... View more
07-21-2017
09:08 AM
Firstly, thank you @J-D to the answer. So what kind of delete I can use instead of : delete from test2p; What's types of logs can I give you to have lighter vision of the problem source ? For the memory, in effect the nodes has memory_limit_hard_bytes=5.5 Go & thanks again.
... View more
07-21-2017
05:36 AM
Hi, When I run delete from kudu table, after deleting some rows, it return an error, and it continues deleting, but not all the rows : > select count(*) from test2p; +----------+ | count(*) | +----------+ | 50000000 | +----------+ > delete from test2p; WARNINGS: Kudu error(s) reported, first error: Timed out: Failed to write batch of 3759 ops to tablet 871c8123905c4e529a233c18751f8154 after 1 attempt(s): Failed to write to server: fb8ca29738c541hc80ed2da98a6e6499 (nod7.exp:7050): Write RPC to X.X.X.X:7050 timed out after 179.998s (SENT) Error in Kudu table 'impala::kudutest.test2p': Timed out: Failed to write batch of 3759 ops to tablet 871c8123905c4e529a233c18751f8154 after 1 attempt(s) : Failed to write to server: fb8ca29738c541hc80ed2da98a6e6499 (nod7.exp:7050): Write RPC to X.X.X.X:7050 timed out after 179.998s (SENT) (1 of 3273 similar) > select count(*) from test2p; +----------+ | count(*) | +----------+ | 38591543 | +----------+ .... > select count(*) from test2p; +----------+ | count(*) | +----------+ | 35220774 | +----------+ In ,pd7.exp log file (/var/log/kudu/kudu-tserver.WARNING), I have this error message: Metrics: {"negotiator.queue_time_us":211,"thread_start_us":193,"threads_started":1}
W0721 12:05:45.431758 28220 negotiation.cc:303] Failed RPC negotiation. Trace:
0721 12:05:45.427384 (+ 0us) reactor.cc:446] Submitting negotiation task for server connection from X.X.X.X:34727
0721 12:05:45.427608 (+ 224us) server_negotiation.cc:167] Beginning negotiation
0721 12:05:45.427609 (+ 1us) server_negotiation.cc:355] Waiting for connection header
0721 12:05:45.428640 (+ 1031us) server_negotiation.cc:363] Connection header received
0721 12:05:45.429483 (+ 843us) server_negotiation.cc:319] Received NEGOTIATE NegotiatePB request
0721 12:05:45.429484 (+ 1us) server_negotiation.cc:404] Received NEGOTIATE request from client
0721 12:05:45.429498 (+ 14us) server_negotiation.cc:331] Sending NEGOTIATE NegotiatePB response
0721 12:05:45.429539 (+ 41us) server_negotiation.cc:188] Negotiated authn=TOKEN
0721 12:05:45.431691 (+ 2152us) negotiation.cc:294] Negotiation complete: Network error: Server connection negotiation failed: server connection from X.X.X.X:34727: BlockingRecv error: Recv() got EOF from remote (error 108) Thnaks in advance.
... View more
Labels:
- Labels:
-
Apache Impala
-
Apache Kudu
07-21-2017
03:23 AM
Hi See the CREATE TABLE AS SELECT paragraph here in documentation: https://www.cloudera.com/documentation/enterprise/latest/topics/impala_create_table.html Good luck.
... View more
07-21-2017
03:00 AM
Hello, In the master of Kudu cluster, when I run the bellow command : kudu cluster ksck 38.95.234.26 I has this messages Connected to the Master
WARNING: Errors gathering consensus info for Tablet Server 2dde98b3e6984f858be32c53g73e6177 (nod1.emd:7050): Remote error: could not fetch all consensus info: Not authorized: unauthorized access to method: GetConsensusState
WARNING: Errors gathering consensus info for Tablet Server 267e129e783446e0ba8ed5cea6c57dec (nod9.hdm1.emd:7050): Remote error: could not fetch all consensus info: Not authorized: unauthorized access to method: GetConsensusState
WARNING: Errors gathering consensus info for Tablet Server 6702cbbe311d4fce93167eam77bc9365 (nod10.hdm1.emd:7050): Remote error: could not fetch all consensus info: Not authorized: unauthorized access to method: GetConsensusState
WARNING: Errors gathering consensus info for Tablet Server 18590d0f0b134dccad08714f6md15e23 (nod8.hdm1.emd:7050): Remote error: could not fetch all consensus info: Not authorized: unauthorized access to method: GetConsensusState
WARNING: Errors gathering consensus info for Tablet Server ab8ca29738c5412c80ed2daj8a6e6499 (nod7.hdm1.emd:7050): Remote error: could not fetch all consensus info: Not authorized: unauthorized access to method: GetConsensusState
Fetched info from all 5 Tablet Servers
Table impala::kudu_test.test_2p is HEALTHY (2 tablet(s) checked)
The metadata for 1 table(s) is HEALTHY
OK Any idea please! Thanks in advance.
... View more
Labels:
- Labels:
-
Apache Kudu
07-20-2017
11:44 AM
1 Kudo
Finaly I found the solution, in Kudu -> Configuration I modify this attribut Kudu Service Advanced Configuration Snippet (Safety Valve) for gflagfile Kudu (Service-Wide) by adding this line : --trusted_subnets=127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,169.254.0.0/16,X.X.X.0/24 Good luck.
... View more
07-20-2017
09:23 AM
I's very easy, go to the service that containe this role -from the CM interface-, then click to instances, check the role in the host you need to remove it, and click remove.
... View more
- « Previous
- Next »