Looks like you don't have the Python development packages installed. On my Ubuntu system that's the libpythonx.y-dev package, where 'x' is 2 and 'y' is 7. On Red Hat or Cent OS systems I think you want to install the python-devel package. On my test Cent OS 6.6 system python-devel provides /usr/lib64/libpython2.6.so. ... View more

You need to download/distribute/activate the KUDU parcel in addition to the CDH parcel. First, add http://archive.cloudera.com/kudu/parcels/5.11.0/ to your parcel repository list in Cloudera Manager. Then a KUDU parcel should show up in the list of available parcels. ... View more

Based on the error message, it sounds like the Cloudera Manager server timed out while trying to access the parcel repo URL you specified. Perhaps double check that you didn't introduce any typos, and that you can download <parcel repo URL>/manifest.json using curl or wget from the machine running the CM server? Since this is an issue with CM and parcel installation, if you can't solve the problem I suggest you use this forum: http://community.cloudera.com/t5/Cloudera-Manager-Installation/bd-p/CMInstall ... View more

There's no workaround for this yet. For the time being, every Kudu table must have a primary key, and it must be populated if loading data from a table without primary keys. ... View more

As you saw in the error message, the "LOAD DATA" statement doesn't work for Kudu tables. Its documentation says as much.   I'm no Impala expert, but perhaps you can build an HDFS table around '/data/feature_matrix_100.csv' first, then SELECT that data into your Kudu table? ... View more

Looking at the documentation, I think "STORED AS KUDU" needs to precede "tblproperties(...)".   Also, I don't think you want "ROW FORMAT..." or "LOCATION ..."; those clauses aren't relevant for Kudu tables. ... View more

Bear in mind that this Python code merely connects and operates on an existing Kudu cluster; it doesn't start one for you. If you haven't yet created and started a Kudu service, you'll need to do that first. If you have, the value for 'host' in kudu.connect() needs to be the valid hostname where you're running the Kudu master. ... View more

Those warnings are likely harmless. Is there any actual failure during the upserts? ... View more

The "rolling extension" process you're suggesting looks reasonable. It should serve as a workaround for Kudu's current inability to add new data directories. ... View more

You need to first install the Kudu client system packages. Check out the documentation here: https://www.cloudera.com/documentation/kudu/latest/topics/kudu_installation.html#install_cmdline Specifically, you need to install the 'kudu-client0' (RPM distros) or 'libkuduclient0' (DEB distros) package. ... View more

Not sure if you've resolved your problem or not, but as a general note, you're using a very old version of Impala, and that version may have interoperability problems with a modern Kudu.   It'd be best if you switched to Kudu and Impala from CDH 5.10 or newer. Beginning with CDH 5.10 the special "Impala-Kudu" release of Impala no longer exists as regular CDH Impala can interoperate with Kudu. ... View more

You can avoid the dependency on ntpd by running Kudu with --use-hybrid-clock=false, but that has a serious effect on transactional consistency so it's not something we recommend. Instead, I'd focus your efforts on figuring out why your servers' time isn't synchronized. It may have to do with your ntp configuration.   Unfortunately I don't know how ntp works; perhaps you can search across past forum posts? If you do manage to fix this, please post your findings here; if it's a general purpose fix (i.e. not particular to your site configuration), we'll include it in the Kudu documentation.   ... View more

HDFS datanodes don't require clock synchronization in the way that Kudu does.   Is NTP running on these nodes? What is the output of the 'ntptime' command? Are these nodes running on physical hardware, or something else? ... View more

Could you get by with the Kudu quickstart VM? I don't think it's as full-featured as the CDH quickstart VM (it may only have HDFS, Hive, Impala, and Kudu installed), but I know it's set up to use CDH 5.10.   You can find documentation for installing it here: https://kudu.apache.org/docs/quickstart.html ... View more

Kudu tables can be range partitioned on multiple columns, though the syntax is slightly different than the Parquet syntax. Take a look at https://kudu.apache.org/docs/kudu_impala_integration.html#_specifying_tablet_partitioning to see what I mean.   https://kudu.apache.org/docs/kudu_impala_integration.html#partitioning_tables also has much more information. ... View more

I reformatted your errors so they're a bit easier to read: W0322 09:40:06.534153 129361 negotiation.cc:240] Failed RPC negotiation. Trace: 0322 09:39:51.533481 (+ 0us) reactor.cc:378] Submitting negotiation task for server connection from 11.136.156.50:39299 0322 09:39:51.533517 (+ 36us) sasl_server.cc:230] Waiting for connection header 0322 09:39:57.673309 (+6139792us) sasl_server.cc:238] Connection header received 0322 09:39:57.673309 (+ 0us) sasl_server.cc:181] Waiting for next SASL message... 0322 09:40:06.534117 (+8860808us) negotiation.cc:231] Negotiation complete: Timed out: Server connection negotiation failed: server connection from 11.136.156.50:39299 Metrics: {"negotiator.queue_time_us":8} W0322 09:40:16.245877 164604 connection.cc:464] server connection from 11.136.156.55:43035 recv error: Network error: recv error: Connection reset by peer (error 104) W0322 09:40:16.246330 164604 connection.cc:378] Connection torn down before Call kudu.tserver.TabletServerService.Write from 11.136.156.55:43035 (ReqId={client: 574a7048c3824c869a1e553825845ba6, seq_no=467150, attempt_no=1}) could send its response W0322 09:27:45.480054 101702 negotiation.cc:240] Failed RPC negotiation. Trace: 0322 09:27:31.439139 (+ 0us) reactor.cc:378] Submitting negotiation task for server connection from 11.1 31.120.17:41515 0322 09:27:31.439526 (+ 387us) sasl_server.cc:230] Waiting for connection header 0322 09:27:32.087993 (+648467us) sasl_server.cc:238] Connection header received 0322 09:27:32.087994 (+ 1us) sasl_server.cc:181] Waiting for next SASL message... 0322 09:27:45.479950 (+13391956us) sasl_server.cc:295] SASL Server: Received NEGOTIATE request from client 0322 09:27:45.479964 (+ 14us) sasl_server.cc:341] Sent NEGOTIATE response 0322 09:27:45.479967 (+ 3us) sasl_server.cc:181] Waiting for next SASL message... 0322 09:27:45.480007 (+ 40us) negotiation.cc:231] Negotiation complete: Network error: Server connection negotiation failed: server connection from 11.131.120.17:41515: BlockingRecv error: Recv() got EOF from remote (error 108) Metrics: {"negotiator.queue_time_us":352,"thread_start_us":344,"threads_started":1} Still, that looks fairly innocuous to me, it's just a connection failure. Have you checked whether all of your data was written? ... View more

The error suggests that your Impala processes can't communicate with the Kudu tservers. Impala gets the addresses of the tservers from the Kudu Master.   Look at the /tablet-servers page in the Kudu Master web UI; are the published tserver addresses/hostnames reasonable? Can you resolve them and connect to them from every machine in the cluster?   Separately, look at the process log for the Kudu Master. If Impala can't connect to the tservers, the Kudu Master won't be able to either, and you may see errors here relating to creation of tablets. ... View more

I'm not familiar with Storm; how exactly does it write to Kudu? The error sounds like one piece of the Storm job abruptly disconnected from one of the Kudu servers. It may be harmless (maybe Storm will retry), or it may indicate a failure in the job. Given that Storm didn't exhibit any errors, it's likely to be the former, though you could always verify that by reading the data out of the table and ensuring that it's all there.   Is there any other associated logging in Kudu about this failure? Maybe a trace message of some kind?   ... View more

By default, CM will warn when 50% of a process' FDs are in use. Also by default, Kudu's block manager system will use 50% of the FDs available to the process. So, after accounting for some additional FDs for WALs, Kudu ends up using a little over 50% of the available FDs and CM warns about it.   If this bothers you, you can: Reconfigure Kudu's block_manager_max_open_files to some fixed value below 16384. The default value of -1 means Kudu will use 50% of what's available (16384 in your case). Reconfigure CM to warn at a higher threshold than 50%. Wait for CDH 5.11, where Kudu's percent usage was dropped from 50% to 40%.   ... View more

How exactly are you setting the flag in CM? Did you restart the Kudu service after setting it? Have you verified that it actually took effect? When a Kudu process starts up, it'll log its non-default command line arguments; you can see if maintenance_manager_num_threads is in that log output.   ... View more