Member since
07-31-2013
1924
Posts
461
Kudos Received
311
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1260 | 07-09-2019 12:53 AM | |
6536 | 06-23-2019 08:37 PM | |
7143 | 06-18-2019 11:28 PM | |
7441 | 05-23-2019 08:46 PM | |
2712 | 05-20-2019 01:14 AM |
08-08-2014
09:09 AM
2 Kudos
The loss of the last majority-making member in any N-member quorum would imply a complete failure of quorum, and the remaining peers will shut off their client serving port and go into a 'waiting' mode for members to reappear to form a majority for leader re-election again. As a result, no clients can connect to the ZK service anymore, for reads or for writes. Clients will simply receive a ECONNREFUSED failure at this point. There's no 'read-only' mode - every operation in ZK requires a presence of an quorum. The ZK overview doc mentions this implicitly http://archive.cloudera.com/cdh5/cdh/5/zookeeper/zookeeperOver.html: """ As long as a majority of the servers are available, the ZooKeeper service will be available. """
... View more
08-07-2014
10:24 PM
2 Kudos
Loss of one node in a 3-member ZooKeeper quorum is tolerable, because 2 out of 3 remaining machines still count as a majority (out of the fully identified quorum of 3). - No loss of functionality will be experienced with the loss of only one of the peers (in three), two peers (in five), etc. - One of the remaining two will automatically be assigned the leader role, in case of a leader failure. - Writes can continue to happen, and no special mode is invoked. - Clients will continue to see the same behaviour they expect out of ZooKeeper, even in such a situation. - No manual intervention is required for this procedure, ZK is automatically HA. The administrator's guide of ZooKeeper covers this BTW: http://archive.cloudera.com/cdh5/cdh/5/zookeeper/zookeeperAdmin.html
... View more
08-05-2014
08:34 PM
You are correct though that this does not exist as a current feature. Please consider filing a HBASE project JIRA upstream requesting (implementation patches welcome too!) this at https://issues.apache.org/jira/browse/HBASE.
... View more
08-03-2014
11:43 PM
1 Kudo
Sorry, the XML appears to have got eaten up. Here's clearer instructions for releases prior to CDH 5.1.0: # SSH to Oozie machine # Log on as root # Do below: cat > /tmp/hive-site.xml <configuration><property><name>hadoop.rpc.protection</name><value>privacy</value></property></configuration> ^D cd /tmp/ jar cf hive-rpc-protection.jar hive-site.xml mv hive-rpc-protection.jar /var/lib/oozie/ # Restart Oozie server, and retry your WF.
... View more
08-03-2014
11:05 PM
Thank you for trying that out! I investigated further into the code and it turns out you are hitting https://issues.apache.org/jira/browse/OOZIE-1593, which we have backported in our CDH 5.1.0 release. With that fix added, the Credentials code will properly load the hadoop.rpc.protection when making HMS connections. But it does not appear do so in prior releases, even if you were to specify it as part of your action configuration. This is the fixed code line, if you are interested in taking a look: https://github.com/cloudera/oozie/blob/cdh5.1.0-release/core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java#L92 (Note the missing line in 5.0.0 sources, at https://github.com/cloudera/oozie/blob/cdh5.0.0-release/core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java#L86) If you are unable to upgrade immediately, you can perhaps try something like the below as one way of workaround for this: # SSH to Oozie machine # Log on as root # Do below: cat > /tmp/hive-site.xml hadoop.rpc.protectionprivacy ^D cd /tmp/ jar cf hive-rpc-protection.jar hive-site.xml mv hive-rpc-protection.jar /var/lib/oozie/ # Restart Oozie server, and retry your WF.
... View more
08-01-2014
09:48 AM
It does lead to this error from another issue we've seen internally (with CM canaries over HMS). Does adding the hadoop.rpc.protection property pair (with "privacy" option set), to your passed config file in the WF help with getting the HMS connection through?
... View more
07-31-2014
07:30 AM
Ugh, very sorry. I replied the above post via email, thinking it was a wholly new question. OK, so looking further in, the Oozie server is already using a TSaslTransport for the HMS client connection, so the property is likely not the problem. Do you perhaps have "hadoop.rpc.protection", on your cluster, set to a non-default value of "privacy" (for traffic encryption in HDFS and the like)?
... View more
07-31-2014
04:47 AM
To have Oozie talk to a secured HiveMetaStore, you need to follow the credentials procedure detailed at http://archive.cloudera.com/cdh5/cdh/5/oozie/DG_UnifiedCredentialsModule.html Basically: 1. Enable Credentials of HCat type at the Oozie Server (via service configuration, requires restart). 2. Add a credentials section to your workflow. This section also configures the HMS location and the SPN. 3. Add the credential name to the action that requires a token for authentication (your hive action).
... View more
07-29-2014
07:19 AM
Given that the 'ls' works and 'cat' is the only problem, the error log and its exception stack trace behind the 500 code would presently be on one of the DataNodes on which the file "/user/weather/defaultBlockSize"'s first block's replicas reside upon. Check the identified DataNode's logs to find the error. Do you by any chance use Cloudera Manager, and have wildcards enabled on the HDFS service? Also, since you are now long on a CDH version that has WebHDFS on it, you can also alternatively try the more modern HTTP interface alternative of webhdfs://, instead of hftp://: hadoop fs -cat webhdfs://bdcx1:50070/user/weather/defaultBlockSize Does that work?
... View more
07-27-2014
06:43 AM
1 Kudo
(1) The "driver" part of run/main code that sets up and submits a job executes where you invoke it. It does not execute remotely. (2) See (1), cause it invalidates the supposition. But for the actual Map and Reduce code execution instead, the point is true. (3) This is true as well. (4) This is incorrect. All output "collector" received data is stored to disk (in an MR-provided storage termed 'intermediate storage') after it runs through the partitioner (which divides them into individual local files pertaining to each target reducer), and the sorter (which runs quick sorts on the whole individual partition segments). (5) Functionally true, but it is actually the Reduce that "pulls" the map outputs stored across the cluster, instead of something sending reducers the data (i.e. push). The reducer fetches its specific partition file from all executed maps that produced one such file, and merge sorts all these segments before invoking the user API of reduce(…) function. The merge sorter does not require that the entire set of segments fit into memory at once - it does the work in phases if it does not have adequate memory. However, if the entire fetched output does not fit into the alloted disk of the reduce task host, the reduce task will fail. We try a bit to approximate and not schedule reduces on such a host, but if no host can fit the aggregate data, then you likely will want to increase the number of reducers (partitions) to divide up the amount of data received per reduce task as a natural solution.
... View more