Member since
07-31-2013
1924
Posts
462
Kudos Received
311
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1950 | 07-09-2019 12:53 AM | |
11787 | 06-23-2019 08:37 PM | |
9066 | 06-18-2019 11:28 PM | |
10024 | 05-23-2019 08:46 PM | |
4440 | 05-20-2019 01:14 AM |
07-27-2015
08:22 AM
Hello Harsh, I tried with ext4 as well. Still the slow block receiver write data to disk warning is logged in datanode log. Do you have any best practice guidelines for Performance turning at OS, Network, HDFS, Server Hardware, filesystem specific? I need performance tuning guidelines for HDFS block from OS to Disk which will help to find the problametic area. can you throw some light? regar
... View more
07-27-2015
03:25 AM
Glad to hear; Please consider marking the topic as resolved so others with similar issues can find it easily.
... View more
07-24-2015
07:18 PM
I'm not sure when I need to access the configuration from a role (or a group) as opposed to using the service directly, but based on a separate communication I used the role. The snippet of code below is based on a working version (on a quickstart vm), and looks like: # peforms the post install oozie service management for the red orbit module
def main():
cmhost = 'quickstart.cloudera'
theUserName = "admin"
thePassword = "admin"
apiResource = ApiResource(cmhost, username=theUserName, password=thePassword)
# A kludgey way of specifying the cluster, but there is only one here
clusters = apiResource.get_all_clusters()
# TODO: This is a kludge to resolve the cluster
if (len(clusters) != 1):
print "There should one cluster, but there are " + repr(len(clusters)) + " clusters"
sys.exit(1)
cluster = clusters[0]
# TODO: These parameters are appropriate for the quick start vm, are they right for our deployment?
hueServiceName = "hue"
oozieServiceName = "oozie"
oozieServiceRoleName = 'oozie-OOZIE_SERVER'
oozieService = cluster.get_service(oozieServiceName)
hueService = cluster.get_service(hueServiceName)
oozieServiceRole = oozieService.get_role(oozieServiceRoleName)
# TODO: Is this always going tobe the oozie service role's name? We may need to use the Role Type to be safe
originalOozieConfig = oozieServiceRole.get_config(view='full')
oozieConfigUpdates = { "oozie_executor_extension_classes" : "org.apache.oozie.action.hadoop.MyCustomActionExecutor",
"oozie_workflow_extension_schemas" : "my-custom-action-0.1.xsd" }
for configUpdateKey in oozieConfigUpdates:
if (configUpdateKey in originalOozieConfig):
print repr(configUpdateKey) + " before update has oozie configured value = " + str(originalOozieConfig[configUpdateKey])
else:
print repr(configUpdateKey) + " not previously configured in oozie"
# stop the oozie service after stoping any services depending on oozie (i.e. hue)
for service in [hueService, oozieService]:
print "stopping service " + repr(service.name)
service.stop().wait() # synchronous stop
print "service " + repr(service.name) + " stopped"
# update the configuration while the servers are quiescent
updatedOozieConfig = oozieServiceRole.update_config(oozieConfigUpdates)
print "updatedOozieConfig = " + repr(updatedOozieConfig)
for configUpdateKey in oozieConfigUpdates:
print 'Config after update for key = ' + repr(configUpdateKey) + " has value = " + repr(updatedOozieConfig[configUpdateKey])
# restart the oozie service before restarting any services depending on oozie (i.e. hue)
for service in [oozieService, hueService]:
print "retarting service " + repr(service.name)
service.restart().wait() # synchronous restart
print "service " + repr(service.name) + " restarted"
# Done!
return
... View more
07-23-2015
05:58 AM
It should be possible to perform SecureBulkLoad without enabling Kerberos, although I have not personally tested this mix. The config steps may involve just configuring the SecureBulkLoad end-point and the staging directory configs on the RSes and clients.
... View more
07-22-2015
09:32 AM
A direct property to change this is now available in the most recent CM 5.4.x releases. Once you upgrade, please remember to remove the env-var workaround so things continue to be configured consistently.
... View more
07-20-2015
11:05 PM
In MultipleOutput's write(K, V, String) method, you can pass a relative path for the baseOutputPath (last) argument such as "my-sub-dir/my-filename-prefix". This will allow you to produce subdirectories under the job output directory. Please ensure to use only relative paths, and not absolute ones, as otherwise your output paths will not be respected by the FileOutputCommitter process.
... View more
07-20-2015
09:21 PM
Thank you Harsh
... View more
07-20-2015
08:36 PM
1 Kudo
Thank you for following up! Please consider marking the thread resolved so others with similar questions may be able to locate topics with solutions faster.
... View more
07-19-2015
04:43 AM
The first byte value is used as the sign indicator, if the number requires more than one byte to encode with. In your specific example with 172 and -172, 1000|1111, or 8f, is used to indicate positive numbers 1000|0111, or 87, is used to indicate negative numbers It helps also if you look at the sources to answer such questions, i.e. at https://github.com/cloudera/hadoop-common/blob/cdh5.4.0-release/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/WritableUtils.java#L271-L298 (serialise) and https://github.com/cloudera/hadoop-common/blob/cdh5.4.0-release/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/WritableUtils.java#L307-L320 (deserialise). Also look at this: https://github.com/cloudera/hadoop-common/blob/cdh5.4.0-release/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/WritableUtils.java#L369-L371 Does this help?
... View more