Cloudera Community

Announcements
Welcome to the upgraded Community! Read this blog to see What’s New!
benoit
user rank
Explorer
My Accepted Solutions
Show All

Re: Spark in YARN with Namenode HA

by Explorer in Support Questions
‎01-31-2018 04:03 PM
‎01-31-2018 04:03 PM
The temporary solution is to fake hostname of the failing Namenode in /etc/revolv.conf (or equivalent) and make it point to the IP of the healthy Namenode, until your Namenode is back to life. ... View more

Ranger Hive Read and Write permissions

by Explorer in Support Questions
‎09-05-2017 10:05 AM
‎09-05-2017 10:05 AM
In HDP-2.6.1.0-129 (Ranger 0.7.0), I'm seeing (apparently new) Read and Write permissions (see screenshot) But I cannot find their counterpart in the documentation (https://cwiki.apache.org/confluence/display/RANGER/Hive+Commands+to+Ranger+Permission+Mapping) What are these permissions for, will they replace select and update permissions? Thanks ... View more
Labels:

Re: hive insert query is failing with following ex...

by Explorer in Support Questions
‎07-15-2017 07:18 PM
‎07-15-2017 07:18 PM
I got this problem once, and it was because the Hive-client component was not installed on the server I was running hive command from. ... View more

Re: Set {USER} user in Ranger policy?

by Explorer in Support Questions
‎05-12-2017 11:02 AM
‎05-12-2017 11:02 AM
I'm using HDP 2.5.3. But thanks I didn't pay attention to the version in the jira (https://issues.apache.org/jira/browse/RANGER-698). I'll upgrade then 🙂 ... View more

Set {USER} user in Ranger policy?

by Explorer in Support Questions
‎05-12-2017 06:18 AM
‎05-12-2017 06:18 AM
https://cwiki.apache.org/confluence/display/RANGER/Support+for+%24username+variable describes an interesting feature to automagically create dynamic rules for users, for instance a personal db for every users. Question is: how can we set this in Ranger admin or API? I got the following answer while calling the API: {"statusCode":1,"msgDesc":"{USER}: user does not exist. policy='test {USER}' service='INT1_hive'"} ... View more
Labels:

Re: [SOLVED] Ambari 2.4.0.1 fails to install HDP 2...

by Explorer in Support Questions
‎09-07-2016 08:16 AM
2 Kudos
‎09-07-2016 08:16 AM
2 Kudos
Replying to my own question 🙂 For some reason, hdp-select was partially installe with a version 2.4.2.0. Running a "yum upgrade hdp-select" on all the nodes did fix the problem of version mismatch. ... View more

[SOLVED] Ambari 2.4.0.1 fails to install HDP 2.5.0...

by Explorer in Support Questions
‎09-07-2016 07:59 AM
‎09-07-2016 07:59 AM
I'm doing a fresh install of Ambari 2.4.0.1 and HDP 2.5.0.0. Ambari is working fine, I can go through the wizard and select the stack (2.5.0.0), do a manual registration of my hosts, assign the components (master/slave) and start the deployment. The "Install, Test and Start" phase fails with the following stacktrace: Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/datanode.py", line 174, in <module> DataNode().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/datanode.py", line 49, in install self.install_packages(env) File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 567, in install_packages retry_count=agent_stack_retry_count) File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__ self.env.run() File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action provider_action() File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 54, in action_install self.install_package(package_name, self.resource.use_repos, self.resource.skip_repos) File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/yumrpm.py", line 49, in install_package self.checked_call_with_retries(cmd, sudo=True, logoutput=self.get_logoutput()) File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 83, in checked_call_with_retries return self._call_with_retries(cmd, is_checked=True, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 91, in _call_with_retries code, out = func(cmd, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 71, in inner result = function(command, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 93, in checked_call tries=tries, try_sleep=try_sleep) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 141, in _call_wrapper result = _call(command, **kwargs_copy) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 294, in _call raise Fail(err_msg) resource_management.core.exceptions.Fail: Execution of '/usr/bin/yum -d 0 -e 0 -y install hadoop_2_4_2_0_258' returned 1. Error: Nothing to do It fails installing hadoop_2_4_2_0_258 while I did select HDP 2.5.0.0 stack ?!?! Looking at the yum repo set during the manual registration, it makes sense that version 2.4.2.0 is not found, since it's properly pointing to 2.5 repo: $ cat /etc/yum.repos.d/HDP.repo [HDP-2.5] name=HDP-2.5 baseurl=http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.0.0 path=/ enabled=1 Why are the wrong version tried to be installed on the nodes? Thanks ... View more

Re: Spark in YARN with Namenode HA

by Explorer in Support Questions
‎02-03-2016 12:50 PM
‎02-03-2016 12:50 PM
@stevel what is your advise to around this behavior? ... View more

Re: Spark in YARN with Namenode HA

by Explorer in Support Questions
‎01-13-2016 02:48 PM
‎01-13-2016 02:48 PM
@Artem Ervits (I'm new to the community forum, not sure if you receive notification to every answers) ... View more

Re: Spark in YARN with Namenode HA

by Explorer in Support Questions
‎01-13-2016 12:43 PM
‎01-13-2016 12:43 PM
Well, Hive and everything else is working fine in this degraded HA mode, the only one having problems is Spark. Active Namenode logs (I just striped out BlockStateChange noise) is given below. There's no stacktrace or anything validating the hypothesis of problem in the HA setup. I reproduced the problem on several environments as follow: Setup a fresh cluster with NameNode HA spark-shell --master yarn-master works fine Figure out which namenode is referred as nn1 in hdfs-site.xml Shutdown this namenode referred as nn1 spark-shell --master yarn-master is failing, waiting infinitely on nn1:8020 If you shutdown the second Namenode in the list of dfs.ha.namenodes.<nameservice>, spark is still working fine. This makes me thinking that Spark HDFS client, as surprising as it could be, is not falling back to the other namenode. After turning on more logging, it looks like by default it is trying 500 times before falling back to the other namenode: There are some tickets speaking about dfs.client.retry.policy.enabled to be turned on or off (https://issues.apache.org/jira/browse/AMBARI-11192 and https://issues.apache.org/jira/browse/AMBARI-12138, as well as https://issues.apache.org/jira/browse/HADOOP-12078), but in the case of Spark, this does not seem to have any impact... My reading of the logs tells me that, for some reason, java.net.NoRouteToHostException is not considered as a failure and the client keep retrying the same namenode, while what I'm expecting is to try with the next in the list... 16/01/13 08:30:21 DEBUG RetryUtils: RETRY 0) policy=MultipleLinearRandomRetry, exception=java.net.NoRouteToHostException: No route to host 16/01/13 08:30:23 INFO Client: Retrying connect to server: daplab-rt-11.fri.lan/10.10.10.111:8020. Already tried 0 time(s); retry policy is RetryPolicy[MultipleLinearRandomRetry[500x2000ms], TryOnceThenFail] 16/01/13 08:30:24 DEBUG RetryUtils: RETRY 1) policy=MultipleLinearRandomRetry, exception=java.net.NoRouteToHostException: No route to host 16/01/13 08:30:25 INFO Client: Retrying connect to server: daplab-rt-11.fri.lan/10.10.10.111:8020. Already tried 1 time(s); retry policy is RetryPolicy[MultipleLinearRandomRetry[500x2000ms], TryOnceThenFail] Active Namenode logs: 2016-01-13 07:30:42,090 INFO hdfs.StateChange (FSNamesystem.java:saveAllocatedBlock(3583)) - BLOCK* allocate blk_1073888940_148127{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-913cec22-84a9-435b-9941-3839a5870b6f:NORMAL:10.10.10.15:50010|RBW], ReplicaUC[[DISK]DS-c982ff94-8e74-4d03-a74c-4b81b48840b5:NORMAL:10.10.10.25:50010|RBW], ReplicaUC[[DISK]DS-34bc38b0-7eec-4521-b756-bfcaaf528f42:NORMAL:10.10.10.23:50010|RBW]]} for /shared/zefix/company/2014/08/18/company_2014-08-18.avro 2016-01-13 07:30:56,522 INFO namenode.FSEditLog (FSEditLog.java:printStatistics(699)) - Number of transactions: 36246 Total time for transactions(ms): 790 Number of transactions batched in Syncs: 568 Number of syncs: 30797 SyncTimes(ms): 311448 238317 2016-01-13 07:32:54,882 INFO namenode.FSEditLog (FSEditLog.java:printStatistics(699)) - Number of transactions: 36253 Total time for transactions(ms): 790 Number of transactions batched in Syncs: 568 Number of syncs: 30804 SyncTimes(ms): 311526 238382 2016-01-13 07:33:54,901 INFO namenode.FSEditLog (FSEditLog.java:printStatistics(699)) - Number of transactions: 36259 Total time for transactions(ms): 790 Number of transactions batched in Syncs: 568 Number of syncs: 30810 SyncTimes(ms): 311593 238438 2016-01-13 07:34:34,561 INFO hdfs.StateChange (FSNamesystem.java:saveAllocatedBlock(3583)) - BLOCK* allocate blk_1073888941_148128{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-0112cfbf-aaca-4294-ae0d-2456935db0b6:NORMAL:10.10.10.26:50010|RBW], ReplicaUC[[DISK]DS-8170fc01-c5dc-4221-9ea2-1f4a3b343b49:NORMAL:10.10.10.32:50010|RBW], ReplicaUC[[DISK]DS-55ab7112-fc8a-428c-a948-cb6ee4efaefe:NORMAL:10.10.10.167:50010|RBW]]} for /user/daplab_prod/.sparkStaging/application_1452430297721_0098/spark-assembly-1.5.2.2.3.4.0-3485-hadoop2.7.1.2.3.4.0-3485.jar 2016-01-13 07:34:35,894 INFO hdfs.StateChange (FSNamesystem.java:saveAllocatedBlock(3583)) - BLOCK* allocate blk_1073888942_148129{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c739e4e5-1545-40b9-a6ef-28989c6cc8eb:NORMAL:10.10.10.32:50010|RBW], ReplicaUC[[DISK]DS-25feb56a-94b7-4520-92dc-c46939414f9a:NORMAL:10.10.10.15:50010|RBW], ReplicaUC[[DISK]DS-31fb3172-c085-43ef-8ccd-f38e72492ad7:NORMAL:10.10.10.23:50010|RBW]]} for /user/daplab_prod/.sparkStaging/application_1452430297721_0098/spark-assembly-1.5.2.2.3.4.0-3485-hadoop2.7.1.2.3.4.0-3485.jar 2016-01-13 07:34:36,910 INFO hdfs.StateChange (FSNamesystem.java:completeFile(3503)) - DIR* completeFile: /user/daplab_prod/.sparkStaging/application_1452430297721_0098/spark-assembly-1.5.2.2.3.4.0-3485-hadoop2.7.1.2.3.4.0-3485.jar is closed by DFSClient_NONMAPREDUCE_186954515_1 2016-01-13 07:34:40,078 INFO hdfs.StateChange (FSNamesystem.java:saveAllocatedBlock(3583)) - BLOCK* allocate blk_1073888943_148130{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-0daa5887-d09e-4e96-888f-757e9c413581:NORMAL:10.10.10.23:50010|RBW], ReplicaUC[[DISK]DS-d3cb3d03-df7d-4211-8b9f-fe95b0e5954f:NORMAL:10.10.10.30:50010|RBW], ReplicaUC[[DISK]DS-09829b26-21f2-4c8b-9936-d69d9a0469a9:NORMAL:10.10.10.32:50010|RBW]]} for /user/daplab_prod/.sparkStaging/application_1452430297721_0098/__spark_conf__8474002130224342241.zip 2016-01-13 07:34:40,118 INFO hdfs.StateChange (FSNamesystem.java:completeFile(3503)) - DIR* completeFile: /user/daplab_prod/.sparkStaging/application_1452430297721_0098/__spark_conf__8474002130224342241.zip is closed by DFSClient_NONMAPREDUCE_186954515_1 2016-01-13 07:34:54,937 INFO namenode.FSEditLog (FSEditLog.java:printStatistics(699)) - Number of transactions: 36286 Total time for transactions(ms): 790 Number of transactions batched in Syncs: 568 Number of syncs: 30831 SyncTimes(ms): 311815 238646 2016-01-13 07:35:54,963 INFO namenode.FSEditLog (FSEditLog.java:printStatistics(699)) - Number of transactions: 36292 Total time for transactions(ms): 790 Number of transactions batched in Syncs: 568 Number of syncs: 30837 SyncTimes(ms): 311862 238700 2016-01-13 07:36:55,010 INFO namenode.FSEditLog (FSEditLog.java:printStatistics(699)) - Number of transactions: 36298 Total time for transactions(ms): 790 Number of transactions batched in Syncs: 568 Number of syncs: 30843 SyncTimes(ms): 311908 238750 2016-01-13 07:37:22,743 INFO hdfs.StateChange (FSNamesystem.java:completeFile(3503)) - DIR* completeFile: /shared/zefix/company/2014/08/18/company_2014-08-18.avro is closed by DFSClient_NONMAPREDUCE_-1471796874_1 2016-01-13 07:37:54,939 INFO hdfs.StateChange (FSNamesystem.java:saveAllocatedBlock(3583)) - BLOCK* allocate blk_1073888944_148131{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-36be5f99-94df-4539-b8e4-c6fcd07cca53:NORMAL:10.10.10.14:50010|RBW], ReplicaUC[[DISK]DS-49d7ed47-bbaf-477c-956e-54cc5dd5d165:NORMAL:10.10.10.24:50010|RBW], ReplicaUC[[DISK]DS-b0436f14-fe85-492d-999e-3f2fe44dd638:NORMAL:10.10.10.32:50010|RBW]]} for /shared/zefix/company/2014/02/14/company_2014-02-14.avro 2016-01-13 07:37:55,010 INFO namenode.FSEditLog (FSEditLog.java:printStatistics(699)) - Number of transactions: 36311 Total time for transactions(ms): 790 Number of transactions batched in Syncs: 568 Number of syncs: 30854 SyncTimes(ms): 312082 238850 2016-01-13 07:38:04,840 INFO hdfs.StateChange (FSNamesystem.java:saveAllocatedBlock(3583)) - BLOCK* allocate blk_1073888945_148132{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-ea1ff4e9-53e5-4e8e-8c02-4a6a5993247a:NORMAL:10.10.10.31:50010|RBW], ReplicaUC[[DISK]DS-7c91c6c5-a0b5-46a6-a963-097150f29878:NORMAL:10.10.10.30:50010|RBW], ReplicaUC[[DISK]DS-d4e2030b-c29d-4e0c-a3b6-3e8b795d3b2e:NORMAL:10.10.10.13:50010|RBW]]} for /app-logs/daplab_prod/logs/application_1452430297721_0098/daplab-wn-31.fri.lan_45454.tmp 2016-01-13 07:38:04,882 INFO hdfs.StateChange (FSNamesystem.java:completeFile(3503)) - DIR* completeFile: /app-logs/daplab_prod/logs/application_1452430297721_0098/daplab-wn-31.fri.lan_45454.tmp is closed by DFSClient_NONMAPREDUCE_-829750558_12726 2016-01-13 07:38:06,441 INFO hdfs.StateChange (FSNamesystem.java:saveAllocatedBlock(3583)) - BLOCK* allocate blk_1073888946_148133{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d4e2030b-c29d-4e0c-a3b6-3e8b795d3b2e:NORMAL:10.10.10.13:50010|RBW], ReplicaUC[[DISK]DS-8ed05f08-aa49-43e8-b1ac-ca60d5cd6196:NORMAL:10.10.10.23:50010|RBW], ReplicaUC[[DISK]DS-dd4bdbf0-dc93-4e42-8009-dc13fe580385:NORMAL:10.10.10.31:50010|RBW]]} for /app-logs/daplab_prod/logs/application_1452430297721_0098/daplab-wn-13.fri.lan_45454.tmp 2016-01-13 07:38:06,490 INFO hdfs.StateChange (FSNamesystem.java:completeFile(3503)) - DIR* completeFile: /app-logs/daplab_prod/logs/application_1452430297721_0098/daplab-wn-13.fri.lan_45454.tmp is closed by DFSClient_NONMAPREDUCE_1188492760_4267 2016-01-13 07:38:06,827 INFO hdfs.StateChange (FSNamesystem.java:saveAllocatedBlock(3583)) - BLOCK* allocate blk_1073888947_148134{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-4c7faccb-83ab-4dbd-a763-f35690bd3767:NORMAL:10.10.10.32:50010|RBW], ReplicaUC[[DISK]DS-0b3b0051-e971-44a4-914c-14ef84528e05:NORMAL:10.10.10.30:50010|RBW], ReplicaUC[[DISK]DS-c26f498e-77dc-4d02-8abc-b1ebc9d25946:NORMAL:10.10.10.23:50010|RBW]]} for /app-logs/daplab_prod/logs/application_1452430297721_0098/daplab-wn-32.fri.lan_45454.tmp 2016-01-13 07:38:06,865 INFO hdfs.StateChange (FSNamesystem.java:completeFile(3503)) - DIR* completeFile: /app-logs/daplab_prod/logs/application_1452430297721_0098/daplab-wn-32.fri.lan_45454.tmp is closed by DFSClient_NONMAPREDUCE_-1454565051_8117 2016-01-13 07:38:55,024 INFO namenode.FSEditLog (FSEditLog.java:printStatistics(699)) - Number of transactions: 36336 Total time for transactions(ms): 790 Number of transactions batched in Syncs: 568 Number of syncs: 30873 SyncTimes(ms): 312254 239031 2016-01-13 07:39:55,046 INFO namenode.FSEditLog (FSEditLog.java:printStatistics(699)) - Number of transactions: 36342 Total time for transactions(ms): 791 Number of transactions batched in Syncs: 568 Number of syncs: 30879 SyncTimes(ms): 312313 239091 2016-01-13 07:40:55,061 INFO namenode.FSEditLog (FSEditLog.java:printStatistics(699)) - Number of transactions: 36349 Total time for transactions(ms): 792 Number of transactions batched in Syncs: 568 Number of syncs: 30886 SyncTimes(ms): 312389 239147 2016-01-13 07:41:55,078 INFO namenode.FSEditLog (FSEditLog.java:printStatistics(699)) - Number of transactions: 36356 Total time for transactions(ms): 792 Number of transactions batched in Syncs: 568 Number of syncs: 30893 SyncTimes(ms): 312468 239203 2016-01-13 07:42:55,092 INFO namenode.FSEditLog (FSEditLog.java:printStatistics(699)) - Number of transactions: 36362 Total time for transactions(ms): 792 Number of transactions batched in Syncs: 568 Number of syncs: 30899 SyncTimes(ms): 312544 239259 2016-01-13 07:43:56,485 INFO namenode.FSEditLog (FSEditLog.java:printStatistics(699)) - Number of transactions: 36370 Total time for transactions(ms): 792 Number of transactions batched in Syncs: 568 Number of syncs: 30907 SyncTimes(ms): 312629 239343 2016-01-13 07:44:56,565 INFO namenode.FSEditLog (FSEditLog.java:printStatistics(699)) - Number of transactions: 36376 Total time for transactions(ms): 794 Number of transactions batched in Syncs: 568 Number of syncs: 30913 SyncTimes(ms): 312690 239390 2016-01-13 07:45:42,857 INFO hdfs.StateChange (FSNamesystem.java:completeFile(3503)) - DIR* completeFile: /shared/zefix/company/2014/02/14/company_2014-02-14.avro is closed by DFSClient_NONMAPREDUCE_-1471796874_1 2016-01-13 07:45:56,612 INFO namenode.FSEditLog (FSEditLog.java:printStatistics(699)) - Number of transactions: 36384 Total time for transactions(ms): 794 Number of transactions batched in Syncs: 568 Number of syncs: 30921 SyncTimes(ms): 312771 239458 2016-01-13 07:46:05,715 INFO hdfs.StateChange (FSNamesystem.java:saveAllocatedBlock(3583)) - BLOCK* allocate blk_1073888948_148135{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-58bd0cf4-a027-4e4c-aa86-fbf49d8ea6f6:NORMAL:10.10.10.24:50010|RBW], ReplicaUC[[DISK]DS-f93cd04c-c780-44fe-91b7-9fbb214c9724:NORMAL:10.10.10.32:50010|RBW], ReplicaUC[[DISK]DS-5809f6ca-32bb-44d1-8367-44ce5a9b6fff:NORMAL:10.10.10.33:50010|RBW]]} for /shared/zefix/company/2014/09/04/company_2014-09-04.avro 2016-01-13 07:47:54,928 INFO namenode.FSEditLog (FSEditLog.java:printStatistics(699)) - Number of transactions: 36394 Total time for transactions(ms): 794 Number of transactions batched in Syncs: 568 Number of syncs: 30929 SyncTimes(ms): 312853 239532 2016-01-13 07:48:54,949 INFO namenode.FSEditLog (FSEditLog.java:printStatistics(699)) - Number of transactions: 36401 Total time for transactions(ms): 794 Number of transactions batched in Syncs: 568 Number of syncs: 30936 SyncTimes(ms): 312950 239591 2016-01-13 07:49:54,979 INFO namenode.FSEditLog (FSEditLog.java:printStatistics(699)) - Number of transactions: 36407 Total time for transactions(ms): 794 Number of transactions batched in Syncs: 568 Number of syncs: 30942 SyncTimes(ms): 313009 239649 ... View more
Contact Me
Online
Offline
Last Visited
‎04-26-2018 07:08 PM
Community Statistics
Member Since ‎01-12-2016 10:00 PM
Last Visited ‎04-26-2018 07:08 PM
Posts 14
Kudos received 5