Hello Community,   I want to add users and groups but when i connect to ambari settings i could see only this. Any idea please? The version is 2.7.4   Many thanks in advance Asma ... View more

Hello   I am trying to use this command    myCluster <- RxSpark(myCluster, consoleOutput=TRUE) rxOptions(computeContext=myCluster)   How can I define the name of my cluster? I configured one with Ambari but when i put name it is not recognized when I use    delayArrCube <- rxCube(ArrDelay ~ F(CRSDepTime):DayOfWeek, data="AirlineData87to08.xdf", blocksPerRead=30)   The calculation is not distributed between compute nodes like in the sample :   ====== CLUSTER-HEAD2 ( process 1 ) has started run at Thu Aug 11 15:56:10 2011 ====== ********************************************************************** Worker Node 'COMPUTE10' has received a task from Master Node 'CLUSTER-HEAD2'.... Thu Aug 11 15:56:10.791 2011 ********************************************************************** Worker Node 'COMPUTE11' has received a task from Master Node 'CLUSTER-HEAD2'.... Thu Aug 11 15:56:10.757 2011 ********************************************************************** Worker Node 'COMPUTE12' has received a task from Master Node 'CLUSTER-HEAD2'.... Thu Aug 11 15:56:10.769 2011 ********************************************************************** Worker Node 'COMPUTE13' has received a task from Master Node 'CLUSTER-HEAD2'.... Thu Aug 11 15:56:10.889 2011 COMPUTE13: Rows Read: 4440596, Total Rows Processed: 4440596, Total Chunk Time: 0.031 seconds COMPUTE11: Rows Read: 4361843, Total Rows Processed: 4361843, Total Chunk Time: 0.031 seconds COMPUTE12: Rows Read: 4467780, Total Rows Processed: 4467780, Total Chunk Time: 0.031 seconds COMPUTE10: Rows Read: 4492157, Total Rows Processed: 4492157, Total Chunk Time: 0.047 seconds COMPUTE13: Rows Read: 4500000, Total Rows Processed: 8940596, Total Chunk Time: 0.062 seconds   > delayArrCube <- rxCube(ArrDelay ~ F(CRSDepTime):DayOfWeek,data="AirlineData87to08.xdf", blocksPerRead=30) I got this Rows Read: 4461826, Total Rows Processed: 4461826, Total Chunk Time: 0.726 seconds Rows Read: 4452096, Total Rows Processed: 8913922, Total Chunk Time: 0.673 seconds Rows Read: 4441200, Total Rows Processed: 13355122, Total Chunk Time: 0.635 seconds Rows Read: 4370893, Total Rows Processed: 17726015, Total Chunk Time: 0.708 seconds Rows Read: 4476925, Total Rows Processed: 22202940, Total Chunk Time: 1.032 seconds Rows Read: 4500000, Total Rows Processed: 26702940, Total Chunk Time: 0.660 seconds Rows Read: 4492157, Total Rows Processed: 31195097, Total Chunk Time: 0.681 seconds Rows Read: 4470501, Total Rows Processed: 35665598, Total Chunk Time: 0.563 seconds Rows Read: 4430048, Total Rows Processed: 40095646, Total Chunk Time: 0.595 seconds Rows Read: 4427435, Total Rows Processed: 44523081, Total Chunk Time: 0.568 seconds Rows Read: 4451983, Total Rows Processed: 48975064, Total Chunk Time: 0.511 seconds Rows Read: 4361843, Total Rows Processed: 53336907, Total Chunk Time: 0.549 seconds Rows Read: 4500000, Total Rows Processed: 57836907, Total Chunk Time: 0.672 seconds Rows Read: 4484721, Total Rows Processed: 62321628, Total Chunk Time: 0.658 seconds     Any  idea please how can  I use  ... View more

Dear Community   I have installed Hadoop Spark Cluster HortonWorks. Then I have installed Machine learning on edge and compute nodes   From an edge node, I launch REvo64 then I want to use rxSparkConnect however i am getting this error :   cc <- rxSparkConnect(reset = TRUE) No Spark applications can be reused. It may take 1 to 2 minutes to launch a new Spark application. Error in rxRemoteExecute(computeContext, shellCmd, schedulerJobInstance) : ERROR: Fail to execute spark-submit. Last 20 lines' log: at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) at com.sun.proxy.$Proxy11.mkdirs(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:653) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)   Am I missing a detail or a configuration?   Thanks and REagrds Asma ... View more

Hello   I have a hadoop cluster on which i have installed MLS but when executing rxSparkConnect I got this error No Spark applications can be reused. It may take 1 to 2 minutes to launch a new Spark application. Error in rxRemoteExecute(computeContext, shellCmd, schedulerJobInstance) : ERROR: Fail to execute spark-submit. Last 20 lines' log: at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) at com.sun.proxy.$Proxy11.mkdirs(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:653) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) ... View more

Dear Community,   First Question Actually we have a cluster Hadoop  HorthonsWorks on Cento7 : Apache Ambari : 2.7.4 HDP 3.1.4 HDFS 3.1.1 YARN 3.1.1 Spark2 2.3.2 MapREduce 3.1.1   I want to install Machine learning on it :   https://docs.microsoft.com/en-us/machine-learning-server/install/r-server-install-supported-platforms   For MLS 9.3 is this only supported with HDP 2.4-2-6? I have the HDP 3.1.4?   Second Question   Actually, we have also a separated machine learning platform on Ubuntu servers : 9.3 Is it possible to integrate an existing MLS platform with an Hadoop Hortonworks ? Or should we install MLS nodes like in the below document?   Thank you very much for your help Asma   ... View more

Dear Community   If I want to have a cluster using ambari hortonWorks with    1. Ambari node 2 Name nodes 4 data nodes  1 edge node   Should I install on data nodes : spark server or spark client? I want that the process will be distributed Which component should be doing the processing server or client spark2?   Thanks in addance ... View more

Hello Community,   I have installed a cluster with ambari hortonworks. I specified a spark server and spark nodes  when I launch this command it is working  /usr/hdp/3.1.4.0-315/spark2/bin/spark-shell --master yarn --deploy-mode client --jars /usr/hdp/3.1.4.0-315/spark2/examples/jars/spark-examples_2.11-2.3.2.3.1.4.0-315.jar However wjeb  I change the cluster mode  /usr/hdp/3.1.4.0-315/spark2/bin/spark-shell --master yarn --deploy-mode cluster --jars /usr/hdp/3.1.4.0-315/spark2/examples/jars/spark-examples_2.11-2.3.2.3.1.4.0-315.jar Error: Cluster deploy mode is not applicable to Spark shells. Run with --help for usage help or --verbose for debug output   Should be something wrong with my conf? Thanks and Regards Asma ... View more

Dear Community   I have installed a cluster with spark server and spark client nodes When trying to access the spark server webi UI     tory:   hdfs:///spark2-history/ Last updated:   2019-11-21 12:50:57 Client local time zone:   Europe/Berlin No completed applications found! Did you specify the correct logging directory? Please verify your setting of   spark.history.fs.logDirectory   listed above and whether you have the permissions to access it. It is also possible that your application did not run to completion or did not stop the SparkContext. Show incomplete applications What should I do please?   I have installed the Spark Server on the same ambari node , should this cause a problem ?   Thanks and Regards Asma ... View more

Hello Community   The installation of Oozie failed ; any idea please? Traceback (most recent call last): File "/var/lib/ambari-agent/cache/stack-hooks/after-INSTALL/scripts/hook.py", line 39, in <module> AfterInstallHook().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute method(env) File "/var/lib/ambari-agent/cache/stack-hooks/after-INSTALL/scripts/hook.py", line 32, in hook setup_stack_symlinks(self.stroutfile) File "/var/lib/ambari-agent/cache/stack-hooks/after-INSTALL/scripts/shared_initialization.py", line 66, in setup_stack_symlinks stack_select.select(package, json_version) File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/stack_select.py", line 312, in select Execute(command, sudo=True) File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__ self.env.run() File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action provider_action() File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 263, in action_run returns=self.resource.returns) File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 72, in inner result = function(command, **kwargs) File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 102, in checked_call tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy, returns=returns) File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapper result = _call(command, **kwargs_copy) File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 314, in _call raise ExecutionFailed(err_msg, code, out, err) resource_management.core.exceptions.ExecutionFailed: Execution of 'ambari-python-wrap /usr/bin/hdp-select set oozie-client 3.1.4.0-315' returned 1. symlink target /usr/hdp/current/oozie-client for oozie already exists and it is not a symlink.   2019-11-19 17:51:28,551 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.4.0-315 -> 3.1.4.0-315 2019-11-19 17:51:28,555 - Using hadoop conf dir: /usr/hdp/3.1.4.0-315/hadoop/conf 2019-11-19 17:51:28,556 - Group['livy'] {} 2019-11-19 17:51:28,557 - Group['spark'] {} 2019-11-19 17:51:28,558 - Group['hdfs'] {} 2019-11-19 17:51:28,558 - Group['hadoop'] {} 2019-11-19 17:51:28,558 - Group['users'] {} 2019-11-19 17:51:28,558 - User['hive'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None} 2019-11-19 17:51:28,559 - User['yarn-ats'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None} 2019-11-19 17:51:28,560 - User['livy'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['livy', 'hadoop'], 'uid': None} 2019-11-19 17:51:28,560 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None} 2019-11-19 17:51:28,561 - User['spark'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['spark', 'hadoop'], 'uid': None} 2019-11-19 17:51:28,561 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None} 2019-11-19 17:51:28,562 - User['oozie'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'users'], 'uid': None} 2019-11-19 17:51:28,563 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'users'], 'uid': None} 2019-11-19 17:51:28,563 - User['tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'users'], 'uid': None} 2019-11-19 17:51:28,564 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hdfs', 'hadoop'], 'uid': None} 2019-11-19 17:51:28,564 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None} 2019-11-19 17:51:28,565 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None} 2019-11-19 17:51:28,566 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2019-11-19 17:51:28,567 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'} 2019-11-19 17:51:28,572 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] due to not_if 2019-11-19 17:51:28,572 - Group['hdfs'] {} 2019-11-19 17:51:28,573 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': ['hdfs', 'hadoop', u'hdfs']} 2019-11-19 17:51:28,573 - FS Type: HDFS 2019-11-19 17:51:28,573 - Directory['/etc/hadoop'] {'mode': 0755} 2019-11-19 17:51:28,585 - File['/usr/hdp/3.1.4.0-315/hadoop/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'} 2019-11-19 17:51:28,586 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 01777} 2019-11-19 17:51:28,601 - Repository['HDP-3.1-repo-2'] {'base_url': 'http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.1.4.0', 'action': ['prepare'], 'components': [u'HDP', 'main'], 'repo_template': '[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0', 'repo_file_name': 'ambari-hdp-2', 'mirror_list': ''} 2019-11-19 17:51:28,607 - Repository['HDP-UTILS-1.1.0.22-repo-2'] {'base_url': 'http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7', 'action': ['prepare'], 'components': [u'HDP-UTILS', 'main'], 'repo_template': '[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0', 'repo_file_name': 'ambari-hdp-2', 'mirror_list': ''} 2019-11-19 17:51:28,609 - Repository['HDP-3.1-GPL-repo-2'] {'base_url': 'http://public-repo-1.hortonworks.com/HDP-GPL/centos7/3.x/updates/3.1.4.0', 'action': ['prepare'], 'components': [u'HDP-GPL', 'main'], 'repo_template': '[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0', 'repo_file_name': 'ambari-hdp-2', 'mirror_list': ''} 2019-11-19 17:51:28,610 - Repository[None] {'action': ['create']} 2019-11-19 17:51:28,611 - File['/tmp/tmpjPFllJ'] {'content': '[HDP-3.1-repo-2]\nname=HDP-3.1-repo-2\nbaseurl=http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.1.4.0\n\npath=/\nenabled=1\ngpgcheck=0\n[HDP-UTILS-1.1.0.22-repo-2]\nname=HDP-UTILS-1.1.0.22-repo-2\nbaseurl=http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7\n\npath=/\nenabled=1\ngpgcheck=0\n[HDP-3.1-GPL-repo-2]\nname=HDP-3.1-GPL-repo-2\nbaseurl=http://public-repo-1.hortonworks.com/HDP-GPL/centos7/3.x/updates/3.1.4.0\n\npath=/\nenabled=1\ngpgcheck=0'} 2019-11-19 17:51:28,611 - Writing File['/tmp/tmpjPFllJ'] because contents don't match 2019-11-19 17:51:28,612 - File['/tmp/tmp9txvIl'] {'content': StaticFile('/etc/yum.repos.d/ambari-hdp-2.repo')} 2019-11-19 17:51:28,612 - Writing File['/tmp/tmp9txvIl'] because contents don't match 2019-11-19 17:51:28,613 - Package['unzip'] {'retry_on_repo_unavailability': False, 'retry_count': 5} 2019-11-19 17:51:28,686 - Skipping installation of existing package unzip 2019-11-19 17:51:28,687 - Package['curl'] {'retry_on_repo_unavailability': False, 'retry_count': 5} 2019-11-19 17:51:28,694 - Skipping installation of existing package curl 2019-11-19 17:51:28,695 - Package['hdp-select'] {'retry_on_repo_unavailability': False, 'retry_count': 5} 2019-11-19 17:51:28,702 - Skipping installation of existing package hdp-select 2019-11-19 17:51:28,706 - The repository with version 3.1.4.0-315 for this command has been marked as resolved. It will be used to report the version of the component which was installed 2019-11-19 17:51:28,908 - Package['zip'] {'retry_on_repo_unavailability': False, 'retry_count': 5} 2019-11-19 17:51:28,987 - Skipping installation of existing package zip 2019-11-19 17:51:28,989 - Package['oozie_3_1_4_0_315'] {'retry_on_repo_unavailability': False, 'retry_count': 5} 2019-11-19 17:51:28,997 - Skipping installation of existing package oozie_3_1_4_0_315 2019-11-19 17:51:29,002 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.4.0-315 -> 3.1.4.0-315 2019-11-19 17:51:29,005 - Using hadoop conf dir: /usr/hdp/3.1.4.0-315/hadoop/conf 2019-11-19 17:51:29,015 - File['/var/lib/ambari-agent/cred/lib/CredentialUtil.jar'] {'content': DownloadSource('http://luapp698.lu.ema.ad.pwcinternal.com:8080/resources/CredentialUtil.jar'), 'mode': 0755} 2019-11-19 17:51:29,015 - Not downloading the file from http://luapp698.lu.ema.ad.pwcinternal.com:8080/resources/CredentialUtil.jar, because /var/lib/ambari-agent/tmp/CredentialUtil.jar already exists 2019-11-19 17:51:29,569 - Directory['/usr/hdp/current/oozie-client/conf'] {'owner': 'oozie', 'create_parents': True, 'group': 'hadoop'} 2019-11-19 17:51:29,569 - File['/usr/hdp/current/oozie-client/conf/oozie-site.jceks'] {'content': StaticFile('/var/lib/ambari-agent/cred/conf/oozie_client/oozie-site.jceks'), 'owner': 'oozie', 'group': 'hadoop', 'mode': 0640} 2019-11-19 17:51:29,570 - Writing File['/usr/hdp/current/oozie-client/conf/oozie-site.jceks'] because contents don't match 2019-11-19 17:51:29,570 - XmlConfig['oozie-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/oozie-client/conf', 'mode': 0664, 'configuration_attributes': {}, 'owner': 'oozie', 'configurations': ...} 2019-11-19 17:51:29,579 - Generating config: /usr/hdp/current/oozie-client/conf/oozie-site.xml 2019-11-19 17:51:29,579 - File['/usr/hdp/current/oozie-client/conf/oozie-site.xml'] {'owner': 'oozie', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0664, 'encoding': 'UTF-8'} 2019-11-19 17:51:29,604 - File['/usr/hdp/current/oozie-client/conf/oozie-env.sh'] {'content': InlineTemplate(...), 'owner': 'oozie', 'group': 'hadoop'} 2019-11-19 17:51:29,604 - Directory['/etc/security/limits.d'] {'owner': 'root', 'create_parents': True, 'group': 'root'} 2019-11-19 17:51:29,607 - File['/etc/security/limits.d/oozie.conf'] {'content': Template('oozie.conf.j2'), 'owner': 'root', 'group': 'root', 'mode': 0644} 2019-11-19 17:51:29,609 - File['/usr/hdp/current/oozie-client/conf/oozie-log4j.properties'] {'content': InlineTemplate(...), 'owner': 'oozie', 'group': 'hadoop', 'mode': 0644} 2019-11-19 17:51:29,614 - File['/usr/hdp/current/oozie-client/conf/adminusers.txt'] {'content': Template('adminusers.txt.j2'), 'owner': 'oozie', 'group': 'hadoop', 'mode': 0644} 2019-11-19 17:51:29,614 - File['/usr/lib/ambari-agent/DBConnectionVerification.jar'] {'content': DownloadSource('http://luapp698.lu.ema.ad.pwcinternal.com:8080/resources/DBConnectionVerification.jar')} 2019-11-19 17:51:29,614 - Not downloading the file from http://luapp698.lu.ema.ad.pwcinternal.com:8080/resources/DBConnectionVerification.jar, because /var/lib/ambari-agent/tmp/DBConnectionVerification.jar already exists 2019-11-19 17:51:29,615 - File['/usr/hdp/current/oozie-client/conf/hadoop-config.xml'] {'owner': 'oozie', 'group': 'hadoop'} 2019-11-19 17:51:29,615 - File['/usr/hdp/current/oozie-client/conf/oozie-default.xml'] {'owner': 'oozie', 'group': 'hadoop'} 2019-11-19 17:51:29,615 - Directory['/usr/hdp/current/oozie-client/conf/action-conf'] {'owner': 'oozie', 'group': 'hadoop'} 2019-11-19 17:51:29,615 - File['/usr/hdp/current/oozie-client/conf/action-conf/hive.xml'] {'owner': 'oozie', 'group': 'hadoop'} 2019-11-19 17:51:29,618 - The repository with version 3.1.4.0-315 for this command has been marked as resolved. It will be used to report the version of the component which was installed 2019-11-19 17:51:29,830 - Using hadoop conf dir: /usr/hdp/3.1.4.0-315/hadoop/conf 2019-11-19 17:51:29,856 - Execute[('ambari-python-wrap', u'/usr/bin/hdp-select', 'set', u'oozie-client', u'3.1.4.0-315')] {'sudo': True} 2019-11-19 17:51:29,881 - The repository with version 3.1.4.0-315 for this command has been marked as resolved. It will be used to report the version of the component which was installed Command failed after 1 tries   Any suggestions please? Thanks   ... View more

Dear Community,   I have choose to install the service Oozie from Ambari, when I checked the installed services , Oozie is green. However ,  I am not able to start it from the ambari web  I get this error : Background Operations / Restart all components with Stale Configs for Oozie / / Restart Oozie Server Task Log stderr: /var/lib/ambari-agent/data/errors-916.txt On host luapp699.lu.ema.ad.pwcinternal.com role OOZIE_SERVER in invalid state. Invalid transition. Invalid event: HOST_SVCCOMP_OP_IN_PROGRESS at INSTALL_FAILED   When I check for oozie server or client    /usr/bin/hdp-select accumulo-client - None accumulo-gc - None accumulo-master - None accumulo-monitor - None accumulo-tablet - None accumulo-tracer - None atlas-client - None atlas-server - None beacon - None beacon-client - None beacon-server - None druid-broker - None druid-coordinator - None druid-historical - None druid-middlemanager - None druid-overlord - None druid-router - None druid-superset - None falcon-client - None falcon-server - None flume-server - None hadoop-client - 3.1.4.0-315 hadoop-hdfs-client - 3.1.4.0-315 hadoop-hdfs-datanode - 3.1.4.0-315 hadoop-hdfs-journalnode - 3.1.4.0-315 hadoop-hdfs-namenode - 3.1.4.0-315 hadoop-hdfs-nfs3 - 3.1.4.0-315 hadoop-hdfs-portmap - 3.1.4.0-315 hadoop-hdfs-secondarynamenode - 3.1.4.0-315 hadoop-hdfs-zkfc - 3.1.4.0-315 hadoop-httpfs - None hadoop-mapreduce-client - 3.1.4.0-315 hadoop-mapreduce-historyserver - 3.1.4.0-315 hadoop-yarn-client - 3.1.4.0-315 hadoop-yarn-nodemanager - 3.1.4.0-315 hadoop-yarn-registrydns - 3.1.4.0-315 hadoop-yarn-resourcemanager - 3.1.4.0-315 hadoop-yarn-timelinereader - 3.1.4.0-315 hadoop-yarn-timelineserver - 3.1.4.0-315 hbase-client - 3.1.4.0-315 hbase-master - 3.1.4.0-315 hbase-regionserver - 3.1.4.0-315 hive-client - 3.1.4.0-315 hive-metastore - 3.1.4.0-315 hive-server2 - 3.1.4.0-315 hive-server2-hive - 3.1.4.0-315 hive-server2-hive2 - None hive-webhcat - 3.1.4.0-315 hive_warehouse_connector - 3.1.4.0-315 kafka-broker - None knox-server - None livy-client - None livy-server - None livy2-client - 3.1.4.0-315 livy2-server - 3.1.4.0-315 mahout-client - None Traceback (most recent call last): File "/usr/bin/hdp-select", line 453, in <module> listPackages(getPackages("all")) File "/usr/bin/hdp-select", line 271, in listPackages os.path.basename(os.path.dirname(os.readlink(linkname))))   However in the ambari it is like they have been installed, should be a space issue?   Best Regards   ... View more

Dear Community   I have a lot of issues on ambari hosts , is it possible to recreate a cluster? or recreate its hosts?   Thanks  ... View more

Dear Community   I try to install hadoop cluster using ambari server. I did not install oozie from the beginning , the nodes have been installed and now I am trying to install clients. Server install fails .. Oozie client fails  stderr: /var/lib/ambari-agent/data/errors-618.txt Python script has been killed due to timeout after waiting 1800 secs stdout: /var/lib/ambari-agent/data/output-618.txt 2019-11-18 11:29:14,797 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.4.0-315 -> 3.1.4.0-315 2019-11-18 11:29:14,802 - Using hadoop conf dir: /usr/hdp/3.1.4.0-315/hadoop/conf 2019-11-18 11:29:14,804 - Group['hdfs'] {} 2019-11-18 11:29:14,805 - Group['hadoop'] {} 2019-11-18 11:29:14,805 - Group['users'] {} 2019-11-18 11:29:14,805 - User['yarn-ats'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None} 2019-11-18 11:29:14,806 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None} 2019-11-18 11:29:14,807 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None} 2019-11-18 11:29:14,807 - User['oozie'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'users'], 'uid': None} 2019-11-18 11:29:14,808 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'users'], 'uid': None} 2019-11-18 11:29:14,808 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hdfs', 'hadoop'], 'uid': None} 2019-11-18 11:29:14,809 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None} 2019-11-18 11:29:14,810 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None} 2019-11-18 11:29:14,810 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2019-11-18 11:29:14,811 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'} 2019-11-18 11:29:14,816 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] due to not_if 2019-11-18 11:29:14,817 - Group['hdfs'] {} 2019-11-18 11:29:14,817 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': ['hdfs', 'hadoop', u'hdfs']} 2019-11-18 11:29:14,818 - FS Type: HDFS 2019-11-18 11:29:14,818 - Directory['/etc/hadoop'] {'mode': 0755} 2019-11-18 11:29:14,831 - File['/usr/hdp/3.1.4.0-315/hadoop/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'} 2019-11-18 11:29:14,831 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 01777} 2019-11-18 11:29:14,846 - Repository['HDP-3.1-repo-2'] {'base_url': 'http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.1.4.0', 'action': ['prepare'], 'components': [u'HDP', 'main'], 'repo_template': '[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0', 'repo_file_name': 'ambari-hdp-2', 'mirror_list': None} 2019-11-18 11:29:14,852 - Repository['HDP-3.1-GPL-repo-2'] {'base_url': 'http://public-repo-1.hortonworks.com/HDP-GPL/centos7/3.x/updates/3.1.4.0', 'action': ['prepare'], 'components': [u'HDP-GPL', 'main'], 'repo_template': '[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0', 'repo_file_name': 'ambari-hdp-2', 'mirror_list': None} 2019-11-18 11:29:14,854 - Repository['HDP-UTILS-1.1.0.22-repo-2'] {'base_url': 'http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7', 'action': ['prepare'], 'components': [u'HDP-UTILS', 'main'], 'repo_template': '[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0', 'repo_file_name': 'ambari-hdp-2', 'mirror_list': None} 2019-11-18 11:29:14,855 - Repository[None] {'action': ['create']} 2019-11-18 11:29:14,856 - File['/tmp/tmpXTNL1e'] {'content': '[HDP-3.1-repo-2]\nname=HDP-3.1-repo-2\nbaseurl=http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.1.4.0\n\npath=/\nenabled=1\ngpgcheck=0\n[HDP-3.1-GPL-repo-2]\nname=HDP-3.1-GPL-repo-2\nbaseurl=http://public-repo-1.hortonworks.com/HDP-GPL/centos7/3.x/updates/3.1.4.0\n\npath=/\nenabled=1\ngpgcheck=0\n[HDP-UTILS-1.1.0.22-repo-2]\nname=HDP-UTILS-1.1.0.22-repo-2\nbaseurl=http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7\n\npath=/\nenabled=1\ngpgcheck=0'} 2019-11-18 11:29:14,857 - Writing File['/tmp/tmpXTNL1e'] because contents don't match 2019-11-18 11:29:14,857 - File['/tmp/tmpuFs8Qh'] {'content': StaticFile('/etc/yum.repos.d/ambari-hdp-2.repo')} 2019-11-18 11:29:14,857 - Writing File['/tmp/tmpuFs8Qh'] because contents don't match 2019-11-18 11:29:14,858 - Package['unzip'] {'retry_on_repo_unavailability': False, 'retry_count': 5} 2019-11-18 11:29:14,938 - Skipping installation of existing package unzip 2019-11-18 11:29:14,938 - Package['curl'] {'retry_on_repo_unavailability': False, 'retry_count': 5} 2019-11-18 11:29:14,947 - Skipping installation of existing package curl 2019-11-18 11:29:14,947 - Package['hdp-select'] {'retry_on_repo_unavailability': False, 'retry_count': 5} 2019-11-18 11:29:14,955 - Skipping installation of existing package hdp-select 2019-11-18 11:29:14,959 - The repository with version 3.1.4.0-315 for this command has been marked as resolved. It will be used to report the version of the component which was installed 2019-11-18 11:29:15,158 - Package['zip'] {'retry_on_repo_unavailability': False, 'retry_count': 5} 2019-11-18 11:29:15,231 - Skipping installation of existing package zip 2019-11-18 11:29:15,233 - Package['oozie_3_1_4_0_315'] {'retry_on_repo_unavailability': False, 'retry_count': 5} 2019-11-18 11:29:15,241 - Installing package oozie_3_1_4_0_315 ('/usr/bin/yum -y install oozie_3_1_4_0_315') Command failed after 1 tries   Any idea please?   Thanks Asma ... View more

Hello Community, When I try to start ambari-agent with root on centos i have permission denied Should this be related to noexec?   # /etc/fstab # Created by anaconda on Thu Nov 7 12:40:54 2019 # # Accessible filesystems, by reference, are maintained under '/dev/disk' # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info # /dev/mapper/centos-root / xfs defaults 0 0 UUID=656e7855-a771-474b-b53d-7b69fc4f0d67 /boot xfs defaults 0 0 /dev/mapper/centos-home /home xfs nodev 0 0 /dev/mapper/centos-tmp /tmp xfs nodev,nosuid,noexec 0 0 /dev/mapper/centos-var /var xfs nodev,nosuid,noexec 0 0 /dev/mapper/centos-var_log /var/log xfs nodev,nosuid,noexec 0 0 /dev/mapper/centos-var_log_audit /var/log/audit xfs nodev,nosuid,noexec 0 0 /dev/mapper/centos-swap swap swap defaults 0 0 /tmp/ /var/tmp none defaults,nodev,nosuid,noexec,bind 0 0 ~ ~ Thanks ^^ ... View more

Dear community   I have installed iâmabri server on a centos7 machine : 2.4.7  it is up and running I have installed an ambari agent but I got Permission denied I am using a root user : [root@luapp698 bin]# sudo ambari-agent status bash: /var/lib/ambari-agent/bin/ambari-agent: Permission denied Is there some wrong configuration that I have made during install? I have the same issue on all the other nodes Please assist Thanks and Regards ... View more

Hello Community   I am trying to start the ambari agent with root account but I got permission denied any idea please? 😞   [root@luapp699 lib]# ambari-agent start bash: /var/lib/ambari-agent/bin/ambari-agent: Permission denied   Thanks and regards ... View more