question Re: Spark2 conf directory is empty in Archives of Support Questions (Read Only)

Spark2 conf directory is empty

facundo — Thu, 26 Oct 2017 23:20:20 GMT

Hi, I just installed Spark2 from Ambari wizard and the Spark2's configuration directory is empty:

> ls -l /etc/spark2/2.6.2.0-205/0
total 0

The installation output is:

14:39:22,875 - Backing up /etc/spark2/conf to /etc/spark2/conf.backup if destination doesn't exist already.
14:39:22,875 - Execute[('cp', '-R', '-p', '/etc/spark2/conf', '/etc/spark2/conf.backup')] {'not_if': 'test -e /etc/spark2/conf.backup', 'sudo': True}
14:39:22,897 - Checking if need to create versioned conf dir /etc/spark2/2.6.2.0-205/0
14:39:22,900 - call[('ambari-python-wrap', u'/usr/bin/conf-select', 'dry-run-create', '--package', 'spark2', '--stack-version', u'2.6.2.0-205', '--conf-version', '0')] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1}
14:39:22,940 - call returned (0, '/etc/spark2/2.6.2.0-205/0', '')
14:39:22,941 - Package spark2 will have new conf directories: /etc/spark2/2.6.2.0-205/0
14:39:22,946 - Checking if need to create versioned conf dir /etc/spark2/2.6.2.0-205/0
14:39:22,952 - call[('ambari-python-wrap', u'/usr/bin/conf-select', 'create-conf-dir', '--package', 'spark2', '--stack-version', u'2.6.2.0-205', '--conf-version', '0')] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1}
14:39:22,987 - call returned (1, '/etc/spark2/2.6.2.0-205/0 exist already', '')
14:39:22,988 - checked_call[('ambari-python-wrap', u'/usr/bin/conf-select', 'set-conf-dir', '--package', 'spark2', '--stack-version', u'2.6.2.0-205', '--conf-version', '0')] {'logoutput': False, 'sudo': True, 'quiet': False}
14:39:23,022 - checked_call returned (0, '/usr/hdp/2.6.2.0-205/spark2/conf -> /etc/spark2/2.6.2.0-205/0')
14:39:23,023 - Ensuring that spark2 has the correct symlink structure
14:39:23,024 - Execute[('cp', '-R', '-p', '/etc/spark2/conf', '/etc/spark2/conf.backup')] {'not_if': 'test -e /etc/spark2/conf.backup', 'sudo': True}
14:39:23,033 - Skipping Execute[('cp', '-R', '-p', '/etc/spark2/conf', '/etc/spark2/conf.backup')] due to not_if
14:39:23,034 - Directory['/etc/spark2/conf'] {'action': ['delete']}
14:39:23,034 - Removing directory Directory['/etc/spark2/conf'] and all its content
14:39:23,035 - Link['/etc/spark2/conf'] {'to': '/etc/spark2/conf.backup'}
14:39:23,035 - Creating symbolic Link['/etc/spark2/conf'] to /etc/spark2/conf.backup
14:39:23,036 - Link['/etc/spark2/conf'] {'action': ['delete']}
14:39:23,036 - Deleting Link['/etc/spark2/conf']
14:39:23,037 - Link['/etc/spark2/conf'] {'to': '/usr/hdp/current/spark2-client/conf'}
14:39:23,037 - Creating symbolic Link['/etc/spark2/conf'] to /usr/hdp/current/spark2-client/conf
14:39:23,037 - /etc/hive/conf is already linked to /etc/hive/2.6.2.0-205/0

I'm using Ambari version 2.5.2.0, HDP version 2.6.2.0-205 and Spark2 version 2.1.1.

Do you know what happened?

There is a way to install again the Spark2 configuration?

Thanks in advance.

Re: Spark2 conf directory is empty

asirna — Fri, 27 Oct 2017 00:06:03 GMT

@Facundo Bianco,

Seems like spark2 client is not installed.

Can you please check this directory (/usr/hdp/2.6.2.0-205/spark2/conf/) . If the directory is not empty you can copy files from the mentioned directory to /etc/spark2/2.6.2.0-205/0.

If the directory is empty , try installing the client and check the directory

yum install spark2_2_6_2_0_205-master spark2_2_6_2_0_205-python

You can also install the client using REST API

curl -k -u {username}:{password} -H "X-Requested-By:ambari" -i -X PUT -d '{"HostRoles": {"state": "INSTALLED"}}'  http://{ambari-host}:{ambari-port}/api/v1/clusters/{clustername}/hosts/{hostname}/host_components/SPARK2_CLIENT

Replace ambari-host, ambari-port , clustername, hostname,username and password with your values

Thanks,

Aditya

Re: Spark2 conf directory is empty

facundo — Fri, 27 Oct 2017 00:29:36 GMT

Hi @Aditya Sirna,

The directory /usr/hdp/2.6.2.0-205/spark2/conf/ is empty but I have installed

spark2_2_6_2_0_205-python-2.1.1.2.6.2.0-205.noarch
spark2_2_6_2_0_205-2.1.1.2.6.2.0-205.noarch

Re: Spark2 conf directory is empty

asirna — Fri, 27 Oct 2017 00:34:32 GMT

@Facundo Bianco

Can you please try running the curl command mentioned above. That should install the client

Re: Spark2 conf directory is empty

facundo — Tue, 31 Oct 2017 03:23:28 GMT

I solved it with a comment on Spark2, Config, Advanced spark2-env. After that, I restarted Spark2 and their clients and the new configuration files were deployed.