@Benjamin Leonhardi Try this https://github.com/hkropp/vagrant-hdp/blob/master/bin/ambari-shell.jar and run it with % java -jar ambari-shell.jar --ambari.host= ... View more

The provided workaround by @Alessio Ubaldi seems to work. But you should also try to first upgrade to 2.3.7. ... View more

The Error message in /var/log/knox/gateway.log says that the certificate used by Knox will be valid starting in the future: Failed to start gateway: org.apache.hadoop.gateway.services.ServiceLifecycleException: Gateway SSL Certificate is not yet valid. Server will not start. -> "not yet valid" Knox refuses to start, because using such a certificate will result in an SSL exception for almost any client. You will need to check the certificate your are using for Knox. This is stored as gateway-identity in gateway.jks under /var/lib/knox/data*/keystore Please refer to this: http://knox.apache.org/books/knox-0-6-0/user-guide.html#Management+of+Security+Artifacts What also should work is, if you simply remove the gateway-identity from the keystore, upon start Knox should create a self-signed certificate for you. Could you share how the certificate was generated? Did you change it after the install? Are you using ntp? ... View more

Can you also provide what you find in /var/log/knox/gateway.log ... View more

Actually this does not quite answer the question, but gives a good hint to dfs.internal.nameservices. The parameter is needed to distinguish between local namservice and other nameservices configured, but does not support distcp between two HA clusters. dfs.internal.nameservices for example is relevant for DNs so they don't register with the other cluster. To support distcp between multiple HA clusters you simply have to define multiple nameservices like this for an example: <configuration> <!-- services --> <property> <name>dfs.nameservices</name> <value>serviceId1,serviceId2</value> </property> <!-- serviceId2 properties --> <property> <name>dfs.client.failover.proxy.provider.nameservices2</name> <value>org.apache.hadoop.hdfs.server .namenode.ha.ConfiguredFailoverProxyProvider</value> </property> <property> <name>dfs.ha.namenodes.serviceId2</name> <value>nn201,nn202</value> </property> <property> <name>dfs.namenode.rpc-address.serviceId2.nn201</name> <value>nn201.pro.net:8020</value> </property> <property> <name>dfs.namenode.servicerpc-address.serviceId2.nn201</name> <value>nn201.pro.net:54321</value> </property> <property> <name>dfs.namenode.http-address.serviceId2.nn201</name> <value>nn201.pro.net:50070</value> </property> <property> <name>dfs.namenode.https-address.serviceId2.nn201</name> <value>nn201.prod.com:50470</value> </property> <property> <name>dfs.namenode.rpc-address.serviceId2.nn202</name> <value>nn202.pro.net:8020</value> </property> <property> <name>dfs.namenode.servicerpc-address.serviceId2.nn202</name> <value>nn202.pro.net:54321</value> </property> <property> <name>dfs.namenode.http-address.serviceId2.nn202</name> <value>nn202.pro.net:50070</value> </property> <property> <name>dfs.namenode.https-address.serviceId2.nn202</name> <value>nn202.prod.net:50470</value> </property> <!—- serviceId1 --> <property> <name>dfs.client.failover.proxy.provider.nameservices1</name> <value>org.apache.hadoop.hdfs.server.namenode.ha. ConfiguredFailoverProxyProvider</value> </property> <property> <name>dfs.ha.namenodes.nameservices1</name> <value>nn101,nn102</value> </property> <property> <name>dfs.namenode.rpc-address.serviceId1.nn101</name> <value>nn101.poc.net:8020</value> </property> <property> <name>dfs.namenode.servicerpc-address.serviceId1.nn101</name> <value>nn101.poc.net:54321</value> </property> <property> <name>dfs.namenode.http-address.serviceId1.nn101</name> <value>nn101.poc.net:50070</value> </property> <property> <name>dfs.namenode.https-address.serviceId1.nn101</name> <value>nn101.poc.net:50470</value> </property> <property> <name>dfs.namenode.rpc-address.serviceId1.nn102</name> <value>nn102.poc.net:8020</value> </property> <property> <name>dfs.namenode.servicerpc-address.serviceId1.nn102</name> <value>nn102.poc.net:54321</value> </property> <property> <name>dfs.namenode.http-address.serviceId1.nn102</name> <value>nn102.poc.net:50070</value> </property> <property> <name>dfs.namenode.https-address.serviceId1.nn102</name> <value>nn102.poc.net:50470</value> </property> </configuration> Adding this to the hdfs-site config makes both nameservices serviceId1,serviceId2 available. ... View more

+1 for the aspect to reuse Spark code itself ... View more