About peter_svarc

peter_svarc · ‎09-02-2019

Oh, I see now. There seems to be problem on the server side when rewriting the docs.hortonworks.com domain to docs.cloudera.com, so the /HDPDocuments appears twice in the URL. Once I'm on the docs.cloudera.com domain, the links are working once again...

peter_svarc · ‎09-02-2019

Does anyone know where the HDP documentation went? The https://docs.hortonworks.com/ gets redirected to https://docs.cloudera.com/index.html. And the link to HDP documentation URL mentioned there (https://docs.cloudera.com/HDPDocuments/index.html) gets redirected back to https://docs.cloudera.com/index.html. BTW, any direct link to original docs gives an HTTP 404 error message, such as: 404 Not Found Code: NoSuchKey Message: The specified key does not exist. Key: HDPDocuments/HDPDocuments/HDPDocuments/HDP3/HDP-3.1.0/authentication-with-kerberos/content/authe_spnego_enabling_browser_access_to_a_spnego_enabled_web_ui.html RequestId: 1F8B4A953909E2F5 HostId: UAdEmAmTrYb4pz3EPxI92SkgHo9kVUlwvXjCnWjRhFzJr3kbvUKHxE9mpDL1hLzdsI5HbGS2MlA=

peter_svarc · ‎08-30-2019

I have dug into it a little deeper and it seems that it is actually an expected behaviour. It all seems to be related to the concept of Hadoop in Secure Mode (see https://cwiki.apache.org/confluence/display/HADOOP/Secure+DataNode and https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SecureMode.html). Enabling Kerberos actually turns the Hadoop Secure Mode on, which requires DataNode to bind to a privileged port. Ambari handles this requirement by issuing DataNode start command as the root user, which leverages the jsvc utility to start the process under he root user, bind to privileged ports, and then drops the privileges. All required configuration seems to be handled by Ambari correctly, so at the end the only missing piece of the puzzle is the required sudo privileges. So just incomplete HDP documentation. After adding the following line to the sudoers file, DataNode starts correctly: ambari ALL=(ALL) NOPASSWD:SETENV: /usr/hdp/*/hadoop/bin/hdfs *, /usr/hdp/*/hadoop-yarn/bin/yarn * Actually, similar issue occurs when starting the YARN Registry DNS component, which binds to privileged port 53 by default. That is the reason why the extra line mentioned above specifies /usr/hdp/*/hadoop-yarn/bin/yarn * as well.

peter_svarc · ‎08-30-2019

Hello everyone, I have installed Ambari 2.7.3 and HDP 3.1.0, setup Ambari to run as non-root, configured sudo rights as described in the documentation, and finally kerberized the cluster running the Kerberos Wizard. Now, the DataNode does not start as the non-root user is not allowed to start the datanode. I have compared the command that Ambari issues to start the Datanode with the same one before the cluster was kerberized and they differ. DataNode starting command before Kerberos setup: 2019-08-27 10:19:15,761 - Execute['ambari-sudo.sh su qpthdfs -l -s /bin/bash -c 'ulimit -c unlimited ; /usr/hdp/3.1.0.0-78/hadoop/bin/hdfs --config /usr/hdp/3.1.0.0-78/hadoop/conf --daemon start datanode''] {'environment': {'HADOOP_LIBEXEC_DIR': '/usr/hdp/3.1.0.0-78/hadoop/libexec'}, 'not_if': '(ambari-sudo.sh -H -E test -f /var/run/hadoop/qpthdfs/hadoop-qpthdfs-root-datanode.pid && ambari-sudo.sh -H -E pgrep -F /var/run/hadoop/qpthdfs/hadoop-qpthdfs-root-datanode.pid) || (ambari-sudo.sh -H -E test -f /var/run/hadoop/qpthdfs/hadoop-qpthdfs-datanode.pid && ambari-sudo.sh -H -E pgrep -F /var/run/hadoop/qpthdfs/hadoop-qpthdfs-datanode.pid)'} DataNode starting command after Kerberos setup: 2019-08-30 10:13:13,772 - Execute['ambari-sudo.sh -H -E /usr/hdp/3.1.0.0-78/hadoop/bin/hdfs --config /usr/hdp/3.1.0.0-78/hadoop/conf --daemon start datanode'] {'environment': {'HADOOP_LIBEXEC_DIR': '/usr/hdp/3.1.0.0-78/hadoop/libexec'}, 'not_if': '(ambari-sudo.sh -H -E test -f /var/run/hadoop/qpthdfs/hadoop-qpthdfs-root-datanode.pid && ambari-sudo.sh -H -E pgrep -F /var/run/hadoop/qpthdfs/hadoop-qpthdfs-root-datanode.pid) || (ambari-sudo.sh -H -E test -f /var/run/hadoop/qpthdfs/hadoop-qpthdfs-datanode.pid && ambari-sudo.sh -H -E pgrep -F /var/run/hadoop/qpthdfs/hadoop-qpthdfs-datanode.pid)'} As far as I understand the difference between the commands, the DataNode gets started as a HDFS technical user (qpthdfs) before Kerberos setup. While after Kerberos setup, Ambari tries to start the DataNode as a root user, which is something I don't want to. Is that an expected behaviour? Also, can anyone point me to the correct place where Ambari is building these commands as I'm afraid I will have to hack that to be able to start DataNode. Thanks, P.S.

peter_svarc · ‎09-21-2017

Hello, I have setup NiFi cluster containing two nodes. Each NiFi node is on a host with two network interfaces, thus with two IP addresses (internal and external networks). I was trying to setup the nifi.web.http.host property with value 0.0.0.0 to make NiFi's Jetty web server listen on both addresses. The nifi.cluster.node.address contains the IP address of the internal network interface. The cluster starts, but both nodes are identified as '0.0.0.0:8090', so when I try to make changes to the flow, I get an error that the change is already being applied on the '0.0.0.0:8090' node. Shouldn't the nifi.cluster.node.address be used for identifying the node in the cluster? Or how these properties get used when NiFi cluster is setup? I'm using NiFi 1.1.2. Regards, Petr Švarc

Date Registered	‎09-21-2017 07:18 AM
Last Visited
Total Messages Posted	5
Total Kudos Received	1

Re: HDP documentation missing

HDP documentation missing

Re: HDP 3.1.0: cannot start hdfs datanode after en...

HDP 3.1.0: cannot start hdfs datanode after enabli...

NiFi - difference between nifi.web.http.host and n...