Member since
01-22-2019
12
Posts
2
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
277 | 01-03-2024 05:39 AM |
01-09-2024
08:19 PM
Hi @Nardoleo, There could be a few issues that can cause delays to jobs, Check the queue you are using to submit the jobs for enough sources. Check the data size and Data skewness which can also lead to delay in job. Also, the network needs to be monitored if that is causing a delay.
... View more
01-09-2024
08:14 PM
@SylvainL It seems your API request has some issues, Can you share the full request you are sending to /flow-files?
... View more
01-06-2024
07:35 PM
@phir1 , Create a queue for each dept and assign the requested resources. Now you know the dept is using how much resources.
... View more
01-03-2024
05:39 AM
@phir1 Generally I have seen customers chargeback on the usage of storage Quota and Resource allocation on the queue level. You can also use the above or chargeback based on node sponsorship. Don't forget to mark it accepted if you are satisfied 🙂
... View more
12-20-2023
10:16 PM
@jayes Can you share the logs, its hard to say without looking at logs but just a hunch it can be because of Kerberos authentication.
... View more
12-20-2023
10:11 PM
@vec Can you check your Zk logs, you will find the actual error in the logs. Seems like ZK is rejecting the RM Connection.
... View more
08-03-2022
10:09 PM
1 Kudo
Hi All, Below are the steps to setup load balancer on top of multiple Knox gateways. I will assume that you have KnoxSSO already set and services are going through it. Install knox gateway on another node Ambari/CM -> Hosts -> Select any host(where you want to install knox) and add knox gateway. Installation of HAproxy and Rsyslog: yum install -y haproxy rsyslog Step 3: Update the Rsyslog for checking the haproxy logs /etc/rsyslog.conf: ## Uncomment below line
$ModLoad imudp
$UDPServerRun 514
##Add following directive in the same file.
# HAProxy Logging
local2.* /var/log/haproxy.log Create SSL certificate for haproxy: openssl genrsa -out /etc/haproxy/haproxy.key 2048
openssl req -new -key /etc/haproxy/haproxy.key -out /etc/haproxy/haproxy.csr -subj "/C=US/ST=North Carolina/L=Raleigh/O=HWX/OU=Support/CN=$(hostname -f)"
openssl x509 -req -days 365 -in /etc/haproxy/haproxy.csr -signkey /etc/haproxy/haproxy.key -out /etc/haproxy/haproxy.crt
cat /etc/haproxy/haproxy.key /etc/haproxy/haproxy.crt > /etc/haproxy/haproxy.pem Update haproxy configuration file /etc/haproxy/haproxy.cfg with both Knox hosts and LB server, In my case c2110-node1 and c2110-node2 are Knox gateway host and c2110-node3 is the LB node: #---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend knox-frontend-443
mode http
option httplog
bind *:443 ssl crt /etc/haproxy/haproxy.pem
option forwardfor
http-request redirect location https://%[req.hdr(Host)]/gateway/homepage/home/ if { path / }
default_backend knox-backend-8443
backend knox-backend-8443
mode http
option httplog
option forwardfor
balance roundrobin
stick-table type ip size 1m expire 24h
stick on src
option httpchk HEAD /gateway/knoxsso/knoxauth/login.html HTTP/1.1\r\nHost:\ c1110-node1:8443
http-check expect status 200
server knox1 c1110-node2:8443 check ssl verify none
server knox2 c1110-node3:8443 check ssl verify none Start the services: systemctl start haproxy
systemctl enable haproxy
systemctl start rsyslog.service set LB url using ambari-server setup-sso command, Only change the url as PEM is already setup for knox1 hosts: [root@c2110-node1 ~]# ambari-server setup-sso
Using python /usr/bin/python
Setting up SSO authentication properties...
Enter Ambari Admin login: admin
Enter Ambari Admin password:
SSO is currently enabled
Do you want to disable SSO authentication [y/n] (n)?
Provider URL (https://c2110-node3.squadron.support.hortonworks.com:8443/gateway/knoxsso/api/v1/websso): https://c2110-node3.squadron.support.hortonworks.com:443/gateway/knoxsso/api/v1/websso
The SSO provider's public certificate has already set. Do you want to change it [y/n] (n)?
Use SSO for Ambari [y/n] (n)?
Manage SSO configurations for eligible services [y/n] (y)?
Use SSO for all services [y/n] (y)?
JWT Cookie name (hadoop-jwt):
JWT audiences list (comma-separated), empty for any ():
Ambari Server 'setup-sso' completed successfully. Generate the gateway-identity file and copy on both nodes: # will create gateway.jks in keystore folder
$GATEWAY_HOME/bin/knoxcli.sh create-cert --hostname localhost
# to check the CN=localhost
keytool -v -list -keystore gateway.jks
# create the gateway-identity.pem certificate from gateway.jks file
keytool -exportcert -alias gateway-identity -rfc -file $certificate_path -keystore $gateway_home/data/security/keystores/gateway.jks
# to check the CN=localhost
openssl x509 -in gateway-identity.pem -text
# scp these two files from one node to another
scp /usr/hdp/3.1.0.0-78/knox/data/security/keystores/gateway* c2110-node2:/usr/hdp/3.1.0.0-78/knox/data/security/keystores/ Thanks for reading.
... View more
01-25-2019
10:15 AM
@Mykyta, Did you check if that was the java version on all the nodemanager?
... View more
12-04-2018
08:44 AM
1 Kudo
@Jay Coder the question is: what happens then? does the failed Namenode becomes a standby node -- TRUE https://stackoverflow.com/questions/33311585/how-does-hadoop-namenode-failover-process-works
... View more