distcp - How to determine number of mappers used by distcp job(s) at cluster level? Sometime we run into network bandwidth issue caused by distcp job(s) running too many mappers or too many distcp jobs. Our plan is to trigger DataDog alert when the total number of mappers used by distcp jobs (at cluster level) reach at defined number (ex: 100). We are open to explore the "-bandwidth" option. We have many users who will be submitting a job from diff edge nodes. so, we don't want to use the "ps" command at server level. Please help us rectify the issue. Thanks in advance. ... View more

Please help me configure a custom script based alert for a non-Hadoop process running on single or multiple nodes I’ve refereed to these articles already. https://github.com/apache/ambari/blob/branch-2.1/ambari-server/docs/api/v1/alert-definitions.md https://cwiki.apache.org/confluence/display/AMBARI/Customizing+the+Alert+Template Script: XYZ-process-chk.sh #!/bin/sh PROCESS_CHK=`code` if [ code ] then echo "process is NOT rounning" fi JSON file: { "AlertDefinition" : { "cluster_name" : "NAME", "alert.hasHostName" : "host1", "host2", # I may need this property "component_name" : "XYZ", "description" : null, "enabled" : true, "help_url" : null, "ignore_host" : false, "interval" : 1, "label" : "XYZ Process", "name" : "XYZ_process", "repeat_tolerance" : 1, "repeat_tolerance_enabled" : false, "scope" : "HOST", "service_name" : "XYZ", "source" : { "path" : "/var/lib/ambari-server/resources/host_scripts/XYZ-process-chk.sh", "type" : "SCRIPT" } } ... View more

I've setup Apace drill 1.10.0 on RHEL 6.7, HDP 2.5.3, kerberos enabled. I'm getting below error while running "drill-conf" as user "drill" which is configured in the "drill-override.conf" file and it has valid keytab/ticket. drill@host:/opt/drill/bin> drill-conf Error: Failure in connecting to Drill: org.apache.drill.exec.rpc.NonTransientRpcException: javax.security.sasl.SaslException: Authentication failed: Server requires authentication using [kerberos, plain]. Insufficient credentials? [Caused by javax.security.sasl.SaslException: Server requires authentication using [kerberos, plain]. Insufficient credentials?] (state=,code=0) java.sql.SQLException: Failure in connecting to Drill: org.apache.drill.exec.rpc.NonTransientRpcException: javax.security.sasl.SaslException: Authentication failed: Server requires authentication using [kerberos, plain]. Insufficient credentials? [Caused by javax.security.sasl.SaslException: Server requires authentication using [kerberos, plain]. Insufficient credentials?] at org.apache.drill.jdbc.impl.DrillConnectionImpl.<init>(DrillConnectionImpl.java:166) at org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:72) at org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69) at org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:143) at org.apache.drill.jdbc.Driver.connect(Driver.java:72) at sqlline.DatabaseConnection.connect(DatabaseConnection.java:167) at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:213) at sqlline.Commands.connect(Commands.java:1083) at sqlline.Commands.connect(Commands.java:1015) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36) at sqlline.SqlLine.dispatch(SqlLine.java:742) at sqlline.SqlLine.initArgs(SqlLine.java:528) at sqlline.SqlLine.begin(SqlLine.java:596) at sqlline.SqlLine.start(SqlLine.java:375) at sqlline.SqlLine.main(SqlLine.java:268) Caused by: org.apache.drill.exec.rpc.NonTransientRpcException: javax.security.sasl.SaslException: Authentication failed: Server requires authentication using [kerberos, plain]. Insufficient credentials? [Caused by javax.security.sasl.SaslException: Server requires authentication using [kerberos, plain]. Insufficient credentials?] at org.apache.drill.exec.rpc.user.UserClient.connect(UserClient.java:157) at org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:432) at org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:379) at org.apache.drill.jdbc.impl.DrillConnectionImpl.<init>(DrillConnectionImpl.java:157) ... 18 more Caused by: javax.security.sasl.SaslException: Authentication failed: Server requires authentication using [kerberos, plain]. Insufficient credentials? [Caused by javax.security.sasl.SaslException: Server requires authentication using [kerberos, plain]. Insufficient credentials?] at org.apache.drill.exec.rpc.user.UserClient$3.mapException(UserClient.java:204) at org.apache.drill.exec.rpc.user.UserClient$3.mapException(UserClient.java:197) at com.google.common.util.concurrent.AbstractCheckedFuture.checkedGet(AbstractCheckedFuture.java:85) at org.apache.drill.exec.rpc.user.UserClient.connect(UserClient.java:155) ... 21 more Caused by: javax.security.sasl.SaslException: Server requires authentication using [kerberos, plain]. Insufficient credentials? at org.apache.drill.exec.rpc.user.UserClient.getAuthenticatorFactory(UserClient.java:285) at org.apache.drill.exec.rpc.user.UserClient.authenticate(UserClient.java:216) ... 22 more apache drill 1.10.0 "this isn't your grandfather's sql" Error from "sqlline.log": 2017-04-07 14:18:22,564 [main] WARN o.a.drill.exec.metrics.DrillMetrics - Removing old metric since name matched newly registered metric. Metric name: drill.allocator.root.used 2017-04-07 14:18:22,565 [main] WARN o.a.drill.exec.metrics.DrillMetrics - Removing old metric since name matched newly registered metric. Metric name: drill.allocator.root.peak 2017-04-07 14:18:22,615 [main] INFO o.a.drill.common.config.DrillConfig - Configuration and plugin file(s) identified in 50ms. Base Configuration: - jar:file:/opt/drill/jars/drill-common-1.10.0.jar!/drill-default.conf Intermediate Configuration and Plugin files, in order of precedence: - jar:file:/opt/drill/jars/drill-java-exec-1.10.0.jar!/drill-module.conf - jar:file:/opt/drill/jars/drill-common-1.10.0.jar!/drill-module.conf - jar:file:/opt/drill/jars/drill-storage-hive-core-1.10.0.jar!/drill-module.conf - jar:file:/opt/drill/jars/drill-memory-base-1.10.0.jar!/drill-module.conf - jar:file:/opt/drill/jars/drill-mongo-storage-1.10.0.jar!/drill-module.conf - jar:file:/opt/drill/jars/drill-jdbc-storage-1.10.0.jar!/drill-module.conf - jar:file:/opt/drill/jars/drill-logical-1.10.0.jar!/drill-module.conf - jar:file:/opt/drill/jars/drill-storage-hbase-1.10.0.jar!/drill-module.conf - jar:file:/opt/drill/jars/drill-gis-1.10.0.jar!/drill-module.conf - jar:file:/opt/drill/jars/drill-kudu-storage-1.10.0.jar!/drill-module.conf - jar:file:/opt/drill/jars/drill-hive-exec-shaded-1.10.0.jar!/drill-module.conf Override File: file:/opt/drill/conf/drill-override.conf 2017-04-07 14:18:22,625 [main] WARN o.a.drill.exec.metrics.DrillMetrics - Removing old metric since name matched newly registered metric. Metric name: drill.allocator.root.used 2017-04-07 14:18:22,625 [main] WARN o.a.drill.exec.metrics.DrillMetrics - Removing old metric since name matched newly registered metric. Metric name: drill.allocator.root.peak 2017-04-07 14:18:22,671 [main] ERROR o.a.drill.exec.client.DrillClient - Connection to host1.com:31010 failed with error javax.security.sasl.SaslException: Authentication failed: Server requires authentication using [kerberos, plain]. Insufficient credentials? [Caused by javax.security.sasl.SaslException: Server requires authentication using [kerberos, plain]. Insufficient credentials?]. Not retrying anymore The drill web console is working with with PAM authentication. i was able to login with local user "drill" Here is my "drill-override.conf" file: drill.exec: { cluster-id: "drillbits1", zk.connect: "host1:2181,host2:2181,host3:2181", security: { user.auth.enabled: true, user.auth.impl: "pam", user.auth.pam_profiles: [ "sudo", "login" ], packages += "org.apache.drill.exec.rpc.user.security", auth.mechanisms: ["KERBEROS","PLAIN"], auth.principal: "drill/labhdp@LAB.COM", auth.keytab: "/opt/drill/.keytab/drill.keytab" } } Please help me resolve the issue. ... View more

Has anyone integrated Ambari traps into IBM Netcool? HDP: 2.4.3, Ambari: 2.4.2, IBM Netcool: Netcool/OMNIbus Object Server - Version 8.1.0 64bit Kerberos: yes MIB file: wget -nv https://raw.githubusercontent.com/apache/ambari/trunk/ambari-server/src/main/resources/APACHE-AMBARI-MIB.txt -O /usr/share/snmp/mibs/APACHE-AMBARI-MIB.txt Manage Alert Notifications configuration: Groups: All Severity: All Method: SNMP Version: SNMPv1 OID: 1.3.6.1.4.1.18060.16.0.1 **** (also tried with 1.3.6.1.4.1.18060.16.1.1) Community: public Host: xyz.com Port: 162 This is what we have accomplished so far: We tested several traps using the Netcool MIB browser and all of them seem to have been parsed correctly via the rules we have setup using the MIB provided to us (though some adjustments to the Summary and other fields are needed based on further review of the traps needed to be shown). The actual traps that you are generating on the device side are all defaulting to one Generic trap (Generic cold start) and getting de-duplicated as a single trap on Netcool end (Netcool Object server uses the Identifier field that gets specifically set using several fields[OIDs and varbinds] from the trap). I've opened a case with HW support requesting more details on the actual traps that are being sent for each service so we can investigate further (Some sample trap PDUs that can help us identify what signifies a specific trap so we can validate the rules get mapped accordingly). ... View more

Everytime I try to access Grafana and/or browse through it, I see below error in "grafana.log": 2017/02/10 15:14:14 [middleware.go:145 initContextWithBasicAuth()] [E] Invalid Basic Auth Header: Invalid basic auth header 2017/02/10 15:14:14 [I] Completed 10.128.211.106 admin "GET /api/datasources/proxy/1/ws/v1/timeline/metrics HTTP/1.1" 401 Unauthorized 39 bytes in 43us If I make "auth.basc=false" I'm getting below error: Connecting (POST) to myserver.com:3000/api/datasources Http response: 403 Forbidden Http data: {"message":"Permission denied"} Ambari Metrics Grafana data source creation failed. It's a new install on HDP stack: 2.4.3.0 with Kerberos and SPNEGO is enabled I had such issue with HDP 2.3.4 but the suggested workaround is not working with 2.4.3.0: https://community.hortonworks.com/questions/59160/grafana-initcontextwithbasicauth-e-invalid-basic-a.html ... View more

Hi, I'm running a distcopy from one cluster to another and getting below error in the destination namenode log file. 2016-10-13 12:48:09,260 WARN blockmanagement.BlockPlacementPolicy (BlockPlacementPolicyDefault.java:chooseTarget(385)) - Failed to place enough replicas, still in need of 1 to reach 2 (unavailableStorages=[DISK], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}, newBlock=true) All required storage types are unavailable: unavailableStorages=[DISK], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]} In the destination cluster, I'm able to write to hdfs. Cluster health is green. I also see these .distcp.tmp* file in the destination cluster: hdfs dfs -ls /mstr/f_opt_wifi_ap_mth_chc/ Found 24 items -rw-r----- 3 hdfs hdfs 0 2016-10-13 11:00 /mstr/f_opt_wifi_ap_mth_chc/.distcp.tmp.attempt_1476280811282_0259_m_000000_0 -rw-r----- 3 hdfs hdfs 0 2016-10-13 11:00 /mstr/f_opt_wifi_ap_mth_chc/.distcp.tmp.attempt_1476280811282_0259_m_000001_0 -rw-r----- 3 hdfs hdfs 0 2016-10-13 11:00 /mstr/f_opt_wifi_ap_mth_chc/.distcp.tmp.attempt_1476280811282_0259_m_000002_0 ... View more