Member since
05-07-2018
331
Posts
45
Kudos Received
35
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
7428 | 09-12-2018 10:09 PM | |
2945 | 09-10-2018 02:07 PM | |
9766 | 09-08-2018 05:47 AM | |
3274 | 09-08-2018 12:05 AM | |
4263 | 08-15-2018 10:44 PM |
06-20-2018
12:57 AM
Awesome @Mahender S As the issue is resolved, hence it will be also great if you can mark this HCC thread as Answered by clicking on the "Accept" Button. That way other HCC users can quickly find the solution when they encounter the same issue.! Thanks 🙂
... View more
06-19-2018
08:46 PM
1 Kudo
Hey @Vladislav Shcherbakov! You can try to use the ExtratText processor and add a parameter for each value that you wanna get using Regex. https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.ExtractText/index.html Hope this helps
... View more
06-19-2018
08:01 PM
Hey @Karthik Chandrashekhar! Unfortunately, i'm not able to see the content of your attach, could you upload it again, please? And regarding, your subdir, could you share the output from the following command: du --max-depth=1 -h /hadoop/ Hope this helps!
... View more
06-19-2018
07:57 PM
1 Kudo
Hi @Suresh Dendukuri! I made a test here, and may serve to you: I have a file called ambari.json with following content: {
"AMBARI_METRICS": {
"service": [
{
"name": "metrics_monitor_process_percent",
"label": "Percent Metrics Monitors Available",
"description": "This alert is triggered if a percentage of Metrics Monitor processes are not up and listening on the network for the configured warning and critical thresholds.",
"interval": 1,
"scope": "SERVICE",
"enabled": true,
"source": {
"type": "AGGREGATE",
"alert_name": "ams_metrics_monitor_process",
"reporting": {
"ok": {
"text": "affected: [{1}], total: [{0}]"
},
"warning": {
"text": "affected: [{1}], total: [{0}]",
"value": 10
},
"critical": {
"text": "affected: [{1}], total: [{0}]",
"value": 30
},
"units" : "%",
"type": "PERCENT"
}
}
}
],
"METRICS_COLLECTOR": [
{
"name": "ams_metrics_collector_autostart",
"label": "Metrics Collector - Auto-Restart Status",
"description": "This alert is triggered if the Metrics Collector has been restarted automatically too frequently in last one hour. By default, a Warning alert is triggered if restarted twice in one hour and a Critical alert is triggered if restarted 4 or more times in one hour.",
"interval": 1,
"scope": "ANY",
"enabled": true,
"source": {
"type": "RECOVERY",
"reporting": {
"ok": {
"text": "Metrics Collector has not been auto-started and is running normally{0}."
},
"warning": {
"text": "Metrics Collector has been auto-started {1} times{0}.",
"count": 2
},
"critical": {
"text": "Metrics Collector has been auto-started {1} times{0}.",
"count": 4
}
}
}
},
{
"name": "ams_metrics_collector_process",
"label": "Metrics Collector Process",
"description": "This alert is triggered if the Metrics Collector cannot be confirmed to be up and listening on the configured port for number of seconds equal to threshold.",
"interval": 1,
"scope": "ANY",
"enabled": true,
"source": {
"type": "PORT",
"uri": "{{ams-site/timeline.metrics.service.webapp.address}}",
"default_port": 6188,
"reporting": {
"ok": {
"text": "TCP OK - {0:.3f}s response on port {1}"
},
"warning": {
"text": "TCP OK - {0:.3f}s response on port {1}",
"value": 1.5
},
"critical": {
"text": "Connection failed: {0} to {1}:{2}",
"value": 5.0
}
}
}
},
{
"name": "ams_metrics_collector_hbase_master_process",
"label": "Metrics Collector - HBase Master Process",
"description": "This alert is triggered if the Metrics Collector's HBase master processes cannot be confirmed to be up and listening on the network for the configured critical threshold, given in seconds.",
"interval": 1,
"scope": "ANY",
"source": {
"type": "PORT",
"uri": "{{ams-hbase-site/hbase.master.info.port}}",
"default_port": 61310,
"reporting": {
"ok": {
"text": "TCP OK - {0:.3f}s response on port {1}"
},
"warning": {
"text": "TCP OK - {0:.3f}s response on port {1}",
"value": 1.5
},
"critical": {
"text": "Connection failed: {0} to {1}:{2}",
"value": 5.0
}
}
}
},
{
"name": "ams_metrics_collector_hbase_master_cpu",
"label": "Metrics Collector - HBase Master CPU Utilization",
"description": "This host-level alert is triggered if CPU utilization of the Metrics Collector's HBase Master exceeds certain warning and critical thresholds. It checks the HBase Master JMX Servlet for the SystemCPULoad property. The threshold values are in percent.",
"interval": 5,
"scope": "ANY",
"enabled": true,
"source": {
"type": "METRIC",
"uri": {
"http": "{{ams-hbase-site/hbase.master.info.port}}",
"default_port": 61310,
"connection_timeout": 5.0
},
"reporting": {
"ok": {
"text": "{1} CPU, load {0:.1%}"
},
"warning": {
"text": "{1} CPU, load {0:.1%}",
"value": 200
},
"critical": {
"text": "{1} CPU, load {0:.1%}",
"value": 250
},
"units" : "%",
"type": "PERCENT"
},
"jmx": {
"property_list": [
"java.lang:type=OperatingSystem/SystemCpuLoad",
"java.lang:type=OperatingSystem/AvailableProcessors"
],
"value": "{0} * 100"
}
}
}
],
"METRICS_MONITOR": [
{
"name": "ams_metrics_monitor_process",
"label": "Metrics Monitor Status",
"description": "This alert indicates the status of the Metrics Monitor process as determined by the monitor status script.",
"interval": 1,
"scope": "ANY",
"source": {
"type": "SCRIPT",
"path": "AMBARI_METRICS/0.1.0/package/alerts/alert_ambari_metrics_monitor.py"
}
}
],
"METRICS_GRAFANA": [
{
"name": "grafana_webui",
"label": "Grafana Web UI",
"description": "This host-level alert is triggered if the Grafana Web UI is unreachable.",
"interval": 1,
"scope": "ANY",
"source": {
"type": "WEB",
"uri": {
"http": "{{ams-grafana-ini/port}}",
"https": "{{ams-grafana-ini/port}}",
"https_property": "{{ams-grafana-ini/protocol}}",
"https_property_value": "https",
"connection_timeout": 5.0,
"default_port": 3000
},
"reporting": {
"ok": {
"text": "HTTP {0} response in {2:.3f}s"
},
"warning":{
"text": "HTTP {0} response from {1} in {2:.3f}s ({3})"
},
"critical": {
"text": "Connection failed to {1} ({3})"
}
}
}
}
]
}
}
And I want to sum the values from the value attrib under the AMBARI_METRICS.service.[0].source.reporting.*.value, so in my flow I'm using GETFILE > EVALUATEJSONPATH > EXTRACTTEXT > EXECUTESTREAMCOMMAND > PUTFIL. In My GetFile I've: In the EvaluateJsonPath: In the In the ExtractText In the ExecuteStreamCommand: And at the end, my putFile. And here's the content of my shell script that Calculates each value from the value attribute and summarize to PutFile Processor. #!/bin/bash
echo $@ | tr ',' '\n' | grep -o -E '[0-9]+' | paste -sd+ | b Hope this helps!
... View more
06-19-2018
02:50 PM
1 Kudo
Hi @Hugo Almeida! Guess you're from Brazil right? As I can see from your insert value (in portuguese hehe). Unfortunately I wasn't able to simulate this issue (follow my test below) on Hive. 0: jdbc:hive2://node3:10000/default> create table special_char (x varchar(11)) stored as orc;
No rows affected (0.393 seconds)
0: jdbc:hive2://node3:10000/default> insert into special_char values('1ºTrimestre');
INFO : Session is already open
INFO : Dag name: insert into special_c...alues('1ºTrimestre')(Stage-1)
INFO : Status: Running (Executing on YARN cluster with App id application_1529333672124_0002)
--------------------------------------------------------------------------------
VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
--------------------------------------------------------------------------------
Map 1 .......... SUCCEEDED 1 1 0 0 0 0
--------------------------------------------------------------------------------
VERTICES: 01/01 [==========================>>] 100% ELAPSED TIME: 5.75 s
--------------------------------------------------------------------------------
INFO : Loading data to table default.special_char from hdfs://Admin-TrainingNS/apps/hive/warehouse/special_char/.hive-staging_hive_2018-06-19_14-41-46_630_2155266942579102600-1/-ext-10000
INFO : Table default.special_char stats: [numFiles=1, numRows=1, totalSize=260, rawDataSize=96]
No rows affected (6.689 seconds)
0: jdbc:hive2://node3:10000/default> select * from special_char;
+-----------------+--+
| special_char.x |
+-----------------+--+
| 1ºTrimestre |
+-----------------+--+
1 row selected (0.109 seconds
But AFAIK, internally the varchar will receive a string value, and if the string can't fit into the varchar pool, then it will truncate. There's a better and clearly explanation than mine in hive cWiki https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Types#LanguageManualTypes-varchar Ps: I'm not sure if this is your case, and if we are running the same version. Hope this helps!
... View more
06-19-2018
02:31 AM
1 Kudo
Hey @Abhijnan Kundu! Did you tried to create like this? CREATE TEMPORARY FUNCTION myupper as 'udf.myupper'; Hope this helps
... View more
06-19-2018
02:24 AM
Awesome @Kishalay Biswas! As the issue is resolved, hence it will be also great if you can mark this HCC thread as Answered by clicking on the "Accept" Button. That way other HCC users can quickly find the solution when they encounter the same issue.
... View more
06-18-2018
11:37 PM
Hi @Suresh Dendukuri! You can use the following flow: GetFile > EvaluateJsonPath > ExecuteStreamCommand or ExecuteScriptCommand > PutFile On EvaluateJsonPath you will add the attributes from your json content, follows attached my example. Ps: In my case i'm getting all value attribute from my json file under the *.service..source.reporting.. path. Hope this helps!
... View more
06-18-2018
06:35 PM
Hey @Kishalay Biswas! On EC2, you have an option to list the old EC2 instances. It's in this page, look for the filter by and change it to all generations. Hope this helps!
... View more