Member since
09-25-2015
101
Posts
51
Kudos Received
25
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1121 | 08-25-2017 01:06 PM | |
1518 | 05-16-2017 12:53 PM | |
1166 | 05-15-2017 05:07 PM | |
1199 | 05-11-2017 04:19 PM | |
879 | 05-05-2017 12:29 AM |
10-19-2015
10:10 PM
Sorry, my question had a 2 part. Sorry I did not make that clear. Setting up different kdcs as suggested above works for configuration and HA. However, the user wants to configure a kdc with a load balancer to ease up the traffic. if 2 kdcs are setup, then users can still connect to the heavy used one instead of being redirected to the idle one via loadbalancer. Does this make sense?
... View more
10-19-2015
06:06 PM
Thanks for the reply all. This was a non issue. Ambari is reading JMX and JMX is incorrectly aggregating GC Times. Similar to: https://issues.apache.org/jira/browse/AMBARI-3178 Thanks for the help
... View more
10-15-2015
09:48 PM
[root@hadm0301 ~]# free -c 5 total used free shared buffers cached Mem: 148308580 121816216 26492364 0 24726180 75655352 -/+ buffers/cache: 21434684 126873896 Swap: 2097144 0 2097144
... View more
10-15-2015
08:14 PM
Yes but it varies heavily depending the day/time. At times, it could be intensive. I am not sure it is heavy at this moment but let me get that
... View more
10-15-2015
07:49 PM
Seeing PSYoungGen GC on YARN/RM nodes aggressively every 8-10 secs: 6171.971: [GC [PSYoungGen: 136875K->512K(144896K)] 252995K->116647K(1542656K), 0.0070630
secs] [Times: user=0.05 sys=0.00, real=0.01 secs]
6180.179: [GC [PSYoungGen: 115578K->480K(142336K)] 231714K->116703K(1540096K), 0.0065030
secs] [Times: user=0.04 sys=0.01, real=0.00 secs]
6190.316: [GC [PSYoungGen: 141974K->736K(142336K)] 258198K->116975K(1540096K), 0.0059270
secs] [Times: user=0.03 sys=0.01, real=0.01 secs]
6200.145: [GC [PSYoungGen: 137784K->672K(140800K)] 254024K->116935K(1538560K), 0.0059910
secs] [Times: user=0.04 sys=0.00, real=0.01 secs]
6209.983: [GC [PSYoungGen: 138483K->704K(139776K)] 254746K->117079K(1537536K), 0.0060980
secs] [Times: user=0.04 sys=0.00, real=0.01 secs]
6220.037: [GC [PSYoungGen: 137973K->544K(138752K)] 254349K->117157K(1536512K), 0.0036380
secs] [Times: user=0.02 sys=0.00, real=0.00 secs]
6228.739: [GC [PSYoungGen: 117752K->768K(137728K)] 234366K->117437K(1535488K), 0.0061930
secs] [Times: user=0.03 sys=0.02, real=0.00 secs]
6237.006: [GC [PSYoungGen: 113053K->768K(136704K)] 229723K->117493K(1534464K), 0.0037210
secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
6244.956: [GC [PSYoungGen: 107217K->672K(135680K)] 223942K->117589K(1533440K), 0.0080520
secs] [Times: user=0.03 sys=0.03, real=0.01 secs]
6255.009: [GC [PSYoungGen: 133601K->512K(134656K)] 250518K->117517K(1532416K), 0.0070160
secs] [Times: user=0.04 sys=0.00, real=0.01 secs]
6265.007: [GC [PSYoungGen: 131612K->672K(134144K)] 248618K->117821K(1531904K), 0.0039770
secs] [Times: user=0.04 sys=0.00, real=0.00 secs]
6274.880: [GC [PSYoungGen: 129679K->815K(132608K)] 246828K->118132K(1530368K), 0.0036740
secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
6284.782: [GC [PSYoungGen: 129427K->608K(132096K)] 246744K->118164K(1529856K), 0.0039850
secs] [Times: user=0.04 sys=0.00, real=0.01 secs]
6294.809: [GC [PSYoungGen: 129803K->576K(130560K)] 247360K->118260K(1528320K), 0.0037790
secs] [Times: user=0.03 sys=0.00, real=0.00 secs]
6297.736: [GC [PSYoungGen: 39607K->544K(129536K)] 157291K->118316K(1527296K), 0.0035240 secs]
[Times: user=0.02 sys=0.00, real=0.00 secs] Is this normal? Is there a way to properly tweak it? Looks like the new size is set to 144M, should this be increased? Thank you !
... View more
Labels:
- Labels:
-
Apache YARN
10-14-2015
01:29 PM
1 Kudo
While trying to configure kdc with a Virtual IP, we are getting the below error Could not find user in Kerberos database Ambari fails the install. VIP is needed for loadbalancing and HA. When single node IP is used for kdc, then the install is successful.
... View more
Labels:
- Labels:
-
Apache Ambari
10-08-2015
03:37 PM
Yes and it gave more insight into what was wrong: FAILED: SemanticException [Error 10007]: Ambiguous column reference visit_num in features_and_label Customer had a duped column. Thank you for the tip.
... View more
10-07-2015
07:18 PM
Hive v13,hitting the semantic error: 2015-10-07 05:12:40,988 ERROR ql.Driver (SessionState.java:printError(547)) - FAILED: NullPointerException null
java.lang.NullPointerException
at org.apache.hadoop.hive.ql.optimizer.ColumnPrunerProcFactory$ColumnPrunerSelectProc.process(ColumnPrunerProcFactory.java:618)
at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)
at org.apache.hadoop.hive.ql.optimizer.ColumnPruner$ColumnPrunerWalker.walk(ColumnPruner.java:166)
at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)
at org.apache.hadoop.hive.ql.optimizer.ColumnPruner.transform(ColumnPruner.java:129)
at org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:146)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9414)
at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:427)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:323)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:980)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1045)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:916)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:906)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:359)
at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:456)
at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:466)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:749)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212) Looks very similar to: https://issues.apache.org/jira/browse/HIVE-9185 Looks like the ability to disable pruner was taken away on Hive13: hive.optimize.cp=false https://issues.apache.org/jira/browse/HIVE-4113 Is there another way to disable the pruner temporarily?
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache Tez
10-05-2015
04:52 PM
Has anyone seen an intermittent error while exporting from HDP to external DB via Teradata Connector: 2015-09-22 13:03:29,969 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : com.teradata.connector.common.exception.ConnectorException: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at java.net.Socket.connect(Socket.java:528)
at java.net.Socket.<init>(Socket.java:425)
at java.net.Socket.<init>(Socket.java:208)
at com.teradata.connector.teradata.TeradataInternalFastloadOutputFormat.getRecordWriter(TeradataInternalFastloadOutputFormat.java:301)
at com.teradata.connector.common.ConnectorOutputFormat$ConnectorFileRecordWriter.<init>(ConnectorOutputFormat.java:84)
at com.teradata.connector.common.ConnectorOutputFormat.getRecordWriter(ConnectorOutputFormat.java:33)
at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:644)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) The connection string is: hadoop jar /usr/hdp/2.2.4.5-1/sqoop/lib/teradata-connector-1.3.5-hadoop220.jar com.teradata.hadoop.tool.TeradataExportTool -url jdbc:teradata://<server>/Database=PR_ODMM_STG -username PR_BATCH_ODMM_SQL -password xxxxx -errortabledatabase PR_ODMM_STG -jobtype hdfs -sourcepaths /user/hive/warehouse/edw_odmm_stg.db/omn_aggregates/batch_id=15/agg_tab_partition=STG_INTERNAL_SEARCH_KEYWORD_1 -nummappers 5 -separator \u001E -targettable STG_INTERNAL_SEARCH_KEYWORD_1 -errortablename NTERNAL_SEARCH_KEYWORD_1 -method internal.fastload &
... View more
Labels:
- Labels:
-
Apache YARN
09-30-2015
05:06 PM
Good find. Thanks
... View more
- « Previous
- Next »