Member since
02-06-2017
20
Posts
4
Kudos Received
2
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
9923 | 08-05-2018 07:38 AM | |
1115 | 06-12-2016 05:15 PM |
06-07-2019
03:34 PM
i really love hdp especially ambari . could somebody answer my question since i noticed no more new hdp release for a while.
... View more
Labels:
01-11-2019
07:11 PM
my final solution is install hbase and my the real base as storage for both ats and ambari metrics .error cleared
... View more
01-08-2019
12:36 AM
hi Geoffrey, I tried by the problem still there .though it's not a big problem for my yarn application .
... View more
01-07-2019
09:40 PM
same thing here ,even after restart everything the "The HBase application reported a 'STARTED' state" is there.
... View more
08-05-2018
07:38 AM
1 Kudo
hi Aditya, Thank you for the response . The issue was related to when using spark to write to hive ,now have to provide the table format as below df.write.format("orc").mode("overwrite").saveAsTable("tt") # this run good
df.write.mode("overwrite").saveAsTable("tt") # this command will fail I didn't change anything on hive tab after hdp 3.0 installed .
... View more
08-03-2018
09:19 PM
Hi ,just doing some testing on newly posted hdp 3.0. and the example failed . I tested same script on previous HDP platform, works fine. can someone advice it's hive's new feature or anything I have done wrong? ./bin/spark-submit examples/src/main/python/sql/hive.py
Hive Session ID = bf71304b-3435-46d5-93a9-09ef752b6c22
AnalysisExceptionTraceback (most recent call last)
/usr/hdp/3.0.0.0-1634/spark2/examples/src/main/python/sql/hive.py in <module>()
44
45 # spark is an existing SparkSession
46 spark.sql("CREATE TABLE IF NOT EXISTS src (key INT, value STRING) USING hive")
47 spark.sql("LOAD DATA LOCAL INPATH 'examples/src/main/resources/kv1.txt' INTO TABLE src")
48
/usr/hdp/3.0.0.0-1634/spark2/python/lib/pyspark.zip/pyspark/sql/session.py in sql(self, sqlQuery)
714 [Row(f1=1, f2=u'row1'), Row(f1=2, f2=u'row2'), Row(f1=3, f2=u'row3')]
715 """
716 return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)
717
718 @since(2.0)
/usr/hdp/3.0.0.0-1634/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py in __call__(self, *args)
1255 answer = self.gateway_client.send_command(command)
1256 return_value = get_return_value(
-> 1257 answer, self.gateway_client, self.target_id, self.name)
1258
1259 for temp_arg in temp_args:
/usr/hdp/3.0.0.0-1634/spark2/python/lib/pyspark.zip/pyspark/sql/utils.py in deco(*a, **kw)
67 e.java_exception.getStackTrace()))
68 if s.startswith('org.apache.spark.sql.AnalysisException: '):
69 raise AnalysisException(s.split(': ', 1)[1], stackTrace)
70 if s.startswith('org.apache.spark.sql.catalyst.analysis'):
71 raise AnalysisException(s.split(': ', 1)[1], stackTrace) AnalysisException: u'org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Table default.src failed strict managed table checks due to the following reason:
Table is marked as a managed table but is not transactional.);' much appreciated!
... View more
07-27-2018
07:12 AM
thanks for the kind answers to both of you
... View more
07-27-2018
06:00 AM
Really love the old UI ,is there a way to switch to the old UI for the latest amberi 2.7?
... View more
Labels:
07-07-2018
02:16 PM
thank you very much ,that' my bad ,I had added some other jars in my class path leading to this error.
... View more
07-07-2018
04:27 AM
Hi ,
I'm using latest HDP ,version is 2.6.5.0-292. spark version is 2.3.0
when I'm trying to run show() from any DataFrame ,it always throw error :
scala> spark.read.csv("/user/a.txt").show() java.lang.NoSuchMethodError: net.jpountz.lz4.LZ4BlockInputStream.<init>(Ljava/io/InputStream;Z)V
at org.apache.spark.io.LZ4CompressionCodec.compressedInputStream(CompressionCodec.scala:122)
at org.apache.spark.sql.execution.SparkPlan.org$apache$spark$sql$execution$SparkPlan$decodeUnsafeRows(SparkPlan.scala:274)
at org.apache.spark.sql.execution.SparkPlan$anonfun$executeTake$1.apply(SparkPlan.scala:366)
at org.apache.spark.sql.execution.SparkPlan$anonfun$executeTake$1.apply(SparkPlan.scala:366)
at scala.collection.TraversableLike$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.TraversableLike$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
at scala.collection.mutable.ArrayOps$ofRef.flatMap(ArrayOps.scala:186)
at org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:366)
at org.apache.spark.sql.execution.CollectLimitExec.executeCollect(limit.scala:38)
at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$collectFromPlan(Dataset.scala:3272)
at org.apache.spark.sql.Dataset$anonfun$head$1.apply(Dataset.scala:2484)
at org.apache.spark.sql.Dataset$anonfun$head$1.apply(Dataset.scala:2484)
at org.apache.spark.sql.Dataset$anonfun$52.apply(Dataset.scala:3253)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:77)
at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3252)
at org.apache.spark.sql.Dataset.head(Dataset.scala:2484)
at org.apache.spark.sql.Dataset.take(Dataset.scala:2698)
at org.apache.spark.sql.execution.datasources.csv.TextInputCSVDataSource$.infer(CSVDataSource.scala:148)
at org.apache.spark.sql.execution.datasources.csv.CSVDataSource.inferSchema(CSVDataSource.scala:63)
at org.apache.spark.sql.execution.datasources.csv.CSVFileFormat.inferSchema(CSVFileFormat.scala:57)
at org.apache.spark.sql.execution.datasources.DataSource$anonfun$8.apply(DataSource.scala:202)
at org.apache.spark.sql.execution.datasources.DataSource$anonfun$8.apply(DataSource.scala:202)
at scala.Option.orElse(Option.scala:289)
at org.apache.spark.sql.execution.datasources.DataSource.getOrInferFileFormatSchema(DataSource.scala:201)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:392)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:239)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:227)
at org.apache.spark.sql.DataFrameReader.csv(DataFrameReader.scala:596)
at org.apache.spark.sql.DataFrameReader.csv(DataFrameReader.scala:473)
I've tried both pyspark and spark-shell on 3 sets of newly installed hdp 2.6.5.0-292. the DataFrame writing function works well ,only show() throws the error. are there anyone encountered same issue as I had? how to fix this problem?
... View more
- Tags:
- Hadoop Core
- hdp-2.6.5
Labels:
05-10-2018
11:32 PM
same problem here ,everything is good except host metric and ambari-metric itself's metric
... View more
03-27-2018
05:13 PM
right now the latest hdp's phoenix version is 4.7 which is really too old to use .phoenix 4.7 is not even support spark2 DataFrame operation and also big improvements made since phoenix 4.10. wondering when the new hdp come with an up to date phoenix?
... View more
11-19-2017
01:15 AM
this is really great I think the key point here is as -Dhdp.version it's is still working for hdp version 2.6.3.0-235
spark.driver.extraJavaOptions -Dhdp.version=2.5.0.0-817 spark.yarn.am.extraJavaOptions -Dhdp.version=2.5.0.0-817
... View more
06-12-2016
06:43 PM
Just checked the pom.xml file for phoenix 4.7 ,it's based on hadoop 2.5.1 which the container id should looks like container_1465095377475_0007_02_000001, while in hadoop 2.7.1 the container id should looks like container_e03_1465095377475_0007_02_000001. So the old version of class org.apache.hadoop.yarn.util.ConverterUtils.toContainerId couldn't handle the new version's container . I should address this problem in phoenix comminity either.
... View more
06-12-2016
05:57 PM
Hi Raj, I already tried that ,I'm using pyspark , added those jars you mentioned in both spark.executor.extraClassPath and spark.driver.extraClassPath and removed phoenix4.7 ,now my spark-submit is working fine ,only the dataframe by specify classname "org.apache.phoenix.spark" is not working . following is I did just now: spark-shell --master yarn-client --jars /usr/hdp/current/phoenix-client/phoenix-client.jar,/usr/hdp/current/phoenix-client/lib/phoenix-spark-4.4.0.2.4.2.0-258.jar
scala> val df = sqlContext.load( "org.apache.phoenix.spark", Map("table" -> "TABLE1", "zkUrl" -> "namenode:2181:/hbase-unsecure"))
warning: there were 1 deprecation warning(s); re-run with -deprecation for details
java.lang.NoClassDefFoundError: org/apache/phoenix/jdbc/PhoenixDriver
at org.apache.phoenix.spark.PhoenixRDD.<init>(PhoenixRDD.scala:40)
at org.apache.phoenix.spark.PhoenixRelation.schema(PhoenixRelation.scala:50)
at org.apache.spark.sql.execution.datasources.LogicalRelation.<init>(LogicalRelation.scala:37)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:125)
at org.apache.spark.sql.SQLContext.load(SQLContext.scala:1153)
at $iwC$iwC$iwC$iwC$iwC$iwC$iwC$iwC.<init>(<console>:25)
at $iwC$iwC$iwC$iwC$iwC$iwC$iwC.<init>(<console>:30)
at $iwC$iwC$iwC$iwC$iwC$iwC.<init>(<console>:32)
at $iwC$iwC$iwC$iwC$iwC.<init>(<console>:34)
at $iwC$iwC$iwC$iwC.<init>(<console>:36)
at $iwC$iwC$iwC.<init>(<console>:38)
at $iwC$iwC.<init>(<console>:40)
at $iwC.<init>(<console>:42)
at <init>(<console>:44)
at .<init>(<console>:48)
at .<clinit>(<console>)
at .<init>(<console>:7)
at .<clinit>(<console>)
at $print(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1346)
at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)
at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)
at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:657)
at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:665)
at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$loop(SparkILoop.scala:670)
at org.apache.spark.repl.SparkILoop$anonfun$org$apache$spark$repl$SparkILoop$process$1.apply$mcZ$sp(SparkILoop.scala:997)
at org.apache.spark.repl.SparkILoop$anonfun$org$apache$spark$repl$SparkILoop$process$1.apply(SparkILoop.scala:945)
at org.apache.spark.repl.SparkILoop$anonfun$org$apache$spark$repl$SparkILoop$process$1.apply(SparkILoop.scala:945)
at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$process(SparkILoop.scala:945)
at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059)
at org.apache.spark.repl.Main$.main(Main.scala:31)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$runMain(SparkSubmit.scala:731)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.phoenix.jdbc.PhoenixDriver
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 51 more if you open the jar file like phoenix-spark-4.4.0.2.4.2.0-258.jar ,you will find there is no jdbc exists ,that's the root reason . if you open the jar for phoenix4.7 ,you will jdbc support classes for spark.
... View more
06-12-2016
05:43 PM
I further checked , the error was actually caused by ConverterUtils.class in phoenix-4.7.0-HBase-1.1-client-spark.jar , which I think it's supporting hadoop 2.7.2 while HDP2.4.2 is still using 2.7.1 , the container id format has been changed .
... View more
06-12-2016
05:15 PM
1 Kudo
Hi Raj, thank you for the response . turns out it's caused by phoenix : I added phoenix-4.7.0-HBase-1.1-client-spark.jar to both spark.executor.extraClassPath and spark.driver.extraClassPath . Now I'm using HDP2.4.2 default jar phoenix-spark-4.4.0.2.4.2.0-258.jar the problem disappeared . However with the default jar , there is no jdbc support to execute statement like below , HDP's phoenix version is too old ! I'm kindly of hoping that HDP could provide a update for phoenix to support JDBC ! df =sqlContext.read.format("org.apache.phoenix.spark").option("table","TABLE1").option("zkUrl","namenode.localdomain:2181:/hbase-unsecure").load() the error rasied by the upper command : java.lang.NoClassDefFoundError: org/apache/phoenix/jdbc/PhoenixDriver
... View more
06-11-2016
09:08 PM
2 Kudos
when I execute spark-submit --master yarn /usr/hdp/current/spark-client/examples/src/main/python/pi.py on hdp2.4.2 ,I got the error below (which doesn't cause any error on hdp2.4.0),according to the following job log ,sounds we got container_e03_1465095377475_0007_02_000001 which don't recognized by spark and caused java.lang.NumberFormatException: For input string: "e03" error SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/hadoop/yarn/local/filecache/11/spark-hdp-assembly.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.4.2.0-258/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
16/06/11 16:30:39 INFO ApplicationMaster: Registered signal handlers for [TERM, HUP, INT]
16/06/11 16:30:39 ERROR ApplicationMaster: Uncaught exception:
java.lang.IllegalArgumentException: Invalid ContainerId: container_e03_1465095377475_0007_02_000001
at org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182)
at org.apache.spark.deploy.yarn.YarnSparkHadoopUtil.getContainerId(YarnSparkHadoopUtil.scala:192)
at org.apache.spark.deploy.yarn.YarnRMClient.getAttemptId(YarnRMClient.scala:92)
at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:142)
at org.apache.spark.deploy.yarn.ApplicationMaster$anonfun$main$1.apply$mcV$sp(ApplicationMaster.scala:672)
at org.apache.spark.deploy.SparkHadoopUtil$anon$1.run(SparkHadoopUtil.scala:69)
at org.apache.spark.deploy.SparkHadoopUtil$anon$1.run(SparkHadoopUtil.scala:68)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:68)
at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:670)
at org.apache.spark.deploy.yarn.ExecutorLauncher$.main(ApplicationMaster.scala:697)
at org.apache.spark.deploy.yarn.ExecutorLauncher.main(ApplicationMaster.scala)
Caused by: java.lang.NumberFormatException: For input string: "e03"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Long.parseLong(Long.java:589)
at java.lang.Long.parseLong(Long.java:631)
at org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137)
at org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177)
... 13 more
16/06/11 16:30:39 INFO ApplicationMaster: Final app status: FAILED, exitCode: 10, (reason: Uncaught exception: java.lang.IllegalArgumentException: Invalid ContainerId: container_e03_1465095377475_0007_02_000001)
16/06/11 16:30:39 INFO ShutdownHookManager: Shutdown hook called
... View more
05-31-2016
11:36 PM
Actually I fixed this problem by install hdp 2.4.2 and download and installed phoenix 4.7.0 without using hdp provided phoenix in /usr/hdp/current/hbase-regionserver/lib folder -rw-r--r-- 1 root root 18263 May 30 22:50 phoenix-server-4.7.0-HBase-1.1.jar -rw-r--r-- 1 root root 3636606 May 30 22:50 phoenix-core-4.7.0-HBase-1.1.jar remember to restart hbase and make sure you had done classpath as following before executing sqlline.py export CLASSPATH=$CLASSPATH:/root/phoenix/phoenix-4.7.0-HBase-1.1-bin/phoenix-4.7.0-HBase-1.1-client.jar
sqline.py namenode:2181:/hbase-unsecure
0: jdbc:phoenix:localhost:2181:/hbase-unsecur> CREATE TABLE TABLE1 (ID BIGINT NOT NULL PRIMARY KEY, COL1 VARCHAR);
No rows affected (1.243 seconds)
0: jdbc:phoenix:localhost:2181:/hbase-unsecur> UPSERT INTO TABLE1 (ID, COL1) VALUES (1, 'test_row_1');
1 row affected (0.074 seconds)
0: jdbc:phoenix:localhost:2181:/hbase-unsecur> UPSERT INTO TABLE1 (ID, COL1) VALUES (2, 'test_row_2');
1 row affected (0.008 seconds)
0: jdbc:phoenix:localhost:2181:/hbase-unsecur> Closing: org.apache.phoenix.jdbc.PhoenixConnection
make sure
spark.executor.extraClassPath
spark.driver.extraClassPath
should like following :
/root/phoenix/phoenix-4.7.0-HBase-1.1-bin/phoenix-spark-4.7.0-HBase-1.1.jar:/root/phoenix/phoenix-4.7.0-HBase-1.1-bin/phoenix-4.7.0-HBase-1.1-client-spark.jar:/usr/hdp/current/hbase-client/lib/*.jar
don't add phoenix client to the path ,it causing conflict with other jars. the jars added all have spark in it's name import org.apache.spark.SparkContext
import org.apache.spark.sql.SQLContext
import org.apache.phoenix.spark._
val sqlContext = new SQLContext(sc)
val df = sqlContext.load(
"org.apache.phoenix.spark",
Map("table" -> "TABLE1", "zkUrl" -> "localhost:2181:/hbase-unsecure")
)
df.filter(df("COL1") === "test_row_1" && df("ID") === 1L).select(df("ID")).show
... View more
05-31-2016
01:46 AM
@jbarnett May I please know the issue got solved or not , I'm facing the exact same issue with my newly installed hdp 2.4.2,even after I pushed the button . I do need to use phoenix badly together with hdp platform , is there any detailed document to make it work?
... View more