Member since
11-15-2016
50
Posts
2
Kudos Received
0
Solutions
02-23-2018
06:15 PM
HI All, I installed a HDP cluster on cloudbreak and am trying to run a simple Spark Job. I open the "pyspark" shell and run the following: ip = "adl://alenzadls1.azuredatalakestore.net/path/to/my/input/directory"
input_data = sc.textFile(ip)
for x in input_data.collect(): print x The print statement returns an error: Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/hdp/current/spark-client/python/pyspark/rdd.py", line 771, in collect
port = self.ctx._jvm.PythonRDD.collectAndServe(self._jrdd.rdd())
File "/usr/hdp/current/spark-client/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py", line 813, in __call__
File "/usr/hdp/current/spark-client/python/pyspark/sql/utils.py", line 45, in deco
return f(*a, **kw)
File "/usr/hdp/current/spark-client/python/lib/py4j-0.9-src.zip/py4j/protocol.py", line 308, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.adl.AdlFileSystem not found Can someone point me to where it is going wrong? I did not find anything related to this online.
... View more
Labels:
02-23-2018
06:09 PM
I did. I just had to change the value of Reserved Space for HDFS in HDFS configs. Thanks for commenting.
... View more
02-21-2018
11:16 PM
I just deployed my cluster on Azure using Cloudbreak. Right now my name node and datanode directories are: /hadoopfs/fs1/hdfs/namenode and /hadoop/hdfs/data,/hadoopfs/fs1/hadoop/hdfs/data,/mnt/resource/hadoop/hdfs/data respectively. I want to change this to a location on my Data Lake Store. When I just paste the path starting with "adl:/" it throws an error. I setup the access and everything. I just want to change the directories to save space on my local FS. Also, I am getting alerts like: Capacity Used:[100%, 28672], Capacity Remaining:[0]. I have 110 GB RAM on my master node. Why is it already used up?
... View more
Labels:
11-30-2017
08:20 PM
@aengineer Hi, thanks for your reply. It seems someone did perform a massive delete operation on the cluster. The issue is resolved.
... View more
11-30-2017
07:46 PM
@Jay Kumar SenSharma Did not help. Please check the logs below: [root@newyorknn ~]# su postgres -c 'psql -c "create database ambarirca" '
ERROR: database "ambarirca" already exists
[root@newyorknn ~]# su postgres -c 'psql -c "create user mapred with password 'mapred'" '
ERROR: syntax error at or near "mapred"
LINE 1: create user mapred with password mapred
^
[root@newyorknn ~]# su postgres -c 'psql -c "GRANT ALL PRIVILEGES ON DATABASE ambarirca TO mapred"'
GRANT
[root@newyorknn ~]# ambari-server upgrade
Using python /usr/bin/python
Upgrading ambari-server
Updating properties in ambari.properties ...
INFO: Can not find ambari-env.sh.rpmsave file from previous version, skipping restore of environment settings. ambari-env.sh may not include any user customization.
Fixing database objects owner
Ambari Server configured for Embedded Postgres. Confirm you have made a backup of the Ambari Server database [y/n] (y)? y
Upgrading database schema
Error output from schema upgrade command:
Exception in thread "main" org.apache.ambari.server.AmbariException: ERROR: schema "ambarirca" does not exist
at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:207)
at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:347)
Caused by: org.postgresql.util.PSQLException: ERROR: schema "ambarirca" does not exist
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:559)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:403)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:395)
at org.apache.ambari.server.orm.DBAccessorImpl.executeQuery(DBAccessorImpl.java:827)
at org.apache.ambari.server.orm.DBAccessorImpl.executeQuery(DBAccessorImpl.java:819)
at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.changePostgresSearchPath(AbstractUpgradeCatalog.java:361)
at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeSchema(AbstractUpgradeCatalog.java:886)
at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:204)
... 1 more
ERROR: Error executing schema upgrade, please check the server logs.
ERROR: Ambari server upgrade failed. Please look at /var/log/ambari-server/ambari-server.log, for more details.
ERROR: Exiting with exit code 11.
REASON: Schema upgrade failed. Then, since database and schema are different things in postgres (?), I did the following: [root@newyorknn ~]# su - postgres
Last login: Thu Nov 30 14:41:51 EST 2017
-bash-4.2$ psql -c "create database ambarirca"
ERROR: database "ambarirca" already exists
-bash-4.2$ psql -c "create schema ambarirca"
CREATE SCHEMA
-bash-4.2$ psql -c "create user mapred with password 'mapred'"
ERROR: role "mapred" already exists
-bash-4.2$ psql -c "GRANT ALL PRIVILEGES ON DATABASE ambarirca TO mapred"
GRANT
-bash-4.2$ psql -c "GRANT ALL PRIVILEGES ON schema ambarirca TO mapred"
GRANT
-bash-4.2$ exit
logout
[root@newyorknn ~]# ambari-server upgrade
Using python /usr/bin/python
Upgrading ambari-server
Updating properties in ambari.properties ...
WARNING: Can not find ambari.properties.rpmsave file from previous version, skipping import of settings
INFO: Can not find ambari-env.sh.rpmsave file from previous version, skipping restore of environment settings. ambari-env.sh may not include any user customization.
Fixing database objects owner
Ambari Server configured for Embedded Postgres. Confirm you have made a backup of the Ambari Server database [y/n] (y)? y
Upgrading database schema
Error output from schema upgrade command:
Exception in thread "main" org.apache.ambari.server.AmbariException: ERROR: schema "ambarirca" does not exist
at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:207)
at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:347)
Caused by: org.postgresql.util.PSQLException: ERROR: schema "ambarirca" does not exist
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:559)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:403)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:395)
at org.apache.ambari.server.orm.DBAccessorImpl.executeQuery(DBAccessorImpl.java:827)
at org.apache.ambari.server.orm.DBAccessorImpl.executeQuery(DBAccessorImpl.java:819)
at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.changePostgresSearchPath(AbstractUpgradeCatalog.java:361)
at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeSchema(AbstractUpgradeCatalog.java:886)
at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:204)
... 1 more
ERROR: Error executing schema upgrade, please check the server logs.
ERROR: Ambari server upgrade failed. Please look at /var/log/ambari-server/ambari-server.log, for more details.
ERROR: Exiting with exit code 11.
REASON: Schema upgrade failed.
... View more
11-30-2017
07:12 PM
I currently have Ambari 2.2.2.0 and want to upgrade to the latest version. My HDP version is 2.4.2. So I am trying to upgrade to 2.4 first (since I cant jump directly to 2.6 from 2.2) and am following the steps given at: https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.0.1/bk_ambari-upgrade/content/upgrade_ambari.html. Everything is good until I rech the "ambari-server upgrade" step. This step fails with the error: [root@newyorknn ~]# ambari-server upgrade
Using python /usr/bin/python
Upgrading ambari-server
Updating properties in ambari.properties ...
WARNING: Original file ambari-env.sh kept
Fixing database objects owner
Ambari Server configured for Embedded Postgres. Confirm you have made a backup of the Ambari Server database [y/n] (y)? y
Upgrading database schema
Error output from schema upgrade command:
Exception in thread "main" org.apache.ambari.server.AmbariException: ERROR: schema "ambarirca" does not exist
at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:207)
at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:347)
Caused by: org.postgresql.util.PSQLException: ERROR: schema "ambarirca" does not exist
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:559)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:403)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:395)
at org.apache.ambari.server.orm.DBAccessorImpl.executeQuery(DBAccessorImpl.java:827)
at org.apache.ambari.server.orm.DBAccessorImpl.executeQuery(DBAccessorImpl.java:819)
at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.changePostgresSearchPath(AbstractUpgradeCatalog.java:361)
at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeSchema(AbstractUpgradeCatalog.java:886)
at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:204)
... 1 more
ERROR: Error executing schema upgrade, please check the server logs.
ERROR: Ambari server upgrade failed. Please look at /var/log/ambari-server/ambari-server.log, for more details.
ERROR: Exiting with exit code 11.
REASON: Schema upgrade failed. Can someone suggest what I should do? I looked at the ambari-server database files that I backed up according to the instructions given at https://community.hortonworks.com/articles/37765/backing-up-the-ambari-database-with-postgres.html (ambari.sql and ambarirca.sql) but none of them have any command called "CREATE SCHEMA AMBARIRCA". I am not sure how to proceed.
... View more
Labels:
- Labels:
-
Apache Ambari
11-30-2017
03:16 PM
One of the disks on one of my data nodes was failing, so I replaced it with the instructions: 1. Stop all services on datanode. 2. shut down the machine. 3. replace the disk 4. power on the machine. 5. mount the disk onto its data point. 6. start all services on HDFS. Now, I get an alert "Pending Deletion Blocks:[276861]" on Ambari. Did I do something wrong? I can I revert it?
... View more
Labels:
- Labels:
-
Apache Hadoop
10-30-2017
05:22 PM
This looks like a bug in Hive LLAP. I get the same error for a simple select count(*) on a table. The query is good in Hive and Spark. This error is what it prints on screen but the actual error when you see the logs is: killed/failed due to:INIT_FAILURE, Fail to create
InputInitializerManager, org.apache.tez.dag.api.TezReflectionException: Unable
to instantiate class with 1 arguments:
org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator
... View more
10-05-2017
04:37 PM
I am running a Sql query in Spark: spark.sql("select person_key, count(*) as count1 from <table_name> group by person_key order by count1 desc").show() This throws a warning: 17/10/05 12:09:03 WARN ReaderImpl: Cannot find field for: person_key in _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9,
17/10/05 12:09:03 WARN ReaderImpl: Cannot find field for: person_key in _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9,
17/10/05 12:09:03 WARN ReaderImpl: Cannot find field for: person_key in _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9,
17/10/05 12:09:03 WARN ReaderImpl: Cannot find field for: person_key in _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9,
17/10/05 12:09:03 WARN ReaderImpl: Cannot find field for: person_key in _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9,
17/10/05 12:09:03 WARN ReaderImpl: Cannot find field for: person_key in _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9,
17/10/05 12:09:03 WARN ReaderImpl: Cannot find field for: person_key in _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, But does give correct results. I want to understand what this means. I did not find anything on the net. I want this resolved because though it is giving the correct results, it is taking very long to execute. (The same query on Hive LLAP takes 3 seconds. Spark numbers are usually comparable to Hive LLAP numbers). I checked person_key does exist in the table (I created it so the table so I know it exists). Not sure why the warning is coming.
... View more
Labels:
- Labels:
-
Apache Spark
10-05-2017
04:23 PM
Thanks, restarting Hive Inseractive Server did help. Regarding Spark, I think it has a problem with the naming convention of the delta file. When I run the same query in Spark after performing a "major compaction", it was able to run the query. But how realistic is preforming major compaction on the table for every update in a production cluster? Is there a work around?
... View more
09-29-2017
05:35 PM
I have a transactional (ACID) table in Hive and can query it fine using the hive shell. However, I want to query it using LLAP/SPark Shell which I am unable to do. When I set all the parameters required for querying ACID tables (transaction manager, compactor threads etc) in the beeline shell I open to connect to Hive Interactive Server, it seems to work but when I run the query, everything just halts and nothing works. After a very long time I get the following error message: ERROR : Status: Failed
ERROR : Dag received [DAG_TERMINATE, SERVICE_PLUGIN_ERROR] in RUNNING state.
ERROR : Error reported by TaskScheduler [[2:LLAP]][SERVICE_UNAVAILABLE] No LLAP Daemons are running
ERROR : Vertex killed, vertexName=Reducer 2, vertexId=vertex_1506697113479_0016_1_03, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_TERMINATED, failedTasks:0 killedTasks:1009, Vertex vertex_1506697113479_0016_1_03 [Reducer 2] killed/failed due to:DAG_TERMINATED]
ERROR : Vertex killed, vertexName=Map 3, vertexId=vertex_1506697113479_0016_1_00, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_TERMINATED, failedTasks:0 killedTasks:1, Vertex vertex_1506697113479_0016_1_00 [Map 3] killed/failed due to:DAG_TERMINATED]
ERROR : Vertex killed, vertexName=Reducer 4, vertexId=vertex_1506697113479_0016_1_01, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_TERMINATED, failedTasks:0 killedTasks:1, Vertex vertex_1506697113479_0016_1_01 [Reducer 4] killed/failed due to:DAG_TERMINATED]
ERROR : Vertex killed, vertexName=Map 1, vertexId=vertex_1506697113479_0016_1_02, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_TERMINATED, failedTasks:0 killedTasks:354, Vertex vertex_1506697113479_0016_1_02 [Map 1] killed/failed due to:DAG_TERMINATED]
ERROR : DAG did not succeed due to SERVICE_PLUGIN_ERROR. failedVertices:0 killedVertices:4
ERROR : FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Dag received [DAG_TERMINATE, SERVICE_PLUGIN_ERROR] in RUNNING state.Error reported by TaskScheduler [[2:LLAP]][SERVICE_UNAVAILABLE] No LLAP Daemons are runningVertex killed, vertexName=Reducer 2, vertexId=vertex_1506697113479_0016_1_03, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_TERMINATED, failedTasks:0 killedTasks:1009, Vertex vertex_1506697113479_0016_1_03 [Reducer 2] killed/failed due to:DAG_TERMINATED]Vertex killed, vertexName=Map 3, vertexId=vertex_1506697113479_0016_1_00, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_TERMINATED, failedTasks:0 killedTasks:1, Vertex vertex_1506697113479_0016_1_00 [Map 3] killed/failed due to:DAG_TERMINATED]Vertex killed, vertexName=Reducer 4, vertexId=vertex_1506697113479_0016_1_01, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_TERMINATED, failedTasks:0 killedTasks:1, Vertex vertex_1506697113479_0016_1_01 [Reducer 4] killed/failed due to:DAG_TERMINATED]Vertex killed, vertexName=Map 1, vertexId=vertex_1506697113479_0016_1_02, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_TERMINATED, failedTasks:0 killedTasks:354, Vertex vertex_1506697113479_0016_1_02 [Map 1] killed/failed due to:DAG_TERMINATED]DAG did not succeed due to SERVICE_PLUGIN_ERROR. failedVertices:0 killedVertices:4
Error: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Dag received [DAG_TERMINATE, SERVICE_PLUGIN_ERROR] in RUNNING state.Error reported by TaskScheduler [[2:LLAP]][SERVICE_UNAVAILABLE] No LLAP Daemons are runningVertex killed, vertexName=Reducer 2, vertexId=vertex_1506697113479_0016_1_03, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_TERMINATED, failedTasks:0 killedTasks:1009, Vertex vertex_1506697113479_0016_1_03 [Reducer 2] killed/failed due to:DAG_TERMINATED]Vertex killed, vertexName=Map 3, vertexId=vertex_1506697113479_0016_1_00, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_TERMINATED, failedTasks:0 killedTasks:1, Vertex vertex_1506697113479_0016_1_00 [Map 3] killed/failed due to:DAG_TERMINATED]Vertex killed, vertexName=Reducer 4, vertexId=vertex_1506697113479_0016_1_01, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_TERMINATED, failedTasks:0 killedTasks:1, Vertex vertex_1506697113479_0016_1_01 [Reducer 4] killed/failed due to:DAG_TERMINATED]Vertex killed, vertexName=Map 1, vertexId=vertex_1506697113479_0016_1_02, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_TERMINATED, failedTasks:0 killedTasks:354, Vertex vertex_1506697113479_0016_1_02 [Map 1] killed/failed due to:DAG_TERMINATED]DAG did not succeed due to SERVICE_PLUGIN_ERROR. failedVertices:0 killedVertices:4 (state=08S01,code=2) Similarly in Spark, I open the spark shell, make the following imports: import org.apache.spark.sql.SparkSession
import spark.implicits._
val spark = SparkSession.builder().appName("Spark SQL basic example").config("spark.sql.parquet.compression.codec", "lzo").getOrCreate()
spark.sqlContext.setConf("spark.sql.parquet.compression.codec", "lzo")
spark.sqlContext.setConf("spark.sql.crossJoin.enabled", "true") and run the query as: val q1 = spark.sql("select * fron <hive_transaction_table")
q1.show() It gives me an error: org.apache.spark.sql.catalyst.errors.package$TreeNodeException: execute, tree:
Exchange SinglePartition
+- *HashAggregate(keys=[], functions=[partial_count(1)], output=[count#159L])
+- HiveTableScan MetastoreRelation avlino_bm, subscriber_dim_test_parquet
at org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:56)
at org.apache.spark.sql.execution.exchange.ShuffleExchange.doExecute(ShuffleExchange.scala:112)
at org.apache.spark.sql.execution.SparkPlan$anonfun$execute$1.apply(SparkPlan.scala:114)
at org.apache.spark.sql.execution.SparkPlan$anonfun$execute$1.apply(SparkPlan.scala:114)
at org.apache.spark.sql.execution.SparkPlan$anonfun$executeQuery$1.apply(SparkPlan.scala:135)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:132)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:113)
at org.apache.spark.sql.execution.InputAdapter.inputRDDs(WholeStageCodegenExec.scala:235)
at org.apache.spark.sql.execution.aggregate.HashAggregateExec.inputRDDs(HashAggregateExec.scala:141)
at org.apache.spark.sql.execution.WholeStageCodegenExec.doExecute(WholeStageCodegenExec.scala:368)
at org.apache.spark.sql.execution.SparkPlan$anonfun$execute$1.apply(SparkPlan.scala:114)
at org.apache.spark.sql.execution.SparkPlan$anonfun$execute$1.apply(SparkPlan.scala:114)
at org.apache.spark.sql.execution.SparkPlan$anonfun$executeQuery$1.apply(SparkPlan.scala:135)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:132)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:113)
at org.apache.spark.sql.execution.SparkPlan.getByteArrayRdd(SparkPlan.scala:225)
at org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:308)
at org.apache.spark.sql.execution.CollectLimitExec.executeCollect(limit.scala:38)
at org.apache.spark.sql.Dataset$anonfun$org$apache$spark$sql$Dataset$execute$1$1.apply(Dataset.scala:2378)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:57)
at org.apache.spark.sql.Dataset.withNewExecutionId(Dataset.scala:2780)
at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$execute$1(Dataset.scala:2377)
at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$collect(Dataset.scala:2384)
at org.apache.spark.sql.Dataset$anonfun$head$1.apply(Dataset.scala:2120)
at org.apache.spark.sql.Dataset$anonfun$head$1.apply(Dataset.scala:2119)
at org.apache.spark.sql.Dataset.withTypedCallback(Dataset.scala:2810)
at org.apache.spark.sql.Dataset.head(Dataset.scala:2119)
at org.apache.spark.sql.Dataset.take(Dataset.scala:2334)
at org.apache.spark.sql.Dataset.showString(Dataset.scala:248)
at org.apache.spark.sql.Dataset.show(Dataset.scala:638)
at org.apache.spark.sql.Dataset.show(Dataset.scala:597)
at org.apache.spark.sql.Dataset.show(Dataset.scala:606)
... 52 elided
Caused by: java.lang.RuntimeException: serious problem
at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1021)
at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1048)
at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:202)
at org.apache.spark.rdd.RDD$anonfun$partitions$2.apply(RDD.scala:252)
at org.apache.spark.rdd.RDD$anonfun$partitions$2.apply(RDD.scala:250)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:250)
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD$anonfun$partitions$2.apply(RDD.scala:252)
at org.apache.spark.rdd.RDD$anonfun$partitions$2.apply(RDD.scala:250)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:250)
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD$anonfun$partitions$2.apply(RDD.scala:252)
at org.apache.spark.rdd.RDD$anonfun$partitions$2.apply(RDD.scala:250)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:250)
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD$anonfun$partitions$2.apply(RDD.scala:252)
at org.apache.spark.rdd.RDD$anonfun$partitions$2.apply(RDD.scala:250)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:250)
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD$anonfun$partitions$2.apply(RDD.scala:252)
at org.apache.spark.rdd.RDD$anonfun$partitions$2.apply(RDD.scala:250)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:250)
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD$anonfun$partitions$2.apply(RDD.scala:252)
at org.apache.spark.rdd.RDD$anonfun$partitions$2.apply(RDD.scala:250)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:250)
at org.apache.spark.ShuffleDependency.<init>(Dependency.scala:91)
at org.apache.spark.sql.execution.exchange.ShuffleExchange$.prepareShuffleDependency(ShuffleExchange.scala:261)
at org.apache.spark.sql.execution.exchange.ShuffleExchange.prepareShuffleDependency(ShuffleExchange.scala:84)
at org.apache.spark.sql.execution.exchange.ShuffleExchange$anonfun$doExecute$1.apply(ShuffleExchange.scala:121)
at org.apache.spark.sql.execution.exchange.ShuffleExchange$anonfun$doExecute$1.apply(ShuffleExchange.scala:112)
at org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:52)
... 85 more
Caused by: java.util.concurrent.ExecutionException: java.lang.NumberFormatException: For input string: "0000035_0000"
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:998)
... 122 more
Caused by: java.lang.NumberFormatException: For input string: "0000035_0000"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Long.parseLong(Long.java:589)
at java.lang.Long.parseLong(Long.java:631)
at org.apache.hadoop.hive.ql.io.AcidUtils.parseDelta(AcidUtils.java:310)
at org.apache.hadoop.hive.ql.io.AcidUtils.getAcidState(AcidUtils.java:379)
at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.call(OrcInputFormat.java:634)
at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.call(OrcInputFormat.java:620)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745) Is anyone else facing the same issue? HOw can I access transactional tables from Hive LLAP or Spark?
... View more
Labels:
- Labels:
-
Apache Spark
09-29-2017
03:55 PM
Hi all, This question is in continuation to: https://community.hortonworks.com/questions/138257/how-to-sync-a-new-secondary-namenode-to-the-cluste.html Scenario: I have a 12 node cluster (machines 01-12) with 8 data nodes. 06, 07 are the NN and SNN respectively. 01 and 12 run the Hive related services. This cluster was upgraded to 12 node from 4 node where 01 was the namenode and 02-05 were the data nodes. So I used the "Move Namenode" and "Move Snamenode" wizards on Ambari to move the nn and snn from 01 to 06 and 07 respectively. I verified all the services running on nn are running on snn as well. Now I want to check if my snn is working properly. So I shut down all the services on nn and tried to connect to Hive from one of the hosts (02) and it failed with the error: Call From <machine_02/ip_address_of_02> to <machine_06:8020> failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused I don't know why it is trying to connect to nn (06) and not snn (07). Can someone point what I am missing here?
... View more
Labels:
- Labels:
-
Apache Hadoop
09-29-2017
03:46 PM
Thanks @Sridhar Reddy. My name node is doing good. I only wanted to move the name node which I did using the "Move Snamenode" wizard on Ambari. I want to sync up the name node with the cluster and am not sure how to do that.
... View more
09-29-2017
03:43 PM
Thanks @Jay SenSharma For some reason I though I have a sudoer account, ambari is not able to install all the user accounts. Anyway, put a request to the client to create those user accounts.
... View more
09-29-2017
03:43 PM
Thanks @chen.yang. Yes, Ambari was not allowing ot install with the same user. Anyway, we submitted a request to the client to create those users for us.
... View more
09-28-2017
05:26 PM
I am installing HDP on client machines with restricted sudo rights.I do not want to install all the services with their respective user accounts, instead want to run then all under one user account. (In fact, I cant do this as I donot have permissions to create all these user accounts. I have to put a request to the client to first create these accounts and then go ahead with the installation). I can see yarn user has to be created since that is declared "final" but can I change all others also to run under "yarn" user? What are the advantages/disadvantages of doing this?
... View more
Labels:
09-23-2017
12:32 AM
Thanks Sonu, that helped 🙂
... View more
09-23-2017
12:31 AM
I have 5 node cluster with 1 name node and 4 data nodes. Right now both my secondary name node and primary name node are on the same machine. I want to add a new secondary name node and sync it up with the rest of the hosts. What is the best possible way to do this? I tried using Ambari's "move secondary name node" wizard and it asked me to copy data into /hadoop/hdfs/namesecondary directory (which is on the boot disk with low disk space) in the new host. But I want to move it into the external hard disks I mounted on /mnt/data1-4. There are about 72 files in /hadoop/hdfs/namesecondary/current which I am not sure how to distribute into the /mnt disks. Can some suggest what is the right way to do this?
... View more
Labels:
- Labels:
-
Apache Hadoop
09-22-2017
04:29 PM
I have a 5 node cluster (machines 01-05) with 4 data nodes (02, 03, 04, 05) which is running good. Now I want to upgrade the cluster from 5 nodes to 12 nodes by adding machines (06-12). Now I want to set 06 and 07 and name node and secondary name nodes instead of 01 (which is currently the name node). Can I do this without losing any data on the cluster? What is the best possible way to do this?
... View more
Labels:
- Labels:
-
Apache Hadoop
09-22-2017
03:25 PM
Thanks a ton @Jay SenSharma. That really helped.
... View more
09-21-2017
09:30 PM
I have a 5 node cluster with 4 data nodes which is in a good condition. But in order to add a secondary name node, I tried to generate the ssh key again instead of copying the already existing key onto the new machine. The new host was not registering with the following error: INFO 2017-09-21 17:24:06,811 NetUtil.py:67 - Connecting to https://asterix01.cem.spirent.com:8440/ca
ERROR 2017-09-21 17:24:06,934 NetUtil.py:93 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
ERROR 2017-09-21 17:24:06,934 NetUtil.py:94 - SSLError: Failed to connect. Please check openssl library versions.
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
WARNING 2017-09-21 17:24:06,937 NetUtil.py:121 - Server at https://asterix01.cem.spirent.com:8440 is not reachable, sleeping for 10 seconds... So I stopped trying to add the new host and checked if the existing hosts are doing any good and restarted ambari-agent of one of them. The logs show failure with the same error as above- which confirms there is an issue with the keys. What can I do to resolve this? I tried to setup passwordless SSH as root user afresh as given in the documentation but it still fails. Can someone help me fix this please? I definitely can ssh without passwords from the command line.
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Hadoop
09-21-2017
07:20 PM
Were you able to resolve it? We have the same issue.
... View more
09-15-2017
02:16 PM
@Vinod Thanks for your reply. Yes, it shows 4 live data nodes. I am surprised how this happened. I have been working on it the whole day yesterday and nothing really happened. Today morning I just restarted the data node process for the failed data node and I can all data nodes are live today. Can you explain this please?
... View more
09-14-2017
04:15 PM
I have a 6 node Centos 7 cluster with 4 datanodes. I have all the datanodes up and running but the dashboard shows only 3/4 datanodes live. I looked at the logs at /var/log/hadoop/hdfs/hadoop-hdfs-datanode-<data_node>.log and it says: 2017-09-14 11:57:04,794 INFO web.DatanodeHttpServer (SimpleHttpProxyHandler.java:exceptionCaught(147)) - Proxy for / failed. cause:
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.UnpooledUnsafeDirectByteBuf.setBytes(UnpooledUnsafeDirectByteBuf.java:447)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:881)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:242)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
at java.lang.Thread.run(Thread.java:745) Not sure what this means. I tried restarting ambari-agent, rebooting the machine itself, restarting ambari-server on namenode. Can someone suggest where else I should look? EDIT: Also, I tried pinging the name node from this particular datanode on that particular port it is listening to (8020- standard port for Hadoop) and it connects. I can see the connection from both, datanode and namenode. I don't understand why the communication is not happening.
... View more
- Tags:
- datanode
- Hadoop Core
Labels:
- Labels:
-
Apache Hadoop
08-07-2017
08:57 PM
I have a Hive table which has to be updated quite often, hence I created it as a "Transactional Table " in ORC Format. I am trying to create a parquet file from it using the following commands in spark: val q= spark.sql("select * from hive_table")
q.write.parquet("hive_table.parquet") I above commands worked for non-transactional tables but don't work for this particular table. I have to have this read into a parquet file because we are using Spark for querying that table (along with many others) Someone please suggest how to do this. Are there any parameters to be set in Spark? I am not even able to do a simple "select count(*) from db_name.hive_table" query on this table from spark. I can do it in Hive after I set some parameters.
... View more
Labels:
- Labels:
-
Apache Spark
07-13-2017
04:15 PM
Thank you @ssathish for your answer, my colleague had set wrong capacity values for the queues and it added up to > 100%. We were able to resolve it.
... View more
07-13-2017
04:13 PM
Hi, I am trying to add extra queues names etl and agg to my already existing defalut and llap queues in Yarn. I have attached both the configurations. The older Scheduler configuration works fine and I am able to use LLAP. But with the 2 extra queues, I get the error: 2017-07-13 12:08:02,548 [main] ERROR main.ServiceLauncher - Exception: Failed to submit application_1499959469167_0009 to YARN : Application application_1499959469167_0009 submitted by user hive to unknown queue: hive That means, the job is not even being submitted. And no, there is no queue called Hive I ever created. Someone please help. Thanks in advance!
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache YARN
07-10-2017
05:08 PM
I added a couple of queues (q1, q2) to the already existing 2 queues (llap and default) in my capacity scheduler and set the parameters accordingly. I divided the cluster resources as: q1: 20, q2: 20, default: 0, llap: 60. Now the resource manager does not start. When I try to restart it, it successfully starts and stops withing seconds. I read other answers to disable the property: yarn.scheduler.capacity.root.accessible-node-labels.default.capacity but I can not even find this property in the xml file nor in Ambari. Can someone please suggest where I should start debugging this?
... View more
Labels:
05-13-2017
08:38 PM
Hi @Bala Vignesh N V I want to partition the table with "integer part of date_key/10000000000". date_key is one of the columns in the table. partitioning it by date_key along will give me millions of part files which I don't want. date_key is of the format: YYYYMMDDHHMMTZ (TZ is the time zone and is 4 digits). So dividing date_key by 10000000000 will make partitions by month. I am trying to migrate data from Vertica to Hadoop. In vertica, the syntax is: PARTITION BY (date_key // 10000000000). The equivalent of // in Hadoop is "DIV". That means if date_key is not perfectly divisible bt the number 10000000000, it returns just the integer part and discards the decimal part. But that is throwing an error for me.
... View more
05-13-2017
07:06 PM
I have a csv file that has huge amount of data which has the data for my hive table. Also, I want to create my hive table as an ORC table with partitions and the partition column is a computed column. So I first created an external table (table1) pointing to the file location, created another table (table2) with orc table properties as I want and inserted the data from table1 to table2. The partition column should be: date_key DIV 10000000000. The script I wrote is: ## to create the orc table:
CREATE TABLE table_name ( key1 type1, key2 type2..... keyn typen) PARTITIONED BY (date_key DIV 10000000000 int) STORED AS ORC tblproperties(........);
INSERT OVERWRITE TABLE table_name PARTITION (date_key DIV 10000000000) SELECT * FROM table_name2;
I am getting an error: FAILED: ParseException line 12:25 cannot recognize input near 'DIV' '10000000000' 'int' in column type Someone please suggest what is wrong here. Am I even doing this right? Is there a different/better way of doing this? Thanks!
... View more
Labels:
- Labels:
-
Apache Hive