HI All, I installed a HDP cluster on cloudbreak and am trying to run a simple Spark Job. I open the "pyspark" shell and run the following: ip = "adl://alenzadls1.azuredatalakestore.net/path/to/my/input/directory" input_data = sc.textFile(ip) for x in input_data.collect(): print x The print statement returns an error: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/hdp/current/spark-client/python/pyspark/rdd.py", line 771, in collect port = self.ctx._jvm.PythonRDD.collectAndServe(self._jrdd.rdd()) File "/usr/hdp/current/spark-client/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py", line 813, in __call__ File "/usr/hdp/current/spark-client/python/pyspark/sql/utils.py", line 45, in deco return f(*a, **kw) File "/usr/hdp/current/spark-client/python/lib/py4j-0.9-src.zip/py4j/protocol.py", line 308, in get_return_value py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe. : java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.adl.AdlFileSystem not found Can someone point me to where it is going wrong? I did not find anything related to this online. ... View more

I just deployed my cluster on Azure using Cloudbreak. Right now my name node and datanode directories are: /hadoopfs/fs1/hdfs/namenode and /hadoop/hdfs/data,/hadoopfs/fs1/hadoop/hdfs/data,/mnt/resource/hadoop/hdfs/data respectively. I want to change this to a location on my Data Lake Store. When I just paste the path starting with "adl:/" it throws an error. I setup the access and everything. I just want to change the directories to save space on my local FS. Also, I am getting alerts like: Capacity Used:[100%, 28672], Capacity Remaining:[0]. I have 110 GB RAM on my master node. Why is it already used up? ... View more

I currently have Ambari 2.2.2.0 and want to upgrade to the latest version. My HDP version is 2.4.2. So I am trying to upgrade to 2.4 first (since I cant jump directly to 2.6 from 2.2) and am following the steps given at: https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.0.1/bk_ambari-upgrade/content/upgrade_ambari.html. Everything is good until I rech the "ambari-server upgrade" step. This step fails with the error: [root@newyorknn ~]# ambari-server upgrade Using python /usr/bin/python Upgrading ambari-server Updating properties in ambari.properties ... WARNING: Original file ambari-env.sh kept Fixing database objects owner Ambari Server configured for Embedded Postgres. Confirm you have made a backup of the Ambari Server database [y/n] (y)? y Upgrading database schema Error output from schema upgrade command: Exception in thread "main" org.apache.ambari.server.AmbariException: ERROR: schema "ambarirca" does not exist at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:207) at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:347) Caused by: org.postgresql.util.PSQLException: ERROR: schema "ambarirca" does not exist at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255) at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:559) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:403) at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:395) at org.apache.ambari.server.orm.DBAccessorImpl.executeQuery(DBAccessorImpl.java:827) at org.apache.ambari.server.orm.DBAccessorImpl.executeQuery(DBAccessorImpl.java:819) at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.changePostgresSearchPath(AbstractUpgradeCatalog.java:361) at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeSchema(AbstractUpgradeCatalog.java:886) at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:204) ... 1 more ERROR: Error executing schema upgrade, please check the server logs. ERROR: Ambari server upgrade failed. Please look at /var/log/ambari-server/ambari-server.log, for more details. ERROR: Exiting with exit code 11. REASON: Schema upgrade failed. Can someone suggest what I should do? I looked at the ambari-server database files that I backed up according to the instructions given at https://community.hortonworks.com/articles/37765/backing-up-the-ambari-database-with-postgres.html (ambari.sql and ambarirca.sql) but none of them have any command called "CREATE SCHEMA AMBARIRCA". I am not sure how to proceed. ... View more

One of the disks on one of my data nodes was failing, so I replaced it with the instructions: 1. Stop all services on datanode. 2. shut down the machine. 3. replace the disk 4. power on the machine. 5. mount the disk onto its data point. 6. start all services on HDFS. Now, I get an alert "Pending Deletion Blocks:[276861]" on Ambari. Did I do something wrong? I can I revert it? ... View more

I am running a Sql query in Spark: spark.sql("select person_key, count(*) as count1 from <table_name> group by person_key order by count1 desc").show() This throws a warning: 17/10/05 12:09:03 WARN ReaderImpl: Cannot find field for: person_key in _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, 17/10/05 12:09:03 WARN ReaderImpl: Cannot find field for: person_key in _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, 17/10/05 12:09:03 WARN ReaderImpl: Cannot find field for: person_key in _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, 17/10/05 12:09:03 WARN ReaderImpl: Cannot find field for: person_key in _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, 17/10/05 12:09:03 WARN ReaderImpl: Cannot find field for: person_key in _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, 17/10/05 12:09:03 WARN ReaderImpl: Cannot find field for: person_key in _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, 17/10/05 12:09:03 WARN ReaderImpl: Cannot find field for: person_key in _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, But does give correct results. I want to understand what this means. I did not find anything on the net. I want this resolved because though it is giving the correct results, it is taking very long to execute. (The same query on Hive LLAP takes 3 seconds. Spark numbers are usually comparable to Hive LLAP numbers). I checked person_key does exist in the table (I created it so the table so I know it exists). Not sure why the warning is coming. ... View more

I have a transactional (ACID) table in Hive and can query it fine using the hive shell. However, I want to query it using LLAP/SPark Shell which I am unable to do. When I set all the parameters required for querying ACID tables (transaction manager, compactor threads etc) in the beeline shell I open to connect to Hive Interactive Server, it seems to work but when I run the query, everything just halts and nothing works. After a very long time I get the following error message: ERROR : Status: Failed ERROR : Dag received [DAG_TERMINATE, SERVICE_PLUGIN_ERROR] in RUNNING state. ERROR : Error reported by TaskScheduler [[2:LLAP]][SERVICE_UNAVAILABLE] No LLAP Daemons are running ERROR : Vertex killed, vertexName=Reducer 2, vertexId=vertex_1506697113479_0016_1_03, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_TERMINATED, failedTasks:0 killedTasks:1009, Vertex vertex_1506697113479_0016_1_03 [Reducer 2] killed/failed due to:DAG_TERMINATED] ERROR : Vertex killed, vertexName=Map 3, vertexId=vertex_1506697113479_0016_1_00, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_TERMINATED, failedTasks:0 killedTasks:1, Vertex vertex_1506697113479_0016_1_00 [Map 3] killed/failed due to:DAG_TERMINATED] ERROR : Vertex killed, vertexName=Reducer 4, vertexId=vertex_1506697113479_0016_1_01, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_TERMINATED, failedTasks:0 killedTasks:1, Vertex vertex_1506697113479_0016_1_01 [Reducer 4] killed/failed due to:DAG_TERMINATED] ERROR : Vertex killed, vertexName=Map 1, vertexId=vertex_1506697113479_0016_1_02, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_TERMINATED, failedTasks:0 killedTasks:354, Vertex vertex_1506697113479_0016_1_02 [Map 1] killed/failed due to:DAG_TERMINATED] ERROR : DAG did not succeed due to SERVICE_PLUGIN_ERROR. failedVertices:0 killedVertices:4 ERROR : FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Dag received [DAG_TERMINATE, SERVICE_PLUGIN_ERROR] in RUNNING state.Error reported by TaskScheduler [[2:LLAP]][SERVICE_UNAVAILABLE] No LLAP Daemons are runningVertex killed, vertexName=Reducer 2, vertexId=vertex_1506697113479_0016_1_03, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_TERMINATED, failedTasks:0 killedTasks:1009, Vertex vertex_1506697113479_0016_1_03 [Reducer 2] killed/failed due to:DAG_TERMINATED]Vertex killed, vertexName=Map 3, vertexId=vertex_1506697113479_0016_1_00, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_TERMINATED, failedTasks:0 killedTasks:1, Vertex vertex_1506697113479_0016_1_00 [Map 3] killed/failed due to:DAG_TERMINATED]Vertex killed, vertexName=Reducer 4, vertexId=vertex_1506697113479_0016_1_01, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_TERMINATED, failedTasks:0 killedTasks:1, Vertex vertex_1506697113479_0016_1_01 [Reducer 4] killed/failed due to:DAG_TERMINATED]Vertex killed, vertexName=Map 1, vertexId=vertex_1506697113479_0016_1_02, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_TERMINATED, failedTasks:0 killedTasks:354, Vertex vertex_1506697113479_0016_1_02 [Map 1] killed/failed due to:DAG_TERMINATED]DAG did not succeed due to SERVICE_PLUGIN_ERROR. failedVertices:0 killedVertices:4 Error: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Dag received [DAG_TERMINATE, SERVICE_PLUGIN_ERROR] in RUNNING state.Error reported by TaskScheduler [[2:LLAP]][SERVICE_UNAVAILABLE] No LLAP Daemons are runningVertex killed, vertexName=Reducer 2, vertexId=vertex_1506697113479_0016_1_03, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_TERMINATED, failedTasks:0 killedTasks:1009, Vertex vertex_1506697113479_0016_1_03 [Reducer 2] killed/failed due to:DAG_TERMINATED]Vertex killed, vertexName=Map 3, vertexId=vertex_1506697113479_0016_1_00, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_TERMINATED, failedTasks:0 killedTasks:1, Vertex vertex_1506697113479_0016_1_00 [Map 3] killed/failed due to:DAG_TERMINATED]Vertex killed, vertexName=Reducer 4, vertexId=vertex_1506697113479_0016_1_01, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_TERMINATED, failedTasks:0 killedTasks:1, Vertex vertex_1506697113479_0016_1_01 [Reducer 4] killed/failed due to:DAG_TERMINATED]Vertex killed, vertexName=Map 1, vertexId=vertex_1506697113479_0016_1_02, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_TERMINATED, failedTasks:0 killedTasks:354, Vertex vertex_1506697113479_0016_1_02 [Map 1] killed/failed due to:DAG_TERMINATED]DAG did not succeed due to SERVICE_PLUGIN_ERROR. failedVertices:0 killedVertices:4 (state=08S01,code=2) Similarly in Spark, I open the spark shell, make the following imports: import org.apache.spark.sql.SparkSession import spark.implicits._ val spark = SparkSession.builder().appName("Spark SQL basic example").config("spark.sql.parquet.compression.codec", "lzo").getOrCreate() spark.sqlContext.setConf("spark.sql.parquet.compression.codec", "lzo") spark.sqlContext.setConf("spark.sql.crossJoin.enabled", "true") and run the query as: val q1 = spark.sql("select * fron <hive_transaction_table") q1.show() It gives me an error: org.apache.spark.sql.catalyst.errors.package$TreeNodeException: execute, tree: Exchange SinglePartition +- *HashAggregate(keys=[], functions=[partial_count(1)], output=[count#159L]) +- HiveTableScan MetastoreRelation avlino_bm, subscriber_dim_test_parquet at org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:56) at org.apache.spark.sql.execution.exchange.ShuffleExchange.doExecute(ShuffleExchange.scala:112) at org.apache.spark.sql.execution.SparkPlan$anonfun$execute$1.apply(SparkPlan.scala:114) at org.apache.spark.sql.execution.SparkPlan$anonfun$execute$1.apply(SparkPlan.scala:114) at org.apache.spark.sql.execution.SparkPlan$anonfun$executeQuery$1.apply(SparkPlan.scala:135) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:132) at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:113) at org.apache.spark.sql.execution.InputAdapter.inputRDDs(WholeStageCodegenExec.scala:235) at org.apache.spark.sql.execution.aggregate.HashAggregateExec.inputRDDs(HashAggregateExec.scala:141) at org.apache.spark.sql.execution.WholeStageCodegenExec.doExecute(WholeStageCodegenExec.scala:368) at org.apache.spark.sql.execution.SparkPlan$anonfun$execute$1.apply(SparkPlan.scala:114) at org.apache.spark.sql.execution.SparkPlan$anonfun$execute$1.apply(SparkPlan.scala:114) at org.apache.spark.sql.execution.SparkPlan$anonfun$executeQuery$1.apply(SparkPlan.scala:135) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:132) at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:113) at org.apache.spark.sql.execution.SparkPlan.getByteArrayRdd(SparkPlan.scala:225) at org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:308) at org.apache.spark.sql.execution.CollectLimitExec.executeCollect(limit.scala:38) at org.apache.spark.sql.Dataset$anonfun$org$apache$spark$sql$Dataset$execute$1$1.apply(Dataset.scala:2378) at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:57) at org.apache.spark.sql.Dataset.withNewExecutionId(Dataset.scala:2780) at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$execute$1(Dataset.scala:2377) at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$collect(Dataset.scala:2384) at org.apache.spark.sql.Dataset$anonfun$head$1.apply(Dataset.scala:2120) at org.apache.spark.sql.Dataset$anonfun$head$1.apply(Dataset.scala:2119) at org.apache.spark.sql.Dataset.withTypedCallback(Dataset.scala:2810) at org.apache.spark.sql.Dataset.head(Dataset.scala:2119) at org.apache.spark.sql.Dataset.take(Dataset.scala:2334) at org.apache.spark.sql.Dataset.showString(Dataset.scala:248) at org.apache.spark.sql.Dataset.show(Dataset.scala:638) at org.apache.spark.sql.Dataset.show(Dataset.scala:597) at org.apache.spark.sql.Dataset.show(Dataset.scala:606) ... 52 elided Caused by: java.lang.RuntimeException: serious problem at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1021) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1048) at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:202) at org.apache.spark.rdd.RDD$anonfun$partitions$2.apply(RDD.scala:252) at org.apache.spark.rdd.RDD$anonfun$partitions$2.apply(RDD.scala:250) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.rdd.RDD.partitions(RDD.scala:250) at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35) at org.apache.spark.rdd.RDD$anonfun$partitions$2.apply(RDD.scala:252) at org.apache.spark.rdd.RDD$anonfun$partitions$2.apply(RDD.scala:250) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.rdd.RDD.partitions(RDD.scala:250) at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35) at org.apache.spark.rdd.RDD$anonfun$partitions$2.apply(RDD.scala:252) at org.apache.spark.rdd.RDD$anonfun$partitions$2.apply(RDD.scala:250) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.rdd.RDD.partitions(RDD.scala:250) at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35) at org.apache.spark.rdd.RDD$anonfun$partitions$2.apply(RDD.scala:252) at org.apache.spark.rdd.RDD$anonfun$partitions$2.apply(RDD.scala:250) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.rdd.RDD.partitions(RDD.scala:250) at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35) at org.apache.spark.rdd.RDD$anonfun$partitions$2.apply(RDD.scala:252) at org.apache.spark.rdd.RDD$anonfun$partitions$2.apply(RDD.scala:250) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.rdd.RDD.partitions(RDD.scala:250) at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35) at org.apache.spark.rdd.RDD$anonfun$partitions$2.apply(RDD.scala:252) at org.apache.spark.rdd.RDD$anonfun$partitions$2.apply(RDD.scala:250) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.rdd.RDD.partitions(RDD.scala:250) at org.apache.spark.ShuffleDependency.<init>(Dependency.scala:91) at org.apache.spark.sql.execution.exchange.ShuffleExchange$.prepareShuffleDependency(ShuffleExchange.scala:261) at org.apache.spark.sql.execution.exchange.ShuffleExchange.prepareShuffleDependency(ShuffleExchange.scala:84) at org.apache.spark.sql.execution.exchange.ShuffleExchange$anonfun$doExecute$1.apply(ShuffleExchange.scala:121) at org.apache.spark.sql.execution.exchange.ShuffleExchange$anonfun$doExecute$1.apply(ShuffleExchange.scala:112) at org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:52) ... 85 more Caused by: java.util.concurrent.ExecutionException: java.lang.NumberFormatException: For input string: "0000035_0000" at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:998) ... 122 more Caused by: java.lang.NumberFormatException: For input string: "0000035_0000" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) at java.lang.Long.parseLong(Long.java:589) at java.lang.Long.parseLong(Long.java:631) at org.apache.hadoop.hive.ql.io.AcidUtils.parseDelta(AcidUtils.java:310) at org.apache.hadoop.hive.ql.io.AcidUtils.getAcidState(AcidUtils.java:379) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.call(OrcInputFormat.java:634) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.call(OrcInputFormat.java:620) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Is anyone else facing the same issue? HOw can I access transactional tables from Hive LLAP or Spark? ... View more

Hi all, This question is in continuation to: https://community.hortonworks.com/questions/138257/how-to-sync-a-new-secondary-namenode-to-the-cluste.html Scenario: I have a 12 node cluster (machines 01-12) with 8 data nodes. 06, 07 are the NN and SNN respectively. 01 and 12 run the Hive related services. This cluster was upgraded to 12 node from 4 node where 01 was the namenode and 02-05 were the data nodes. So I used the "Move Namenode" and "Move Snamenode" wizards on Ambari to move the nn and snn from 01 to 06 and 07 respectively. I verified all the services running on nn are running on snn as well. Now I want to check if my snn is working properly. So I shut down all the services on nn and tried to connect to Hive from one of the hosts (02) and it failed with the error: Call From <machine_02/ip_address_of_02> to <machine_06:8020> failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused I don't know why it is trying to connect to nn (06) and not snn (07). Can someone point what I am missing here? ... View more

I am installing HDP on client machines with restricted sudo rights.I do not want to install all the services with their respective user accounts, instead want to run then all under one user account. (In fact, I cant do this as I donot have permissions to create all these user accounts. I have to put a request to the client to first create these accounts and then go ahead with the installation). I can see yarn user has to be created since that is declared "final" but can I change all others also to run under "yarn" user? What are the advantages/disadvantages of doing this? ... View more

I have 5 node cluster with 1 name node and 4 data nodes. Right now both my secondary name node and primary name node are on the same machine. I want to add a new secondary name node and sync it up with the rest of the hosts. What is the best possible way to do this? I tried using Ambari's "move secondary name node" wizard and it asked me to copy data into /hadoop/hdfs/namesecondary directory (which is on the boot disk with low disk space) in the new host. But I want to move it into the external hard disks I mounted on /mnt/data1-4. There are about 72 files in /hadoop/hdfs/namesecondary/current which I am not sure how to distribute into the /mnt disks. Can some suggest what is the right way to do this? ... View more

I have a 6 node Centos 7 cluster with 4 datanodes. I have all the datanodes up and running but the dashboard shows only 3/4 datanodes live. I looked at the logs at /var/log/hadoop/hdfs/hadoop-hdfs-datanode-<data_node>.log and it says: 2017-09-14 11:57:04,794 INFO web.DatanodeHttpServer (SimpleHttpProxyHandler.java:exceptionCaught(147)) - Proxy for / failed. cause: java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:192) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at io.netty.buffer.UnpooledUnsafeDirectByteBuf.setBytes(UnpooledUnsafeDirectByteBuf.java:447) at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:881) at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:242) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) at java.lang.Thread.run(Thread.java:745) Not sure what this means. I tried restarting ambari-agent, rebooting the machine itself, restarting ambari-server on namenode. Can someone suggest where else I should look? EDIT: Also, I tried pinging the name node from this particular datanode on that particular port it is listening to (8020- standard port for Hadoop) and it connects. I can see the connection from both, datanode and namenode. I don't understand why the communication is not happening. ... View more

I have a Hive table which has to be updated quite often, hence I created it as a "Transactional Table " in ORC Format. I am trying to create a parquet file from it using the following commands in spark: val q= spark.sql("select * from hive_table") q.write.parquet("hive_table.parquet") I above commands worked for non-transactional tables but don't work for this particular table. I have to have this read into a parquet file because we are using Spark for querying that table (along with many others) Someone please suggest how to do this. Are there any parameters to be set in Spark? I am not even able to do a simple "select count(*) from db_name.hive_table" query on this table from spark. I can do it in Hive after I set some parameters. ... View more

Hi, I am trying to add extra queues names etl and agg to my already existing defalut and llap queues in Yarn. I have attached both the configurations. The older Scheduler configuration works fine and I am able to use LLAP. But with the 2 extra queues, I get the error: 2017-07-13 12:08:02,548 [main] ERROR main.ServiceLauncher - Exception: Failed to submit application_1499959469167_0009 to YARN : Application application_1499959469167_0009 submitted by user hive to unknown queue: hive That means, the job is not even being submitted. And no, there is no queue called Hive I ever created. Someone please help. Thanks in advance! ... View more

I added a couple of queues (q1, q2) to the already existing 2 queues (llap and default) in my capacity scheduler and set the parameters accordingly. I divided the cluster resources as: q1: 20, q2: 20, default: 0, llap: 60. Now the resource manager does not start. When I try to restart it, it successfully starts and stops withing seconds. I read other answers to disable the property: yarn.scheduler.capacity.root.accessible-node-labels.default.capacity but I can not even find this property in the xml file nor in Ambari. Can someone please suggest where I should start debugging this? ... View more

I have a csv file that has huge amount of data which has the data for my hive table. Also, I want to create my hive table as an ORC table with partitions and the partition column is a computed column. So I first created an external table (table1) pointing to the file location, created another table (table2) with orc table properties as I want and inserted the data from table1 to table2. The partition column should be: date_key DIV 10000000000. The script I wrote is: ## to create the orc table: CREATE TABLE table_name ( key1 type1, key2 type2..... keyn typen) PARTITIONED BY (date_key DIV 10000000000 int) STORED AS ORC tblproperties(........); INSERT OVERWRITE TABLE table_name PARTITION (date_key DIV 10000000000) SELECT * FROM table_name2; I am getting an error: FAILED: ParseException line 12:25 cannot recognize input near 'DIV' '10000000000' 'int' in column type Someone please suggest what is wrong here. Am I even doing this right? Is there a different/better way of doing this? Thanks! ... View more