question Re: New Cluster Install (Design Question) in Support Questions

New Cluster Install (Design Question)

oakdemirci — Thu, 14 Jan 2016 16:25:09 GMT

Hi all,

I downloaded and ran sandbox but I want to install Hadoop on a server on a test environment. I tried a standalone Windows installation by following the guide but failed. I think the guide has some missing parts, like formatting hdfs, configuring derby with hive etc.

I need some help about arcihtecture I think. If I install ambari on a centOS VM, can I add windows nodes with ambari.

Thanks

Regards 😉

Özgür

Re: New Cluster Install (Design Question)

oakdemirci — Thu, 14 Jan 2016 18:03:41 GMT

PS: I'll be happy if you can give a complete tutorial if exists 😉

Re: New Cluster Install (Design Question)

aervits — Thu, 14 Jan 2016 21:08:30 GMT

@Özgür Akdemirci

feel free to paste the actual problems as questions on HCC. I believe it is still not possible to have a mixed OS environment.

Re: New Cluster Install (Design Question)

nsabharwal — Thu, 14 Jan 2016 21:18:57 GMT

@Özgür Akdemirci

If I install ambari on a centOS VM, can I add windows nodes with ambari. - No

I recommend having centos image and configure all the networking pieces and then add nodes

Re: New Cluster Install (Design Question)

oakdemirci — Thu, 14 Jan 2016 23:28:21 GMT

Thank you @Neeraj Sabharwal & @Artem Ervits 😉

Before cluster, I want to end Windows single node installation with success. Below are the errors.

And what must be the next step for me when trying hadoop with windows standalone. Should I follow hello world tutorial of Sandbox?

Really appreciate for your help.

Regards

After running smoke tests, I got errors like:

1- Hive Smoke Test NestedThrowablesStackTrace: The query returned more than one instance BUT either unique is set to true or on ly aggregates are to be returned, so should have returned one result maximum org.datanucleus.store.query.QueryNotUniqueException: The query returned more tha n one instance BUT either unique is set to true or only aggregates are to be ret urned, so should have returned one result maximum at org.datanucleus.store.query.Query.executeQuery(Query.java:1822) at org.datanucleus.store.query.Query.executeWithArray(Query.java:1672) at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:243) at org.apache.hadoop.hive.metastore.ObjectStore.getMDatabase(ObjectStore .java:540) at org.apache.hadoop.hive.metastore.ObjectStore.convertToMTable(ObjectSt ore.java:1085) at org.apache.hadoop.hive.metastore.ObjectStore.createTable(ObjectStore. java:813) at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces sorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.j ava:114) at com.sun.proxy.$Proxy0.createTable(Unknown Source) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_tabl e_core(HiveMetaStore.java:1416) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_tabl e_with_environment_context(HiveMetaStore.java:1449) at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces sorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHM SHandler.java:107) at com.sun.proxy.$Proxy18.create_table_with_environment_context(Unknown Source) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$cr eate_table_with_environment_context.getResult(ThriftHiveMetastore.java:9200) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$cr eate_table_with_environment_context.getResult(ThriftHiveMetastore.java:9184) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedPr ocessor.java:110) at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedPr ocessor.java:106) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma tion.java:1657) at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBased Processor.java:118) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadP oolServer.java:285) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor. java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor .java:615) at java.lang.Thread.run(Thread.java:744) )

Run-HiveSmokeTest : Hive Smoke Test: FAILED At line:1 char:1 + Run-HiveSmokeTest + ~~~~~~~~~~~~~~~~~ + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorExcep tion + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorExceptio n,Run-HiveSmokeTest

HiveServer2 smoke test - drop table, create table and describe table Connecting to jdbc:hive2://XYZXYZ:10001/ 16/01/14 16:59:59 INFO jdbc.Utils: Supplied authorities: XYZXYZ:10001 16/01/14 16:59:59 INFO jdbc.Utils: Resolved authority: XYZXYZ:10001 16/01/14 16:59:59 INFO jdbc.HiveConnection: Will try to open client transport wi th JDBC Uri: jdbc:hive2://XYZXYZ:10001/ Connected to: Apache Hive (version 1.2.1.2.3.0.0-2557) Driver: Hive JDBC (version 1.2.1.2.3.0.0-2557) Transaction isolation: TRANSACTION_REPEATABLE_READ No rows affected (2.429 seconds) Beeline version 1.2.1.2.3.0.0-2557 by Apache Hive Closing: 0: jdbc:hive2://XYZXYZ:10001/ Connecting to jdbc:hive2://XYZXYZ:10001/ 16/01/14 17:00:03 INFO jdbc.Utils: Supplied authorities: XYZXYZ:10001 16/01/14 17:00:03 INFO jdbc.Utils: Resolved authority: XYZXYZ:10001 16/01/14 17:00:03 INFO jdbc.HiveConnection: Will try to open client transport wi th JDBC Uri: jdbc:hive2://XYZXYZ:10001/ Connected to: Apache Hive (version 1.2.1.2.3.0.0-2557) Driver: Hive JDBC (version 1.2.1.2.3.0.0-2557) Transaction isolation: TRANSACTION_REPEATABLE_READ Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:For direct Me taStore DB connections, we don't support retries at the client level.) (state=08 S01,code=1) Closing: 0: jdbc:hive2://XYZXYZ:10001/

Run-HiveServer2SmokeTest : HiveServer2 Smoke Test: FAILED At line:1 char:1 + Run-HiveServer2SmokeTest + ~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorExcep tion + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorExceptio n,Run-HiveServer2SmokeTest

Re: New Cluster Install (Design Question)

nsabharwal — Sun, 17 Jan 2016 09:04:44 GMT

@Özgür Akdemirci

Windows install: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4-Win/index.html

You asked: If I install ambari on a centOS VM, can I add windows nodes with ambari.

My answer is No because there is no ambari for windows.

Re: New Cluster Install (Design Question)

oakdemirci — Sun, 17 Jan 2016 21:11:45 GMT

Thanks @Neeraj Sabharwal again, =D

what I meant is by following the Windows tutorial, during smoke tests, I got some errors about hive, and I gave the errors with my below message. When I check the link you gave (http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4-Win/index.html) I realized that HDP-2.3.4 exists (???), but when I start a download, it says it'll download 2.3.0. Is there an update to 2.3.0?

I'll try windows standalone one more time.

Regards.

Re: New Cluster Install (Design Question)

nsabharwal — Mon, 19 Aug 2019 12:15:28 GMT

@Özgür Akdemirci

Its downloading 2.3.4.0.zip

Re: New Cluster Install (Design Question)

oakdemirci — Mon, 19 Aug 2019 12:15:20 GMT

=D, murphy on the job.I'm doing sth wrong. By using this page http://hortonworks.com/hdp/downloads/ the result is on the image =D

Can I have a direct link. Thanks a lot.

Re: New Cluster Install (Design Question)

oakdemirci — Sun, 17 Jan 2016 22:12:05 GMT

I found the 2.3.4 installation link in documentation. Thanks 😉

is it my mistake or does the main page need an update ?

😄

Really thanks for the help.

Regards

Re: New Cluster Install (Design Question)

nsabharwal — Sun, 17 Jan 2016 22:51:00 GMT

🙂 Murphy on the job ..lol @Özgür Akdemirci

http://public-repo-1.hortonworks.com/HDP-Win/2.3/2.3.4.0/hdp-2.3.4.0.zip

Re: New Cluster Install (Design Question)

nsabharwal — Sun, 17 Jan 2016 23:08:05 GMT

@Özgür Akdemirci

I will send a note to docs team. Thanks for find this out.

Re: If I install ambari on a centOS VM, can I add windows nodes with ambari.

Answer is No because there is no ambari for windows. Bits are different

Re: New Cluster Install (Design Question)

oakdemirci — Fri, 22 Jan 2016 18:04:44 GMT

@Neeraj Sabharwal

really appreciate for your useful directions, I gave up windows and installed I have a 6 virtual node centos 7 cluster (on windows virtualbox). The cluster has the following nodes.

1 edge, 1 namenode, 1 secondary namenode, 3 datanodes,

and it's working now ;), I can use ambari without any red errors.

I began hello world tutorial,

In hive section of the tutorial, I created geolocation_stage table with a CREATE statement, the statement is executing without an error. But I could not see my staging table under default database,

An interesting point is, if I use search database textbox, I can show the staging table.

I verified the table with an sql statement to mysql hive database, "select * from TBLS;"

Am I on the right way ? 😄

Thanks

Regards

Özgür

Re: New Cluster Install (Design Question)

oakdemirci — Fri, 22 Jan 2016 19:43:34 GMT

What I meant is the table I created is not shown under default database, refresh also not works.

Re: New Cluster Install (Design Question)

nsabharwal — Fri, 22 Jan 2016 19:46:39 GMT

@Özgür Akdemirci Could you do me a favor please? 🙂 If you can post this as new question then it will helpful as its new issue. Accept one of the answers in this thread only if it was helpful.

When you post a new question then post the output of

hive

show tables;