Member since
11-17-2015
33
Posts
12
Kudos Received
6
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
4575 | 06-20-2017 02:10 PM | |
83013 | 08-26-2016 01:14 PM | |
2625 | 07-03-2016 06:10 AM | |
37225 | 05-05-2016 02:58 PM | |
3123 | 05-04-2016 08:00 PM |
12-01-2016
03:00 PM
1 Kudo
In the Ambari Hive view, there is a "Saved Queries" tab, where are these queries saved? Are they in the Ambari DB? A local file on the Ambari node? HDFS?
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Hive
08-26-2016
01:14 PM
1 Kudo
Hi @Simran Kaur, You can run the following command to execute a HQL file in hive: hive -f filename.hql Be sure each of your queries in your HQL are terminated with a semi-colon ; Here's the help output of hive too: $ hive -H
WARNING: Use "yarn jar" to launch YARN applications.
hivusage: hive
-d,--define <key=value> Variable subsitution to apply to hive
commands. e.g. -d A=B or --define A=B
--database <databasename> Specify the database to use
-e <quoted-query-string> SQL from command line
-f <filename> SQL from files
-H,--help Print help information
--hiveconf <property=value> Use value for given property
--hivevar <key=value> Variable subsitution to apply to hive
commands. e.g. --hivevar A=B
-i <filename> Initialization SQL file
-S,--silent Silent mode in interactive shell
-v,--verbose Verbose mode (echo executed SQL to the
console)
And here is great documentation for all hive cli options: https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Cli
... View more
08-15-2016
09:06 PM
1 Kudo
After setting this property to false, we are no longer seeing extremely high ulimits for the hive user account. hive.server2.logging.operation.enabled=false Any idea what this is doing to consume so many open files?
... View more
08-04-2016
01:06 PM
Thanks @Hajime for the suggestion, netstat looks OK, not too many connections opened.
... View more
08-02-2016
06:48 PM
Thanks @Benjamin Leonhardi. That's what I am wondering as well. Something doesn't seem right. Can you point me to the ATS issue you are referring to?
... View more
08-02-2016
06:47 PM
Thanks @Scott Shaw We have found that the user's managed version of the limits conf (/etc/security/limits.d/hive.conf) is being used instead of the limits.conf file. Changing the hive.conf file manually gets overwritten once ambari sees that it has changed. But the real question is, should hive really have 64k+ files open?
... View more
07-30-2016
06:56 PM
1 Kudo
By default, hive's ullimit (managed through ambari) is 32000. We reached that limit last week on our hiveserver2 server and decided to increase this value in ambari to 64000. We just hit the 64k nofile ulimit. This leads me to believe that hiveserver2 is not cleaning up connections like it should and files are not being released. Has anyone else experienced this issue? Any suggestions on what to check? What do you have your hive user's nofile limit set too? # lsof -u hive | wc -l 64450 We are on HDP 2.4.2, Ambari 2.2.2 Should hive really have that many files open? Update: We're approaching the 64k nofile ulimit setting again for the hive user. # lsof -u hive | wc -l 57090 After digging through the output of lsof, I see a lot of temporary operation_logs. /tmp/hive/operation_logs/658c3930-8975-47db-ad7f-7cbef6279b11/acc2043a-d3bb-4a8c-9a7d-d0b743b9ce5d Here is the total number of open operation_logs files open right now. # lsof -u hive | grep operation_logs | wc -l 56102 These files are 3 to 4 days old.
... View more
Labels:
- Labels:
-
Apache Hive
07-03-2016
06:17 AM
Hi @Sunile Manjee, Thank you for your response. This is the documentation I followed to setup this environment: https://doc.lucidworks.com/lucidworks-hdpsearch/2.3/Guide-Install.html I will be testing performance against HDFS indexing with NRT setup. I have local SSD disks setup as a fallback if this isn't fast enough or too unreliable. Thanks, Jon
... View more
07-03-2016
06:10 AM
After more digging, I discovered the solrconfig.xml in ZK was not the correct version. I did a series of downconfig and upconfig to load the correct configs and verify everything is OK. After loading the correct solrconfig.xml and restarting each solr node, the create collection command succeeded. /opt/lucidworks-hdpsearch/solr/server/scripts/cloud-scripts/zkcli.sh -cmd downconfig -d collection -z $zk_quorum:2181/solr -n collection
/opt/lucidworks-hdpsearch/solr/server/scripts/cloud-scripts/zkcli.sh -cmd upconfig -d $path_to_configs -z $zk_quorum:2181/solr -n collection
... View more
07-02-2016
06:14 AM
Hello, I am trying to setup and configure HDPSearch. I have 4 solr boxes running 6 instances of solr. I have setup HDFS with NN HA. All 4 boxes can successfully reach HDFS using the NN HA name. However, I am receiving the below error when trying to create a collection in solr. What is solr missing that it can't connect to HDFS?
126330 ERROR (qtp59559151-22) [c:collection s:shard23 r:core_node86 x:collection_shard23_replica3] o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: Error CREATEing SolrCore 'collection_shard23_replica3': Unable to create core [collection_shard23_replica3] Caused by: NN_HA_Name.
.. 31 more
Caused by: java.net.UnknownHostException: NN_HA_Name
... 45 more
Here is the command to start solr cloud:
solr -c -p 8983 -z $zk_quorum:2181/solr -Dsolr.directoryFactory=HdfsDirectoryFactory -Dsolr.lock.type=hdfs-Dsolr.hdfs.home=hdfs://NN_HA_Name/apps/solr
Here is the command to create the collection:
solr create -c collection -d collection -n collection -s 48 -rf 3
Here are my solrconfig.xml DirectoryFactory Settings:
<directoryFactory name="DirectoryFactory" class="solr.HdfsDirectoryFactory">
<str name="solr.hdfs.home">hdfs://NN_HA_Name/apps/solr</str>
<str name="solr.hdfs.confdir">/etc/hadoop/conf</str>
<bool name="solr.hdfs.blockcache.enabled">true</bool>
<int name="solr.hdfs.blockcache.slab.count">1</int>
<bool name="solr.hdfs.blockcache.direct.memory.allocation">true</bool>
<int name="solr.hdfs.blockcache.blocksperbank">16384</int>
<bool name="solr.hdfs.blockcache.read.enabled">true</bool>
<bool name="solr.hdfs.nrtcachingdirectory.enable">true</bool>
<int name="solr.hdfs.nrtcachingdirectory.maxmergesizemb">16</int>
<int name="solr.hdfs.nrtcachingdirectory.maxcachedmb">192</int>
</directoryFactory>
I have installed the hdfs clients on the solr nodes and can successfully hdfs dfs -ls hdfs://NN_HA_Name/apps/solr I also see core-site.xml and hdfs-site.xml (with the correct NN configurations) in the /etc/hadoop/conf directory. Thanks,
Jon
... View more
Labels:
- Labels:
-
Apache Solr
- « Previous
-
- 1
- 2
- Next »