Member since
01-25-2017
119
Posts
7
Kudos Received
2
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
8865 | 04-11-2017 12:36 PM | |
1564 | 01-18-2017 10:36 AM |
12-28-2018
12:09 PM
This issue occurred after mainboard change. Do you think it is related with this change? Or nothing to do with it?
... View more
12-28-2018
11:16 AM
Hello @scharan , Thanks for your reply. I have a feeling that renewal of agent keys (maybe both on agent and server) would be the proper way. Do you aggree? Regardless of that, of course I accept this answer! Agent can connect now and works fine! Thanks a lot @scharan! Best regards. Have a nice day and new year!
... View more
12-27-2018
02:06 PM
Hello, We had a problem with one of our node's mainboard and it was changed. As we re-opened the node ambari agent could not connect to ambari server with below error: INFO 2018-12-27 16:59:24,790 NetUtil.py:70 - Connecting to https://master01:8440/ca
ERROR 2018-12-27 16:59:24,797 NetUtil.py:96 - EOF occurred in violation of protocol (_ssl.c:618)
ERROR 2018-12-27 16:59:24,797 NetUtil.py:97 - SSLError: Failed to connect. Please check openssl library versions.
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
WARNING 2018-12-27 16:59:24,797 NetUtil.py:124 - Server at https://master01:8440 is not reachable, sleeping for 10 seconds... My humble guess is that old keys were not accepted by ambari server with new hardware. Guys who installed the mainboard says they updated the seriel with the old one. How can I get back this node? Is there any way to renew keys? PS: There are no files in path /var/lib/ambari-agent/keys/ Thanks in advance.
... View more
- Tags:
- ambari-agent
- Hive
- ssl
Labels:
- Labels:
-
Apache Ambari
-
Apache Hive
10-19-2018
01:25 PM
For my case, previously I had given my anaconda installation. Reverting it back to /usr/bin/python2.7 fixed the problem.
... View more
08-16-2018
01:52 PM
@Sampath Kumar, @SHAIKH FAIROZ AHMED, @Jack Marquez, did you find the solution to this problem? I am facing it too.
... View more
07-11-2018
03:23 PM
Thanks for your answer and also for the warning about the version in the stack. Current Spark2 version is 2.2.0. I am going to correct it on question. And also both answers are good news to me. Thanks again.
... View more
07-10-2018
10:13 PM
My team needs Spark v2.3 for new features. We have HDP 2.6.3 installed which has Spark 2.0 (Correction:2.2.0) within stack. Is that enough to comply such version requirement if I use a docker container as Spark Driver which has Spark 2.3 and configure it so as to use Yarn of current HDP installation? Or do i need all workers Spark 2.3 installed? The thing I need to understand is does workers (or nodemanagers) need new Spark libraries once job is submitted to Yarn? Following note in Spark Cluster overview page led me to think it may not be mandatory: "The user's jar
should never include Hadoop or Spark libraries, however, these will be added at runtime." Thanks in advance...
... View more
Labels:
- Labels:
-
Apache Spark
-
Apache YARN
06-14-2018
09:58 AM
Does physical data really remain in the node? For my case I saw lots of (thousands and repeating) HDFS log lines (on the node) about deleting blocks. Are these lines unexpected (already have broken RAID controller)? They keep being scheduled and deleted. 2018-06-14 11:58:53,005 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-1789482724-9.1.10.22-1491814552298 blk_1155905937_82210162 file /grid/2/hadoop/hdfs/data/current/BP-1789482724-9.1.10.22-1491814552298/current/finalized/subdir229/subdir185/blk_1155905937
... View more
05-24-2018
09:13 AM
What I did was editing the code giving '/tmp/hive/hive' as the root directory. I should have given '/tmp' at the first place.
... View more
03-23-2018
08:51 PM
Good idea. Felt bad and lazy for not inspecting the code before. Got again OOM however. Thank you anyways. I will try to spare some time to take a look at the code. I will update the question if I can progress.
... View more
03-20-2018
06:43 PM
Hello @Aditya Sirna Thank you for your answer. I have added the parameter with a value of 0 but got an exception (HDP 2.6.3.0 on CentOS 7.2) 2018-03-20 21:09:48,207 ERROR namenode.FSNamesystem (FSNamesystem.java:<init>(913)) - FSNamesystem initialization failed. java.lang.IllegalArgumentException: Cannot set dfs.namenode.fs-limits.max-directory-items to a value less than 1 or greater than 6400000 Thus, I doubled the old value (4194304) and now it works. Will HDFS be removing the tmp dir? Is there any preset period of configuration for that? Otherwise may tmp dir exceed the new limit? Or may hdfs get OOM exception while cleaning it like i got trying to clean manually? You can check my other question if you have a comment on it. https://community.hortonworks.com/questions/179904/having-issue-with-tmp-directory-removal.html
... View more
03-20-2018
04:08 PM
Initial solution i tried was a script in below answer https://community.hortonworks.com/answers/92110/view.html I modified this script to work in /tmp/hive/hive path but it received OOM error.
... View more
03-20-2018
03:38 PM
Hello I need to remove /tmp/hive/hive directory content which has too many folders which i guess most of them are empty. bash-4.2$ hadoop fs -count /tmp/hive/hive 2097194 18 2581710 /tmp/hive/hive I tried using a script which referred before in below question https://community.hortonworks.com/questions/21684/today-tfh-refused-to-start-with-message-of-orgapac.html Script is located in Github: https://github.com/nmilford/clean-hadoop-tmp Script is written in Ruby. When i run this script I receive an exception. How can I delete them? Do you have any suggestions? Exception with Ruby script is: -bash-4.2$ ./clean-hadoop-tmp Dropping lock file. (/var/tmp/clean-hadoop-tmp.lock) Scanning for directories in HDFS' /tmp older than 1800 seconds. ./clean-hadoop-tmp:38:in `block in <main>': undefined method `each' for nil:NilClass (NoMethodError) from ./clean-hadoop-tmp:35:in `each' from ./clean-hadoop-tmp:35:in `<main>' Should I modify this script and how? Is there any other ways?
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache Hive
03-20-2018
03:20 PM
Hello, I am having an issue with /tmp/hive/tmp directory: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException): The directory item limit of /tmp/hive/hive is exceeded: limit=1048576 items=1048576 With a short search i find out that it is a setting with parameter "dfs.namenode.fs-limits.max-directory-items" which is in hdfs-default.xml file. However it is not available for Ambari. Which file should I update? What is the right path? Should i update it on both hosts for HA mode?
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Hadoop
03-14-2018
01:34 PM
Follow up...
... View more
03-13-2018
08:55 PM
Hello, This question can be trivial but I am stuck at it when enabling SQLStdAuth. I have enabled SQLStdAuth in Hive config in Ambari. Then i followed this article: https://community.hortonworks.com/content/supportkb/48940/how-to-setup-sql-based-authorization-in-hive-with.html However if i create a connection with the username that i set admin role without providing any password, i was able to drop a table after setting current role to admin. Saying that, having only the username but not the password is enough. The username that i have set admin role is an OS user that i created. Am i missing that part? With which layer or component does hive check user's password in jdbc connection? Or is this the expected behaviour and there is no password check? Thanks in advance
... View more
Labels:
- Labels:
-
Apache Hive
03-12-2018
07:49 PM
@Benjamin Leonhardi if I use SQL Authentication using this method, how should I assign passwords to users? For using in DB connection for instance... Will Hive consider OS level user passwords? If so, should I set a password also for 'hive' user? Does it affect other operations?
... View more
12-05-2017
02:24 PM
@Geoffrey Shelton Okot Unfortunately, I couldn't start HDFS services this way, neither. Thank you very much though.
... View more
12-05-2017
08:09 AM
Hi @Geoffrey Shelton Okot, I had tried hadoop namenode -format before but tried again and received the same exception: 17/12/05 09:46:25 ERROR namenode.NameNode: Failed to start namenode.
org.apache.hadoop.hdfs.qjournal.client.QuorumException: Could not format one or more JournalNodes. 2 exceptions thrown:
10.0.109.11:8485: Directory /hadoop/hdfs/journal/testnamespace is in an inconsistent state: Can't format the storage directory because the current directory is not empty.
at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.checkEmptyCurrent(Storage.java:482)
at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:558)
at org.apache.hadoop.hdfs.qjournal.server.JNStorage.format(JNStorage.java:185)
at org.apache.hadoop.hdfs.qjournal.server.Journal.format(Journal.java:217)
at org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.format(JournalNodeRpcServer.java:145)
at org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolServerSideTranslatorPB.format(QJournalProtocolServerSideTranslatorPB.java:145)
at org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos$QJournalProtocolService$2.callBlockingMethod(QJournalProtocolProtos.java:25419)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2347) This time additionally I deleted content of /hadoop/hdfs/journal/testnamespace but nothing changed. Command ended up with the same exception.
... View more
12-04-2017
01:40 PM
So.. What are the steps for reinstall? Is there any way to start from only HDP installation but keeping OS level changes as prerequisite and also ambari installation? Does command ambari-server reset work for that?
... View more
12-04-2017
10:55 AM
If recovery steps will take more than re-install and/or give me an unstable cluster then its better to reinstall. What I anticipate from your answer you mean such costs, right?
... View more
12-01-2017
02:11 PM
Hello, After a mass disk operation on our test environment, we have lost all the data in /data dir which was assigned as storage directory for Zookeeper, Hadoop and Falcon (the list yet we know) Since it was our test cluster, data is not important but I don't want to reinstall all the components. I also want to learn how to recover the cluster running from this state. In /data dir we only have folders but no files. After struggling a little on ZKFailoverController, I was able to start it with -formatZK flag. Now however, I am unable to start namenode(s) getting below exception: 10.0.109.12:8485: Directory /hadoop/hdfs/journal/testnamespace is in an inconsistent state: Can't format the storage directory because the current directory is not empty. I have tried; - removing lost+found folder on mount root, - changing ownership of all folders under /data/hadoop/hdfs to hdfs:hadoop - changing permission of all folders to 777 /data/hadoop/hdfs PS: I have updated ownership of path /hadoop/hdfs/ which contains journal folder and it led me to move one step forward: 17/12/01 14:20:26 ERROR namenode.NameNode: Failed to start namenode.
java.io.IOException: Cannot remove current directory: /data/hadoop/hdfs/namenode/current PS: I have removed contents of /data/hadoop/hdfs/namenode/current and now it keeps checking 8485 ports of all Journal quorum nodes. 17/12/01 16:04:35 INFO ipc.Client: Retrying connect to server: bigdata2/10.0.109.11:8485. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) and keeps printing below line in hadoop-hdfs-zkfc-bigdata2.out file Proceed formatting /hadoop-ha/testnamespace? (Y or N) Invalid input: Do you have any suggestion? Or should I give up?
... View more
Labels:
- Labels:
-
Apache Hadoop
11-28-2017
01:04 PM
Hello @Aditya Sirna, Thank you, it worked. Cluster is not Kerberized. 1) Value was: "org.apache.zeppelin.notebook.repo.GitNotebookRepo,org.apache.zeppelin.notebook.repo.VFSNotebookRepo" I have added also FileSystemNotebookRepo and after restart it updated the directory with the new notebook. May this requirement be missing in upgrade documentation?
... View more
11-28-2017
11:52 AM
After HDP 2.6.3 upgrade, I expect seeing Zeppelin starts updating its HDFS directory with new notes created since I was noticed in upgrade doc: https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-upgrade/content/upgrading_HDP_prerequisites.html However, I don't see files or folders modified dates updated either new was created in HDFS. I can't see any errors related to HDFS in Zeppelin log. HDFS log also does not contain any new lines for Zeppelin. What I remember is I forgot copying notebook folder in HDFS in prerequisites step but did it after upgrade was completed. Then i restarted Zeppelin. Do you have any idea about how I can make it write on HDFS? Thanks in advance...
... View more
Labels:
- Labels:
-
Apache Zeppelin
11-24-2017
01:55 PM
1 Kudo
For a complete YARN administration I would recommend below famous book by Sam R. Alapati: https://www.amazon.com/Expert-Hadoop-Administration-Addison-Wesley-Analytics/dp/0134597192 Chapter 13 describes how YARN works and Chapter 18 explains how you can make tuning. Below page describes not only Hive parameters for Hive tuning but also mentions what to do on Yarn. https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/bk_hive-performance-tuning/content/ch_hive-perf-tuning-intro.html I would suggest reading this to understand how capacity scheduler works. This way you understand how it works and what to do to share resources: https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html
... View more
11-21-2017
01:17 PM
We have just overcame such issue. I assume you have enabled pre-emption on YARN. After that, a little bit of Hive tuning is kind of prerequisite for all users when enabling LLAP (interactive query component). Most common problem with starting interactive server is conflicts between YARN and Hive for resource usage. There are some main parameters which are mostly effective on starting (and running stable) LLAP: hive.llap.daemon.yarn.container.mb: Suggestion on Ambari is pretty clear: Total memory used by individual LLAP daemons (YARN Container size). This includes memory
for the cache as well as for the query execution. Should be larger than the sum of
the Daemon cache size and the daemon heap size, and should leave some headroom
after this (In most cases: cache size + heap size + headroom = Memory Per Daemon) We also tried to keep it slightly below yarn.scheduler.maximum-allocation-mb value of yarn config. This way we prevent resource conflict with YARN in basic level. llap_headroom_space : Maximum headroom reserved from the YARN container running LLAP daemons.
This is an upper limit used during automatic size calculations, and the actual
value may be lower. We left this as recommended value which is very low compared to others. It is about %6 of llap_heap_size llap_heap_size: LLAP Daemon Heap Size in MB. hive.llap.io.memory.size: The amount of memory reserved for Hive's optimized in-memory cache. In memory cache size should be %20 of llap heap size And below parameters are number of executors which are set in default according to number of CPU's num_llap_nodes_per_llap_daemons: Number of Node(s) on which Hive LLAP daemon runs.<br> We set this value to number of our nodes (nodemanagers / datanodes). hive.llap.daemon.num.executors: The number of fragments that a single LLAP daemon will run concurrently.
Usually, this will be the same as the number of available CPUs<br> We set this to %40 of number of CPUs hive.llap.io.threadpool.size: Specify the number of threads to use for low-level IO thread pool.<br> We also set this one with the same value of number of executors. After updating these values, you can save configs and start interactive server. Afterwards, you may need to update queue capacities on Yarn. If you are using Ambari, Yarn Queue Manager view is ideal as it provides ability to view and adjust the values visually. Here you should check if LLAP uses more resource over default queue. If you don't have YARN Queue Manager, you can adjust below parameters and variants of the for default queue. yarn.scheduler.capacity.root.llap.capacity=12.0 yarn.scheduler.capacity.root.llap.maximum-am-resource-percent=1 yarn.scheduler.capacity.root.llap.maximum-capacity=12.0
... View more
11-21-2017
11:52 AM
1 Kudo
If it is only one datanode failing but not all of them, it may be failing due to a volume failure. You can check Namenode web-ui to see if you are facing any volume failures: http://<active-namenode-host>:50070/dfshealth.html#tab-datanode-volume-failures or just http://<active-namenode-host>:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystemState For plain JMX data to see whether or not you're having volume failures (it is not physical disk error all the time. It may occur after logical write issues).
... View more
11-14-2017
02:29 PM
Right, @Sedat Kestepe
... View more
11-06-2017
09:09 AM
@Deepesh, @Tamil Selvan K thank you both for your responses. And I thought it was installed as another component like Spark2 came with Spark 1.6 etc. So that means, Hive 2 will replace old 1.2, right? And what about other components listed in my question? How am I supposed to add them?
... View more
11-03-2017
02:37 PM
Hello, I have upgraded HDP from 2.5.3 to 2.6.2.14 Getting benefits of Hive 2.1.0 was one of the motivation points in my upgrade plan. However, even i kept myself optimistic every step i did not see it Hive 2 was not there as upgrade finalised eventually. Furthermore, it is not there as I want to add new services. What am i missing? Info: Centos 7.3.1611 Cluster of 4 VMs Ambari was upgraded from 2.4.2 to 2.5.2 Services able to add: Ranger 0.7.0, Ranger KMS 0.7.0, Druid 0.10.1, Storm 1.1.0 Services that i expected to see but missing: Apache Calcite 1.2.0, Apache DataFu 1.3.0, Apache Hive 2.1.0, Apache Phoenix 4.7.0, Cascading 3.0.0, Hue 2.6.1
... View more
Labels:
- Labels:
-
Apache Hive