Member since
09-22-2017
37
Posts
0
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2298 | 11-16-2019 10:29 AM |
12-01-2021
07:01 AM
I have seen some of my jobs are using only one reducer in the end. Does it mean we have one output file? Or it depends?
... View more
Labels:
- Labels:
-
Apache Hive
12-01-2021
06:07 AM
Our user is running a job and which is a hive query and number of mapper is always 6 and not changing even the data size change. It is a insert query. How do I change number of mappers ? Which parameter determine number of mappers?
... View more
Labels:
- Labels:
-
Apache Hive
10-13-2021
04:58 PM
I can do query through spark using spark.sql, but when I tried via beeline ranger is blocking the access because of the policy. How does it work via spark query? Doesn't it check HDFS level permission? I
... View more
Labels:
- Labels:
-
Apache Ranger
-
Apache Spark
-
HDFS
04-19-2020
08:23 AM
Hello Guys,
If we are moving the active directory server from one to another without changing REALM or anything. Can we just migrate the keytabs and user information to new server? Or do we need to re-generate the keytabs?
... View more
Labels:
- Labels:
-
Kerberos
11-20-2019
09:00 AM
Hello,
I am trying to setup oozie high availability. Can I run both as active servers ? Submit jobs to each server? Or does it works as primary and secondary?
... View more
Labels:
- Labels:
-
Apache Oozie
11-16-2019
05:02 PM
Thank you for the info. Yes, I have created with backup with another directory and I was about to boot restart the namenode from that image.
... View more
11-16-2019
04:57 PM
Hello,
I copied the llap jdbc string from ambari to connect to hive interactive server, but it shows as connected to port number 10000 which belongs to hiveserver. Isn't suppose to connect to port 10500? Am I missing some thing ?
INFO jdbc.HiveConnection: Connected to server1.hiverserever.net:10000
... View more
Labels:
- Labels:
-
Apache Hive
11-16-2019
10:29 AM
Thank you for the information. If I have multiple directories configured for dfs.namenode.name.dir. Defualt: /hadoop/hdfs/namenode I can configure /backup/namenode /goldbackup/namenode In this case I will have multiple copy of FSimage and Editlog and in case one the directory is corrupted, I can use another one. Does it make sense?
... View more
11-15-2019
03:35 PM
Hello,
I have a basic question about about namenode format. If we perform namenode format, will clean up all the metadata info in namenode like directory structure, etc. After it completed, if we restart namenode, would it rebuild the directory structure from datanode information? Or we are lost.
... View more
Labels:
- Labels:
-
Apache Hadoop
12-01-2018
03:34 PM
Hello, I would like to replicate hive data from one cluster to another cluster, in the beginning I can do snapshot and I can us distcp to copy to other cluster, but for me to keep in sync rather than running disctcp cronjob. Can I have use hbase replication feature to sync the hive data? Since it is in hdfs I just need to replicate /apps/warehouse/hive/dir1 Is some thing I can do using habase?
... View more
Labels:
- Labels:
-
Apache HBase
05-25-2018
01:24 PM
Hello, When NN writes data(block) to DN or when DN send block over to next DN, what protocol they use? RPC or HTTP?
... View more
Labels:
- Labels:
-
Apache Hadoop
05-24-2018
02:31 PM
Thanks a lot for giving great explanation. Just a last question, delegation token is given by NN, from NN what package or what tool provides the delegation token. Is it part of namenode package?
... View more
05-24-2018
02:25 PM
Thank you very much, we are using centos we should be able to install using yum?
... View more
05-24-2018
02:07 PM
As far as I know after kerberos authentication is established, we can get delegation token and even if we kdestroy the tickets, we can still access using delegation token. Is delegation token part of kerberos or just depend on kereberos? Is it just a separate pacakge?
... View more
Labels:
- Labels:
-
Kerberos
05-24-2018
01:51 PM
Hello, I need to plan a edge node or utility node to run hive jobs and few other standard jobs. What are the standard packages needed for client? I know it all depends on the jobs, but just wondering what are the base packages needed.
... View more
- Tags:
- utility
Labels:
- Labels:
-
Apache Hive
05-24-2018
01:39 PM
I'm just curious. Why do hive jobs need the home directory to run the job?
... View more
05-22-2018
03:39 PM
I have kerbertized our environtment with default priveleges. I created another user called tom, but haven't given any provileges, but user can execute all the commands in our cluster such as hdfs, etc.. Do I need to prevent in acl ?
... View more
05-21-2018
11:49 AM
Hello, From my understanding namenode and datanodes uses RPC to protocol to exchange data between name node and datanodes (HDFS data). Is this correct Or Do they used different protocol?
... View more
- Tags:
- data-nodes
Labels:
- Labels:
-
HDFS
05-16-2018
06:17 PM
Hi Geoffery, Does all the journal node manages their own edit log in their local disk ?
... View more
05-16-2018
04:28 PM
Thank you for your answer. When we do upgrade, does it only touch the binaries of Yarn, MR and HDFS or does it touch other components as well?
... View more
05-15-2018
08:54 PM
Hello, I have a question regarding "shared edits dir" location. It is in /hadoop/hdfs/journal/clustername/current/ it is stored in hdfs file system. Is it suppose to work or can we configure in any other file sytems like NFS?
... View more
- Tags:
- hadoop
- Hadoop Core
Labels:
- Labels:
-
Apache Hadoop
05-15-2018
02:33 PM
Hello, We have HDP 2.2, how do I upgrade it to 2.6. Can we directly upgrade it to 2.6? Or do we need to upgrade it to 2.4? And 2.6?
... View more
Labels:
- Labels:
-
Hortonworks Data Platform (HDP)
01-23-2018
09:19 PM
We are planning to migrate all the data nodes from RHEL 6 to RHEL 7 , but looks like direct upgrade is not supporting and we are not confident that it will go smooth. Even if it goes well , there may be a future problem. Have anyone done RHEL to RHEL 7 upgrade? We are planning to migrate data to other cluster nodes and do fresh installation. The problem is it might take longer to migrate. Any specific method to upgrade?
... View more
09-25-2017
04:12 PM
That was typo on cut and paste, but it is there in right format as you mentioned. How do it start ambari server? How do we force it to hostname instead of localhost ?
... View more
09-25-2017
03:58 PM
cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 52.72.242.100 mokkan1.myserver.com 54.15.189.202 mokkan2.myserver.com [root@mokkan1 ~]# hostname -f mokkan1.myserver.com
[root@mokkan1 ~]# python -c 'import socket;print socket.getfqdn()' mokkan1.myserver.com [root@mokkan1 ~]# netstat -tnlpa | grep -i AmbariServer
... View more
09-25-2017
03:37 PM
Here is ambari server output. mokkan1.myserver.com
[root@mokkan1 sysconfig]# hostname
mokkan1.myserver.com
[root@mokkan1 sysconfig]# netstat -tnlpa | grep 8440
tcp6 0 0 :::8440 :::* LISTEN 2308/java
[root@mokkan1 sysconfig]# netstat -tnlpa | grep 8441
tcp6 0 0 :::8441 :::* LISTEN 2308/java
[root@mokkan1 sysconfig]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 52.75.242.100 mokkan1.myserver.com 54.15.189.202 mokkan2.myserver.com [root@mokkan1 sysconfig]# telnet localhost 8440
Trying ::1...
Connected to localhost.
Escape character is '^]'.
^CConnection closed by foreign host.
[root@mokkan1 sysconfig]# telnet mokkan1.myserver.com 8440
Trying 52.70.242.100
... View more
09-25-2017
02:25 PM
The problem is I can do telent to localhost 8440, but when I try with telent ambari-servername 8440, it is not connecting. When the service starts, it is binding with localhost rather than host name. How do I force it to bind with hostname rather than localhost?
... View more
09-25-2017
02:19 PM
Thank you for your reply. The problem is this I can't telnet to ambari server from agent, but ambari server is up and running. From ambari server, I can do telnet localhost 8440, it works, but I can't do telnet ambariserver 8440, some how when it starts, it is binding with localhost, not with server name.
... View more