Member since
09-24-2015
816
Posts
488
Kudos Received
189
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2634 | 12-25-2018 10:42 PM | |
12096 | 10-09-2018 03:52 AM | |
4174 | 02-23-2018 11:46 PM | |
1860 | 09-02-2017 01:49 AM | |
2177 | 06-21-2017 12:06 AM |
03-23-2016
04:51 AM
3 Kudos
Hi @Lubin Lemarchand, for simple, single script, independent jobs you can also use cron. For jobs consisting of 2 or more scripts you can use Oozie which is the scheduling tool of choice in Hadoop ecosystem. Initially, Oozie didn't support Hive but now it does together with Pig, MR, Sqoop, Spark and other, so-called "actions" or steps. Long term, it's definitely worth learning more about Oozie: quick start.
... View more
03-21-2016
01:50 PM
1 Kudo
I'm having the same issue, after Kafka upgrade from 0.8.1 to 0.9.0. Everything works against newly created topics, but not against old ones. I checked the details and found that IDs are now numbered from 1001, while before they were starting from 0. For example: $ ./kafka-topics.sh --zookeeper zk1.example.com:2181 --describe --topic k81c
Topic:k81c PartitionCount:2 ReplicationFactor:2 Configs:
Topic: k81c Partition: 0 Leader: 0 Replicas: 0,1 Isr: 0
Topic: k81c Partition: 1 Leader: 0 Replicas: 1,0 Isr: 0
$ ./kafka-topics.sh --zookeeper zk1.example.com:2181 --describe --topic k9a
Topic:k9a PartitionCount:2 ReplicationFactor:1 Configs:
Topic: k9a Partition: 0 Leader: 1002 Replicas: 1002 Isr: 1002
Topic: k9a Partition: 1 Leader: 1001 Replicas: 1001 Isr: 1001
I also checked ZK's entries for /topics/ids, there are (in my case, with 2 brokers) only 2 of them for 1001 and 1002.
... View more
03-20-2016
08:48 AM
You need HDP-2.3.4, 2.3.4.7, or 2.4. HDP-2.3.2 has Kafka-0.8.2 IIRC. Edit: Sorry, what I meant is that Kafka-0.9.0.0 is included in the above versions.
... View more
03-18-2016
06:59 AM
Can you try to add that final quote through Ambari, not directly in the oozie-env file. The value looks good, at least the "unexpected EOF" error should go away. By the way, in my 2.3.2 version the count is set to 5 instead of 2.
... View more
03-12-2016
01:24 AM
2 Kudos
Your solution indicates an unhealthy cluster, and/or erroneous kerberization. Normally, there is no need to touch these files, unless you are installing ZK manually, but you are talking about Ambari. There is also no need for any j2 files. In healthy cluster it's enough to set JVMFLAGS and invoke zkCli.sh. I tested the other day, before filing my answer.
... View more
03-12-2016
12:39 AM
2 Kudos
@Cassandra Spencer, the answers are simple: (1) No, it can be any other group. Actually, I don't use this setting, rather set groups per proxy-user (to "*" or particular groups). (2) Human users: No, human users can be in any group. I usually keep service accounts in the hadoop group, like what you plan to do. (3) Ranger will require its own service account, by default called "ranger". It can be in its own group, I use the "hadoop" group. If your cluster is kerberized you'll need one more account usually called "rangerlookup" to facilitate autocompletion of databases, tables etc, with a headless principal and a password (keytab unsupported). The docs talk about a rangerlookup account per service (hdfs, hbase, etc.) but I use only one.
... View more
03-11-2016
03:05 AM
2 Kudos
Usually no problems, provided port 8670 is available for the agent to listen to, and iptables or a firewall etc is not blocking the agent to connect to Ambari server on ports 8080, 8440 and 8441. Also need Python-2.6.x.
... View more
03-11-2016
02:14 AM
1 Kudo
That's easy, if you are on CentOS/RHEL yum remove ambari-agent ... make sure it's gone: yum list ambari-agent
yum install ambari-agent Before removal save the config file if needed, it's in /etc/ambari-agent/conf/ambari-agent.ini. By the way, you can upgrade it by "yum upgrade" after providing a new repo file in /etc/yum.repos.d
... View more
03-10-2016
04:30 AM
1 Kudo
Hi @R M, login to Ambari, on the left side select Hive, and on the right at the top select Summary. Than click on HiveServer2 to see the host where it's running.
... View more
03-10-2016
04:02 AM
2 Kudos
Ping is not enough. If you run iptables, stop them for a while to see if some ports are blocked. Agents are accessing the server at 8080, 8440 and 8441, and the server is pinging agents at 8670. Details are here. And, well, you haven't mentioned starting ambari-agent, but I'm sure you did it.
... View more