Druid Installation

[#1]

Hello Everyone - We have a 16 node cluster [12 data nodes, 2 Master nodes, 2 edge nodes].All the servers have 32 cores & 252GB RAM.

For Installing druid, how to choose the following : Cordinator,Superset,Broker,Overlord,Router.Can I have all these sitting on 1 instance?

And for the Slaves & client: Druid Historical & Druid MiddleManager, how to pick on what servers these to be installed.

Once the installation is done, are there other steps involved to integrate this with hive or is it done as part of installation?

As part of Druid Installation, are there any other services that gets impacted or needs to be restarted?

I have gone through this article: https://community.hortonworks.com/questions/108316/how-to-choose-servers-for-druid.html

Please let me know if I miss anything or any other steps involved prior to the installation.

Thanks in advance.

[#2 · Accepted solution]

yes, you can have a bundle of broker router and coordinator overlord over one physical node and you can have a couple of this. For historicals and middle manager, it depends on your use case, if you have more historical data to serve then you need more historical and vise versa. For the hive integration, you need to set those.

set hive.druid.metadata.username=${DRUID_USERNAME};
set hive.druid.metadata.password=${DRUID_PASSWORD};
set hive.druid.metadata.uri=jdbc:mysql://${DRUID_HOST}/${DATA_BASE_NAME};
set hive.druid.broker.address.default=${DRUID_HOST}:8082;
set hive.druid.coordinator.address.default=${DRUID_HOST}:8081;
set hive.druid.storage.storageDirectory=/apps/hive/warehouse;

make sure that ,

/apps/hive/warehouse

Is readable by hadoop group.

Starting from HDP 2.6.3 all those properties will be set automatically.

[#3]

If you set these properties by hand, take into account that in HDP 3.x the route for the Hive Warehouse has been changed to:

/warehouse/tablespace/managed/hive

Also in the Ambari configuration for Druid 3.1 the property is set to:

hive.druid.storage.storageDirectory =  {{druid_storage_dir}}

which is expanded to /apps/druid/warehouse, a different path from the Hive Warehouse.

[#4]

@Slim For historicals and middle manager, do they need to be installed on Data nodes or Edge nodes from where users generally access the cluster?

[#5]

User is not going to query the middle manager or historical so no need to have it on Edge nodes.

[#6]

@Slim

I started the Druid installation process, but ran into some errors

ExecutionFailed: Execution of '/usr/bin/yum -d 0 -e 0 -y install superset_2_6_0_3_8' returned 1. Error: Package: superset_2_6_0_3_8-0.15.0.2.6.0.3-8.x86_64 (HDP-2.6) Requires: libffi-devel

I tried uninstalling "libffi-devel" but it has dependencies, and not able to uninstall

root@Host:~ # rpm -e libffi-3.0.5-3.2.el6.x86_64

error: Failed dependencies:

libffi.so.5()(64bit) is needed by (installed) python-libs-2.6.6-66.el6_8.x86_64

libffi.so.5()(64bit) is needed by (installed) python34-libs-3.4.5-1.el6.x86_64

root@host:~ # /usr/bin/yum -d 0 -e 0 -y install superset_2_6_0_3_8 --skip-broken

Packages skipped because of dependency problems:

openblas-0.2.19-2.el6.x86_64 from CDP

openblas-devel-0.2.19-2.el6.x86_64 from CDP

openblas-openmp-0.2.19-2.el6.x86_64 from CDP

openblas-threads-0.2.19-2.el6.x86_64 from CDP

superset_2_6_0_3_8-0.15.0.2.6.0.3-8.x86_64 from HDP-2.6

[#7]

can you start a new thread and add more information about the install process and the stack trace ?