question Re: Druid Installation in Archives of Support Questions (Read Only)

Druid Installation

BharathN — Fri, 16 Sep 2022 15:47:02 GMT

Hello Everyone - We have a 16 node cluster [12 data nodes, 2 Master nodes, 2 edge nodes].All the servers have 32 cores & 252GB RAM.

For Installing druid, how to choose the following : Cordinator,Superset,Broker,Overlord,Router.Can I have all these sitting on 1 instance?

And for the Slaves & client: Druid Historical & Druid MiddleManager, how to pick on what servers these to be installed.

Once the installation is done, are there other steps involved to integrate this with hive or is it done as part of installation?

As part of Druid Installation, are there any other services that gets impacted or needs to be restarted?

I have gone through this article: https://community.hortonworks.com/questions/108316/how-to-choose-servers-for-druid.html

Please let me know if I miss anything or any other steps involved prior to the installation.

Thanks in advance.

Re: Druid Installation

sbouguerra — Thu, 05 Oct 2017 04:40:27 GMT

yes, you can have a bundle of broker router and coordinator overlord over one physical node and you can have a couple of this. For historicals and middle manager, it depends on your use case, if you have more historical data to serve then you need more historical and vise versa. For the hive integration, you need to set those.

set hive.druid.metadata.username=${DRUID_USERNAME};
set hive.druid.metadata.password=${DRUID_PASSWORD};
set hive.druid.metadata.uri=jdbc:mysql://${DRUID_HOST}/${DATA_BASE_NAME};
set hive.druid.broker.address.default=${DRUID_HOST}:8082;
set hive.druid.coordinator.address.default=${DRUID_HOST}:8081;
set hive.druid.storage.storageDirectory=/apps/hive/warehouse;

make sure that ,

/apps/hive/warehouse

Is readable by hadoop group.

Starting from HDP 2.6.3 all those properties will be set automatically.

Re: Druid Installation

BharathN — Thu, 05 Oct 2017 05:14:40 GMT

@Slim For historicals and middle manager, do they need to be installed on Data nodes or Edge nodes from where users generally access the cluster?

Re: Druid Installation

sbouguerra — Thu, 05 Oct 2017 05:17:59 GMT

User is not going to query the middle manager or historical so no need to have it on Edge nodes.

Re: Druid Installation

BharathN — Thu, 05 Oct 2017 05:56:35 GMT

@Slim

I started the Druid installation process, but ran into some errors

ExecutionFailed: Execution of '/usr/bin/yum -d 0 -e 0 -y install superset_2_6_0_3_8' returned 1. Error: Package: superset_2_6_0_3_8-0.15.0.2.6.0.3-8.x86_64 (HDP-2.6) Requires: libffi-devel

I tried uninstalling "libffi-devel" but it has dependencies, and not able to uninstall

root@Host:~ # rpm -e libffi-3.0.5-3.2.el6.x86_64

error: Failed dependencies:

libffi.so.5()(64bit) is needed by (installed) python-libs-2.6.6-66.el6_8.x86_64

libffi.so.5()(64bit) is needed by (installed) python34-libs-3.4.5-1.el6.x86_64

root@host:~ # /usr/bin/yum -d 0 -e 0 -y install superset_2_6_0_3_8 --skip-broken

Packages skipped because of dependency problems:

openblas-0.2.19-2.el6.x86_64 from CDP

openblas-devel-0.2.19-2.el6.x86_64 from CDP

openblas-openmp-0.2.19-2.el6.x86_64 from CDP

openblas-threads-0.2.19-2.el6.x86_64 from CDP

superset_2_6_0_3_8-0.15.0.2.6.0.3-8.x86_64 from HDP-2.6

Re: Druid Installation

sbouguerra — Thu, 05 Oct 2017 06:00:58 GMT

can you start a new thread and add more information about the install process and the stack trace ?

Re: Druid Installation

lvazquez — Mon, 01 Apr 2019 22:23:58 GMT

If you set these properties by hand, take into account that in HDP 3.x the route for the Hive Warehouse has been changed to:

/warehouse/tablespace/managed/hive

Also in the Ambari configuration for Druid 3.1 the property is set to:

hive.druid.storage.storageDirectory =  {{druid_storage_dir}}

which is expanded to /apps/druid/warehouse, a different path from the Hive Warehouse.