Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Is it necessary to install HDP along with HDF for working streaming analytics manager ??

Is it necessary to install HDP along with HDF for working streaming analytics manager ??

Contributor
6 REPLIES 6

Re: Is it necessary to install HDP along with HDF for working streaming analytics manager ??

Mentor

@Aditya Gary

In a nutshell, NO Schema Analytics Manager is part of HDF and doesn't need HDP to operationalize. I have a 3 node HDF cluster running SAM its drag and drop future akin of NiFI UI helps to build a working streaming application without writing a single line of code.

It allows you to add/connect to any remote HDF/HDP cluster by adding the Ambari URL

See attached screenshot



nifi02.png

Re: Is it necessary to install HDP along with HDF for working streaming analytics manager ??

Contributor

Thanks for the quick reply @Geoffrey Shelton Okot its extremely kind of you !!
Please have a look at this issue I am having:-
https://community.hortonworks.com/questions/233052/sam-cannot-store-and-run-any-test-case-gives-nosu...

Re: Is it necessary to install HDP along with HDF for working streaming analytics manager ??

Mentor

@Aditya Gary

I have seen your other post too how can I reproduce your case? As reiterated I set up a 3 node cluster with SAM and if I could reproduce your use case, basically there are 3 ways to install HDF see attached screenshot

  • Installing an HDF Cluster
  • Installing HDF Services on an existing HDP cluster
  • Installing HDF Services on a new HDP cluster

Please share your walk through


hdf-sam2.png
Highlighted

Re: Is it necessary to install HDP along with HDF for working streaming analytics manager ??

Contributor

Thanks @Geoffrey Shelton Okot for responding it was extremely kind of you
I have setup the HDF cluster on a single machine, with all the services installed on the same host. No HDP installed.
I created a single Kafka source and sink in SAM.
This is my schema
{ "namespace": "hdf.heap.com", "type": "record", "name": "PatientField", "fields": [ { "name": "Patient_name", "type": "string" } ] }
and this is my test data
{ "Patient_name": "john" }
I get an exception when I try to run my application (NoSuchMethodException)
When try to run my test case it gives me a storageException
SAM is not allowing me to even delete the test cases whilst giving the same exception !!

I am using a console producer to talk to the kafka source(consumer) inside SAM but that is also not showing any activity.
Any help is much appreciated !!!

Re: Is it necessary to install HDP along with HDF for working streaming analytics manager ??

Contributor

@Geoffrey Shelton OkotHow many kafka brokers are required for a kafka source and sink in the dataflow that i created ??

Re: Is it necessary to install HDP along with HDF for working streaming analytics manager ??

Mentor

@Aditya Gary

All depends on the resilence your application is built to meet. The basic config is one broker. As a rule of thumb just like for the zookeeper ensemble to avoid the split brain decision you will need at least 3 brokers A three(3) node {zk or Kafka broker} cluster can survive the loss of 1 node see attached screenshot.

99451-3-brokers.png

I have set up a 3 node HDF cluster (3 zookeeper & 3 brokers ) using virtual box each with 6 GB of RAM for test purposes.

Don't have an account?
Coming from Hortonworks? Activate your account here