Is it necessary to install HDP along with HDF for working streaming analytics manager ??

[#1]

@Suresh Kota @Sandeep Nemuri @pshah @Arun Mahadevan @ARUN

@George Vetticade

[#2]

@Aditya Gary

In a nutshell, NO Schema Analytics Manager is part of HDF and doesn't need HDP to operationalize. I have a 3 node HDF cluster running SAM its drag and drop future akin of NiFI UI helps to build a working streaming application without writing a single line of code.

It allows you to add/connect to any remote HDF/HDP cluster by adding the Ambari URL

See attached screenshot

[#3]

Thanks for the quick reply @Geoffrey Shelton Okot its extremely kind of you !!
Please have a look at this issue I am having:-
https://community.hortonworks.com/questions/233052/sam-cannot-store-and-run-any-test-case-gives-nosu...

[#4]

@Aditya Gary

I have seen your other post too how can I reproduce your case? As reiterated I set up a 3 node cluster with SAM and if I could reproduce your use case, basically there are 3 ways to install HDF see attached screenshot

• Installing an HDF Cluster
• Installing HDF Services on an existing HDP cluster
• Installing HDF Services on a new HDP cluster

Please share your walk through

[#5]

Thanks @Geoffrey Shelton Okot for responding it was extremely kind of you
I have setup the HDF cluster on a single machine, with all the services installed on the same host. No HDP installed.
I created a single Kafka source and sink in SAM.
This is my schema
{ "namespace": "hdf.heap.com", "type": "record", "name": "PatientField", "fields": [ { "name": "Patient_name", "type": "string" } ] }
and this is my test data
{ "Patient_name": "john" }
I get an exception when I try to run my application (NoSuchMethodException)
When try to run my test case it gives me a storageException
SAM is not allowing me to even delete the test cases whilst giving the same exception !!

I am using a console producer to talk to the kafka source(consumer) inside SAM but that is also not showing any activity.
Any help is much appreciated !!!

[#6]

@Geoffrey Shelton OkotHow many kafka brokers are required for a kafka source and sink in the dataflow that i created ??

[#7]

@Aditya Gary

All depends on the resilence your application is built to meet. The basic config is one broker. As a rule of thumb just like for the zookeeper ensemble to avoid the split brain decision you will need at least 3 brokers A three(3) node {zk or Kafka broker} cluster can survive the loss of 1 node see attached screenshot.

I have set up a 3 node HDF cluster (3 zookeeper & 3 brokers ) using virtual box each with 6 GB of RAM for test purposes.