Support Questions
Find answers, ask questions, and share your expertise
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

What tool to use for IoT Ingestion (AMQP, MQTT)

What tool to use for IoT Ingestion (AMQP, MQTT)

What is best tool for sensor data ingestion to Cloudera Data Hub? Need to support AMQP and MQTT protocols, security access keys, device registry and two way communication.


Nifi+Kafka? Or does Cloudera have own tool?


Re: What tool to use for IoT Ingestion (AMQP, MQTT)

New Contributor

We Use Kafka and Flume, together, to collect data from third party servers into CDH cluster and process with spark streaming, to obtain parquet files persisted in Hadoop as Apache Hive Tables.


 Is rudimentar bot works 95% of the time. Some use cases, is better just to run batch and extract data from some ftp / NFS file systems, S3, and other, instead of using MQ systems.


Re: What tool to use for IoT Ingestion (AMQP, MQTT)

Expert Contributor

Since the question was asked, the situation has changed. As soon as Hortonworks and Cloudera merged, NiFi became supported by Cloudera.


Shortly after the integrations with CDH were also completed, so that NiFi is now a fully supported and integrated component.


Hence the question already contains the answer: Please look into NiFi+Kafka for solving this usecase.


You may also be interested in MiNiFi, for pushing/evaluating the data on the edge.

If this answer helped, please mark it as 'solved' and/or if it is valuable for future readers please apply 'kudos'.
Don't have an account?
Coming from Hortonworks? Activate your account here