Member since
06-02-2020
331
Posts
67
Kudos Received
49
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 4097 | 07-11-2024 01:55 AM | |
| 11362 | 07-09-2024 11:18 PM | |
| 8558 | 07-09-2024 04:26 AM | |
| 8574 | 07-09-2024 03:38 AM | |
| 7503 | 06-05-2024 02:03 AM |
04-29-2022
05:29 AM
Hi @JoeR Spark will support reading files with multiple file formats like parquet, orc, json, xml, avro,csv etc. I think there is no direct mechanism to read the data from the payload. If I found a different solution, I will share it with you.
... View more
04-05-2022
12:46 AM
1 Kudo
In this post, we will learn how to create a Kafka topic and produce and consume messages from a Kafka topic. After testing the basic producer and consumer example, we will test it with Spark using spark-examples.jar file.
Creating a Kafka topic:
# kafka bootstrap server
KAFKA_BROKERS="localhost:9092"
# kafka topic name
TOPIC_NAME="word_count_topic"
# group name
GROUP_NAME="spark-kafka-group"
# creating a topic
/opt/cloudera/parcels/CDH/lib/kafka/bin/kafka-topics.sh --create --topic ${TOPIC_NAME} --bootstrap-server ${KAFKA_BROKERS}
# describing a topic
/opt/cloudera/parcels/CDH/lib/kafka/bin/kafka-topics.sh --describe --topic ${TOPIC_NAME} --bootstrap-server ${KAFKA_BROKERS}
Producing messages to Kafka topic:
# producing kafka messages
/opt/cloudera/parcels/CDH/lib/kafka/bin/kafka-console-producer.sh --topic ${TOPIC_NAME} --broker-list ${KAFKA_BROKERS}
Consuming messages from Kafka topic:
# consuming kafka messages
/opt/cloudera/parcels/CDH/lib/kafka/bin/kafka-console-consumer.sh --bootstrap-server ${KAFKA_BROKERS} --group ${GROUP_NAME} --topic ${TOPIC_NAME} --from-beginning
Submitting the Spark KafkaWordCount example:
spark-submit \
--master yarn \
--deploy-mode client \
--packages org.apache.spark:spark-streaming-kafka-0-10_2.11:2.4.7.7.1.7.0-551 \
--repositories https://repository.cloudera.com/artifactory/cloudera-repos/ \
--class org.apache.spark.examples.streaming.DirectKafkaWordCount \
/opt/cloudera/parcels/CDH/lib/spark/examples/jars/spark-examples_*.jar ${KAFKA_BROKERS} ${GROUP_NAME} ${TOPIC_NAME}
... View more
Labels:
02-22-2022
10:35 PM
Hi @Rajeshhadoop I think it is the not right way to ask set of questions in single community article. Please create a new thread for any kind of questions.
... View more
02-08-2022
02:44 PM
1 Kudo
Looking at the serialized data, that seems like the Java binary serialization protocol. It seems to me that the producer is simply writing the HashMap java object directly to Kafka, rather than using a proper serializer (Avro, JSON, String, etc.) You should look into modifying your producer so that you can properly deserialize the data that you're reading from Kafka.
... View more
02-08-2022
04:01 AM
Hi @loridigia If cluster/application is not enabled dynamic allocation and if you set --conf spark.executor.instances=1 then it will launch only 1 executor. Apart from executor, you will see AM/driver in the Executor tab Spark UI.
... View more
12-07-2021
10:29 PM
1 Kudo
In this article, we will learn how to integrate Zeppelin JDBC (Phoenix) interpreter example.
1. Configuring the JDBC (Phoenix) interpreter: Login to Zeppelin UI -> Click on the user name (in my case, admin) at the right-hand corner. It will display a menu > click on Interpreter.
Click on + Create at the right-hand side of the screen.
It will display a popup menu. Enter Interpreter Name as jdbc and select Interpreter Group as jdbc. Then, it will populate Properties in table format.
Click on + button and add the Phoenix-related properties according to your cluster, and click on the Save button.
phoenix.driver
org.apache.phoenix.jdbc.PhoenixDriver
phoenix.url
jdbc:phoenix:localhost:2181:/hbase
phoenix.user
phoenix.password
2. Creating the Notebook:
Click Notebook dropdown menu in the top left-hand corner and select Create new note and enter Note Name as Phoenix_Test,and select Default Interpreter as jdbc. Finally, click on Create button.
3. Running the Phoenix queries using jdbc (Phoenix) interpreter in Notebook:
%jdbc(phoenix)
CREATE TABLE IF NOT EXISTS Employee (
id INTEGER PRIMARY KEY,
name VARCHAR(225),
salary FLOAT
)
%jdbc(phoenix)
UPSERT INTO Employee VALUES(1, 'Ranga Reddy', 24000)
%jdbc(phoenix)
UPSERT INTO Employee (id, name, salary) VALUES(2, 'Nishantha', 10000)
%jdbc(phoenix)
select * from Employee
4. Final Results:
Happy Learning.
... View more
Labels:
11-09-2021
10:48 PM
@EBH, Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.
... View more
10-28-2021
11:34 PM
@SimonBergerard, Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.
... View more
10-28-2021
12:05 AM
Hi @Marwn Please check the application logs to identify why application startup is taking X mins. Without providing application logs very difficult to provide.
... View more
10-21-2021
10:36 AM
@LegallyBind Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.
... View more