- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Spark and llap jobs process
- Labels:
-
Apache Hive
-
Apache Spark
-
Apache YARN
Created 02-14-2018 02:11 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I'm running spark jobs in server-1 and if i can run llap daemons on the same server is there any impact on running spark jobs.
Created 02-14-2018 04:13 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@kishore sanchina LLAP is a long-running service so it will preempt memory for the llap queue. The best practice is to dedicate nodes to LLAP workloads.
You can utilize the LLAPContext in Spark which will stream data from HDFS to the spark executor but this is more of a Hive process and not Spark which can incorporate some masking and filtering security features but you may see a 3x-4x performance degradation.
Created 02-14-2018 04:13 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@kishore sanchina LLAP is a long-running service so it will preempt memory for the llap queue. The best practice is to dedicate nodes to LLAP workloads.
You can utilize the LLAPContext in Spark which will stream data from HDFS to the spark executor but this is more of a Hive process and not Spark which can incorporate some masking and filtering security features but you may see a 3x-4x performance degradation.