Support Questions
Find answers, ask questions, and share your expertise

Can YARN logs provide detailed debug info for spark jobs

Highlighted

Can YARN logs provide detailed debug info for spark jobs

New Contributor

Spark jobs that are INSERTing data into NoSql DB having performance issues . It is not clear how much time is spent in Spark processing Vs NoSql INSERTs. Application code does NOT have additional logging steps. However, Dev-team claims YARN has detailed logs that can provide execution time of INSERT statements. The question is : Can YARN produce such detailed application level debug logs (eg., execution time for an INSERT) , while the application itself is NOT logging anything other than the default level of logging available in the framework (like job start and end times).

Thanks

usav