Hello, I am using Zeppelin and Spark to run query data sitting on Amazon S3. The code itself is so simple which is something like querying Hive external table on S3 with SparkSQL. Here I have some difficulty with it and suspecting I am exceeding S3 rate limit as described here: http://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate-perf-considerations.html If so that could be confirmed by error message from S3 which says 'rate limit exceeded' with HTTP Status Code: 503. So I would like to enable logging in aws java sdk which is used in Spark. I put the line below in my log4j.properties of Spark according to the AWS document: http://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/java-dg-logging.html log4j.logger.com.amazonaws.request=DEBUG
I'm assuming logs can be found in container(executor)'s log so would be aggregated by YARN's log aggregation services, but it's not emitted in that log. Where can I find these logs? Or Am I going wrong way?
... View more
