Support Questions
Find answers, ask questions, and share your expertise

How to download profile.out heap dump (hprof)

Highlighted

How to download profile.out heap dump (hprof)

New Contributor

Hi

 

I run my java MR job with parameters:

 

 

-Dmapreduce.task.profile=true \
-Dmapreduce.task.profile.maps=0 \
-Dmapreduce.task.profile.params="-agentlib:hprof=cpu=samples,heap=sites,force=n,thread=y,format=b,file=%s" \

Looks good, in webUI i see 

 

 

Log Type: profile.out
Log Upload Time: Wed Nov 09 17:53:39 +0300 2016
Log Length: 2184156
Showing 4096 bytes of 2184156 total. Click here for the full log.
����(��9�����( A/�}�	��'��'���	��'��'�����'x�'x����4'pw;0����5'pw;0����%p�P1��<$�'' 9�A�C$�'' 9�Q�#��#��� ��" /�-�8��!��!������!��!���{�� �J`S�	�� @A@�"�� (�<�2
�#�^  �!�6 `0�!�n `0�!�s `0�!�d `0�!�i `0
�#��  �!ȯ  �!ȴ  


 

 

It looks file created, but i can't download it. I want to open it in visualVM.

 

When i try 

 

 

hadoop fs -ls /var/log/hadoop-yarn/apps/a.pilipenko/logs/application_1478695140324_0107/
Found 2 items
-rw-r-----   2 a.pilipenko hadoop    2189451 2016-11-09 17:53 /var/log/hadoop-yarn/apps/a.pilipenko/logs/application_1478695140324_0107/srv108_57010
-rw-r-----   2 a.pilipenko hadoop      52479 2016-11-09 17:53 /var/log/hadoop-yarn/apps/a.pilipenko/logs/application_1478695140324_0107/srv37_33143

 

It's 2 logs - from 1 mapper and from AM, no profile dump, on local server directories there is not such file.

 

But when i try

yarn logs -applicationId application_1478695140324_0107 -nodeAddress srv108 -containerId container_1478695140324_0107_01_000008

It shows

 

LogType:profile.out
Log Upload Time:Wed Nov 09 17:53:39 +0300 2016
LogLength:2184156
Log Contents:
JAVA PROFILE 1.0.1XI?*?,8G??java.lang.ExceptionG???P??G??java.util.WeakHashMap$EntryG???P??G?!?java.util.WeakHashMap$Entry[]G???P??G??java.lang.invoke.MethodTypeG???P??G??java.lang.CloneableG???P??G??java.util.SetG??1P??G??java.lang.System$2G???P??G?java.io.PrintStreamG????G??	java.lang.ThreadLocalG??(P	??	G?"?
java.lang.ref.PhantomReferenceG???P
??
G?0?
    java.util.Collections$SynchronizedCollectionG??]P
                                                     ??
                                                       G?9?
                                                           java.security.ProtectionDomain$JavaSecurityAccessImplG???P
                                                                                                                     ??
G??java.lang.LinkageErrorG???P??G??sun.misc.CleanerG???P??G?java.security.PrincipalG???P??G??java.security.Principal[]G???P??G?6?sun.reflect.generics.repository.AbstractRepositoryG??DP??G?%?java.lang.SystemClassLoaderActionG???P??G??java.lang.reflect.ModifierG??HP??G??java.util.Hashtable$EntryG????G??java.util.Hashtable$Entry[]G???P??G?'?java.lang.reflect.InvocationHandlerG??sP??G?)?java.lang.ClassLoader$ParallelLoadersG???P??G?#?java.nio.charset.CharsetEncoderG???P??G??java.util.Map$EntryG??P??G??ava.util.Map$Entry[]G???P???java.util.MapG???P??G??java.util.ComparatorG??P??G?"?java.security.cert.CertificateG??

and so on. It looks like my binary hprof file.

But i can't download it separate.

Any ideas?

 

 

 

 

2 REPLIES 2

Re: How to download profile.out heap dump (hprof)

Rising Star

 

 

Dear apilipenko,

 

Here is my custom awk code to do this:

 

yarn logs -applicationId application_1478695140324_0107 -nodeAddress srv108 -containerId container_1478695140324_0107_01_000008 | awk 'BEGIN {a=0}; /^JAVA PROFILE/{a=1;print $0}; a==1 && !/^LogType/{print $0}; /^LogType/{a=0}' > profile.out

 

Gabor

Highlighted

Re: How to download profile.out heap dump (hprof)

New Contributor

Thank you!

 

I've tried script like this:

#!/usr/bin/perl
#

$p = 0;

while (<STDIN>) {
    if ($_ =~ /^JAVA PROFILE/) { $p = 1; }
    if ($_ =~ /^LogType\:stderr/) { $p = 0; }
    print $_ if $p;
}

and than

head --bytes -1 profile.hprof.1 > profile.hprof.2

or 

head --bytes -2 profile.hprof.1 > profile.hprof.3

if dump can't be open correct because of last bites.

 

Thank you one more time!

 

Don't have an account?