How to download profile.out heap dump (hprof)

[#1]

Hi

I run my java MR job with parameters:

-Dmapreduce.task.profile=true \
-Dmapreduce.task.profile.maps=0 \
-Dmapreduce.task.profile.params="-agentlib:hprof=cpu=samples,heap=sites,force=n,thread=y,format=b,file=%s" \

Looks good, in webUI i see 

Log Type: profile.out
Log Upload Time: Wed Nov 09 17:53:39 +0300 2016
Log Length: 2184156
Showing 4096 bytes of 2184156 total. Click here for the full log.
����(��9��
���( 
A/�
}
�	��'��'��
�	��'��'��
���'x
�'x
�
���4'pw;0�
���5'pw;0�
���%p�P
1
��<$�
'' 
9
�A�C$�
'' 
9
�Q�#��#��
� ��" /�-
�8��!��!��
����!��!��
�{�� �
J`S
�	�� @A@
�"�� (�<�
2

�#�^ 
 

�!�6 
`0
�!�n 
`0
�!�s 
`0
�!�d 
`0
�!�i 
`0

�#�� 
 

�!ȯ 
 

�!ȴ 
 

It looks file created, but i can't download it. I want to open it in visualVM.

When i try 

hadoop fs -ls /var/log/hadoop-yarn/apps/a.pilipenko/logs/application_1478695140324_0107/
Found 2 items
-rw-r-----   2 a.pilipenko hadoop    2189451 2016-11-09 17:53 /var/log/hadoop-yarn/apps/a.pilipenko/logs/application_1478695140324_0107/srv108_57010
-rw-r-----   2 a.pilipenko hadoop      52479 2016-11-09 17:53 /var/log/hadoop-yarn/apps/a.pilipenko/logs/application_1478695140324_0107/srv37_33143

It's 2 logs - from 1 mapper and from AM, no profile dump, on local server directories there is not such file.

But when i try

yarn logs -applicationId application_1478695140324_0107 -nodeAddress srv108 -containerId container_1478695140324_0107_01_000008

It shows

LogType:profile.out
Log Upload Time:Wed Nov 09 17:53:39 +0300 2016
LogLength:2184156
Log Contents:
JAVA PROFILE 1.0.1XI?*?,8G??java.lang.ExceptionG???P??G??java.util.WeakHashMap$EntryG???P??G?!?java.util.WeakHashMap$Entry[]G???P??G??java.lang.invoke.MethodTypeG???P??G??java.lang.CloneableG???P??G??java.util.SetG??1P??G??java.lang.System$2G???P??G?java.io.PrintStreamG????G??	java.lang.ThreadLocalG??(P	??	G?"?
java.lang.ref.PhantomReferenceG???P
??
G?0?
    java.util.Collections$SynchronizedCollectionG??]P
                                                     ??
                                                       G?9?
                                                           java.security.ProtectionDomain$JavaSecurityAccessImplG???P
                                                                                                                     ??
G??java.lang.LinkageErrorG???P??G??sun.misc.CleanerG???P??G?java.security.PrincipalG???P??G??java.security.Principal[]G???P??G?6?sun.reflect.generics.repository.AbstractRepositoryG??DP??G?%?java.lang.SystemClassLoaderActionG???P??G??java.lang.reflect.ModifierG??HP??G??java.util.Hashtable$EntryG????G??java.util.Hashtable$Entry[]G???P??G?'?java.lang.reflect.InvocationHandlerG??sP??G?)?java.lang.ClassLoader$ParallelLoadersG???P??G?#?java.nio.charset.CharsetEncoderG???P??G??java.util.Map$EntryG??P??G??ava.util.Map$Entry[]G???P???java.util.MapG???P??G??java.util.ComparatorG??P??G?"?java.security.cert.CertificateG??

and so on. It looks like my binary hprof file.

But i can't download it separate.

Any ideas?

[#2]

Dear apilipenko,

Here is my custom awk code to do this:

yarn logs -applicationId application_1478695140324_0107 -nodeAddress srv108 -containerId container_1478695140324_0107_01_000008 | awk 'BEGIN {a=0}; /^JAVA PROFILE/{a=1;print $0}; a==1 && !/^LogType/{print $0}; /^LogType/{a=0}' > profile.out

Gabor

[#3]

Thank you!

I've tried script like this:

#!/usr/bin/perl
#

$p = 0;

while (<STDIN>) {
    if ($_ =~ /^JAVA PROFILE/) { $p = 1; }
    if ($_ =~ /^LogType\:stderr/) { $p = 0; }
    print $_ if $p;
}

and than

head --bytes -1 profile.hprof.1 > profile.hprof.2

or 

head --bytes -2 profile.hprof.1 > profile.hprof.3

if dump can't be open correct because of last bites.

Thank you one more time!