question Re: IMPALA Profile Log Format in Support Questions

IMPALA Profile Log Format

VenkyJagannath — Fri, 16 Sep 2022 10:11:42 GMT

Hello fellow members,

I see that there is a line entered into the "/var/log/impalad/profiles/" log folder.

I see the format of the log is kind of cryptic. The first long number looks like it is an epoch timestamp. The second string looks lilke a base64 encoded string, but it does not represent anything after I decode it. I read the documentation "http://www.cloudera.com/documentation/enterprise/5-5-x/topics/impala_logging.html"

It looks like these logs are compressed with zlib? I checked the actual bytes - but none of it makes it look like a zlib compressed string. Can anyone help me with this? I tried to go through the code in C++ and python, but I did not get any clue as to how it is formatted.

Here is a sample line:

1458322151442 24496831f65ac754:2ba8e34b4c441c92 eJytVM9T00AUDj9GaoFiHcF2gJn14sAMZJLdTdJ28MDQMHRExAbw4mW7u4Ed0wQ2GxCvXtQ/wBn/Cw8ePPoHON714p+hR7etQOkAJ0/Je5t97/u+972UV0pLzzIuT8GCYI8gxlW3guzQdQj1HFyDLVLhCLcwxdimVbg4PVpeKRVW09OY7iSKRDuizafzM4ZRKjZiQpU45gP525fjmaHirFHOzeqTsSBrt4k8nTb+Q83i28JaEsecKs7AbsqlMb6WJJKJmKhEPmRcP1vIbjFi2abiqTKVIGSZSh6aidyvQWhZVq5e3wQ7p4d8ZDfw83UekixSoN4aY73XQp1HfJ+ct8j5MQMdGPPQst1lCy3bFWCjGqzWbNvEGCKnU3ZuQ1MIuDzmEoJtmaiEJhHY4zIVSTy85xYa7UMSkbNMILohA8e9GEATmtYyZQcOaPqb/mrgg4VWJiIGYBgyDxEMHYfykDnQRq7OcZuGLeRhF4WuF3oOXZza4uokkS/BKmOSp+ld2zIhQiaGpu3WMLLtynjPB4HS/HLrja1GsOHXJy6SWTr89HG+F3c10mLlA12rA7FRf1DB2IOug0KEEbGqqOZVsEdbFrMoI8yusomzjzu3xzcae37gN/f8JpwMjqJe3zaP1WSWcnAYkThMaJbmdV6qG0S2Kp7XEXm0O5Hyn2Ifpksdb5xCX5vzofZx7dfmkiQDo+uCGDDigGkGJ9Fns8s69Pv33Jh6d5ZKRvlWOWeUCv/waaiRiHl5/cuPkZ+/vs59+vh7/vvnN6C8MdVjxV9xmikNr7itZY1FvA9CEYv0gLM7TX6U6W24SOzGku+LVHEJjjr19bpN9Ej2tNPb+uLqbb23FgmNe50revCcCNU5kfro24d3w9fv8v1mcvJEU5aCROI16cA8u3j932PovbFwZbvrqhnGX43whcc=

Thanks for your help in advance.

Venky

Re: IMPALA Profile Log Format

Bharathv — Wed, 30 Mar 2016 14:54:08 GMT

Hey Venky,

Its a zlib compressed TRuntimeProfileTree object. You might want to use the thrift bindings to decode it. HTH.

- Bharath

Re: IMPALA Profile Log Format

VenkyJagannath — Wed, 30 Mar 2016 16:20:49 GMT

Thanks for the tip Bharath. I tried to write a small program to get this string out and read it using thrift bindings. I still get the zip header exception. Any ideas on what I am doing wrong?

package org.tiaa.impala.log;

import java.io.BufferedInputStream;
import java.io.ByteArrayInputStream;

import org.apache.commons.codec.binary.Base64;
import org.apache.thrift.protocol.TBinaryProtocol;
import org.apache.thrift.transport.TIOStreamTransport;
import org.apache.thrift.transport.TZlibTransport;

import com.cloudera.impala.thrift.TRuntimeProfileTree;

public class ImpalaLogParser {
private static String dataCompressed = "4c4d52afea4a40a1:802034563d1cfc93 eJztPH1sG9d9pPwlU7JsOZFK1ln7VmOxVEj03fFL1CwPFElZWiRKISk7aTYYp+OjdDB5VO6OspUVmLplmZs/PGdxDDfdBgMrjGB1M89eh6TbHK/IsHTtVs9og67dMmAIPC8ZAi/AUq8w1v3euzvy7ngnUV7q2AsPAfXuff2+v947J3B7t3/g0SqWl1GfWBgJC+FChOOLmA/zYYZnh4cYjgmFI9FQgRWKQjzU37M5sM/flVCWJSFfUflSXizjHl+vx+PvnpB4QRWXsK1/q/W919u9yxNo3wUjW3LVcpmXl3s8H8KePce7khVJwoKKC2hWwbKnI1mpyAVR4tWK/HABw9+5EDtX4Bk2qGJFDaoizw8KMi4GK/L8MMcxDONL4SJfLakoNbeloDW7UriE5/naru1pqYAI5J/jGDY6yIQGuThiQ8Ph2HA4HIzE4lwoDDv1phVVLNNlM1geHK8oKprC5a2hKDfExLh4NOAw4WCyImPFy3Wkj2FBZ87NLb7pRSwTIpD27CZzFWgkluYpJtCc4o/pzd3ZylEYhN2DRnsG80cIcL23/prCKi+WkO3xDd7dx8dEhxMHDmTTBxL5tIEEC/+FmOBQKFZWLE19sPYw8BoNMgx6ZNToGmTJO7yOTWQSkxOfSyPL42Miw+nHkuOJzIH6CN1yKMhFmKpiazbAs7dq8GYzM4lsfiI/MZ1Jp0zwOGf6IrFgNBym9NWbTvRxFvpYhrxOjSKXx8eEHeEhxAXDEU4DYmqaadLaoSAbqgNYG17IhZ9gD0NhjYmmZiO8xl7tdTyRG+/L5TL9NnisCz/jIDRNX+pNJ3hxIH499DHDuWQig8ZTYzkzfXFgYmyIAjE1HeExdXihUNSAt1jipWJFqCpBLMmVUqmMJTDsh8bB4eWwvIRlDs3IFbUiVEroIJYVsSK1HYx2TZQX+RJv9ORE+lpAS9o7iBb0ZVAoLERQNj2ZTuTSqG+uKpYKiCsWC7EQH+YiEQEXwdOz4I6KRQy+fS4UC0dDxWisGIsI/dszWD1akY+gRKEAHknZCTziQqFgmAuy0eEwx3HxjTOA/EudDc7C5+DXsvjJqihjSt4wcT0VeXkkFCZ8ADZoXm+E8/nGwFJmJoHTY9nEgal0Jo+esJjUr/qAn2Zn8YRh4WTk8whVqupiVR1GQqUqqcNlLM9jqj/a6ALxmyMsAnc6SNqDZVweqUr8EjhBfq6EtVlqdbGEB8UCYITkytFBRXwKjwyNIoGnwaQkqssjDJlKkDF5kkZUUVMgUbMggT2snT01CyHwUol8IpfPphNTKDeReQS4o88bAcYOIAPTESYygBr5anJSDsxci43UiKZG74CFYTvceblSXURzy8MIYK4fJFsHyUVcYJoclpWFbiJjRu2SWhUMiIqxiyqbyKSmp9aSE2uWU2gAWbAzO767xiuz83vC0WMNoDpt8Czysiqq4Ilg/71hVBRLmECiUMLEj0zpzFyUcUEUwFWAU+gj3DqcmZ0aTWfRRAb17WEZJsQxewaQ1mJrLa7WCtVakVorWmvF9vT3D6Dk5ARw1th5BO1hmKFQhKykECfTB9OTh9OZ7PTkJBHB4Vw+kZ/NkYnJShn4RBxZukYrLDPNzT8+kz6cSUylNYynpZIoYQJ/hgfu08ZCBXr6LatSxHPtBwAkgWRCLLencXhffXgIhkG4KDOd18DEjIcAiBvPnn5dQ4lbQYrKE1fLxphQJEaNUEEqyZm1SRBQqmXJmMWXSquovhavphoMgKnrDhth7cpjDwsdWpGRA4i4Hbz2RG48neqsd1aVtulHfNp7fnkRb3p0Np19vJNEDsjV0UylUurUM/LBRXjx5SAskWg3kfp5PhKOzeGQAHGMZ6NFZjgSY/lwkWOG2EgxMhcXOo3JZOeO8YmD6Vw6ezCd5bblnixpOBHh/nijryH45yCCJvPIt7VhJDk9m8n3pSZy+YkMzKBpSgLyg7ooE5PpbJ5Oc1g+lp2eoio4Np2czQXrq3IOkw+Np7NpVPPYhydSICWwOavZiJLJbGpWUzOams3UTKZmMXWDcaKUALKY0YhhRC6T78iyXPa6E3Oz7+OydZ/d6kbT+UPpdMZkmXRe3RKBPY47udmn2Tx/sWHpVqPHByooq25VZSgYG+LCcVKfbqQlaOBWj0n9Lcq9au5oAlOrYk0mZzZRi2XaEk6KhK3YtlXJ9vzRVFdbTc5co9OM0qVudq+WLS7CXDjv8gQG/DvInhKWKa1EXQLFp3943PvOiZPeC5f/znvmCze9x194tu29C99uCyx0J8B9LSuiAjFLEpUFXHgoDYnrEl/CkoCRUOIVBSvgPMuQF+GCPydK8+BrpUoB01QeCTImKPpToqLK4hyZZBnomQQM+HkMllqs1PbppjjCXjWwHn+XLhYD66Xzxze+91vf3nD19o0V7yt/+sKfbfjbr7/zX5ve+OC9Dzbd+MK7P94UOLZdky4GDlRJ9G3c9tNZzBeWIQ4Qxw9TISOvqBgVZX6exnF/1tahzcOFLnqOUEtau8dEGRgO7h8VsSrAzjtmJRnPA9XA6ScJ5h6Pv1PTGk0ZezyBX3E+4nkwWRIB1hjZ6BAvqmREhqHzF6973Q+APgEYTQFLZRECzlM8oddY+Ad9rmdO3mc9fY7w3Lbz+ENpg5/EnEgug9Y8JOvxBSZdzrPGaIS0ItzsOdcP/vu3399EaWk77h/TZaSLsgTIS4KIlTmt9EGQ74O6QoOBio2LR4YkSNPKoqR1MPGhKO3gj9lmYF6fwsZjHOlR1EIBL5E+8D902SfBAlGGKL1xkgTlHJg9KPYT5mO2kOsxWx8bCYZIRdyPmjuX62P6UaDPlehVMCIyd2C7x+83eR5U2xgKpZ7Nge84S++BBGgy7JlfAHMu5CtHsKT0dPauaM/NX3eX43aCl4YWxapnc+/KlQ2wIyBOfJk+bIzBio4ZGUMijfU9Tr75z58HkyJ2CGpYqArgTDy9bdBFISUXq/rE19+9dHYj2Aft1r1wFguY4KTN+J+3v38NLOtB84wclgomk6RDOWAMoGPYiINC3vzqP3y3zWxcZ5w4ZKfdiWYztRYiLeS5EeVIiiMR/QP+DosYem5sCTzsWbkCDOnQkNYGPGQA+Ovxt4+WKsKRqXniwm5vd1aMXjrnkCyC/k5XVdBOCTRrHiQEg9vooJLUQwDt69L6gIZloWR0do5Wi0UMNdGMCHqldy3DlmRj0HRNMdxUrHOKP0Z3zUFGTvRrZaXdY1A7KZZFlSrdpevf2eCoj0S/6ZYaFlaxayNpSZCXFw0b0h0xHZmQVDwPWC4nF7BwpD66k46SsENRW/NSAHToDRdmWtloY6CFdRamWdhi5kaDWjoQ70S2G8EOpHr8W5LglA6A6DyB33NWnQ4yYx4bHD330mdpX3kRIo3Rt/LsRne5t09WeMN2X/8XEHnXFJhOCY/BBroq3Lj1JfAUO6aBhrLJCcKCs2e+u9Dga0597Y9+zSahld/4nX/3mqW0YMbbjG8NHRsaDdDNQEGNjHMNUt1kplNpeh0V7e9pczW6raPkTFN/f+cBIPwAVrNYgTRTMfF3nFcWRqvCEazqNuWaUkzy8jzEnxnj+AL8lAAxQV8FelQbmcRLuKT1b89Uy1lcP/LQZ9emWu3eKQ6sgGy6ic8zbVNzCVq/WpUlw+HvMHdlYeuetl6iHjmQAFjCjA0Tixj/4+1XicPu7N7lPe4jic00VevtuiyhGiPpXSHgNw1Sm6zz2sZkM3vdGNjIOjvTGtllZ1QjgyysaWBKIzsgrBkHa3UFi/SDZbpE+23UjehhpgCSQuAHoeqBCKcCuFEeEkh99gcdkE+ksKLnj9g8TDK8k095/AGaNNPuhCyTosLkYt96+1vX3LLdixdovHZ22U0qCLiFB0hghAKOeidcMCWfu2wjBmy577P9Dt763J+8Cfj0+nbWor6VU048cmWOO1fs9K4ubyfqVqOr/6EG+dKI7wFqe/QkpmBOC9me9ruUFp5yzwuvvnd+e4O7Pn36lTPepnLDi/967vW2VXPDi9/7zQtt7rnhTy7tbD47/Ps3T35rA40ZG4/vELQTH1JGQRjASoaUIuF4PMiG6K0ZKUTMr7QM0Ts4jv4Jwl8oBciwXpPQkgQeyDFrVS+SyXnyfrI9vdbbq2BB2978SrY3vZv2oz3bpGoZanSSeghY8bIdymJJVOkJtvIZvYoyiqcBqIvn6d9apRQYNy9ooN2OrBXafZFH/yzSYpKD34N58coKamXG90BmfPraM20NqfHtrz3/ieZS41Nf71xfanz5xJXnvQ3O9oeX3jrltQnp7a/86ErbzzI53pXiVT6ngnzLWkhDfQVFPWzkL+fd8mMi7BzJYGnu4salB3WDB19Tqc4vLFZVEvfbe79yeQcsmiaOqFSqD8LI9Rc/557K7swZgd6cIV17FOIqbCIW1bzMS0pZUyMtw3jmz7vsOcY12Mk/KxHXSc6PIcjrWYMuN2Tm97E6qY7ENBJhR94JaUdsXXFyK2G4pkuY09vv2xKGuO67VMKc+90fvQFZxca6V/x/UZs4K0+YKs/OJpTnQgAkA+Rn8FFzPFl5o+veVyrk4kyeg4C8BcKW6fi8SR3zOOuYp0kdu3Dhjy+Cmm6dlRZr0J/Z6ultr5U8t7bXuW9n+0ekhQan/u/aWKfbsWYOradm9rjWzBd8q9fMnjVK5ltfpaXKOkqrdSnKh1YuX/8JwbN3831dLrsUxcxduiu5+vKXOtdXFL9+6YUXHYpiT2NRfPW1lT9c/cLE414QP7vBvR7+5nPf6LTpwvsvf/Omd9WSOBSKBMPRuFESm19pSax3xDiO/AnG6O1YUyXxL5PtWTYejJNPe+tlsb2LgLH1rac8jmnlMf3OCzbQq+T6Ky2W668fq5p53TXPW9f+ydtQ8zzzxXcGmqt5Tv6lb301z5kvn/uLxprnxInbDTXPm5ev/vQjqnlCzdY8q+Q0bjWPx6XicXZLZ2fdyh3PKtWOPUBcveVdvdbx3C+1Dtt0uvq997sc89Wzr963+eqFf3x160eUrz792hdvem356ouQr7Z1f+zy1R7ymfJh8sFyXS0Z4jBuf8pZLZEeNcYLRYUcUWnRA3IyoSrLWBKWIWmBaZ/Wp+UE+jmZy6ytembHk9T3pZuvtZEzUKOLODTyjUqSFxb0dLSrNjhZEfhSbVGg1q99kTUr4WOL9GM7bV1PbTy3UJHVpCgLVXqOqS3vTmHDcOu+/cbTf715FaOpf/2E8+QT37q/+sG7P/0l8Mcg8mTNG+TxMZU48kkszasLGlI+UIEk/a6Y6CikPjugIyUqR5SEINBVWvXvh24LG5Wc9pWZNuxoWlB9+T8FyleXUJY/SkRm8dInzv/NdQJXYxqBkeUl0F/NZNo1TeIJnBO/H1yPFe6s72V8uqoh22clZEJaAgZIQM4y6IYKTModFckHcjoKPTa6l00ODlnHqHwOgSNPmtzjqZMvQ8jptc4k31vq41f+7a8OePwPW8cPuqP0SSPv1blJ2VsT/Cunv+E1Drh1xlsiosUHXf7PM48BjVqV0wYeZ23LWsum6tbkYkQ281nFahztpdFMVlNyk3o3KLarStuVeQ0dbtDcms6u7hMd9LNpzVxT75pUJzdFctKfNWEOufojRwtysQgwccsNbtuH90+XH5vQa481vwaNoD7yTylGmvsysr91zXzH18z39UeIIJHnN9zFjxBbt62t29bWbWvrtrV123rvHjTcS7etH8cPRluXsvfHpez6dPNjcxZ2f97dnr5Ex1r3t3frc+dOy4XuR1IjR9dZI7dune/s1tl7PE5OZ5B+00r+7w+obx85GSpjJBb2D+/brY0p+/fu0yaV6OGXsr9/GzvM7jUublHAf+c73cdF+tnvP+depLeud1vXu63r3db17l283m1lvq1b4NYtcOsW+J67Bd5odky7DMeENcc0jBjyPxdElSJiO4kCobGKXIYUsnsmkX10Np3fm8skZmYeH46i3TTLJHCRRggymRvSHNRBZpj5BcSSH478hMhPmPxEyE+U/MTIzxD5idPJ2hK6hqWLWLqKpctYuu5DS5XH73ynZsi3cNDM9dYtfOsW/h64he9/yBIRjXMHz/8C+1Y8fw==";

public static void main(String[] args) throws Exception {
byte[] decoded = Base64.decodeBase64(dataCompressed);
System.out.println(decoded);

ByteArrayInputStream bais = new ByteArrayInputStream(decoded);
BufferedInputStream bufferedIn = new BufferedInputStream(bais, 2048);

TBinaryProtocol binaryIn = new TBinaryProtocol(new TZlibTransport(
new TIOStreamTransport(bufferedIn)));

System.out.println("Hello");
TRuntimeProfileTree aTree = new TRuntimeProfileTree();
aTree.read(binaryIn);

bais.close();
}
}

Hello
Exception in thread "main" org.apache.thrift.transport.TTransportException: java.util.zip.ZipException: incorrect header check
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at org.apache.thrift.protocol.TBinaryProtocol.readByte(TBinaryProtocol.java:285)
at org.apache.thrift.protocol.TBinaryProtocol.readFieldBegin(TBinaryProtocol.java:243)
at com.cloudera.impala.thrift.TRuntimeProfileTree$TRuntimeProfileTreeStandardScheme.read(TRuntimeProfileTree.java:349)
at com.cloudera.impala.thrift.TRuntimeProfileTree$TRuntimeProfileTreeStandardScheme.read(TRuntimeProfileTree.java:1)
at com.cloudera.impala.thrift.TRuntimeProfileTree.read(TRuntimeProfileTree.java:289)
at org.tiaa.impala.log.ImpalaLogParser.main(ImpalaLogParser.java:28)
Caused by: java.util.zip.ZipException: incorrect header check
at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:164)
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
... 8 more

Re: IMPALA Profile Log Format

Bharathv — Thu, 31 Mar 2016 09:46:27 GMT

Why TZlibTransport? Can try java.util.zip.InflaterInputStream, something like

InflaterInputStream in =
          new InflaterInputStream(new ByteArrayInputStream(compressedProfile));
      ByteArrayOutputStream out = new ByteArrayOutputStream();
      IOUtils.copy(in, out);
      out.toByteArray();

Re: IMPALA Profile Log Format

VenkyJagannath — Thu, 31 Mar 2016 13:51:53 GMT

Yes I had tried that first.

Exception in thread "main" java.util.zip.ZipException: incorrect header check
at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:164)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1025)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:999)
at org.tiaa.impala.log.ImpalaLogParser.testTRuntimeProfileTreeSerialization(ImpalaLogParser.java:33)
at org.tiaa.impala.log.ImpalaLogParser.main(ImpalaLogParser.java:25)

That is what led me to this website:

http://www.cloudera.com/documentation/enterprise/5-5-x/topics/impala_logging.html

Formerly, the logs contained the query profile for each query, showing low-level details of how the work is distributed among nodes and how intermediate and final results are transmitted across the network. To save space, those query profiles are now stored in zlib-compressed files in/var/log/impala/profiles. You can access them through the Impala web user interface. For example, at http://impalad-node-hostname:25000/queries, each query is followed by a Profile link leading to a page showing extensive analytical data for the query execution.

I looked through the impala www code - to see how they deserialize this, but could not figure out.

Thanks once again for your help!

Venky

Re: IMPALA Profile Log Format

Bharathv — Tue, 05 Apr 2016 17:42:38 GMT

Sorry I missed this. Impala prepends the query hash to the base64 string while logging to the file. In your case that is 4c4d52afea4a40a1:802034563d1cfc93. Just remove it from dataCompressed string and it should work.

Re: IMPALA Profile Log Format

VenkyJagannath — Tue, 05 Apr 2016 21:29:39 GMT

Thanks once again!

I was able to remove the query ID from the first part, decode from base64 and get the bytes. I saved it into a file - and it gives me data that I can see. However the thrift binding lookup fails - it says it can't deserialize based on bytes.

Exception in thread "main" org.apache.thrift.protocol.TProtocolException: Required field 'nodes' was not present! Struct: TRuntimeProfileTree(nodes:null)
at com.cloudera.impala.thrift.TRuntimeProfileTree.validate(TRuntimeProfileTree.java:315)
at com.cloudera.impala.thrift.TRuntimeProfileTree$TRuntimeProfileTreeStandardScheme.read(TRuntimeProfileTree.java:381)
at com.cloudera.impala.thrift.TRuntimeProfileTree$TRuntimeProfileTreeStandardScheme.read(TRuntimeProfileTree.java:1)
at com.cloudera.impala.thrift.TRuntimeProfileTree.read(TRuntimeProfileTree.java:289)
at org.apache.thrift.TDeserializer.deserialize(TDeserializer.java:81)
at org.apache.thrift.TDeserializer.deserialize(TDeserializer.java:67)
at org.tiaa.impala.log.ImpalaLogParser.testTRuntimeProfileTreeSerialization(ImpalaLogParser.java:29)
at org.tiaa.impala.log.ImpalaLogParser.main(ImpalaLogParser.java:17)

I saved the bytes off to a file - and I can get some information, but it is all encoded with funky characters,

ü#,Query (id=4c4d52afea4a40a1:802034563d1cfc93)<AsyncTotalTime
InactiveTotalTime
TotalTime
Summary <AsyncTotalTime
InactiveTotalTime
TotalTime
ˆConnected User Coordinator&denatb31bda01.test.tiaa-cref.org:22000
Default DbdefaultDelegated User End Time2016-03-29 13:47:44.579234000Estimated Per-Host Mem 362807296Estimated Per-Host VCores2ExecSummaryð
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
---------------------------------------------------------------------------------------------------------------
06:AGGREGATE 1 130.837ms 130.837ms 1 0 16.00 KB -1.00 B FINALIZE
05:EXCHANGE 1 8.250us 8.250us 1 0 0 -1.00 B UNPARTITIONED
02:AGGREGATE 1 157.644ms 157.644ms 1 0 12.00 KB 10.00 MB
04:AGGREGATE 1 2.452ms 2.452ms 0 0 3.13 MB 10.00 MB
03:EXCHANGE 1 4.584us 4.584us 0 0 0 0 HASH(SSN)
01:AGGREGATE 1 159.257ms 159.257ms 0 0 9.45 MB 10.00 MB
00:SCAN HDFS 1 92.478ms 92.478ms 0 0 9.05 MB 336.00 MB planfocus.enrollments HiveServer2 Protocol VersionV6Impala VersionSimpalad version 2.2.0-cdh5 RELEASE (build 2ffd73a4255cefd521362ffe1cfb37463f67f75c)Network Address10.233.42.16:42229Plan¨
----------------

I used this code - and it fails.

private static void testTRuntimeProfileTreeSerialization() throws Exception {
byte[] decodedBytes = Base64.decodeBase64(dataCompressed);
InflaterInputStream in = new InflaterInputStream(
new ByteArrayInputStream(decodedBytes));
ByteArrayOutputStream out = new ByteArrayOutputStream();
IOUtils.copy(in, out);
byte[] uncompressedBytes = out.toByteArray();
TRuntimeProfileTree aTree = new TRuntimeProfileTree();
TDeserializer dez = new TDeserializer();
dez.deserialize(aTree, uncompressedBytes);
}

Anything else going on in the serialization process???

Thanks so much for your insights.

Venky

Re: IMPALA Profile Log Format

Bharathv — Wed, 06 Apr 2016 05:45:24 GMT

Yep, try using the TCompactProtocol to deserialize (Default is TBinaryProtocol). Initialize the deserializer like,

new TDeserializer(TCompactProtocol.Factory())

Re: IMPALA Profile Log Format

VenkyJagannath — Wed, 06 Apr 2016 13:35:25 GMT

Yes that was it!

TCompactProtocol was the thing to use!

Thanks so much for your help

Venky