Reply
Explorer
Posts: 44
Registered: ‎07-24-2014

Impala performance issue and bottleneck identification

Hi,

I was trying to understand what is the major bottleneck for one of the queries found taking 76 sec time.

As far as I see based on the following it is about network performance while coordinator fragment collecting data. Is my understanding correct?

Query Timeline
      Start execution: 39796
      Planning finished: 4467867
      Submit for admission: 4923118
      Completed admission: 4992759
      Rows available: 70564213398
      First row fetched: 70606365909
      Unregister query: 75603663472
  ImpalaServer
    - AsyncTotalTime: 0
    - ClientFetchWaitTimer: 60188808
    - InactiveTotalTime: 0
    - RowMaterializationTimer: 59937
    - TotalTime: 0
  Execution Profile 98469b2167af16ff:b1ff6a3acead989a
    Per Node Peak Memory Usage: lrdna0ooxbd86.bankofamerica.com:22000(2.56 GB) lrdna0ooxbd19.bankofamerica.com:22000(789.99 MB) lrdna0ooxbd87.bankofamerica.com:22000(2.56 GB) lrdna0ooxbd22.bankofamerica.com:22000(2.54 GB) lrdna0opxbd01.bankofamerica.com:22000(40.02 KB) lrdna0ooxbd58.bankofamerica.com:22000(773.76 MB) lrdna0ooxbd23.bankofamerica.com:22000(2.72 GB) lrdna0ooxbd27.bankofamerica.com:22000(2.72 GB) lrdna0ooxbd80.bankofamerica.com:22000(2.56 GB) lrdna0ooxbd10.bankofamerica.com:22000(754.44 MB) lrdna0ooxbd35.bankofamerica.com:22000(100.27 MB) lrdna0ooxbd46.bankofamerica.com:22000(794.15 MB) lrdna0ooxbd09.bankofamerica.com:22000(30.04 MB) lrdna0ooxbd79.bankofamerica.com:22000(2.56 GB) lrdna0ooxbd44.bankofamerica.com:22000(2.52 GB) lrdna0ooxbd05.bankofamerica.com:22000(705.09 MB) lrdna0ooxbd81.bankofamerica.com:22000(1.52 GB) lrdna0ooxbd54.bankofamerica.com:22000(2.55 GB) lrdna0ooxbd45.bankofamerica.com:22000(2.85 GB) lrdna0ooxbd04.bankofamerica.com:22000(99.81 MB) lrdna0ooxbd03.bankofamerica.com:22000(88.17 MB)
    - AsyncTotalTime: 0
    - FinalizationTimer: 0
    - InactiveTotalTime: 0
    - TotalTime: 75537756243
    Coordinator Fragment F02
      - AsyncTotalTime: 0
      - AverageThreadTokens: 0.0
      - InactiveTotalTime: 0
      - PeakMemoryUsage: 40976
      - PrepareTime: 28765
      - RowsProduced: 8
      - TotalCpuTime: 314949972
      - TotalNetworkReceiveTime: 75258519918
      - TotalNetworkSendTime: 0
      - TotalStorageWaitTime: 0
      - TotalTime: 75258590005
      CodeGen
        - AsyncTotalTime: 0
        - CodegenTime: 0
        - CompileTime: 50871943
        - InactiveTotalTime: 0
        - LoadTime: 25656515
        - ModuleFileSize: 369500
        - TotalTime: 77381852
      EXCHANGE_NODE (id=5)
        - AsyncTotalTime: 0
        - BytesReceived: 347
        - ConvertRowBatchTime: 14984
        - DeserializeRowBatchTimer: 239645
        - FirstBatchArrivalWaitTime: 70286281742
        - InactiveTotalTime: 75258438855
        - PeakMemoryUsage: 0
        - RowsReturned: 8
        - RowsReturnedRate: 0
        - SendersBlockedTimer: 0
        - SendersBlockedTotalTimer(*): 0
        - TotalTime: 75258579148
   

New Contributor
Posts: 2
Registered: ‎03-15-2016

Re: Impala performance issue and bottleneck identification

Could you please attche the FULL  profile file.   Since a slow single node  may be the role.

Announcements