VMware 에서 실행되는 Windows VM이 시스템 행 현상이 있다고 하여 분석을 진행해 보았습니다.


먼저 !vm 명령을 싱행해서 메모리 상태를 확인해 보았는데 별다른 문제가 확인되지 않았습니다.

0: kd> !vm

Page File: \??\C:\pagefile.sys

  Current:   1048576 Kb  Free Space:    807844 Kb

  Minimum:   1048576 Kb  Maximum:     10485500 Kb

 

Physical Memory:          4194156 (   16776624 Kb)

Available Pages:          3479978 (   13919912 Kb)

ResAvail Pages:           4063051 (   16252204 Kb)

Locked IO Pages:                0 (          0 Kb)

Free System PTEs:      4294989313 (17179957252 Kb)

Modified Pages:             13432 (      53728 Kb)

Modified PF Pages:          12747 (      50988 Kb)

Modified No Write Pages:        0 (          0 Kb)

NonPagedPool    0:             33 (        132 Kb)

NonPagedPoolNx  0:          32181 (     128724 Kb)

NonPagedPool    1:              0 (          0 Kb)

NonPagedPoolNx  1:              0 (          0 Kb)

NonPagedPool Usage:           168 (        672 Kb)

NonPagedPoolNx Usage:       40172 (     160688 Kb)

NonPagedPool Max:      4294967296 (17179869184 Kb)

PagedPool  0:               79908 (     319632 Kb)

PagedPool  1:               51553 (     206212 Kb)

PagedPool  2:                   0 (          0 Kb)

PagedPool Usage:           131461 (     525844 Kb)

PagedPool Maximum:     4160749568 (16642998272 Kb)

Processor Commit:             917 (       3668 Kb)

Session Commit:             16179 (      64716 Kb)

Syspart SharedCommit 0

Shared Commit:              51347 (     205388 Kb)

Special Pool:                   0 (          0 Kb)

Kernel Stacks:               8889 (      35556 Kb)

Pages For MDLs:               342 (       1368 Kb)

Pages For AWE:                  0 (          0 Kb)

NonPagedPool Commit:        38599 (     154396 Kb)

PagedPool Commit:          131461 (     525844 Kb)

Driver Commit:               9397 (      37588 Kb)

Boot Commit:                50253 (     201012 Kb)

System PageTables:            997 (       3988 Kb)

VAD/PageTable Bitmaps:       6871 (      27484 Kb)

ProcessLockedFilePages:        12 (         48 Kb)

Pagefile Hash Pages:          206 (        824 Kb)

Sum System Commit:         315470 (    1261880 Kb)

Total Private:             473164 (    1892656 Kb)

Misc/Transient Commit:       4291 (      17164 Kb)

Committed pages:           792925 (    3171700 Kb)

Commit limit:             4456300 (   17825200 Kb)




Storage에는 Pending 된 IO가 있는지 확인을 해 보았지만 Pending 된 것이 확인되지 않았습니다.

0: kd> !storunit

STORPORT Units:

==================

Product                 SCSI ID  Object            Extension         Pnd Out Ct  State

--------------------------------------------------------------------------------------

VMware     Virtual di   0  0  0  ffffab06b03a4060  ffffab06b03a41b0    0   0  0  Working

VMware     Virtual di   0  1  0  ffffab06b0399060  ffffab06b03991b0    0   0  0  Working

 

0: kd> !storunit ffffab06b03a4060

   DO: ffffab06b03a4060   Ext: ffffab06b03a41b0   Adapter: ffffab06b03cb1a0   Working

   Vendor: VMware    Product: Virtual disk      SCSI ID: (0, 0, 0)

   Claimed Enumerated

   SlowLock: Free  RemLock: 1  PageCount: 1

   QueueTagList: ffffab06b03a42b0     Node 0 Outstanding: Head: 0000000000000000  Tail: 0000000000000000  Timeout: 0 (Ticking Down)

   Node 1 Outstanding: Head: 0000000000000000  Tail: 0000000000000000  Timeout: 0 (Ticking Down)

   DeviceQueue: ffffab06b03a4340  Depth: 64  Status: Not Frozen   PauseCount: 0  BusyCount: 0

   IO Gateway: Busy Count: 0  Pause Count: 0

   Requests: Outstanding: 0  Device: 0  ByPass: 0

 

 

[Device-Queued Requests]

 

IRP               SRB Type   SRB               XRB               Command           MDL               SGList            Timeout

-----------------------------------------------------------------------------------------------------------------------------------

 

 

[Bypass-Queued Requests]

 

IRP               SRB Type   SRB               XRB               Command           MDL               SGList            Timeout

-----------------------------------------------------------------------------------------------------------------------------------

 

 

[Outstanding Requests]

 

IRP               SRB Type   SRB               XRB               Command           MDL               SGList            Timeout

-----------------------------------------------------------------------------------------------------------------------------------

 

 

[Completed Requests]

 

IRP               SRB Type   SRB               XRB               Command           MDL               SGList            Timeout

-----------------------------------------------------------------------------------------------------------------------------------




커널에서 동기화 객체를 기다리는 것이 있는지 확인해 보았지만 없었습니다.

0: kd> !locks

**** DUMP OF ALL RESOURCE OBJECTS ****

KD: Scanning for held locks.........

89274 total locks




그렇다면 CPU가 바빠서 발생하는 이슈일지 모른다고 생각해서 CPU의 상태를 확인해보았으니 IDLE 상태 였습니다.

0: kd> !cpuinfo

CP  F/M/S Manufacturer  MHz PRCB Signature    MSR 8B Signature Features

 0  6,79,1 GenuineIntel 2295 0b00002a00000000                   311b3fff

 1  6,79,1 GenuineIntel 2295 0b00002a00000000                   311b3fff

 2  6,79,1 GenuineIntel 2295 0b00002a00000000                   311b3fff

 3  6,79,1 GenuineIntel 2295 0b00002a00000000                   311b3fff

                      Cached Update Signature 0b00002a00000000

                     Initial Update Signature 0b00002a00000000

0: kd> !running

Process PID Thread           Id Pri Base Pri Next CPU CSwitches User          Kernel State              Time Reason

======= === ================ == === ======== ======== ========= ==== =============== ======= =============== ==============

Idle      0 fffff8035d3d5940  0   0        0        0  24448367    0   23h:56:06.250 Running 1d.00:48:53.796 Executive

Idle      0 ffffdf005f631000  0   0        0        1  20088217    0   23h:48:42.703 Running          3s.984 WrCalloutStack

Idle      0 ffffdf005f6b9000  0   0        0        2  18610602    0   23h:58:19.172 Running 1d.00:48:53.796 Executive

Idle      0 ffffdf005f741000  0   0        0        3  25365439    0 1d.00:00:03.219 Running 1d.00:48:53.796 Executive

 

Count: 4 | Show Unique Stacks

 

0: kd> !mex.us -cpu

4 threads: ffffdf005f6b9000 ffffdf005f741000 fffff8035d3d5940 ffffdf005f631000

    fffff8035d223ff1 nt!PpmIdleGuestExecute+0x15

    fffff8035d109eac nt!PpmIdleExecuteTransition+0xcbc

    fffff8035d10903a nt!PoIdle+0x33a

    fffff8035d17afac nt!KiIdleLoop+0x2c

 

1 stack(s) with 4 threads displayed (4 Total threads)




lpc의 상태를 확인해보니 svchost.exe가 explorer에 lpc message를 보내고 기다리는 것이 확인 됩니다.

0: kd> !lpcwait

Server Process ffffab06`b8137800  Connection Port ffffab06`b853c540  (\Device\HarddiskVolume1\Windows\explorer.exe)

 

    Client Thread        Client Wait Time     Client Message       Server Thread   

    =================    =================    =================    =================

    ffffab06`b1b28800           1:25.406      ffffce0e`5b04dcf0                                                          (\Device\HarddiskVolume1\Windows\System32\svchost.exe)

 

    1 thread waiting




svchost.exe의 Thread를 확인해보니 logoff를 처리하던 것으로 보입니다.

0: kd> kc

  *** Stack trace for last set context - .thread/.cxr resets it

 # Call Site

00 nt!KiSwapContext

01 nt!KiSwapThread

02 nt!KiCommitThreadWait

03 nt!KeWaitForSingleObject

04 nt!AlpcpSignalAndWait

05 nt!AlpcpReceiveSynchronousReply

06 nt!AlpcpProcessSynchronousRequest

07 nt!NtAlpcSendWaitReceivePort

08 nt!KiSystemServiceCopyEnd

09 ntdll!NtAlpcSendWaitReceivePort

0a RPCRT4!LRPC_BASE_CCALL::DoSendReceive

0b RPCRT4!I_RpcSendReceive

0c combase!CMessageCall::CallI_RpcSendReceive

0d combase!ThreadSendReceive

0e combase!CSyncClientCall::SwitchAptAndDispatchCall

0f combase!CSyncClientCall::SendReceive2

10 combase!SyncClientCallRetryContext::SendReceiveWithRetry

11 combase!CSyncClientCall::SendReceiveInRetryContext

12 combase!DefaultSendReceive

13 combase!CSyncClientCall::SendReceive

14 combase!CClientChannel::SendReceive

15 combase!NdrExtpProxySendReceive

16 RPCRT4!NdrpClientCall3

17 combase!ObjectStublessClient

18 combase!ObjectStubless

19 combase!RemoteReleaseRifRefHelper

1a combase!RemoteReleaseRifRef

1b combase!CStdMarshal::DisconnectCliIPIDs

1c combase!CStdMarshal::Disconnect

1d combase!CStdIdentity::{dtor}

1e combase!CStdIdentity::CInternalUnk::Release

1f netprofmsvc!ATL::IConnectionPointImpl<CImplINetworkListManager,&IID_INotifyNetworkGlobalCostEvents,ATL::CComDynamicUnkArray>::Unadvise

20 netprofmsvc!CImplINetworkListManager::UnadviseAndRemoveSessionCookie<&IID_INotifyNetworkGlobalCostEvents>

21 netprofmsvc!CImplINetworkListManager::PurgeClients

22 netprofmsvc!EnterPurgeClientsCallback

23 combase!EnterForCallback

24 combase!SwitchForCallback

25 combase!PerformCallback

26 combase!CObjectContext::InternalContextCallback

27 combase!CObjectContext::ContextCallback

28 combase!CContextSwitcher::ContextCallback

29 netprofmsvc!NetProfileManUserLogOff

2a ntdll!TppSimplepExecuteCallback

2b ntdll!TppWorkerThread

2c KERNEL32!BaseThreadInitThunk

2d ntdll!RtlUserThreadStart




Explorer는 파일을 닫는 중이었는데 백신 프로그램의 필터 드라이버인 TmXPFlt가 IO가 더 진행되지 못하게 막고 있었습니다.

0: kd> !us -p ffffab06b8137800

1 thread [stats]: ffffab06b8121080

    fffff8035d17e576 nt!KiSwapContext+0x76

    fffff8035d0d74fd nt!KiSwapThread+0x17d

    fffff8035d0d6f9f nt!KiCommitThreadWait+0x14f

    fffff8035d0d8d77 nt!KeWaitForSingleObject+0x377

    fffff8022e82bce3 TmXPFlt+0x1bce3

    fffff8022e814884 TmXPFlt+0x4884

    fffff8022e814d65 TmXPFlt+0x4d65

    fffff8022e814f65 TmXPFlt+0x4f65

    fffff8022e4b3f8d TmPreFlt!TmpQueryFullName+0x1e85

    fffff8022e4b40d8 TmPreFlt!TmpQueryFullName+0x1fd0

    fffff8022c603d15 FLTMGR!FltpPerformPostCallbacks+0x2a5

    fffff8022c603756 FLTMGR!FltpPassThroughCompletionWorker+0x76

    fffff8022c605299 FLTMGR!FltpLegacyProcessingAfterPreCallbacksCompleted+0x239

    fffff8022c603146 FLTMGR!FltpDispatch+0xb6

    fffff8035d4e75a0 nt!IopCloseFile+0x150

    fffff8035d5142c5 nt!ObCloseHandleTableEntry+0x245

    fffff8035d43098b nt!NtClose+0xcb

    fffff8035d187503 nt!KiSystemServiceCopyEnd+0x13

    00007ffe863c5ce4 ntdll!NtClose+0x14

    00007ffe83335aa2 KERNELBASE!CloseHandle+0x62

    00007ffe82f02560 SHCORE!CGenericFileHandle::Release+0x160

    00007ffe82f02082 SHCORE!CFileStream::~CFileStream+0xd2

    00007ffe82f00d35 SHCORE!CFileStream::Release+0x35

    00007ffe84b0d273 SHELL32!IconCacheSave+0x18965b

    00007ffe84983a05 SHELL32!SHDesktopMessageLoop+0x45

    00007ff694781468 Explorer!wWinMain+0x714

    00007ff694810727 Explorer!__wmainCRTStartup+0x1c7

    00007ffe83ce8364 KERNEL32!BaseThreadInitThunk+0x14

    00007ffe8638e851 ntdll!RtlUserThreadStart+0x21


1 stack(s) with 1 threads displayed (1 Total threads)



분석 결과를 보면 사용자는 Logoff를 하려고 하였는데 백신 제품이 IO를 잡고 있어서 시스템이 멈춘 것으로 보인 증상이었습니다.


'Debugging' 카테고리의 다른 글

System Hang 분석  (0) 2018.11.24
Symbol Server 설정 (공유, HTTP)  (0) 2018.03.04
[디버거 명령]!Mex.p  (0) 2017.09.25
[디버깅 명령]!mex.help  (0) 2017.09.24

Mex Extension 명령 중 프로세스의 정보를 보여주는 !Mex.p 에 대해서 알아보도록 하겠습니다.


우선 !Mex.p의 사용법을 확인하기 위해서 /help를 사용해 보았습니다.


0: kd> !mex.p /help

Failed converting value '/help' to System.UInt64 for argument "Process Address" (internal name = processAddress)

!p - Displays process details


Usage:

    !p [-t] [-z] [-p <PID>] [<Process Address>] 

        -t|-threads        : Show Threads

        -z                 : Show Terminated (zombie) threads

        -p|-pid <PID>      : Finds a process via its Process ID (PID)

        Process Address    : Address of _EPROCESS Object


    !p 

        !p with no params assumes current implicit process (set with .process)


    !p [-?|-h] 

        -?|-h|-help    : Display this help text


Note: In usermode, you may not specify a PID of process Address, only the current process can be displayed

Keywords: process, pid

Current Owner: mexfeedback



Mex는 파라미터로 PID나 Process Address를 받아서 Process의 상세 정보를 출력 합니다. 그럼 Explorer의 정보를 확인하기 위해 아래 명령을 사용하여 Explorere의 Prcess Address를 확인하도록 합니다.



0: kd> !process 0 0 explorer.exe

PROCESS ffffe00100fda900

    SessionId: 1  Cid: 0c8c    Peb: 7ff66a125000  ParentCid: 0db8

    DirBase: 119a35000  ObjectTable: ffffc001ce3e1c80  HandleCount: <Data Not Accessible>

    Image: explorer.exe



Explorer의 주소는 ffffe00100fda900 로 확인 되었고 Process Address 파라미터를 사용하여 !Mex.p를 실행 합니다.



0: kd> !mex.p ffffe00100fda900

Name         Address                  Ses PID          Parent       PEB              Create Time                Mods Handle Thrd User Name

============ ======================== === ============ ============ ================ ========================== ==== ====== ==== ================

explorer.exe ffffe00100fda900 (E|K|O)   1 c8c (0n3212) db8 (0n3512) 00007ff66a125000 07-10-2017 08:39:08.606 오전  184      0   47 MyPC\admin


Command Line: C:\Windows\Explorer.EXE


Memory Details:


    VM   Peak Work Set  Commit Size PP Quota NPP Quota

    ==== ==== ========= =========== ======== =========

    2 TB 2 TB 145.94 MB    58.53 MB  1.34 MB   91.5 KB


Explorer's main thread


Show LPC Port information for process


Show Threads: Unique Stacks    !mex.listthreads (!lt) ffffe00100fda900    !process ffffe00100fda900 7



실행 결과 이름, 주소, Session, PID, 부모 process, PEB, Create Time, Module 수, Handle 수 Thread 수 그리고 사용자 이름이 출력 됩니다.

프로세스의 메모리에 대한 정보가 추가로 출력 됩니다. Windows 운영체제에 추가된 보안 기능 때문에 VM이 2TB로 보입니다.

그리고 몇가지 링크가 출력되는데 Process의 상세 정보를 개별 명령을 입력해서 확인할 필요 없이 간단하게 링크를 클릭해서 결과를 확인할 수 있습니다.



0: kd> !mex.lt ffffe00100fda900

Process      PID Thread             Id State        Time Reason

============ === ================ ==== ======= ========= =============

explorer.exe c8c ffffe00101344080  fe8 Waiting   42s.937 WrUserRequest

explorer.exe c8c ffffe001014d6080  cc4 Waiting    1s.640 WrUserRequest

explorer.exe c8c ffffe00100666080  fac Waiting 6m:30.718 UserRequest

explorer.exe c8c ffffe00101522080  a54 Waiting 1m:08.265 UserRequest

explorer.exe c8c ffffe00101534700  198 Waiting 6m:30.718 UserRequest

explorer.exe c8c ffffe00101536800  3d8 Waiting 6m:03.656 UserRequest

explorer.exe c8c ffffe00101545080  418 Waiting     765ms UserRequest

explorer.exe c8c ffffe00101542800  410 Waiting     765ms UserRequest

explorer.exe c8c ffffe0010154a080  40c Waiting 6m:30.718 UserRequest

explorer.exe c8c ffffe0010155b080  31c Waiting 3m:58.531 UserRequest

explorer.exe c8c ffffe001014f9880  d94 Waiting 1m:20.468 WrQueue

explorer.exe c8c ffffe00101516080  ddc Waiting 6m:30.718 UserRequest

explorer.exe c8c ffffe0010151c080  d40 Waiting   42s.937 UserRequest

explorer.exe c8c ffffe00101564080  d34 Waiting 1m:17.968 UserRequest

explorer.exe c8c ffffe0010155c080 1004 Waiting 3m:21.046 WrUserRequest

explorer.exe c8c ffffe0010155e080 1018 Waiting 6m:30.718 UserRequest

explorer.exe c8c ffffe00101568080 1020 Waiting 1m:17.968 UserRequest

explorer.exe c8c ffffe00101573080 1034 Waiting 1m:17.968 UserRequest

explorer.exe c8c ffffe00101574080 1038 Waiting 1m:17.968 UserRequest

explorer.exe c8c ffffe00101575080 103c Waiting   42s.937 UserRequest

explorer.exe c8c ffffe00101578080 1040 Waiting 1m:17.984 UserRequest

explorer.exe c8c ffffe0010157a080 1044 Waiting 6m:30.718 WrUserRequest

explorer.exe c8c ffffe0010157c080 1050 Waiting 2m:04.281 UserRequest

explorer.exe c8c ffffe0010157d080 1054 Waiting 1m:58.546 WrUserRequest

explorer.exe c8c ffffe00101586080 1060 Waiting 1m:17.984 WrUserRequest

explorer.exe c8c ffffe00101598500 1068 Waiting   12s.843 UserRequest

explorer.exe c8c ffffe001015ae080 1080 Waiting 3m:58.593 UserRequest

explorer.exe c8c ffffe001015b5080 1088 Waiting 1m:17.968 WrUserRequest

explorer.exe c8c ffffe000ffb25080 11d0 Waiting 6m:30.718 UserRequest

explorer.exe c8c ffffe0010014b080 11e4 Waiting 6m:30.718 WrQueue

explorer.exe c8c ffffe001020b8080  f94 Waiting   42s.937 WrUserRequest

explorer.exe c8c ffffe000ff760080  360 Waiting 3m:21.187 UserRequest

explorer.exe c8c ffffe00107423880 1578 Waiting 1m:17.968 UserRequest

explorer.exe c8c ffffe00107cba380  c48 Waiting 1m:17.968 UserRequest

explorer.exe c8c ffffe00109594080 1744 Waiting     765ms UserRequest

explorer.exe c8c ffffe00109cd6880  ba4 Waiting   53s.171 UserRequest

explorer.exe c8c ffffe0010af95880 12e8 Waiting 6m:30.718 UserRequest

explorer.exe c8c ffffe0010aad1080 1714 Waiting 1m:17.984 UserRequest

explorer.exe c8c ffffe0010b1e1880 1640 Waiting   18s.265 WrQueue

explorer.exe c8c ffffe000ffe232c0  ca8 Waiting   42s.937 UserRequest

explorer.exe c8c ffffe0010b114880 148c Waiting   18s.265 WrQueue

explorer.exe c8c ffffe000ffe98880  d10 Waiting   42s.937 WrQueue

explorer.exe c8c ffffe001025f0080  b08 Waiting     765ms WrQueue

explorer.exe c8c ffffe001019d1240  3dc Waiting   13s.796 WrQueue

explorer.exe c8c ffffe00101932880  a5c Waiting   42s.937 WrQueue

explorer.exe c8c ffffe0010b1e8880  6b4 Waiting   18s.843 WrQueue

explorer.exe c8c ffffe0010abba780  f68 Waiting   42s.937 WrQueue


Thread Count: 47



위의 결과는 Mex의 list thread 명령의 출력으로 각 Thread의 주소, TID, Thread 상태, Wait 상태로 전환된지 얼마나 지났는지, 그리고 Wait 상태로 전환된 이유에 대한 설명을 출력 합니다.



0: kd> !mex.fems -s Explorer!.*WinMain !mex.t

Process                         Thread                       CID       TEB              UserTime KernelTime ContextSwitches Wait Reason      Time State   COM-Initialized

explorer.exe (ffffe00100fda900) ffffe00101344080 (E|K|W|R|V) c8c.fe8   00007ff66a12e000   5s.641    11s.984          939044 WrUserRequest 42s.937 Waiting APTKIND_APARTMENTTHREADED (STA)


WaitBlockList:

    Object           Type                 Other Waiters

    ffffe001005352f0 SynchronizationEvent             0


Priority:

    Current Base Decrement ForegroundBoost IO Page

    10      8    0         0               0  5


# Child-SP         Return           Call Site

0 ffffd000215846b0 fffff801feaa5f1e nt!KiSwapContext+0x76

1 ffffd000215847f0 fffff801feaa5999 nt!KiSwapThread+0x14e

2 ffffd00021584890 fffff801feaa4f60 nt!KiCommitThreadWait+0x129

3 ffffd00021584910 fffff96000180288 nt!KeWaitForMultipleObjects+0x3a0

4 ffffd000215849c0 fffff9600017b361 win32k!xxxRealSleepThread+0x278

5 ffffd00021584a80 fffff96000259bcf win32k!xxxSleepThread+0xc1

6 ffffd00021584ad0 fffff801feb6bab3 win32k!NtUserWaitMessage+0x20

7 ffffd00021584b00 00007ffd8399104a nt!KiSystemServiceCopyEnd+0x13

8 00000000008ff358 00007ffd821a1527 USER32!NtUserWaitMessage+0xa

9 00000000008ff360 00007ffd82356b7d SHELL32!CDesktopBrowser::_MessageLoop+0x112

a 00000000008ff3f0 00007ff66ac08498 SHELL32!SHDesktopMessageLoop+0x3d

b 00000000008ff420 00007ff66abe8c21 Explorer!wWinMain+0x5f4

c 00000000008ff710 00007ffd81fa13d2 Explorer!CApplicationUsageTracker::OnPowerBroadcastMessage+0x334

d 00000000008ff7e0 00007ffd846854e4 KERNEL32!BaseThreadInitThunk+0x22

e 00000000008ff810 0000000000000000 ntdll!RtlUserThreadStart+0x34


Explorer 의 Main Thread를 확인하는 링크를 클릭하여 나온 결과 입니다. GUI를 가진 프로세스는 WinMain 이라는 Entry 함수를 가지는데 Mex에서는 각 Thread의 Stack을 확인하여 WinMain 함수를 가진 Thread를 찾고 정보를 출력 합니다. GUI를 가진 Process가 "응답 없음" 상태가 되는 경우 Message 처리를 하지 못하는 것인데 이 경우 WinMain Thread가 어떤 동작을 하고 있는지 확인해서 원인을 찾을 수 있습니다. 



0: kd> !kdexts.alpc /lpp ffffe00100fda900


Ports created by the process ffffe00100fda900:


ffffe001014fab00('OLE68D95701484F75BAC07FBACD02B9') 0, 5 connections

ffffe00101505e40 0 ->ffffe001014f2bb0 0 ffffe000ffda2080('svchost.exe')

ffffe001015474a0 0 ->ffffe00101547910 0 ffffe000ffd736c0('svchost.exe')

ffffe001015ac7f0 0 ->ffffe001015aca20 0 ffffe000fffe3900('svchost.exe')

ffffe001091b5e40 0 ->ffffe00101fdba80 0 ffffe001091e3300('dllhost.exe')

ffffe001012e3780 0 ->ffffe0010a9b0e40 0 ffffe00101fca900('rdpclip.exe')


Ports the process ffffe00100fda900 is connected to:


ffffe000fffaa330 0 -> ffffe000fe6fdda0('ApiPort') 0 ffffe000ffcde080('csrss.exe')

ffffe000ffd4edb0 0 -> ffffe000fffe8900('ThemeApiPort') 0 ffffe000fe6a5900('svchost.exe')

ffffe000ffb494c0 0 -> ffffe000ffdbc1e0('LSMApi') 20 ffffe000ffd736c0('svchost.exe')

ffffe001014c7570 0 -> ffffe000ffd608a0('lsasspirpc') 0 ffffe000ffd4c900('lsass.exe')

ffffe001015059e0 0 -> ffffe000ffd9a8c0('epmapper') 0 ffffe000ffda2080('svchost.exe')

ffffe00101508e40 0 -> ffffe000fffcee40('DwmApiPort') 0 ffffe000ffdd8840('dwm.exe')

ffffe00101537070 0 -> ffffe000ffd78c10('ntsvcs') 38 ffffe000ffcae900('services.exe')

ffffe00101549c70 0 -> ffffe000fef70070('PdcPort') 0 ffffe000fe69d040('System')

ffffe001015415b0 0 -> ffffe000ffd9c090('actkernel') 0 ffffe000ffd736c0('svchost.exe')

ffffe00101555e40 0 -> ffffe000fef70070('PdcPort') 0 ffffe000fe69d040('System')

ffffe00101527d70 0 -> ffffe001001b7e40('FontCachePort') 0 ffffe000fffe3900('svchost.exe')

ffffe001013d7cd0 0 -> ffffe000ffda89e0('umpo') 0 ffffe000ffd736c0('svchost.exe')

ffffe000ffdafe40 0 -> ffffe000fef70070('PdcPort') 0 ffffe000fe69d040('System')

ffffe000ffd80660 0 -> ffffe001014e5c80('webcache_{7329ea82-0845-4e4c-bd18-02b67ac065cc}_S-1-5-21-2726512140-888677503-3125549036-1001') 0 ffffe001014d4900('dllhost.exe')

ffffe000ffd1be40 0 -> ffffe000ffb22090('OLE9F792F2B42A5C2F467737E8AEF83') 0 ffffe001014d4900('dllhost.exe')

ffffe000ffcb7a30 0 -> ffffe000fffcee40('DwmApiPort') 0 ffffe000ffdd8840('dwm.exe')

ffffe001014fb840 0 -> ffffe000fffe5e40('OLE613E2E13198237E01302C92BBC53') 0 ffffe000fffe3900('svchost.exe')

ffffe001014c9cf0 0 -> ffffe000fffc7e40('eventlog') 30 ffffe000fe6a7900('svchost.exe')

ffffe00101526a30 0 -> ffffe000fef70070('PdcPort') 0 ffffe000fe69d040('System')

ffffe001015cc380 0 -> ffffe001015eca50('msctf.serverDefault1') 0 ffffe000fe73e080('taskhostex.exe')

ffffe001016193d0 0 -> ffffe001015eca50('msctf.serverDefault1') 0 ffffe000fe73e080('taskhostex.exe')

ffffe0010161ca50 0 -> ffffe001015eca50('msctf.serverDefault1') 0 ffffe000fe73e080('taskhostex.exe')

ffffe0010161a770 0 -> ffffe001015eca50('msctf.serverDefault1') 0 ffffe000fe73e080('taskhostex.exe')

ffffe0010165a070 0 -> ffffe001015eca50('msctf.serverDefault1') 0 ffffe000fe73e080('taskhostex.exe')

ffffe00101608e40 0 -> ffffe000fe9a82c0('SessEnvPrivateRpc') 0 ffffe000fe6a5900('svchost.exe')

ffffe001096c76b0 0 -> ffffe001015eca50('msctf.serverDefault1') 0 ffffe000fe73e080('taskhostex.exe')

ffffe001096658d0 0 -> ffffe001090c32b0('OLE2B514DB3A33CF461692869C92314') 0 ffffe001091e3300('dllhost.exe')

ffffe0010afd4700 0 -> ffffe00100b4c1a0('TermSrvApi') 0 ffffe000fe9c3640('svchost.exe')

ffffe0010ac22e40 0 -> ffffe001015eca50('msctf.serverDefault1') 0 ffffe000fe73e080('taskhostex.exe')

ffffe001093a46b0 0 -> ffffe000ffd944b0('OLE6FAB873CEB8A01A900BB6C89EA7E') 0 ffffe00101fca900('rdpclip.exe')

ffffe00100f51660 0 -> ffffe00100e50090('OLE5BB8085653D110B414538A51314E') 0 ffffe001019f3900('TSTheme.exe')



Process의 LPC 정보를 나열 하는 링크를 클릭한 결과 입니다. LPC는 Local Procedure Call 로 간단히 이야기 하면 Process 간에 통신을 하는것입니다. Explorer 프로세스가 만든 Port에 connect 되어 있는 Process와 Explorer 가 connect 한 Process의 상태를 확인할 수 있습니다.



0: kd> !us -p ffffad8ad052a080

1 thread [stats]: ffffad8ad0528080

    fffff801f79f52f6 nt!KiSwapContext+0x76

    fffff801f78b7a9a nt!KiSwapThread+0x16a

    fffff801f78b7461 nt!KiCommitThreadWait+0x101

    fffff801f78b6d78 nt!KeWaitForSingleObject+0x2b8

    fffffbb43e64d6f8 win32kfull!xxxRealSleepThread+0x2d8

    fffffbb43e64d377 win32kfull!xxxSleepThread2+0x97

    fffffbb43e6e4242 win32kfull!NtUserWaitMessage+0x42

    fffff801f79fb413 nt!KiSystemServiceCopyEnd+0x13

    00007ffc14211204 0x7ffc14211204


1 thread [stats]: ffffad8ad22ac080

    fffff801f79f52f6 nt!KiSwapContext+0x76

    fffff801f78b7a9a nt!KiSwapThread+0x16a

    fffff801f78b7461 nt!KiCommitThreadWait+0x101

    fffff801f78b56b7 nt!KeWaitForMultipleObjects+0x217

    fffffbb43e64d6f8 win32kfull!xxxRealSleepThread+0x2d8

    fffffbb43e64d377 win32kfull!xxxSleepThread2+0x97

    fffffbb43e6509c9 win32kfull!xxxRealInternalGetMessage+0x919

    fffffbb43e64df5c win32kfull!NtUserGetMessage+0x8c

    fffff801f79fb413 nt!KiSystemServiceCopyEnd+0x13

    00007ffc14211144 0x7ffc14211144


4 threads [stats]: ffffad8ad0677080 ffffad8ad0a8d080 ffffad8ad0a8e080 ffffad8ad0a95700

    fffff801f79f52f6 nt!KiSwapContext+0x76

    fffff801f78b7a9a nt!KiSwapThread+0x16a

    fffff801f78b7461 nt!KiCommitThreadWait+0x101

    fffff801f78b6d78 nt!KeWaitForSingleObject+0x2b8

    fffff801f7d0cdb8 nt!NtWaitForSingleObject+0xf8

    fffff801f79fb413 nt!KiSystemServiceCopyEnd+0x13

    00007ffc177b5424 0x7ffc177b5424


5 threads [stats]: ffffad8ad0688080 ffffad8acdfe3080 ffffad8ad09f3080 ffffad8ad1b0a700 ffffad8ad061f080

    fffff801f79f52f6 nt!KiSwapContext+0x76

    fffff801f78b7a9a nt!KiSwapThread+0x16a

    fffff801f78b7461 nt!KiCommitThreadWait+0x101

    fffff801f78b6d78 nt!KeWaitForSingleObject+0x2b8

    fffff801f7d0c7d1 nt!ObWaitForMultipleObjects+0x2c1

    fffff801f7d0c4d9 nt!NtWaitForMultipleObjects+0xf9

    fffff801f79fb413 nt!KiSystemServiceCopyEnd+0x13

    00007ffc177b5ef4 0x7ffc177b5ef4


7 threads [stats]: ffffad8ad1de1080 ffffad8ad0aa6080 ffffad8ad05dd080 ffffad8acdfcd080 ffffad8acce87080 ffffad8ad08a3080 ffffad8acd0b8700

    fffff801f79f52f6 nt!KiSwapContext+0x76

    fffff801f78b7a9a nt!KiSwapThread+0x16a

    fffff801f78b7461 nt!KiCommitThreadWait+0x101

    fffff801f78b6d78 nt!KeWaitForSingleObject+0x2b8

    fffffbb43e64d6f8 win32kfull!xxxRealSleepThread+0x2d8

    fffffbb43e64d377 win32kfull!xxxSleepThread2+0x97

    fffffbb43e6509c9 win32kfull!xxxRealInternalGetMessage+0x919

    fffffbb43e64df5c win32kfull!NtUserGetMessage+0x8c

    fffff801f79fb413 nt!KiSystemServiceCopyEnd+0x13

    00007ffc14211144 0x7ffc14211144


10 threads [stats]: ffffad8acccfa700 ffffad8ad06fc080 ffffad8ac8ed1700 ffffad8ad15f0080 ffffad8ad0365080 ffffad8ad085c700 ffffad8ad0a63080 ffffad8ad0354080 ffffad8ace0ee080 ffffad8ad2736080

    fffff801f79f52f6 nt!KiSwapContext+0x76

    fffff801f78b7a9a nt!KiSwapThread+0x16a

    fffff801f78b7461 nt!KiCommitThreadWait+0x101

    fffff801f78b6d78 nt!KeWaitForSingleObject+0x2b8

    fffff801f789f104 nt!KiSchedulerApc+0x304

    fffff801f78b94fe nt!KiDeliverApc+0x23e

    fffff801f79f39a3 nt!KiApcInterrupt+0xc3

    fffff801f7cc0405 nt!PspUserThreadStartup+0x35

    fffff801f79f5a86 nt!KiStartUserThread+0x16

    fffff801f79f5a00 nt!KiStartUserThreadReturn

    00007ffc17780d30 0x7ffc17780d30


22 threads [stats]: ffffad8ad4141080 ffffad8ad412d040 ffffad8acba56080 ffffad8ad06a5080 ffffad8ad249b080 ffffad8ad09f6700 ffffad8ad1b8c700 ffffad8ace697080 ffffad8acba54080 ffffad8ad40a1080 ...

    fffff801f79f52f6 nt!KiSwapContext+0x76

    fffff801f78b7a9a nt!KiSwapThread+0x16a

    fffff801f78b7461 nt!KiCommitThreadWait+0x101

    fffff801f78b62e8 nt!KeRemoveQueueEx+0x238

    fffff801f78b5dfd nt!IoRemoveIoCompletion+0x8d

    fffff801f78b4beb nt!NtWaitForWorkViaWorkerFactory+0x30b

    fffff801f79fb413 nt!KiSystemServiceCopyEnd+0x13

    00007ffc177b8c34 0x7ffc177b8c34


30 threads [stats]: ffffad8ad06a1080 ffffad8ad08bc280 ffffad8ad09f4080 ffffad8ad06a6080 ffffad8acd02c080 ffffad8acba60080 ffffad8ad02b1080 ffffad8acba68080 ffffad8ad11c5700 ffffad8ad06643c0 ...

    fffff801f79f52f6 nt!KiSwapContext+0x76

    fffff801f78b7a9a nt!KiSwapThread+0x16a

    fffff801f78b7461 nt!KiCommitThreadWait+0x101

    fffff801f78b56b7 nt!KeWaitForMultipleObjects+0x217

    fffff801f7d0c7d1 nt!ObWaitForMultipleObjects+0x2c1

    fffff801f7d0c4d9 nt!NtWaitForMultipleObjects+0xf9

    fffff801f79fb413 nt!KiSystemServiceCopyEnd+0x13

    00007ffc177b5ef4 0x7ffc177b5ef4


8 stack(s) with 80 threads displayed (100 Total threads)

20 stack(s) were not displayed because we could not switch to thread context, or stack trace was empty


!us 명령은 흥미로운 Mex 명령 입니다. 디버깅을 하다보면 반복되는 콜스택이 화면 전체를 채워버리는 경우를 많이 보았을 것 입니다. 커널스택에서 Thread가 Wait 상태에 들어가면 ObWaitForMultipleObjects 함수를 호출하는 콜스택이 반복적으로 보이게 됩니다. 이 경우 !us 명령은 아주 유용합니다.

!us 는 반복적으로 보이는 콜스택은 1번만 보여주고 각 Thread의 address를 출력하여 어떤 Thread 들이 동일한 콜스택을 보이고 있는지 보여 줍니다.


!Mex.p 명령을 통해서 나오는 출력들을 사용해서 많은 정보를 확인할 수 있는데 각 Case study에 대해서는 나중에 설명하도록 하겠습니다.

'Debugging' 카테고리의 다른 글

System Hang 분석  (0) 2018.11.24
Symbol Server 설정 (공유, HTTP)  (0) 2018.03.04
[디버거 명령]!Mex.p  (0) 2017.09.25
[디버깅 명령]!mex.help  (0) 2017.09.24

+ Recent posts