今日、問題が発生しました。本番 Web サーバーの iis 接続が非常に高く、CPU とメモリは通常のレベルにあります。Zabbix (オープンソースの監視ソフトウェア、サーバー/パフォーマンス カウンターからデータを収集し、Web 上に表示) から、多くの「ASP.NET Requests Queued (パフォーマンス カウンター)」が表示されます。サーバーからダンプを取得します。DebugDiag2.2 を使用して、ダンプを自動的に分析します。それは次のように報告しています。
このスレッドは HttpWebRequest を作成し、リモート サーバーからの応答を待機しています。これらの要求の 1 つ以上が、使用可能な接続の最大数の少なくとも半分を使用しています。
警告、利用可能な接続の少なくとも半分が使用されています
HttpRequest URI: http://ws.product.aaa.bbb.com/product.search/api/x-protobuf/SearchData ServicePoint - ConnectionLimit:32 CurrentConnections:31
しばらくグーグルで検索すると、「asp-net-performance-case-study-web-service-calls-taking-forever」と「.NET で Web リクエストを作成するときのタイムアウト」のリンクが見つかります。
私のサーバーは .net 4 を使用しています。手順に従って、Zabbix で描画された iis 接続である waitList をダンプしようとしています。
0:048> !dumpheap -type System.Net.ConnectStream
Address MT Size
000000d2395e0fe8 00007ffbc6052c70 176
000000d2395e1148 00007ffbc6055440 24
000000d2395f5780 00007ffbc6052c70 176
/*dump the ConnectStream to find m_connection */
0:048> !do 000000d2395e0fe8
Name: System.Net.ConnectStream
MethodTable: 00007ffbc6052c70
EEClass: 00007ffbc5d29f78
Size: 176(0xb0) bytes
File: C:\Windows\Microsoft.Net\assembly\GAC_MSIL\System\v4.0_4.0.0.0__b77a5c561934e089\System.dll
Fields:
MT Field Offset Type VT Attr Value Name
...
00007ffbc6f737c8 4001d16 78 System.Int32 1 instance 3 m_CallNesting
00007ffbc60649e0 4001d17 20 ...tterGatherBuffers 0 instance 000000d2395e1098 m_BufferedData
00007ffbc6f6f370 4001d18 9c System.Boolean 1 instance 0 m_SuppressWrite
00007ffbc6f6f370 4001d19 9d System.Boolean 1 instance 1 m_BufferOnly
00007ffbc6f92ac0 4001d1a 68 System.Int64 1 instance 0 m_BytesLeftToWrite
00007ffbc6f737c8 4001d1b 7c System.Int32 1 instance 0 m_BytesAlreadyTransferred
00007ffbc6051148 4001d1c 28 ...em.Net.Connection 0 instance 000000d53828cf18 m_Connection
00007ffbc6f766d0 4001d1d 30 System.Byte[] 0 instance 0000000000000000 m_ReadBuffer
00007ffbc6f737c8 4001d1e 80 System.Int32 1 instance 0 m_ReadOffset
00007ffbc6f737c8 4001d1f 84 System.Int32 1 instance 0 m_ReadBufferSize
00007ffbc6f92ac0 4001d20 70 System.Int64 1 instance 0 m_ReadBytes
00007ffbc6f6f370 4001d21 9e System.Boolean 1 instance 0 m_Chunked
00007ffbc6f737c8 4001d22 88 System.Int32 1 instance 2 m_DoneCalled
00007ffbc6f737c8 4001d23 8c System.Int32 1 instance 2 m_ShutDown
00007ffbc6f70e08 4001d24 38 System.Exception 0 instance 0000000000000000 m_ErrorException
00007ffbc6f6f370 4001d25 9f System.Boolean 1 instance 0 m_ChunkEofRecvd
00007ffbc6057300 4001d26 40 ...m.Net.ChunkParser 0 instance 0000000000000000 m_ChunkParser
00007ffbc6071f28 4001d27 90 System.Int32 1 instance 3 m_HttpWriteMode
00007ffbc6f737c8 4001d28 94 System.Int32 1 instance 300000 m_ReadTimeout
00007ffbc6f737c8 4001d29 98 System.Int32 1 instance 300000 m_WriteTimeout
00007ffbc6052760 4001d2a 48 ...estLifetimeSetter 0 instance 0000000000000000 m_RequestLifetimeSetter
00007ffbc6f7c8c8 4001d2b 50 System.AsyncCallback 0 instance 000000d2395e10c8 m_ReadCallbackDelegate
00007ffbc6f7c8c8 4001d2c 58 System.AsyncCallback 0 instance 000000d2395e1108 m_WriteCallbackDelegate
00007ffbc60a4850 4001d2f 60 ...et.HttpWebRequest 0 instance 000000d2395e0338 m_Request
00007ffbc6f6f370 4001d31 a0 System.Boolean 1 instance 0 m_IgnoreSocketErrors
00007ffbc6f6f370 4001d32 a1 System.Boolean 1 instance 0 m_ErrorResponseStatus
/*find m_connection, then to dump the waitlist */
0:048> !do 000000d53828cf18
Name: System.Net.Connection
MethodTable: 00007ffbc6051148
EEClass: 00007ffbc5cc1ff8
Size: 296(0x128) bytes
File: C:\Windows\Microsoft.Net\assembly\GAC_MSIL\System\v4.0_4.0.0.0__b77a5c561934e089\System.dll
Fields:
MT Field Offset Type VT Attr Value Name
....
00007ffbc6f6f370 4001c6d 5c System.Boolean 1 instance 0 m_CheckLifetime
00007ffbc6f87580 4001c6e 60 System.TimeSpan 1 instance 000000d53828cf78 m_Lifetime
00007ffbc6f87af8 4001c6f 68 System.DateTime 1 instance 000000d53828cf80 m_CreateTime
00007ffbc6f6f370 4001c70 5d System.Boolean 1 instance 0 m_ConnectionIsDoomed
00007ffbc6075a30 4001c71 20 ...et.ConnectionPool 0 instance 0000000000000000 m_ConnectionPool
00007ffbc6f874e0 4001c72 28 System.WeakReference 0 instance 000000d53828d040 m_Owner
00007ffbc6f737c8 4001c73 58 System.Int32 1 instance -1 m_PooledCount
00007ffbc6f6f370 4001c74 5e System.Boolean 1 instance 0 m_Initalizing
00007ffbc6099378 4001c75 30 System.Net.IPAddress 0 instance 000000d238ee5040 m_ServerAddress
00007ffbc60546e0 4001c76 38 ...ets.NetworkStream 0 instance 000000d53828d058 m_NetworkStream
00007ffbc609b3c0 4001c77 40 ...et.Sockets.Socket 0 instance 0000000000000000 m_AbortSocket
00007ffbc609b3c0 4001c78 48 ...et.Sockets.Socket 0 instance 0000000000000000 m_AbortSocket6
00007ffbc6f6f370 4001c79 5f System.Boolean 1 instance 1 m_JustConnected
00007ffbc6053ad0 4001c7a 50 ...eralAsyncDelegate 0 instance 000000d53828d218 m_AsyncCallback
00007ffbc6099bc8 4001ccb e0 System.Int32 1 instance 0 m_Error
00007ffbc6f70e08 4001ccc 70 System.Exception 0 instance 0000000000000000 m_InnerException
00007ffbc6f737c8 4001ccd e4 System.Int32 1 instance 7 m_IISVersion
00007ffbc6f766d0 4001cce 78 System.Byte[] 0 instance 000000d5390cca00 m_ReadBuffer
00007ffbc6f6f370 4001ccf 104 System.Boolean 1 instance 1 m_ReadBufferFromPinnableCache
00007ffbc6f737c8 4001cd0 e8 System.Int32 1 instance 0 m_BytesRead
00007ffbc6f737c8 4001cd1 ec System.Int32 1 instance 0 m_BytesScanned
00007ffbc6f737c8 4001cd2 f0 System.Int32 1 instance 505 m_TotalResponseHeadersLength
00007ffbc6f737c8 4001cd3 f4 System.Int32 1 instance 65536 m_MaximumResponseHeadersLength
00007ffbc6f92ac0 4001cd4 d8 System.Int64 1 instance -1 m_MaximumUnauthorizedUploadLength
00007ffbc6053fc8 4001cd5 80 ....CoreResponseData 0 instance 000000d239a4a930 m_ResponseData
00007ffbc605fd38 4001cd6 f8 System.Int32 1 instance 0 m_ReadState
00007ffbc6056670 4001cd7 88 ...+StatusLineValues 0 instance 000000d53828d168 m_StatusLineValues
00007ffbc6f737c8 4001cd8 fc System.Int32 1 instance 5 m_StatusState
00007ffbc5c8b0c8 4001cd9 90 ...istItem, System]] 0 instance 000000d53828d098 m_WaitList
00007ffbc6f87260 4001cda 98 ...ections.ArrayList 0 instance 000000d53828d0c0 m_WriteList
00007ffbc6f81430 4001cdb a0 System.IAsyncResult 0 instance 0000000000000000 m_LastAsyncResult
00007ffbc6063a50 4001cdc a8 ...TimerThread+Timer 0 instance 000000d53828d190 m_RecycleTimer
00007ffbc607e258 4001cdd 110 ...Net.WebParseError 1 instance 000000d53828d028 m_ParseError
00007ffbc6f6f370 4001cde 105 System.Boolean 1 instance 1 m_AtLeastOneResponseReceived
00007ffbc6056328 4001ce3 b0 ...HttpAbortDelegate 0 instance 000000d53828d0e8 m_AbortDelegate
00007ffbc6056100 4001ce4 b8 ...t.ConnectionGroup 0 instance 000000d2384cdc18 m_ConnectionGroup
00007ffbc6056f58 4001ce5 c0 ...onnectionDelegate 0 instance 000000d53828d128 m_ConnectionUnlock
00007ffbc6f87af8 4001ce6 118 System.DateTime 1 instance 000000d53828d030 m_IdleSinceUtc
00007ffbc60a4850 4001ce7 c8 ...et.HttpWebRequest 0 instance 0000000000000000 m_LockedRequest
00007ffbc60a4850 4001ce8 d0 ...et.HttpWebRequest 0 instance 0000000000000000 m_CurrentRequest
00007ffbc6f6f370 4001ce9 106 System.Boolean 1 instance 1 m_CanPipeline
00007ffbc6f6f370 4001cea 107 System.Boolean 1 instance 1 m_Free
00007ffbc6f6f370 4001ceb 108 System.Boolean 1 instance 1 m_Idle
00007ffbc6f6f370 4001cec 109 System.Boolean 1 instance 1 m_KeepAlive
00007ffbc6f6f370 4001ced 10a System.Boolean 1 instance 0 m_Pipelining
00007ffbc6f737c8 4001cee 100 System.Int32 1 instance 0 m_ReservedCount
00007ffbc6f6f370 4001cef 10b System.Boolean 1 instance 1 m_ReadDone
00007ffbc6f6f370 4001cf0 10c System.Boolean 1 instance 1 m_WriteDone
00007ffbc6f6f370 4001cf1 10d System.Boolean 1 instance 0 m_RemovedFromConnectionList
00007ffbc6f6f370 4001cf2 10e System.Boolean 1 instance 0 m_NonKeepAliveRequestPipelined
00007ffbc6f6f370 4001cf3 10f System.Boolean 1 instance 0 m_IsPipelinePaused
ウエストリストを見つけましたが、サイズが 0 です。
0:048> !do 000000d53828d098
Name: System.Collections.Generic.List`1[[System.Net.Connection+WaitListItem, System]]
MethodTable: 00007ffbc5c8b0c8
EEClass: 00007ffbc69b8c48
Size: 40(0x28) bytes
File: C:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
Fields:
MT Field Offset Type VT Attr Value Name
00007ffbc6f71250 4000cd1 8 System.Object[] 0 instance 000000d43805cb70 _items
00007ffbc6f737c8 4000cd2 18 System.Int32 1 instance 0 _size
00007ffbc6f737c8 4000cd3 1c System.Int32 1 instance 0 _version
00007ffbc6f711b8 4000cd4 10 System.Object 0 instance 0000000000000000 _syncRoot
foreach ループを使用してすべての待機リストをダンプします。
.foreach ( obj {!dumpheap -mt 00007ffbc6051148 -short}){.echo ******************; !do poi(${obj}+0x90) }
....
MT Field Offset Type VT Attr Value Name
00007ffbc6f71250 4000cd1 8 System.Object[] 0 instance 000000d43805cb70 _items
00007ffbc6f737c8 4000cd2 18 System.Int32 1 instance 0 _size
....
出力では、_size=0 および _items =000000d43805cb70 です。
今のところ、「Queued Requests」がどこにあるのかわかりません。調べるのを手伝ってくれませんか？