C# で記述され、Mono 3.2 (最新の AWS .net SDK 1.5.28.2 を使用) で実行されている EC2 インスタンス AWS でリサイクル サービスを実行しています。サービスが dynamodb にヒットし、dynamo との通信中に常に低レベルのネットワーク エラーが発生します。私の現在の回避策は、AmazonServiceExceptionをキャッチして再試行することです。これはうまくいくようです
私の推測では、これはある種の Mono バグである可能性がありますが、この問題をデバッグして原因を見つける方法がわかりません。これらの例外を修正するために調査する原因または何かを提案できますか?
例外の例を次に示します。
2013-08-08 09:54:21.3991 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure ---> System.Net.WebException: Error getting response stream (ReadDone1): ReceiveFailure ---> System.IO.IOException: EndRead failure ---> System.Net.Sockets.SocketException: Connection reset by peer
at System.Net.Sockets.Socket.EndReceive (IAsyncResult result) [0x00000] in <filename unknown>:0
at System.Net.Sockets.NetworkStream.EndRead (IAsyncResult ar) [0x00000] in <filename unknown>:0
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.EndRead (IAsyncResult ar) [0x00000] in <filename unknown>:0
at Mono.Security.Protocol.Tls.SslStreamBase.InternalReadCallback (IAsyncResult result) [0x00000] in <filename unknown>:0
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.EndGetResponse (IAsyncResult asyncResult) [0x00000] in <filename unknown>:0
at System.Net.HttpWebRequest.GetResponse () [0x00000] in <filename unknown>:0
at Amazon.Runtime.AmazonWebServiceClient.getResponseCallback (IAsyncResult result) [0x00000] in <filename unknown>:0
--- End of inner exception stack trace ---
at Amazon.Runtime.AmazonWebServiceClient.handleHttpWebErrorResponse (Amazon.Runtime.Internal.AsyncResult asyncResult, System.Net.WebException we) [0x00000] in <filename unknown>:0
at Amazon.Runtime.AmazonWebServiceClient.getResponseCallback (IAsyncResult result) [0x00000] in <filename unknown>:0
そして、このエラーが発生する頻度を示します:
2013-08-07 20:31:04.4475 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-07 20:34:14.0017 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-07 20:45:06.9636 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-07 20:53:21.4654 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-07 20:56:16.8788 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-07 21:14:20.7060 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-07 21:21:17.3771 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-07 21:39:09.1383 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-07 21:57:11.6650 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-07 22:07:08.2615 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-07 22:44:16.6803 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-07 22:53:08.8771 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-07 22:53:09.1383 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-08 09:37:28.9755 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-08 09:42:28.3976 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
2013-08-08 09:54:21.3991 Warn Got service error Amazon.Runtime.AmazonServiceException: Error getting response stream (ReadDone1): ReceiveFailure
これはコミットしようとしているコードです。ddb.BatchWriteItem 呼び出しによってエラーがスローされています。
private void BatchPutWithRetry(AmazonDynamoDBClient ddb, string table, IEnumerable<Dictionary<string, AttributeValue>> items)
{
var request = new BatchWriteItemRequest
{
RequestItems = new Dictionary<string, List<WriteRequest>>{
{table, items.Select(item => new WriteRequest{ PutRequest = new PutRequest{ Item = item }}).ToList()},
},
};
Log.Trace("BatchWriteItem {0}", request.RequestItems.First().Value.Count);
var totalSends = 0;
var unsentItems = request.RequestItems;
while (true)
{
try
{
totalSends++;
request.RequestItems = unsentItems;
var response = ddb.BatchWriteItem(request);
unsentItems = response.BatchWriteItemResult.UnprocessedItems;
}
catch (ProvisionedThroughputExceededException)
{
Log.Warn("ProvisionThroughputExceeded");
}
catch (Amazon.Runtime.AmazonServiceException ex)
{
if (totalSends > 1000)
throw;
Log.WarnException("Got service error", ex);
}
if (unsentItems.Count == 0)
return;
var delay = TimeSpan.FromMilliseconds(Random.Next(500, 3000));
Log.Trace("Unprocessed {0}, waiting {1:0.000} seconds",
unsentItems.First().Value.Count,
delay.TotalSeconds);
Thread.Sleep(delay);
}
}