Service Bus for Windows Server “Blind Alley”

Damir Dobric — Tue, 09 Jul 2013 07:00:00 GMT

These days we detected some harmful information message in the Service Bus event log.

ServiceBus Inner Management Channel has faulted. Type: PartitionId:0, ContainerId:1.ree

When something like this happen, you should hope that this was a temporary issue, which somehow has been automatically covered. In most cases this is exactly what happen.
But if you after a while notice that nothing works, you should immediately try to fix the issue which does not exist. Yes, this is exactly what all Windows Event Logs will give for free: “No any issue”.

If you are a SharePoint guy you will probably find this instead:

Error processing management request. Method: PUT, RequestUri: http://****:12291/WorkflowManagement/$Workflows/YOURSCOPENAME/, Error: System.TimeoutException: Could not successfully create subscription for topic ******/WFTOPIC' despite multiple retries over a timespan of 00:05:00.2220000.. The exception of the last retry is: The request has timed out after 60000 milliseconds. The successful completion of the request cannot be determined. Additional queries should be made to determine whether or not the operation has succeeded.. ---> System.TimeoutException: The request has timed out after 60000 milliseconds. The successful completion of the request cannot be determined. Additional queries should be made to determine whether or not the operation has succeeded. ---> System.Net.WebException: The request was aborted: The request was canceled.

at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)

at Microsoft.ServiceBus.Messaging.ServiceBusResourceOperations.CreateOrUpdateAsyncResult`1.EndGetResponse(CreateOrUpdateAsyncResult`1 thisPtr, IAsyncResult ar)

at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult result)

--- End of inner exception stack trace ---

Server stack trace:

Exception rethrown at [0]:

at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)

at Microsoft.ServiceBus.NamespaceManager.CreateOrUpdateSubscriptionAsyncResult.OnCreateSubscription(IAsyncResult result)

at Microsoft.ServiceBus.Common.AsyncResult.AsyncCompletionWrapperCallback(IAsyncResult result)

Exception rethrown at [1]:

at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)

at Microsoft.Workflow.Gateway.CreateOrUpdateWorkflowAsyncResult.CreateSubscriptionRetryAsyncResult.EndAsyncResult(IAsyncResult result)

at Microsoft.Workflow.Common.BackoffRetryAsyncResult.IsolateWithRetry(IAsyncResult result)

--- End of inner exception stack trace ---

If you are Service Bus guy you will notice that no any entity can be created.

When you see something like this do no try to reinstall or to repair the node, because this will happen on all nodes. The issue is related to Service Bus database which holds Service Bus Gatewayzy thing about this israc information.
All these errors can also be noticed when using Workflow Management Client. In this case you might get various errors like “404 Forbidden”, “500 Internal Server Error” or even “403 Not Found”.
The most crazy about this is that all errors are totally undeterministic.

The root of the problem is database. Open SQL Management studio and try to execute any command which would write something in Transaction Log. For example delete of workflow instance or even some service bus entity.
I know we should never ever do something like this. But unfortunately I didn’t find any better way.

If you do that following will happen:

Error Message: The transaction log for database 'SbGatewayDatabase' is full. To find out why space in the log cannot be reused, see the log_reuse_wait_desc column in sys.databases

Correct the errors and attempt to delete the row again or press ESC to cancel the change(s).

Then go to this database and take a look on transaction log. And, don’t be surprised if the transaction is not full. The problem is possibly not in Service Bus database. Usually on on SQL server some other databases are hosted too.
If can be that other database has full transaction log. Or the drive of the SQL server machine is full.

The solution of the problem is to make a free place on the drive which hosts Service Bus databases.

Damir Dobric Posts : work

Service Bus for Windows Server “Blind Alley”