AppFabric error “not able to activate any of the hosts”

When working with long running processes (workflows) in AppFabric you will probably from time to tome get following errors:

Error 1:

WMS was not able to activate any of the hosts - backing off. Store name 'yourStoreName ('Root).

Error 2:

Failed to invoke service management endpoint at 'net.pipe://dado-nb1/TestServices/ServiceManagement.svc' to activate service '/YourAppVirtualPath/YourService.xamlx'.\rException: 'The message could not be dispatched because the service at the endpoint address 'net.pipe://yourhostname/YourAppVirtualPath/ServiceManagement.svc' is unavailable for the protocol of the address.'

Both errors appear in pairs after the application pool has been stopped for any reason and persisted workflow instance is still in run able state. Because Workflow Management Service which is responsible to reload persisted instance has no knowledge of ApplicationPool i tries to reload the Workflow Instance. If the AppFabric is installed din cluster this will happen at all nodes. If no any of nodes (hosts) is able to reload instance this error will happen.

In common system log you will notice following two entries:

Error:

Application pool 'yourpoolname' is being automatically disabled due to a series of failures in the process(es) serving that application pool.

Warning:

A process serving application pool 'yourpoolname' suffered a fatal communication error with the Windows Process Activation Service. The process id was '1216'. The data field contains the error number.