What will happen if WorkflowManager crashes?

If The Workflow Manager Backend service terminates unexpectedly all workflow’s persisted data will remain persisted in database, but the workflow itself will stop the execution. In this case there is an event id 7031 in System Event Log: The Workflow Manager Backend service terminated unexpectedly.

If there is one node in cluster only, the machine which runs Workflow Backend Service (this is the service which executes the workflow) will try to restart the service. This will also happen if there are more nodes in cluster. However in this case some other node will probable schedule execution of the workflow. In a case of one node in cluster the Workflow Backend Service will try to restart in 30 seconds. This action will be repeated over and over again. If you want to change this behavior you can do it in Service Properties. Following picture shows default settings:

In fact, temporary crashing of the process which executes the workflow will just temporary stop the execution of the workflow. Even if the recovery takes years, the persisted instance of the workflow will remain in the WorkflowState=’Started’ as long one Workflow Manager host takes it out of database and schedule execution.

Persistence

But what is with unpersisted data? This is right question on right place.

Persistence in Workflow Manager is a feature which works automatically out-of-the box. PersistActivity which is just most trivial possibility to explicitly persist the instance is in general not required. Anyhow it is not a part of trusted surface.
For example, Workflow Manager will automatically persist the workflow when workflow instance enters DelayActivity or any of awaiting activities which cause listening for message.
Whenever such activities are scheduled Manager will enforce persistence of workflow instance. States of all variables which has been changed between two activities which cause persistence will be lost! Keep that in mind.
If you want to enforce persistence, just schedule Delay activity of any interval. Unlike to AppFabric Server Workflow Manager will persist on any delay interval. AppFabric Service has a configuration is would block CPU for all other tenants.