Search results matching tag 'SharePoint'

Power of Windows Azure PaaS: State Share and Persistence in the cloud

ddobric — Mon, 22 Apr 2013 13:00:00 GMT

If you are experienced enterprise developer who targets the Microsoft platform, you have probably at least once in your have a requirement to share some state across different instance of an application.
But, if you are hobby developer or any other professional developer, who possible do not focus distributed systems, you will might find the term “State” in context of persistence a pure theory thing.
So who ever you are, let’s start with a simple theory thing.
Imaging you have in theory a trivial application (today I call such applications simply “apps”). Assume this is an app running on smart phone or any other mobile device. This app has a variable ‘A’ and value assigned to this variable ‘1’. This value is not the default value. It is rather the value which user has assigned at some point of time.

Now, imagine the application crashes and you start it again. The value of variable A is obviously lost. If you want to keep this value somewhere, so it can be loaded even if application crashes. We call that usually persistence. Depending on application and your skill, there are several ways to persists the value of an variable. For example you can use any kind of local cache, like a file system.

But what if your application has to survive the hardware damage? Or what if your application is consisted multiple clients which have to share the same value? In this case you will have to use some kind of database.
As long you are in control of your infrastructure (means application is deployed in your enterprise) you can use some database to persist the value. The good example of this is Workflow Foundation. Long running processes require by default the state persistence.
Unfortunately we have more often scenarios where applications are distributed across the web. In such cases the only sharing artifact between such applications is the cloud. Following picture shows how two apps (or applications) share their state through the cloud.

This is a nice theoretical case which is practically a horror to implement. I strongly believe that today we are implicitly preparing for dominance of devices in the web. In this hypothetical scenario both devices (see picture above) should act as services to be able to share their state. If you are distributed application or integration developer you know that connectivity us such scenarios is a big, big issue. Or simply, it is the fact, devices in the web have a private IP address and because of that they are not reachable by anybody. So if the microscope on the left cannot reach the smart-phone on the right, the state from left to right cannot be propagated and vice versa.

If this to theoretical to you, imagine just a service which is running distributed across cloud. This might be WCF service, Windows Service (running in SCM) or even Web Api based service. Whatever technology you peek, it does not matter. The only important thing is that you want run multiple instances of the service, but on two machines which are not in the same environment (if they are in the same environment even better). So that means, independent on a case, devices or services or both, we have the same requirement:

1. Sharing of the state between applications distributed in the web
2. Persisting of the state without database.

You believe or not, this problem can be solved by using Windows Azure Service Bus, which in this case plays a role of cloud artifact. To illustrate how this can work, take a look on following example.
This is a console application in C#. The state which should be shared and persisted across many applications is stared in variable State.

class Program{

public static WebState<string> State { get; set; }

static void Main(string[] args)
{

State = new WebState<string>(ConfigurationManager.AppSettings["SbConnStr"], "BusinessDomain", ConfigurationManager.AppSettings["Node"]);

           State.StateChanged += State_StateChanged;

            Console.WriteLine("I'm node with name: " + ConfigurationManager.AppSettings["Node"]);

Console.WriteLine("Running with state: {0}", State.Value);

           while (true)
            {
                State.Value = Console.ReadLine();
            }
        }

        static void State_StateChanged(object sender, WebStateEventArg<string> e)
        {
            Console.WriteLine("State changed by {0} at value: {1}", e.InstanceName, e.NewValue);
        }

}

As you see this variable is of type WebState, which is implemented by me. When you start this application, enter some text on the console. The text will be immediately persisted in the cloud after following line of code is executed:

State.Value = …

Now stop the application gracefully and start it again. You will notice that application write out the value which was as last one persisted. In fact event with one participant in the system you have achieved the persistence.
The magic behind this is implemented on top of Windows Azure Service Bus queues. Each time the value is changed, the WebState implementation receives the existing message from the queue and put back the new message in the queue which basically holds the serialized value of variable State, and so on. Because access to the queue is session based, Service Bus guaranties that no any other applications which shares this value, will access the queue in the same time. I don’t want to call this distributed transaction, but it behaves in persistence this scenario like it would be one. In fact the WebState class uses the Queue as a unit of storage. By reading and immediate writing back of the new value, it is guarantied that the queue holds one and only one message all the time.
Following picture illustrate this. Each time the value i changed the old value is stored in the queue as a message. The old value is received from queue and the new message with new value is sent to the queue. At the end the old one is completed, because PeekAndLock receive is used.When the new instance of the State is created (for example when application is started) the same cycle with receive/send is done. This is how multiple applications obtain there actual state.

Additionally the implementation of the WebState takes a care about state sharing. For this purpose there is a topic used to share state changes. All participants in this distributed system have to create a subscription on that topic.
In this case (see picture below) the new value is published as a message to the topic. After that all subscribers can receive the message with the new value. The publishing process additionally updates the new value in the persistence queue from previous example.

Because topic-subscriptions are used, the application which was not running for a while, will automatically receive all state changes which have happened in the mean time.

This examples shows how complex patterns can be implemented by using connectivity service in Windows Azure. I know that majority of people out there still think a..bout cloud as an old thing. There is also a group of people who thinks that cloud is unsafe think to keep the data. If you are one them them please rethink your thinking. The scenario and solution shown above is a new way of thinking and using cloud services. And the nice think here is that practically no real data is stored in the cloud.
As you see, the cloud is more than solely putting of some application online. :)

The code for this will be published soon.

Welcome to WinDays 2013

ddobric — Sun, 21 Apr 2013 14:06:16 GMT

I'm very happy that 01. Mai is on the front. This is my favorite Barbie Queue day.
But till than one more public conference only. WinDays is probably after Tech Ed the biggest developer conference in Europe.

Traditionally I will give few talks there. As always I will talk about things which publicly never been covered from perspective I usually do.
I’m ready for new year of more or less new fiscal strategies. This time my focus will be story about Devices and Service in context of cloud.
A good short introduction in upcoming content will be presented in my keynote.
Of course as MVP and VTSP I will focus Windows Azure Cloud and my personal expertise area around Distributed Systems, heavy architectures, Service Bus, Workflow Manager etc.

Here is short summary:

Devices and Service Intro

22.04.: Keynote
To be covered very common and useful scenarios for devices in the cloud. What can you do with Service Bus and why is Service Bus one of core PaaS components of Windows Azure.

How to build Services for Devices
24.04: Distributed Applications for Windows, Cloud & SharePoint (400)
srijeda15:05 - 15:50Coral 5

Runtime for Devices

24.04: Runtime 8 Fight-Club
srijeda18:20 - 19:05Coral 5

Cumulative Updates for Workflow Manager, Service Bus (SharePoint 2013 – Workflow Engine)

ddobric — Thu, 28 Feb 2013 10:23:12 GMT

The first Cumulative Update for Workflow Manager and Service Bus or formally known SharePoint workflow is online. which actually includes Suspend/Resume of instances support. (It will also be available via WebPI soon).

WF Manager:

KB article (KB2799754): here

Download page: here

Service Bus:

KB article (KB2799752): here

Download page: here

Deploying of the new version of SharePoint Workflow

ddobric — Tue, 04 Dec 2012 07:58:00 GMT

Assume you have an activity which is executing for very long time. This is so called Long-Running workflow. In fact long-running scenario is one of scenarios in Commuter Sciences which practically can very easy be solved by Using Microsoft Workflow Foundation.
Following workflow show never ending workflow which is often used in Integration scenarios where some kind of listening or long-polling need to be implemented.

The question in this post is: “After this workflow is started, what happens with this workflow when SharePoint deploys the new version?”. This is usually the case when some bug has been fixed or some new feature has been implemented.
During deployment of the new version SharePoint will not take a care about running instances. Instead, all running instances of the version N will be terminated with following error:

The instance has been terminated because the workflow associated with this instance is no longer available.

After this has happened, the new version N+1 is (should be) deployed and ready to start. Currently it is not clear (to me at least) if there is a way at all in SharePoint to figure out such issues. Fortunately Workflow Manager provides way to figure out such issues.
Following code shows how we do that. Just call enlistInstancesInState(…) to obtain all workflows in specified state.

private static void enlistInstancesInState(WorkflowInstanceStatus status)
{

                enlistAllScopes(null, (client) =>
                {
                    var cnt = client.Instances.GetCount();
                    if (cnt > 0)
                    {
                        foreach (var wfInst in client.Instances.Get(0, cnt))
                        {

if (WorkflowInstanceStatus.NotSpecified != status && wfInst.WorkflowStatus == status)
traceOutWorkflow(wfInst);

                            else if (WorkflowInstanceStatus.NotSpecified == status)
                                traceOutWorkflow(wfInst);
                        }
                    }
                });
        }

        /// <summary>
        /// Enlists all scopes starting at root.
        /// </summary>
        /// <param name="scopeRoot">Scope to be enlisted.</param>
        private static void enlistAllScopes(WorkflowManagementClient client, Action<WorkflowManagementClient> action)
        {

if(client == null)
client = getWfMgmClient(null);

if (action != null)
action(client);

var scopes = client.CurrentScope.GetChildScopes();

            foreach (var childScope in scopes)
            {
                enlistAllScopes(getWfMgmClient(Path.Combine(m_Root.TrimEnd('/') + childScope.Path)), action);
            }
        }

        /// <summary>
        /// Creates the instance of the WF-client.
        /// </summary>
        private static WorkflowManagementClient getWfMgmClient(string scope)
        {

WorkflowManagementClient client;

            if (scope == null)
                client = new WorkflowManagementClient(m_Root);
            else
                client = new WorkflowManagementClient(scope);

return client;
}

How do SharePoint 2013 Workflows look like under the hub?

ddobric — Thu, 29 Nov 2012 07:40:00 GMT

To get a feeling how SharePoint Workflows look like under the hub take a look on following example: http://developers.de/media/p/422263.aspx.
This is the original XML which fully (natively) describe the workflow deployed by SharePoint.

Following picture shows a snapshot of the XML which can be downloaded above:

Note that the dump shown above describe the WorkflowDescription without activities. However notice that description contains the reference to activity ActivityPath.
Similarly you can dump activities as follows:

http://host:12291/SharePoint/default/eac2871d-0758-4a2e-8b03-c02598c63c0b/8256fccd-21e8-4435-8327-2affb3234c23/$Activities

This call will return list of activities by their name. For example: WorkflowXaml_78ba2e34_c23f_4ea9_94fb_86d3efcd3c80
That name can be used to obtain the original XAML
http://host:12291/SharePoint/default/eac2871d-0758-4a2e-8b03-c02598c63c0b/8256fccd-21e8-4435-8327-2affb3234c23/$Activities/WorkflowXaml_78ba2e34_c23f_4ea9_94fb_86d3efcd3c80

And finally, here is the result which dups out the XAML of the workflow:

Examples shown above has been taken out of Workflow Manager which is running on Premise. However same or similar workflow descriptions are used when running in Office 365 and Windows Azure.
I will describe the meaning of all of required artifacts in my future blog posts.

What will happen if WorkflowManager crashes?

ddobric — Tue, 27 Nov 2012 11:20:00 GMT

If The Workflow Manager Backend service terminates unexpectedly all workflow’s persisted data will remain persisted in database, but the workflow itself will stop the execution. In this case there is an event id 7031 in System Event Log: The Workflow Manager Backend service terminated unexpectedly.

If there is one node in cluster only, the machine which runs Workflow Backend Service (this is the service which executes the workflow) will try to restart the service. This will also happen if there are more nodes in cluster. However in this case some other node will probable schedule execution of the workflow. In a case of one node in cluster the Workflow Backend Service will try to restart in 30 seconds. This action will be repeated over and over again. If you want to change this behavior you can do it in Service Properties. Following picture shows default settings:

In fact, temporary crashing of the process which executes the workflow will just temporary stop the execution of the workflow. Even if the recovery takes years, the persisted instance of the workflow will remain in the WorkflowState=’Started’ as long one Workflow Manager host takes it out of database and schedule execution.

Persistence

But what is with unpersisted data? This is right question on right place.

Persistence in Workflow Manager is a feature which works automatically out-of-the box. PersistActivity which is just most trivial possibility to explicitly persist the instance is in general not required. Anyhow it is not a part of trusted surface.
For example, Workflow Manager will automatically persist the workflow when workflow instance enters DelayActivity or any of awaiting activities which cause listening for message.
Whenever such activities are scheduled Manager will enforce persistence of workflow instance. States of all variables which has been changed between two activities which cause persistence will be lost! Keep that in mind.
If you want to enforce persistence, just schedule Delay activity of any interval. Unlike to AppFabric Server Workflow Manager will persist on any delay interval. AppFabric Service has a configuration is would block CPU for all other tenants.

SharePoint Workflow TrustedSurface

ddobric — Mon, 26 Nov 2012 07:23:00 GMT

As I described here Windows Azure Workflow (Workflow Manager) is originally designed to run in the cloud. That means it designed as multitenant application. because of that the Workflow Manager must fulfill several very sensitive security requirements. One of used approaches is so called “Reduced Trusted Surface”. This means that subset of common primitive types is supported only. Unfortunately SharePoint requires many activities which are not the part of Trusted Surface of Workflow Manager.
For this reason, SharePoint team has extended the Type System of the Workflow Manager.
The list shown below is the full list of all SharePoint custom activities which extends the default Trusted Surface described in the post above:

MatchesString IsLessThanOrEqualDateTime IsLessThanDateTime IsGreaterThanOrEqualDateTime IsGreaterThanDateTime IsEqualUser IsEqualStringIgnoreCase IsEqualDynamicValue IsEqualDateTime IsEqualDate ContainsStringIgnoreCase WriteToHistory WorkflowInterop WordsInTitle WaitForItemEvent WaitForFieldChange	WaitForCustomEvent UpdateListItem UndoCheckOutItem TranslateDocument SetWorkflowStatus SetTimeField SetRelatedItem SetField ReplaceEmailTokens RegisterForSharePointEvent ModifiedInRange ModifiedBy LoopNTimes LookupWorkflowContextProperty LookupSPUserProperty LookupSPUser
LookupSPTaskListItemAssignedTo LookupSPTaskAssignedToDisplayName LookupSPPrincipalProperty LookupSPPrincipalId LookupSPPrincipal LookupSPFields LookupSPField LookupSPListProperty LookupSPListItemStringProperty LookupSPListItemSPFieldLookupProperty LookupSPListItemSPFieldLookupMultiProperty LookupSPListItemPropertyNameInREST LookupSPListItemProperty LookupSPListItemInt32Property LookupSPListItemGuid LookupSPListItemDynamicValueProperty LookupSPListItemDoubleProperty LookupSPListItemDateTimeProperty LookupSPListItemBooleanProperty LookupSPListItem	LookupSPList LookupSPGroupMembers LookupSPGroup LookupSPChoiceFieldIndex JoinSPPrincipalPropertyFromInitFormParamUserMulti JoinSiteUserInfoListPropertyFromSPFieldUserMulti JoinIdOrValueFromSPFieldLookupMulti JoinChoicesFromSPFieldMultiChoice IsValidUser IsSPListItemPropertyLookupToDocsField GetHistoryListId GetTaskListId GetCurrentListId GetCurrentItemGuid GenerateEmailDynamicValue ExtractSubstringFromIndexLength ExtractSubstringFromIndex ExtractSubstringFromStart ExtractSubstringFromEnd ExpandInitFormUsers
ExpandGroupToUsers Email DeleteListItem DelayUntil DelayFor DefaultRetryPolicy DateInterval IsNull WebUri CreateListItem CreatedInRange CreatedBy CopyItem ConvertPropertiesForSPListItem CompositeTaskHelper	CompositeTask Comment CheckOutItem ConvertTimeZoneFromUtcToSPLocal BuildSPUri ConvertTimeZoneFromSPLocalToUtc CheckInItem CallHTTPWebService Calc BuildSPListItemWebLink AtomicTaskItemUpdatedHelper SingleTask AppOnlySequence

All of activities are deployed to following scope:
http://spswfmanagermachine:12291/sharepoint

Note that SharePoint Workflow Manager by default also installs the scope ‘default’ (http://spswfmanagermachine:12291/sharepoint/default)
This scope currently does not contain any activity.

When working with Workflows in Office 365, these activities are present by default too.

Agenda for SharePoint Workflow Engine Deep Dive

ddobric — Sun, 18 Nov 2012 21:05:01 GMT

Announcing agenda for session “SharePoint Workflow Engine Deep Dive” at conference “Knowledge Days rund um SharePoint - DAS Event zu Microsoft SharePoint 2013” in Munic at
Nov. 20 and Nov.21. 2012.

AGENDA

Deployment of Workflows in Workflow Manager

ddobric — Sun, 18 Nov 2012 10:48:51 GMT

When deploying Workflows within Workflow Manager as long activities inside of trusted surface are used, there is no assembly which should be referenced or published to the host.
All required is contained in a XAML.

Usually one workflow is WorkflowDefinition which basically defines the activation envelope of the around the activity.
That means one workflow is defined by two XAML-s: Activit.Xaml and Workflow.Xaml.
Activity.Xaml fully implements the workflow in XAML. Workflow.Xaml defines configuration, activation, etc. and sets the reference to Activity.xaml which will be executed.

So, Deployment is two-phase process:

client.Activities.Publish(Activity.Xaml) and

client.Workflows.Publish(Workflow.Xaml);

As we see, first root activity is published to the manager database and then the workflow envelope (workflow description) itself. This is because, Workflow has a dependency to root activity, so activity has to be deployed first.
As long you use build-in activities defined by trusted surface there is no need for any assembly to be deployed. All you have and need is declarative XAML.
However in a case of extending of trusted surface, fist assemblies which implement custom types have to be deployed. After they are deployed, workflow publishing works as already described above.

To get a feeling about publishing open the browser and paste following:

GET //SharePoint/default/eac2871d-0758-4a2e-8b03-c02598c63c0b/8256fccd-21e8-4435-8327-2affb3234c23/$Workflows?$skip=0&$top=100

Note, the underlined part is the scope path of workflow at your machine (this one is example from my machine). This statement is a REST call to Workflow Manager which will return up to 100 installed workflow definitions.
Similarly following REST call would get the full Workflow Description of workflow with name=’ d1991cf0-0fa3-46d7-9cf7-32014141c6aa‘:

GET //SharePoint/default/eac2871d-0758-4a2e-8b03-c02598c63c0b/8256fccd-21e8-4435-8327-2affb3234c23/$Workflows/d1991cf0-0fa3-46d7-9cf7-32014141c6aa

There is also a REST call which can dump-out the whole XAML implementation.

http://host:12291/yourscope/$Activities/WorkflowXaml_78ba2e34_c23f_4ea9_94fb_86d3efcd3c80

Validate that file name before uploading a file to the SharePoint library!

ajugo — Thu, 29 Oct 2009 13:36:00 GMT

This blog post has moved to a new address. Please read it there.