Using Azure Cognitive Services FACE API in UWP App

Indraneel Pole — Wed, 03 May 2017 13:41:29 GMT

I have been playing around with Cognitive Services since sometime now, as I find them mind blowing APIs to do a lot of Artificial Intelligence tasks without spending much time building AI algorithms yourself. Take FACE API for example. Normally the most common way to do face detection/recognition is to use Eigenface classification algorithm. For this you need to know basics of AI such as Regression and Classification, as well as basics of algorithms such as SVD, Neural Networks and so on. Even if you use a library such as OpenCV, you still need some knowledge of Artificial Intelligence to make sure you use the correct set of parameters from the library.

Cognitive Services however, make this demand completely obsolete and put the power of AI truly in the hands of common developer. You don’t need to learn complicated mathematics to use the AI tools anymore. This is good because as AI becomes more and more common in software world, it should also be accessible to the lowest common denominator of developers world who might not have the high degrees in science and mathematics, but who know how to code. The downside of it? The APIs are somewhat restricted in what they can do. But this might change in future (Or, so we should hope for).

Today I am going to talk about Using the Face Api from Microsoft Azure Cognitive Services, to build a simple UWP application that can tell some of the characteristics of a face (Such as age, emotion, smile, facial hair etc.)

To do so, first open Visual Studio –> Create New Project –> Select Windows Universal Blank App. This is your UWP application. To use Azure Cognitive Services in this project you would have to do two things.

1. Subscribe to Face API in Azure Cognitive Services portal – To do so, go to Cognitive Services Subscription, log on with your account, and Face API. Once you do, you will see following

You will need the key from here later in the code to subscribe to Face API client.

2. Go To Solution Explorer in Visual Studio –> Right Click on Project –> Manage Nuget Packages –> Search for Microsoft.ProjectOxford.Face and Install it. Then in MainPage.xaml.cs, add following lines of code at the top -

        1: using Microsoft.ProjectOxford.Face;

       2: using Microsoft.ProjectOxford.Face.Contract;

       3: using Microsoft.ProjectOxford.Common.Contract;

Now you are all set to create the Universal App for Face Detection using Cognitive Services.

To do so, you need to do following things -

1. Access the device camera

2. Run the camera stream in the app

3. Capture the image

4. Call Face API.

We will look into them one by one.

First, let us access the camera and start streaming the video in the app. To do so, in MainPage.xaml add CaputreElement. See the following code snippet -

       1: <CaptureElement Name="PreviewControl" Stretch="Uniform" Margin="0,0,0,0" Grid.Row="0"/>

Then create a Windows.Media.Capture.MediaCapture object and initialize it. Add this as a source to CaptureElement and call the StartPreviewAsync() method. Following code snippet might make it clearer

       1: try

       2:    {

       3:      m_mediaCapture = new MediaCapture();

       4:      await m_mediaCapture.InitializeAsync();

       5:  

       6:      m_displayRequest.RequestActive();

       7:      DisplayInformation.AutoRotationPreferences = DisplayOrientations.Landscape;

       8:      

       9:      PreviewControl.Source = m_mediaCapture;

      10:      await m_mediaCapture.StartPreviewAsync();

      11:      m_isPreviewing = true;

      12:    }

      13:   catch(Exception ex)

      14:   {

      15:       //Handle Exception

      16:   }

This will start video streaming from the camera in your application. Now to capture the image and process it, create a button in MainPage.xaml, and add a event handler to this button. In the event handler, call the FaceServiceClient that you should initialize in the initializing code of your app.

       1: FaceServiceClient fClient = new FaceServiceClient("Here your subscription key");

And then, use this InMemoryRandomAccessStream object to capture the Photo in JPG encoding and then call DetectAsync from the FaceServiceClient to get the information about faces.

         1: using (var captureStream = new InMemoryRandomAccessStream())

         2: {

         3:     await m_mediaCapture.CapturePhotoToStreamAsync(ImageEncodingProperties.CreateJpeg(), captureStream);

         4:     captureStream.Seek(0);

         5:     var faces = await fClient.DetectAsync(captureStream.AsStream(), returnFaceLandmarks: true, returnFaceAttributes: new FaceAttributes().GetAll());

         6: };

returnFaceLandmarks and returnFaceAttributes are two important properties that you need to take care of in order to get the full Detect information from the API. When returnFaceLandmarks is set to true, you get all the information of location of your face parts such as Pupils, Nose, Mouth and so on. The information that comes back looks like following

-color:#f4f4f4;"> "faceLandmarks": { "pupilLeft": { "x": 504.4, "y": 202.8 }, "pupilRight": { "x": 607.7, "y": 175.9 }, "noseTip": { "x": 598.5, "y": 250.9 }, "mouthLeft": { "x": 527.7, "y": 298.9 }, "mouthRight": { "x": 626.4, "y": 271.5 }, "eyebrowLeftOuter": { "x": 452.3, "y": 191 }, "eyebrowLeftInner": { "x": 531.4, "y": 180.2 }, "eyeLeftOuter": { "x": 487.6, "y": 207.9 }, "eyeLeftTop": { "x": 506.7, "y": 196.6 }, "eyeLeftBottom": { "x": 506.8, "y": 212.9 }, "eyeLeftInner": { "x": 526.5, "y": 204.3 }, "eyebrowRightInner": { "x": 583.7, "y": 167.6 }, "eyebrowRightOuter": { "x": 635.8, "y": 141.4 }, "eyeRightInner": { "x": 592, "y": 185 }, "eyeRightTop": { "x": 607.3, "y": 170.1 }, "eyeRightBottom": { "x": 612.2, "y": 183.4 }, "eyeRightOuter": { "x": 626.6, "y": 171.7 }, "noseRootLeft": { "x": 549.7, "y": 201 }, "noseRootRight": { "x": 581.7, "y": 192.9 }, "noseLeftAlarTop": { "x": 557.5, "y": 241.1 }, "noseRightAlarTop": { "x": 603.7, "y": 228.5 }, "noseLeftAlarOutTip": { "x": 549.4, "y": 261.8 }, "noseRightAlarOutTip": { "x": 616.7, "y": 241.7 }, "upperLipTop": { "x": 593.2, "y": 283.5 }, "upperLipBottom": { "x": 594.1, "y": 291.6 }, "underLipTop": { "x": 595.6, "y": 307 }, "underLipBottom": { "x": 598, "y": 320.7 } }

In faceAttributes properties you should give the FaceAttributeType that you want to have such as following. In my application I created a class from which I return all of them in a list with a method called GetAll().

       1: FaceAttributeType.Age,

       2: FaceAttributeType.Emotion,

       3: FaceAttributeType.FacialHair,

       4: FaceAttributeType.Gender,

       5: FaceAttributeType.Glasses,

       6: FaceAttributeType.HeadPose,

       7: FaceAttributeType.Smile

The result will look similar to this

       1: "faceAttributes": {

       2:       "age": 23.8,

       3:       "gender": "female",

       4:       "headPose": {

       5:         "roll": -16.9,

       6:         "yaw": 21.3,

       7:         "pitch": 0

       8:       },

       9:       "smile": 0.826,

      10:       "facialHair": {

      11:         "moustache": 0,

      12:         "beard": 0,

      13:         "sideburns": 0

      14:       },

      15:       "glasses": "ReadingGlasses",

      16:       "emotion": {

      17:         "anger": 0.103,

      18:         "contempt": 0.003,

      19:         "disgust": 0.038,

      20:         "fear": 0.003,

      21:         "happiness": 0.826,

      22:         "neutral": 0.006,

      23:         "sadness": 0.001,

      24:         "surprise": 0.02

      25:       }

      26:     }

These are float values between 0 and 1 with 1 being maximum and 0 is being least. In my application I wrote a basic threshold method that can display if I am smiling, angry, happy, etc in UI based on these values. The final result looks like this -

It failed to detect my age (Because when I tested it, I was using a warm light bulb for lighting. Tip: Lighting matters a lot in age detection from Face API. Use cold lights if you want to look younger ). But apart from that the other information was quite correct. I was indeed smiling, my face was happy, I wasn’t wearing glasses and have some beard.

At the first look, Face API looks really interesting. You can do a lot with other endpoints of API such as Verification, Identification etc. I would try to cover these other functions in next posts.

Till then.

Creating custom blob filter using AForge .NET in UWP Application

Indraneel Pole — Tue, 24 May 2016 13:08:30 GMT

As you might already know, AForge is a .NET library for Image Processing. A lot of people prefer using .NET wrapper for OpenCV known as EmguCV, and no doubt it has stronger image processing capabilities. However, being a wrapper, it has its own limitations and sometimes AForge proves to be a better choice than EmguCV, for example while trying to process an image in Universal Windows Application. I am not going to discuss the comparison between AForge and EmguCV, I am just going to talk about how to use AForge with custom filters for object detection.

AForge comes with a lot of out of the box shape detection methods such as whether the blob is circle, triangle and so on. On the other hand it also has BlobsFiltering method that can detect object based on minimum width, minimum height, maximum width, and maximum height. But what if you have to detect a shape based on custom features. Such as ratio of height to width, or specific color, or anything else. In such case, custom filters come handy.

AForge provides Filtering class called BlobsFiltering. While one can use this class directly to filter blobs using width and height of blob. However, you can pass a custom filter that inherits from IBlobsFilter interface.

Before diving into the code, some background – I have used AForge not in usual .NET environment but the new .NET Core environment which is the environment for UWP applications. The library for .NET Core is available on GitHub. Why is this important? Because it gives us the infinite opportunity to use AForge with not only usual Forms or backend applications, but also with Phones, IoT devices such as Raspberry Pi, and even HoloLens to not only detect and identify objects, but to use machine learning algorithms such as supervised perceptron learning to make these vast variety of devices “Intelligent”.

Now, I have created a sample filter called BlobFilter that inherits from IBlobsFilter interface. Here is how it looks -

I am checking if the blob that is detected is triangle, circle, quadrilateral or convex polygon, and if it is, then I remove these from the image. If the check method returns true, it means the object should be removed from the image.

Then I create a BlobsFiltering object with this filter. And then, I just apply the filter on the image. It gives me the image without the shapes that I mentioned.

To show the example, here I uploaded the sample image from AForge website in my application -

And when I click on Remove shapes button, I get the output -

The two circle like objects are actually elliptical and thus are not removed from the image.

Using machine learning functions, this custom filter can be made really advanced where it can remove more complex figures such as numbers, characters, or even real world objects like cars, chairs, etc.

Next time I will talk about how we can use supervised learning algorithm in AForge to create the custom filters that can remove more complex shapes.

Tip – If you are new to Image processing and confused about what does it mean by blob, well, a blob is any object that is detected by the algorithm inside the image.

Developer docs by Indraneel Pole : UWP

Using Azure Cognitive Services FACE API in UWP App

Creating custom blob filter using AForge .NET in UWP Application