Introduction

Daany is .NET data analytics library written in C# and it supposed to be a tool
for data preparation, feature engineering and other kinds of data
transformations prior to creating ml-ready data set. It is .NET Core based
library with ability to run on Windows Linux based distribution and Mac. It is
based on .NET Standard 2.1.

Besides data analysis, the library implements a set of statistics or data
science features e.g. time series decompositions, optimization performance
parameters and similar.

Currently Daany project consists of four main
components:

• Daany.DataFrame,
• Daany.Stats,
• Daany.MathStuff and
• Daany.DataFrame.Ext

The main Daany component is Daany.DataFrame - a data frame implementation for
data analysis. It is much like Pandas but the component is not going to follow
pandas implementation. It is suitable for doing data exploration and preparation
with C# Jupyter Notebook. In order to create or load data into data frame it
doesn’t require any predefined class type. In order to defined relevant value type of each column all data are parsed internally during data frame creation. The Daany.DataFrame implements set of powerful features for data manipulation, handling missing values, calculated columns, merging two or more data frames into one, and similar. It is handy for extracting its rows or columns as series of elements and put into the chart to visualizing the data.

Daany.Stat is a collection of statistics features e.g. time series
decompositions, optimization, performance parameters and similar.

Daany.Math is a component within data frame with implementation of od matrix and related
linear algebra capabilities. It also contains some implementation of other great open source projects. The component is not going to be separate NuGet package.

Daany.DataFrame.Ext contains extensions for Daany.DataFrame component, but they are related to other projects mostly to ML.NET. The Daany.DataFrame should not be dependent on ML.NET and other libraries. So, any future data frame feature which depends on something other than Daany.Math, should be placed in Daany.Ext.

The project is developed as a need to have a set of data transformation features
in one library while I am working with machine learning. So, I thought it might
help to others. Currently, the library has pretty much data transformation
features and might be your number one data analytics library on .NET
platform. Collaboration to the project is also welcome.

How to start with Daany

Daany is 100% .NET Core component and can be run on any platform .NET Core supports, from the Windows x86/x64 to Mac or Linux based OS. It can be used by Visual Studio or Visual Studio Code. It consisted of 3 NuGet packages, so
the easiest way to start with it is to install the packages in your .NET
application. Within Visual Studio create or open your .NET application and open
NuGet packages window. Type Daany in the browse edit box and hit enter. You can
find four packages starting with Daany. You have few options to install the
packages.

1. Install Daany.DataFrame – only. Use this option if you want only data
   analysis by using data frame. Once you click Install button, Daany.DataFrame
   and Daany.Math will be installed into your project app.

2. Install Daany.Stat package. This package already contains DataFrame, as well as time series decomposition and related statistics features.

Once you install the packages, you can start developing your app using Daany
packages.

Using Daany as assembly reference

Since Daany has no dependency to other libraries you can copy three dlls and add them as reference to your project.

In order to do so clone the project from http://github.com/bhrnjica/daany,build it and copy Daany.DataFrame.dll, Daany.Math.dll and Daany.Stat.dll to your project as assembly references. Whole project is just 270 KB.

Using Daany with .NET Jupyter Notebook

Daany library is ideal with .NET Jupyter Notebook, and some of the great notebooks are implemented already, and can be viewed at http://github.com/bhrnjica/notebooks. The GitHub project contains the code necessary to run the notebooks in Binder, a Jupyter Virtual Environment, and try Daany without any local installation. So the first recommendation is to try Daany with already implemented notebooks using Binder.com.

Namespaces in Daany

Daany project contains several namespaces for separating different
implementation. The following list contains relevant namespaces:

• using Daany – data frame and related code implementation,
• using Daany.Ext – data frame extensions, used with dependency on third party
  library,
• using Daany.MathStuff – math related stuff implemented in Daany,
• using Daany.Optimizers – set of optimizers like SGD,
• using Daany.Stat – set of statistics implementations in the project.

That's all for this post. Next blog posts will show more exciting implementation using Daany.

With ANNdotNET 1.2 the user is able to create and train deep learning models for image classification. Image classification module provides minimum of GUI actions in order to fully prepare data set. In this post, we are going to create and train deep learning model for CIFAR-10 data set, and see how it easy to do that with ANNdotNET v1.2.

In order to prepare data we have to download CIFAR-10 data set from official web site . The CIFAR-10 data set is provided in 6 binary batch files that should be extracted and persisted on your local machine. Number 10 in the name means that data set is created for 10 labels.The following image shows 10 labels of CIFAR-10 data set each label with few sample images.

The data set contains 60 000 (50 000 for training and validation, and 10 000 for test) tinny colored images dimensions of 32x32. There is also bigger version of the data set CIFAR-100 with 100 labels. Our task is to create deep learning model capable of recognizing only one of 10 predefined labels from each image.

Data preparation

In order to prepare images, we need to do the following:

• download compressed file (http://www.cs.toronto.edu/~kriz/cifar-10-binary.tar.gz)
• decompress the file and  store each image in 10 different folders that are represent image label.
• each folder contains images for one label.

The following image shows extracted data set persisted in 10 label folders. The bird folder is opened and shows all images labeled for bird. The test folder contains all images created for testing the model once the model is trained.

In order to properly save all images, we need to create simple C# Console application which should extract and save all 60 000 images. Complete C# program can be downloaded from here.

In order to successfully extract the images, we have to see how those images are stored in binary files. From the official site we can see that there are 5 for training and 1 for test binary files: data_batch_1.bin, data_batch_2.bin, …, data_batch_5.bin, as well as test_batch.bin. Each of these files is formatted as follows so that the first byte of the array is label index, and the next 3072 represent the image. Each batch contains 10000 images.

Important to know is that images are stored in CHW format which means that 1d image array is created so that the first 1024 bytes are the red channel values, the next 1024 the green, and the final 1024 the blue. The values are stored in row-major order, so the first 32 bytes are the red channel values of the first row of the image. So all those information have been carried out when implementing the Extractor application. The most important methods are reshaping the 1D byte array into [3, height, width] image tensor, and creating the image from the byte tensor. The following implementation shows how 1D byte array transformed into 3channel bitmap tensor.

static int[][][] reshape(int channel, int height, int width,  byte[] img)
{
    var data = new int[channel][][];
    int counter = 0;
    for(int c = 0; c < channel; c++)
    {
        data[c] = new int[height][];
        for (int y = 0; y < height; y++)
        {
            data[c][y] = new int[width];
            for (int x = 0; x < width; x++)
            {
                data[c][y][x] = img[counter];
                counter++;
            }
        }
    }
    return data;
}

Once the 1D byte array is transformed into tensor, the image can be created and persisted on disk. The following method iterates through all 10000 images in one batch extract them and persist on disk.

public static void extractandSave(byte[] batch, string destImgFolder, ref int imgCounter)
{
    var nStep = 3073;//1 for label and 3072 for image
    //
    for (int i = 0; i < batch.Length; i += nStep)
    {
        var l = (int)batch[i];
        var img = new ArraySegment<byte>(batch, i + 1, nStep - 1).ToArray();
// data in CIFAR-10 dataset is in CHW format, which means CHW: RR...R, GG..G, BB..B;

        // while HWC: RGB, RGB, ... RGB
        var reshaped = reshape(3, 32, 32, img);
        var image = ArrayToImg(reshaped);
        //check if folder exist
        var currentFolder = destImgFolder + classNames[l];

        if (!Directory.Exists(currentFolder))
            Directory.CreateDirectory(currentFolder);

        //save image to specified folder
        image.Save(currentFolder + "\\" + imgCounter.ToString() + ".png");

        imgCounter++;
   }
}

Now that we have 60000 tiny images on disk arranged by label we can start creating deep learning model.

Create new Image Classification annproject file in ANNdotNET

Open the latest ANNdotNET v1.1 and select New-> Image Classification project. Enter CIFAR project name and press save button. The following image shows CIFAR new ann-project:

Once we have new project, we can start defining image labels by pressing Add button. For each 10 labels we need to add new label item in the list. In each item the following fields should be defined:

• Image label
• Path to images with the label.
• Query - in case we need to get all images within the specified path with certain part of the name. In case all images withing the specified path are images that indicate one label, query should be empty string.

Beside Label item, image transformation should be defined in order to define the size of the images, as well as how many images create validation/test data set.

Assuming the CIFAR-10 data set is extracted at c:/data folder, the following image shows how label items should be defined:

In case label item should be removed from the list, this is done by selecting the item, and then pressing Remove button. Beside image properties, we should defined how many images belong to validation data set. As can be seen 20% of all extracted images will be created validation data set. Notice that images from the test folder are not part of those two data set. they will be used for testing phase once the model is trained. Now that we done with data preparation we can move to the next step: creating mlconifg file.

Create mlconfig in ANNdotNET

By selecting New MLConfig command the new mlconfig file is created within the project explorer. Moreover by pressing F2 key on selected mlconfig tree item, we can easily change the name into "CIRAF-10-ConvNet". The reason why we gave such name is because we are going to use convolution neural network in order to build model.

In order to define mlconfig file we need to define the following:

• Network configuration using Visual Network Designer
• Define Learning parameters
• Define training parameters

Create Network configuration

By using Visual Network Designer we can quickly create network model. For this CIFAR-10 dataset we are going to create 9 layers model with two Convolutional, 2 Pooling, 2 DropOut and 2 Dense layer, all followed by Scale layer:

Scale (1/255)->Conv2D(32,[3,3])->Pooling2d([2,2],2)->DropOut(0.2)->Conv2D(64,[3,3])->Pooling2d([2,2],2)->DropOut(0.2)->Dense(512, ReLU)->Dense(10,Softmax)

This network can be created so that selecting appropriate layer from the VND combo box and clicking on the Add button. The first layer is Scale layer, since we need to normalize input values to be in interval (0,1). Then we created two sequence of Convolution, Pooling and Drop out layers. Once we done with that, we can add one Dense layer with 512 neurons and ReLU activation function. The last layer is output layer that must follow the output dimension, and Softmax activation function.

Once network model is defined, we can move to the next step: Setting learning and training parameters.

Learning parameters can be defined through the Learning parameters interface: For this model we can select:

• AdamLearner with 0.005 rate and 0.9 momentum value. Loss function is Classification Error, and the evaluation function is Classification Accuracy

In order to define the training parameters we switch to Training tabe page and setup:

• Number of epoch
• Minibatch size
• Progress frequency
• Randomize minibatch during training
  

Now we have enough information to start training model. The training process is started by selecting Run command from the application ribbon. In order to get good model we need to train the model at least few thousands epoch. The following image shows trained model with training history charts.

The model is trained with exactly of 1144 epoch, with network parameters mentioned above. As can be seen from the upper chart, minibatch loss function was CrossEntropyWithSoftmax, while the evaluation was classification accuracy.  The bottom chart shows performance of the training and validation data sets for each 1144 epoch. We can also recognize that validation data set has roughly the same accuracy as training data set which indicates the model is trained well.  More details about model performance can be seen on the next image:

Upper charts of the image above show actual and predicted values for training (left) and validation (right). Most of the point values are blue and overlap the orange which indicates that most of value are correctly predicted. The charts can be zoomed and view details of each value.The bottom part of the evaluation show performance parameters of the model for corresponded data set. As can be seen the trained model has 0.989 accuracy for training data set and 0.986 accuracy for validation data set, which indicate high accuracy of the model. Moreover the next two images shows confusion matrix for the both data set, which in details shows how model predict all 10 labels.

The last part of the post is testing model for test data set. For that purpose we selected 10 random images from each label of the test set, and evaluate the model. The following images shows the model correctly predicted all 10 images.

Conclusion

ANNdotNET v1.2 image classification module offers complete data preparation and model development for image classification. The user can prepare data for training, create network model with Neural Network Designer, and perform set of statistical tools against trained model in order to validate and evaluate model. The important note is that the data set of images must be stored on specific location in order to use this trained model shown in the blog post. The trained model, as well as mlcofig files, can be load directly into ANNdotNET project explorer by doublick on CIFAR-10.zip feed example.

In this blog post, step by step instruction is going to be described in order to prepare clean Windows based machine (virtual) with GPU for deep learning with CNTK, Tensorflow and Keras. Installation of OS is not covered in the post, and this is task is assumed it is already completed.

Preparing the machine

Once you have up and running clean Windows machine, there are several things you should concider:
1. Physical machine with NVIDIA compatible graphics card.
This requirement will provide deep learning frameworks to train models on GPU, which speedups the training process rapidly.

2. Virtual Machine with GPU.
In case you plan to prepare virtual machine, or Azure virtual machine, be aware that (for my knowledge) only Windows Server 2016 based virtual machine recognize GPU card. So if you install Windows 10 or lower version on virtual machine, you will not be able to use GPU for training deep learning models.

3. Azure N-Series VM
In case you plan to select one of Azure virtual machine, only N-series support GPU.

Installation for NVIDIA driver and related stuff

In this blog post only NVIDIA related driver will be described, and no other installation driver will be considered. In case of other driver installation, please refer to related vendor site.

For this blog post, drivers and related stuff for NVIDIA Tesla K80 graphics card will be explained. For other NVIDIA cards the installation process is almost the same.

1. First you have to know what NVIDIA graphics card is installed your machine.

2. Then go to NVIDIA official site, and select appropriate information before driver download. In my case the following information are selected:

3. Press search and download the driver.

Once you download the driver, install it on your machine.

Once you have driver installed, you have to download and install two more NVIDIA software components:

1. CUDA Toolkit 9.0
2. cuDNN 7.4

Those two software components are used by deep learning frameworks (CNTK and TensorFlow) for GPU based computation.
The CUDA 9.0 is compatible with the latest versions of CNTK 2.6 and Tensorflow 1.12, so it makes easier to used one CUDA version for both frameworks, which was not the case in the past.

Installation of CUDA 9.0

In order to install CUDA Toolkit, go to CUDA download page and select appropriate information of your machine. The following information I have selected in order to download it:

Once you select the right information, press download button. Once the CUDA 9.0 is downloaded on you machine install it by performing Express installation option.

Installation of cuDNN 7.4

Download the cuDNN from the official site, and then press Download cuDNN button.

Once you press it, the following page should appear. Notice also notice that login page might appear before download page.

Once the cuDNN is downloaded unzip it. Only three files are exist in the installation, and those should be copied on the right place. In order to successfully install cuDNN, perform the following files copy:

1. cudnn64_7.dll to C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\bin

2. cudnn.lib to C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\lib\x64

3. cudnn.h to C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\include

Once you've done that, the NVIDIA related stuff installation has been completed, and you can switch to installation Python related stuff.

Installation Python development environment

CNTK and TensorFlow support various python environments, but you should always see official site for the compatibility. In order to use CNTK and Tensor flow on the same python environment, it is recommended to use Anaconda3 version 4.1.1 environment.

First download the Anaconda3 v4.1.1 from the official site:

- Go to http://repo.continuum.io/archive/ and download Anacoda3 4.1.1 for 64 bit.

Once the Anaconda is downloaded install it, on standard way using installer.

Prepare python environment for the installation

Once Anaconda3 4.1.1 has been installed several commands needs to be performed in order to install all necessary software. Before start, we need to upgrade pip since Anaconda3 4.1.1 is little bit old. So run the Anaconda Command Prompt from the Start->Anacoda->Anaconda Prompt

Once the Anaconda Prompt is running, type the following command:
python -m pip install --upgrade pip

Now we are ready to install CNTK, Tensorflow and Keras. But before that we should create separate python environment with python 3.5. Once we have the environment we can install those frameworks to it. The new environment must be relies on python 3.5. So type the following command into Anaconda Prompt:
conda create --name mlenv1218 python=3.5
We have created environment named "mlenv1218".

Now, don't forget to activate the environment before installing software. Type the following commands in order to activate environment.
activate mlenv1218
Once we've done that, the Anaconda Prompt should looks like (active environment is shown on the left site):

Installation of CNTK, Tensorflow and Keras

It is very important to properly install NVIDIA related stuff, before installation of deep learning libraries, because most of the installation problems are related to it. Once we have NVIDIA and Python environment installed properly, the installation process for deep learning frameworks is very easy.
In Anaconda Prompt, with activate "mlenv2118" environment, type the following command in order to install CNTK: pip install cntk-gpu

The type the following python code to test CNTK installation:
python -c "import cntk; print(cntk.version)"

Once you've done that, type the following command in order to install Tensorflow:
pip install tensorflow-gpu

Type the following command in order to test installation:
python -c "import tensorflow as tf; tf.enable_eager_execution(); print(tf.reduce_sum(tf.random_normal([1000, 1000])))"

At the end type the following command to install Keras:

pip install keras

In addition it is useful to install the following packages:

• matplotlib
• opencv

Those packages are installed using:

pip install matplotlib, and
pip install python-opencv

That is all to install in order to run CNTK, TensorFlow and Keras.

Install Visual Studio Code to write python code

In order to write python code for deep learning you have two options among many other:

• Install Visual Studio 2017
• Install Visual Studio Code

Visual Studio Code can be downloaded from official site. Download it and install. Once you install the VS Code, run it. Press Extension button on the lest side and type python in search box. Select on Python extension and press Install.

• Restart VS Code, and
• Select File->New File
• Save file as python_test.py
• Change current python environment to "mlenv1218" (by double click)
• Run python code, by right click on python code and select "Run Python File in Terminal"

import cntk import tensorflow as tf import keras print("CNTK:",cntk.__version__) print("TensorFlow:",tf.__version__) print("Keras:",keras.__version__)

the following output produced the above code:

When building deep learning models, it is often required to check the model for consistency and proper parameters definition. In ANNdotNET, ml network models are designed using Visual Network Designer (VND), so it is easy to see the network configuration. Beside VND, in ANNdotNET there are several visualization features on different level: network preparation, model training phase, post training evaluation, performance analysis, and export results. In this blog post we will learn how to use those features when working with deep learning models.

Visualization during network preparation and model training

When preparing network and training parameters, we need information about data sets, input format and output type. This information is relevant for selecting what type of network model to configure, what types of layers we will use, and what learner to select. For example the flowing image shows  network configuration containing of 2 embedding layers, 3 dense layers and 2 dropout layers. This network configuration is used to train CNTK model for mushroom data set. As can be seen network layers are arranged as listbox items, and the user has possibility to see, on the highest level, how neural networks looks like, which layers are included in the network, and how many dimensions each layer is defined. This is very helpful, since it provides the way of building network very quickly and accurately, and it requires much less times in comparisons to use traditional way of coding the network in python, or other programming language.

ANNdotNET Network Settings page provides pretty much information about the network, input and output layers, what data set are defined, as well as whole network configuration arranged in layers. Beside network related information, the Network Settings tab page also provides the learning parameters for the network training. More about Visual Network Designer the ready can find on one of the previous blog post.

Since ANNdotNET implements MLEngine which is based on CNTK, so all CNTK related visualization features could be used. The CNTK library provides rich set of visualizations. For example you can use Tensorboard in CNTK  for visualization not just computational graph, but also training history, model evaluation etc. Beside Tensorboard, CNTK provides logger module which uses Graphviz tool for visualizing network graph.The bad news of this is that all above features cannot be run on C#, since those implementation are available only in python.

This is one of the main reason why ANNdotNET provides rich set of visualizations for .NET platform. This includes: training history, model evaluation for training and validation data set, as well as model performance analysis. The following image show some of the visualization features: the training history (loss and evaluation) of minibatches during training of mushroom model:

Moreover, the following image shows evaluation of training and validation set for each iteration during training:

The graphs are generated during training phase, so the user can see what is happening with the model.  This is of tremendous help, when deciding when to stop the training process, or are training parameters produce good model at all, or this can be helpful in case when can stop and change parameters values. In case we need to stop the training process immediately, ANNdotNET provides Stop command which stops training process at any time.

Model performance visualization

Once the model is trained, ANNdotNET provides performance analysis tool for all three types of ML problems: regression, binary and multi class classification.

Since the mushrooms project is binary ML problem the following image shows the performance of the trained model:

Using Graphviz to visualize CNTK network graph in C#

We have seen that ANNdotNET provides all types of visualizations CNTK models, and those features are provided by mouse click through the GUI interfaces. One more feature are coming to ANNdotNET v1.1 which uses Grpahviz to visualize CNTK network graph. The feature is implemented based on original CNTK python implementation with some modification and style.

In order to use Graphviz to visualize network computation graph the following requirements must be met:

• Install Graphviz on you machine.
• Register Graphviz path as system variable. (See image below)
  

Now that you have install Graphviz tool, you can generate nice image of your network model directly in ANNdotNET just by click on Graph button above the Visual Network Designer (see image 1).

Here is some of nice graphs which can be generate from ANNdotNET preclaculated models.

In case you like this nice visualization features go to http://github.com/bhrnjica/anndotnet, download the latest version from release section or just download the source code and try it with Visual Studio, but don't forget to give a star.

In the next blog post I will show you how visualization of CNTK
computational graph is implemented, so you will be able to use it in
your custom solutions.

The October 2018 issue of MSDN magazine brings the article "Sentiment Analysis Using CNTK" written by James McCaffrey. I was wondering if I can implement this solution in ANNdotNET as Dr. McCaffrey written in the magazine. Indeed I have implemented complete solution in less than 5 minutes.

In this blog post I am going to walk you through this very good and well written MSDN article example. I am not going to repeat the text written in the MSDN article, so it is recommendation to read the article first, and back here and implement the example in ANNdotNET. Since the ANNdotNET is GUI tool, it is interesting to see all great visualizations during the model training and evaluation. Also the ANNdotNET provides complete binary model evaluation by providing the confusion matrix, ROC Curve, and other binary performance parameters, this example makes more interesting and valuable to read.

Whole example is implemented in five steps.

Step 1: Prepare files and folder structure

First we need to create several folders and files in order to create empty annproject. This manual creation of folders are necessary because ANNdotNET v1.0 has not option to create Empty project. This will be added in the next version.

So first, create the following set of hierarchically ordered folders:

SentimentAnalysis
    MoveReview
        data

The following figure shows this set of folder.

Only thing we need from the MSDN article is train and test data sets. The data can be downloaded from the MSDN sample: Code_McCaffreyTestRun1018.zip. Once the zip file is downloaded unzip the sample, and copy files: imdb_sparse_train_50w.txt and indb_sparse_test_50w.txt to data folder as image above shows.

Step 3: Create MoviewReview.ann and LSTM-Net.mlconfig files

Open Notepad and create file with the following content:

project:|Name:MovieReview |Type:NoRawData |MLConfigs:LSTM-Net
data:|RawData:MovieReview_rawdata.txt
parser:|RowSeparator:rn |ColumnSeparator: ; |Header:0 |SkipLines:0

Save file in SentimenAnalysis folder as MovieReview.ann. The following picture shows saved annproject file on disk.

Now open Notepad again, create a new empty file. The empty file is supposed to be mlconfig file with the content shown below. Don't worry about the content of the file, since all those details will be visible once we open it with ANNdotNET. If you want to know more about structure of the mlconfig file, please refer to this wiki page of the ANNdotNET project.

configid:msdn-oct-2018-issue-sentiment-analysis-article
metadata:|Column02:y;Category;Label;Random;0;1
features:|x 129892 1
labels:|y 2 0
network:|Layer:Embedding 50 0 0 None 0 0 |Layer:LSTM 25 25 0 TanH 1 1 |Layer:Dense 2 0 0 Softmax 0 0
learning:|Type:AdamLearner |LRate:0.01 |Momentum:0.85 |Loss:CrossEntropyWithSoftmax |Eval:ClassificationAccuracy |L1:0 |L2:0
training:|Type:Default |BatchSize:250 |Epochs:400 |Normalization:0 |RandomizeBatch:0 |SaveWhileTraining:0 |FullTrainingSetEval:1 |ProgressFrequency:1 |ContinueTraining:0 |TrainedModel:
paths:|Training:data\imdb_sparse_train_50w.txt |Validation:data\imdb_sparse_test_50w.txt |Test:data\imdb_sparse_test_50w.txt |TempModels:temp_models |Models:models |Result:LSTM-Net_result.csv |Logs:log

The file should be saved in the MovieReview folder with LSTM-Net.mlconfig file name. The next image shows where mlconfig file is stored.

Now we have setup everything in order to open and train sentiment analysis example with ANNdotNET. Since ANNdotNET implements MLEngine which is based on CNTK, data sets are compatible and can be read by the trainer. In order to get better result we have changed learning parameter a little bit. Instead of SGD we used AdamLearner.

In case you don't have ANNdotNET tool installed on your machine, just go to release section and download the latest version. Or clone the GitHub repository and run it within the Visual Studio. All information about how to run ANNdotNET as standalone application or as the Visual Studio solution can be found at GitHub page https://github.com/bhrnjica/anndotnet.

After simple unzipping binaries of the ANNdotNET on your machine, run it by simply selecting anndotnet.wnd.exe file. Once the ANNdotNET is running, click the Open application command and select the MoveReview.ann file. In a second the application loads the project with corresponded mlconfig file. From the project explorer, click on LSTM-NET three item, and similar content as image below should be appeared.

Everything we have written into mlconfig file are now shown in the Network settings tab page.

Input layer with 129892 dimensions
Output layer with 2 dimension (binary problem)
Learning parameters:
    AdamLearner, with 0.01 lr and 0.85 momentum,
    Loss Function is CrossEntropywithSoftmax
    Evaluation function is ClassificationAccuracy
NNetwork Designer shows typical LSTM recurrent network

Step 5. Training and Evaluation of the Example

Now that we reviewed the network settings, we can switch to the train tab page, and review the training parameters. Since we already setup training parameters in the mlconfig file, we don't need to change anything.

Start training process by click on the Run application command. After some time we should see the following result:

If we switch to Evaluation page we can perform some statistics analysis in order to evaluate if the model is good or not. Once the evaluation tab page is shown, click on Refresh button to evaluate the model against training and validation data stets.

The left statistics are for the training dataset, and the left side is for the validation data set. As can be seen, the model perfectly predicted all data from the training data set, and about 70% of accuracy described the validation data set. Off cource, the model is not good as we expected for the production, but for this demonstration is good enough. There are also two buttons to show ROC curve, and other binary performance parameters, for both data sets, which the reader my taste.

That's all needed in order to have complete Sentiment Analysis exemple setup and running. In case you want complete ANNdotNET project, it can be downloaded from here.

ANNdotNET

ANNdotNET is windows desktop application written in C# for creating and training ANN models. The application relies on Microsoft Cognitive Toolkit, CNTK, and it is supposed to be GUI tool for CNTK library with extensions in data preprocessing, model evaluation and exporting capabilities. It is hosted at http://github.com/bhrnjica/anndotnet

Currently supported Network Types of:

• Simple Feed Forward NN
• Deep Feed Forward NN
• Recurrent NN with LSTM

The process of creating, training, evaluating and exporting models is provided from the GUI Application and does not require knowledge for supported programming languages. The ANNdotNET is ideal for engineers which are not familiar with programming languages.

Software Requirements

ANNdotNET is x64 Windows desktop application which is running on .NET Framework 4.7.1. In order to run the application, the following requirements must be met:

• Windows 7, 8 or 10 with x64 architecture
• .NET Framework 4.7.1

How to run application

In order to run the application there are two possibilities:

1. Clone the GitHub repository and open it in Visual Studio 2017. Change build architecture into x64, build and run the application.
2. Download released version unzip and run ANNdotNET.exe.

The following three short videos quickly show how to create, train and evaluate reression, binary and multiclass classification models.

Regression model
Binary classification model
Multiclass classification model

More info at https://bhrnjica.net/anndotnet/

Regardless of machine learning library you use, the data preparation is the first and one of the most important step in developing predictive models. It is very often case that the data supposed to be used for the training is dirty with lot of unnecessary columns, full of missing values, un-formatted numbers etc. Before training the data must be cleaned and properly defined in order to get good model. This is known as data preparation. The data preparation consist of cleaning the data, defining features and labels, deriving the new features from the existing data, handling missing values, scaling the data etc. It can be concluded that the total time we spend in ML modelling,the most of it is related to data preparation.

In this blog post I am going to present the simple tool which can significantly reduce the preparation time for ML. The tool simply loads the data in to GUI, and then the user can define all necessary information. Once the data is prepared user can store the data it to files which can be then directly imported into ML algorithm such as CNTK.
The following image shows the ML Data Preparation Tool main window.

From the image above, the data preparation can be achieved in several steps.
From the image above, the data preparation can be achieved in several steps.

• Load dirty data into ML Prep Tool, by pressing Import Data button
• Transform the data by providing the flowing table:

• Defines the testing data set size by providing information of row numbers or percent.
  • Defines export options,
  • Press Export Button.

As can be seen this is straightforward workflow of data preparation.

Besides the general export options which can be achieved by selecting different delimiter options, you can export data set in to CNTK format, which is very handy if you play with CNTK.

After data transformations, the user need to check CNTK format in the export options group box and press Export button in order to get CNTK training and testing files, which can be directly used in the code without any modifications.

Some of examples will be provided in the next blog post.

The project is hosted at GitHub, where the source code can be freely downloaded and used at this location:https://github.com/bhrnjica/MLDataPreparationTool.

In case you want only binaries, the release of version v1.0 is published here: https://github.com/bhrnjica/MLDataPreparationTool/releases/tag/v1.0.

In this blog post we are going to implement training and evaluation ANN model based on Iris data set using CNTK and Python. The Iris data set has categorical output value which contains three classes : Sentosa, Virglica and Versicolor. The features consist of the 4 real value inputs. The Iris data set can be easily found on the internet. One of the places is on http://kaggle.com

Usually, the Iris data is given in the flowing format:

Since we are going to use CNTK we should prepare the data in cntk file format, which is far from the format we can see on the previous image. This format has different structure and looks like on the flowing image:

The difference is obvious. To transform the previous file format in to the cntk format it tooks me several minutes and now we can continue with the implementation.

First, lets implement simple python function to read the cntk format. For the implementation we are going to use CNTK MinibatchSource, which is specially developed to handle file data. The flowing python code reads the file and return the MinibatchSource.

import cntk

# The data in the file must satisfied the following format:
# |labels 0 0 1 |features 2.1 7.0 2.2 - the format consist of 4 features and one 3 component hot vector
# represents the iris flowers
def create_reader(path, is_training, input_dim, num_label_classes):

# create the streams separately for the label and for the features
labelStream = cntk.io.StreamDef(field='label', shape=num_label_classes, is_sparse=False)
featureStream = cntk.io.StreamDef(field='features', shape=input_dim, is_sparse=False)

# create deserializer by providing the file path, and related streams
deserailizer = cntk.io.CTFDeserializer(path, cntk.io.StreamDefs(labels = labelStream, features = featureStream))

# create mini batch source as function return
mb = cntk.io.MinibatchSource(deserailizer, randomize = is_training, max_sweeps = cntk.io.INFINITELY_REPEAT if is_training else 1)
return mb

The code above take several arguments:

-path - the file path where the data is stored,

-is_training - Boolean variable which indicates if the data is for training or testing. In case of training the data will be randomized.

• input_dim, num_label_classes are the numbers of the input features and the output hot vector size. Those two arguments are important in order to properly parse the file.

The first method creates the two streams , which are passed as argument in order to create deserializer, and then for minibatchsource creation. The function returns minibatchsource object which the trainer uses for data handling.

Once that we implemented the data reader, we need the python function for model creation. For the Iris data set we are going to create 4-50-3 feed forward neural network, which consist of one input layer with 4 neurons, one hidden layer with 50 neurons and the output layer with 4 neurons. The hidden layer will contain tanh- activation function.

The function which creates the NN model will looks like on the flowing code snippet:

#model creation
# FFNN with one input, one hidden and one output layer 
def create_model(features, hid_dim, out_dim):
    #perform some initialization 
    with cntk.layers.default_options(init = cntk.glorot_uniform()):
        #hidden layer with hid_def number of neurons and tanh activation function
        h1=cntk.layers.Dense(hid_dim, activation= cntk.ops.tanh, name='hidLayer')(features)
        #output layer with out_dim neurons
        o = cntk.layers.Dense(out_dim, activation = None)(h1)
        return o

As can be seen Dense function creates the layer where the user has to specify the dimension of the layer, activation function and the input variable. When the hidden layer is created, input variable is set to the input data. The output layer is created for the hidden layer as input.

The one more helper function would be showing the progress of the learner. The flowing function takes the three arguments and prints the current status of the trainer.

# Function that prints the training progress
def print_training_progress(trainer, mb, frequency):
    training_loss = "NA"
    eval_error = "NA"

    if mb%frequency == 0:
        training_loss = trainer.previous_minibatch_loss_average
        eval_error = trainer.previous_minibatch_evaluation_average
        print ("Minibatch: {0}, Loss: {1:.4f}, Error: {2:.2f}%".format(mb, training_loss, eval_error*100))   
    return mb, training_loss, eval_error

Once we implemented all three functions we can start with CNTK learning on the Iris data.

At the beginning, we have to specify some helper variable which we will use later.

#setting up the NN type
input_dim=4
hidden_dim = 50
num_output_classes=3
input = cntk.input_variable(input_dim)
label = cntk.input_variable(num_output_classes)

Create the reader for data batching.

# Create the reader to training data set
reader_train= create_reader("C:/sc/Offline/trainData_cntk.txt",True,input_dim, num_output_classes)

Then create the NN model, with Loss and Error functions:

#Create model and Loss and Error function
z= create_model(input, hidden_dim,num_output_classes);
loss = cntk.cross_entropy_with_softmax(z, label)
label_error = cntk.classification_error(z, label)

Then we defined how look like the trainer. The trainer will be with Stochastic Gradient Decadent learner, with learning rate of 0.2

# Instantiate the trainer object to drive the model training
learning_rate = 0.2
lr_schedule = cntk.learning_parameter_schedule(learning_rate)
learner = cntk.sgd(z.parameters, lr_schedule)
trainer = cntk.Trainer(z, (loss, label_error), [learner])

Now we need to defined parameters for learning, and showing results.

# Initialize the parameters for the trainer
minibatch_size = 120 #mini batch size will be full data set
num_iterations = 20 #number of iterations 

# Map the data streams to the input and labels.
input_map = {
label  : reader_train.streams.labels,
input  : reader_train.streams.features
} 
# Run the trainer on and perform model training
training_progress_output_freq = 1

plotdata = {"batchsize":[], "loss":[], "error":[]}

As can be seen the batchsize is set to dataset size which is typical for small data sets. Since we defined minibach to dataset size, the iteration should be very small value since Iris data is very simple and the learner will find good result very fast.

Running the trainer looks very simple. For each iteration, the reader load the batch size amount of the data, and pass to the trainer. The trainer performs the learning process using SGD learner, and returns the Loss and the error value for the current iteration. Then we call print function to show the progress of the trainer.

for i in range(0, int(num_iterations)):
        # Read a mini batch from the training data file
        data=reader_train.next_minibatch(minibatch_size, input_map=input_map) 
        trainer.train_minibatch(data)
        batchsize, loss, error = print_training_progress(trainer, i, training_progress_output_freq)
        if not (loss == "NA" or error =="NA"):
            plotdata["batchsize"].append(batchsize)
            plotdata["loss"].append(loss)
            plotdata["error"].append(error)

Once the learning process completes, we can perform some result presentation.

# Plot the training loss and the training error
import matplotlib.pyplot as plt

plt.figure(1)
plt.subplot(211)
plt.plot(plotdata["batchsize"], plotdata["loss"], 'b--')
plt.xlabel('Minibatch number')
plt.ylabel('Loss')
plt.title('Minibatch run vs. Training loss')

plt.show()

plt.subplot(212)
plt.plot(plotdata["batchsize"], plotdata["error"], 'r--')
plt.xlabel('Minibatch number')
plt.ylabel('Label Prediction Error')
plt.title('Minibatch run vs. Label Prediction Error')
plt.show()

We plot the Loss and Error function converted in to total accuracy of the classifier. The folowing pictures shows those graphs.

The last part of the ML procedure is the testing or validating the model. FOr the Iris data set we prepare 20 samples which will be used for the testing. The code i similar to the previous, except we call create_reader with different file name. Then we try to evaluate the model and grab the Loss and error values, and print out.

# Read the training data
reader_test = create_reader("C:/sc/Offline/testData_cntk.txt",False, input_dim, num_output_classes)

test_input_map = {
    label  : reader_test.streams.labels,
    input  : reader_test.streams.features,
}

# Test data for trained model
test_minibatch_size = 20
num_samples = 20
num_minibatches_to_test = num_samples // test_minibatch_size
test_result = 0.0

for i in range(num_minibatches_to_test):
    
    data = reader_test.next_minibatch(test_minibatch_size,input_map = test_input_map)
    eval_error = trainer.test_minibatch(data)
    test_result = test_result + eval_error

# Average of evaluation errors of all test minibatches
print("Average test error: {0:.2f}%".format(test_result*100 / num_minibatches_to_test))

Full sample with python code and data set can be found here.

Introduction

1. N64 Nintendo emulator which can be found here,
2. Mario Kart 64 ROM, which can be found on internet as well,
3. CNTK - Microsoft Cognitive Toolkit
4. .NET Framework and C#

Also the N64 emulator is positioned to Top-Left corned of screen, so it is easier to capture the images.

Data collection for training data set

In .NET and C# it is implemented image capture from the specific position of screen, as well as it is recorded which keys are pressed during game play. In order to record keys press, the code found here is modified and used.

The flowing image shows the position of N64 emulator with playing Mario Kart game (1), the windows which is capture and transform the image (2), and the application which collect images, and key press action and generated training data set into file(3).

The data is generated on the following way:

• each image is captured, resized to 100x74 pixels and gray scaled prior to be transformed and persisted to data set training file.
• before image is persisted the hotkey of action key press is recorded and connected to image.

1. |label - which represent 5 component hot vector, indicate: Forward, Break, Forward-Left, Forward-Right and None (1 0 0 0 0)
2. |features consist of 100x74 numbers which represent pixels of the images.

|label 1 0 0 0 0 |features 202 202 202 202 202 202 204 189 234 209 199...
|label 0 1 0 0 0 |features 201 201 201 201 201 201 201 201 203 18...
|label 0 0 1 0 0 |features 199 199 199 199 199 199 199 199 199 19...
|label 0 0 0 1 1 |features 199 199 199 199 199 199 199 199 199 19...

 await GenerateData.Start();

Training Model to play the game

By running the following code, the training process is started with provided training data:

 CNTKDeepNN.Train(DeviceDescriptor.GPUDevice(0));

Playing the game with CNTK model

var dev = DeviceDescriptor.CPUDevice;
MarioKartPlay.LoadModel("../../../../training/mario_kart_modelv1", dev);
MarioKartPlay.PlayGame(dev);

How it looks like on my case, you can see on this youtube video: