.NET: CNN v1.0 for Supervised Deep Learning Example

See the CNN version 2.0 which uses backpropagation to create filters, a.k.a. kernels. The CNN on this page is not deprecated. It learns quickly, but for more difficult patterns that must be learned, try version 2.0.

Supervised learning is a machine learning paradigm for problems where the available data consists of labeled examples, meaning that each data point contains features (covariates) and an associated label. The goal of supervised learning algorithms is learning a function that maps feature vectors (inputs) to labels (output), based on example input-output pairs.

See unsupervised learning version, or MNIST number recognition tool, which was trained to over 98.5% using this CNN.

Learn about convolutional neural networks.

This code is built upon the neural network C# example (CceNeuralNetwork.cs) which is used for the fully connected layers. It implements both Categorical Cross-Entropy (CCE) and Sparse Categorical Cross-Entropy (SCCE).

This neural network code is for the fully connected hidden layers is available in a C++ library.

Download these files including example training and the MNIST image files with labels: NEURALNETWORK.zip. Experiment with the number and type of CNN filters (a.k.a. kernels). This example usage code requires SixLabors.ImageSharp (NuGet package).

About CNNs

About CNN Kernels/Filters

Choosing Kernels/Filters

Max Pooling

Filter/Kernel Effects

While not required by the network, the decision was made to convert the images to black and white. Filters applied to color images look quite different because they are applied to each color channel. Linear activation seems to have the most data, but it may not be what works best for all networks as ReLU activation does chop a lot of the noise out.

Dog - Linear Activation

Identity

BoxBlur

BoxBlurLarge

HorizontalEdge

VerticalEdge

DiagonalEdgeTLtoBR

DiagonalEdgeTRtoBL

Gaussian

GaussianLarge

Laplacian

LaplacianLarge

Sharpen

SharpenLarge

SobelX

SobelY

GaussianLaplacian

Dog - ReLU Activation

Identity

BoxBlur

BoxBlurLarge

HorizontalEdge

VerticalEdge

DiagonalEdgeTLtoBR

DiagonalEdgeTRtoBL

Gaussian

GaussianLarge

Laplacian

LaplacianLarge

Sharpen

SharpenLarge

SobelX

SobelY

GaussianLaplacian

Filter Visualizer

Use the filter visualizer to help choose the right filter.

Fully Connected/Hidden Layers

This CNN is useless without "fully connected" layers a.k.a. hidden layers. The neural network code for the fully connected layers can be found here in C# and here in C++.

The convolutional neural network code:

// Sobel Filter
// The Sobel filter is used to detect edges in images based on the first-order derivatives in the horizontal
// and vertical directions. It emphasizes edges at points where the gradient of the intensity is maximum.
// How it Works
// Sobel Operator: The Sobel operator uses two separate 3x3 matrices (kernels), one for detecting changes in
// horizontal gradients (Sobel X) and one for vertical gradients (Sobel Y). The operator applies these
// matrices to the image to calculate the approximate derivatives.

// Laplacian Filter
// The Laplacian filter is a second-order derivative filter, used extensively in image processing to detect
// regions of rapid intensity change and is especially sensitive to noise.
// How It Works
// Laplacian Operator: This operator calculates the second derivative of the image, providing a measurement
// of the rate of change in gradients. Unlike the Sobel filter, which calculates the gradient in specific
// directions, the Laplacian detects edges in all directions.

// Comparison and Usage
// Noise Sensitivity: The Laplacian filter is more sensitive to noise compared to the Sobel filter. Therefore,
// it is often used in combination with Gaussian smoothing (as Laplacian of Gaussian, or LoG) to reduce
// sensitivity to noise.
// Edge Detection: Sobel is generally preferred when direction of the edges matters, while Laplacian is
// suitable for scenarios where detailed edge detection is required regardless of the edge orientation.

// SUMMARY
// Both filters have their own strengths and are chosen based on the specific requirements of the application.
// In many real-world applications, these filters can be used in combination to enhance the performance of
// edge detection and other image processing tasks.


// ConvolutionalNeuralNetwork.cs
using System;

namespace ML
{
	namespace CNN1D
	{
		public class Kernel1D
		{
			public float[] filter;
			public float divisor;
			public Kernel1D(float[] filter, float divisor = 1)
			{
				this.filter = filter;
				this.divisor = divisor;
			}
		}
		public static class ConvolutionalNetwork
		{
			// for 1D CNNs
			private static readonly float[] twoGram1DFilter = new float[] { 1, 1 };
			private static readonly float[] threeGram1DFilter = new float[] { 1, 1, 1 };
			private static readonly float[] fourGram1DFilter = new float[] { 1, 1, 1, 1 };

			// for 1D CNNs
			public static Kernel1D TwoGram1DFilter = new Kernel1D(twoGram1DFilter);
			public static Kernel1D ThreeGram1DFilter = new Kernel1D(threeGram1DFilter);
			public static Kernel1D FourGram1DFilter = new Kernel1D(fourGram1DFilter);

			// for 1D CNNs
			public static void ApplyFilter(float[,] vectors, Kernel1D kernel, float[,] filtered, CNN.IActivation activation)
			{
				if (filtered.GetLength(0) != vectors.GetLength(0) || filtered.GetLength(1) != vectors.GetLength(1) - (kernel.filter.Length - 1))
					throw new ArgumentException($"The filtered and vectors arguments are not a match, or the wrong kernel is being applied.");

				for (int row = 0; row < vectors.GetLength(0); row++)
					for (int i = 0; i < filtered.GetLength(1); i++)
					{
						float result = 0;
						for (int j = 0; j < kernel.filter.Length; j++)
							result += vectors[row, i + j] * kernel.filter[j];
						filtered[row, i] = activation.Function(result / kernel.divisor);
					}
			}
			// for 1D CNNs
			public static void ApplyMaxOverTimePooling(float[,] filtered, float[,] pooled, int pooledColumn)
			{
				if (pooled.GetLength(0) != filtered.GetLength(0))
					throw new ArgumentException("The pooled argument must have a rows length equal to the rows length of the filtered argument.");

				float max;
				for (int row = 0; row < filtered.GetLength(0); row++)
				{
					max = filtered[row, 0];
					for (int col = 0; col < filtered.GetLength(1); col++)
						max = Math.Max(max, filtered[row, col]);
					pooled[row, pooledColumn] = max;
				}
			}
			// for 1D CNNs
			public static void FlattenPooled(float[,] pooled, float[] flattened, out float min, out float max)
			{
				if (flattened.Length < pooled.GetLength(0) * pooled.GetLength(1))
					throw new ArgumentException("The flattened argument must have a length greater than or equal to the pooled argument's dimensions multiplied.");

				min = max = pooled[0, 0];
				// Loop through the rows and columns of the image
				for (int row = 0; row < pooled.GetLength(0); row++)
					for (int column = 0; column < pooled.GetLength(1); column++)
					{
						var v = pooled[row, column];
						min = Math.Min(min, v);
						max = Math.Max(max, v);
						flattened[row * pooled.GetLength(1) + column] = v;
					}
				for (int i = pooled.GetLength(0) * pooled.GetLength(1); i < flattened.Length; i++)
					flattened[i] = 0;
			}

		}
	}
	namespace CNN2D
	{
		public class Kernel2D
		{
			public float[,] filter;
			public float divisor;
			public Kernel2D(float[,] filter, bool summationForDivisor = false) // summation for divisor is usually used by blur filters
			{
				this.filter = filter;
				if(summationForDivisor)
				{
					divisor = 0;
					int FILTER_2D_SIZE = filter.GetLength(0);
					for (int i = 0; i < FILTER_2D_SIZE; i++)
						for (int j = 0; j < FILTER_2D_SIZE; j++)
							divisor += filter[i, j];
				}
				else
					divisor = 1;
			}
		}
		public static class ConvolutionalNetwork
		{
			// for 2D CNNs
			public const int POOL_2D_SIZE = 2;
			private static readonly float[,] vertEdge2DFilter3x3 = new float[,] {
				{ -1, 0, 1 },
				{ -1, 0, 1 },
				{ -1, 0, 1 } };
			private static readonly float[,] horzEdge2DFilter3x3 = new float[,] {
				{ -1, -1, -1 },
				{ 0, 0, 0 },
				{ 1, 1, 1 } };
			private static readonly float[,] diagEdgeTopLeftToBottomRight2DFilter3x3 = new float[,] {
				{ 1, 0, -1 },
				{ 0, 0, 0 },
				{ -1, 0, 1 } };
			private static readonly float[,] diagEdgeTopRightToBottomLeft2DFilter3x3 = new float[,] {
				{ -1, 0, 1 },
				{ 0, 0, 0 },
				{ 1, 0, -1 } };
			private static readonly float[,] sobelX2DFilter3x3 = new float[,] {
				{ -1, 0, 1 },
				{ -2, 0, 2 },
				{ -1, 0, 1 } };
			private static readonly float[,] sobelY2DFilter3x3 = new float[,] {
				{ -1, -2, -1 },
				{ 0, 0, 0 },
				{ 1, 2, 1 } };
			private static readonly float[,] identity2DFilter3x3 = new float[,] { // does nothing
				{ 0, 0, 0 },
				{ 0, 1, 0 },
				{ 0, 0, 0 } };
			private static readonly float[,] laplacian2DFilter3x3 = new float[,] { // used for detecting edges and sharp intensity changes in an image, regardless of their direction
				{ -1, -1, -1 },
				{ -1, 8, -1 },
				{ -1, -1, -1 } };
			private static readonly float[,] sharpen2DFilter3x3 = new float[,] {
				{ 0, -1, 0 },
				{ -1, 5, -1 },
				{ 0, -1, 0 } };
			private static readonly float[,] gaussian2DFilter3x3 = new float[,] { // must apply a divisor of 16 (the sum of the filter elements)
				{ 1, 2, 1 },
				{ 2, 4, 2 },
				{ 1, 2, 1 } };
			private static readonly float[,] boxBlur2DFilter3x3 = new float[,] { // must apply a divisor of 9 (the sum of the filter elements)
				{ 1, 1, 1 },
				{ 1, 1, 1 },
				{ 1, 1, 1 } };
			private static readonly float[,] sharpen2DFilter5x5 = new float[,] {
				{-1, -1, -1, -1, -1},
				{-1, 2, 2, 2, -1},
				{-1, 2, 8, 2, -1},
				{-1, 2, 2, 2, -1},
				{-1, -1, -1, -1, -1} };
			private static readonly float[,] gaussian2DFilter5x5 = new float[,] { // must apply a divisor of 273 (the sum of the filter elements)
				{1, 4, 7, 4, 1},
				{4, 16, 26, 16, 4},
				{7, 26, 41, 26, 7},
				{4, 16, 26, 16, 4},
				{1, 4, 7, 4, 1} };
			private static readonly float[,] boxBlur2DFilter5x5 = new float[,] { // must apply a divisor of 25 (the sum of the filter elements)
				{1, 1, 1, 1, 1},
				{1, 1, 1, 1, 1},
				{1, 1, 1, 1, 1},
				{1, 1, 1, 1, 1},
				{1, 1, 1, 1, 1} };
			private static readonly float[,] laplacian2DFilter5x5 = new float[,] {
				{0, 0, -1, 0, 0},
				{0, -1, -2, -1, 0},
				{-1, -2, 16, -2, -1},
				{0, -1, -2, -1, 0},
				{0, 0, -1, 0, 0} };

			// Small 3x3 filters for 2D CNNs
			public static readonly Kernel2D VerticalEdge2DFilter = new Kernel2D(vertEdge2DFilter3x3);
			public static readonly Kernel2D HorizontalEdge2DFilter = new Kernel2D(horzEdge2DFilter3x3);
			public static readonly Kernel2D DiagonalEdgeTopLeftToBottomRight2DFilter = new Kernel2D(diagEdgeTopLeftToBottomRight2DFilter3x3);
			public static readonly Kernel2D DiagonalEdgeTopRightToBottomLeft2DFilter = new Kernel2D(diagEdgeTopRightToBottomLeft2DFilter3x3);
			public static readonly Kernel2D SobelX2DFilter = new Kernel2D(sobelX2DFilter3x3);
			public static readonly Kernel2D SobelY2DFilter = new Kernel2D(sobelY2DFilter3x3);
			public static readonly Kernel2D Identity2DFilter = new Kernel2D(identity2DFilter3x3);
			public static readonly Kernel2D Laplacian2DFilter = new Kernel2D(laplacian2DFilter3x3);
			public static readonly Kernel2D Gaussian2DFilter = new Kernel2D(gaussian2DFilter3x3, true);
			public static readonly Kernel2D Sharpen2DFilter = new Kernel2D(sharpen2DFilter3x3);
			public static readonly Kernel2D BoxBlur2DFilter = new Kernel2D(boxBlur2DFilter3x3, true);

			// Larger 5x5 filters for 2D CNNs
			public static readonly Kernel2D Gaussian2DFilterLarge = new Kernel2D(gaussian2DFilter5x5, true);
			public static readonly Kernel2D Sharpen2DFilterLarge = new Kernel2D(sharpen2DFilter5x5);
			public static readonly Kernel2D BoxBlur2DFilterLarge = new Kernel2D(boxBlur2DFilter5x5, true);
			public static readonly Kernel2D Laplacian2DFilterLarge = new Kernel2D(laplacian2DFilter5x5);

			// for 2D CNNs
			public static void ApplyFilterToRightWithZeroPadding(float[,] image, Kernel2D kernel, float[,] filtered, CNN.IActivation activation, bool mergeMaximum = false)
			{
				if (image.GetLength(0) != image.GetLength(1))
					throw new ArgumentException("The image argument must be square.");
				if (filtered.GetLength(0) != image.GetLength(0) || filtered.GetLength(1) != image.GetLength(1))
					throw new ArgumentException("The filtered argument must be same size as image argument.");

				int FILTER_2D_SIZE = kernel.filter.GetLength(0);
				// Calculate padding size
				int padding = FILTER_2D_SIZE / 2;
				int image_size = image.GetLength(0);

				// Loop through the image with padding
				for (int j = padding; j < image_size - padding; j++)
					for (int i = padding; i < image_size - padding; i++)
					{
						float sum = 0;

						// Convolve the filter with the image region
						for (int fi = 0; fi < FILTER_2D_SIZE; fi++)
							for (int fj = 0; fj < FILTER_2D_SIZE; fj++)
								sum += image[i + fi - padding, j + fj - padding] * kernel.filter[fi, fj];

						// Store the result in the output matrix
						if (mergeMaximum)
							filtered[i, j] = Math.Max(activation.Function(sum / kernel.divisor), filtered[i, j]);
						else
							filtered[i, j] = activation.Function(sum / kernel.divisor);
					}
			}
			// for 2D CNNs
			public static void ApplyFilterToBottomWithZeroPadding(float[,] image, Kernel2D kernel, float[,] filtered, CNN.IActivation activation, bool mergeMaximum = false)
			{
				if (image.GetLength(0) != image.GetLength(1))
					throw new ArgumentException("The image argument must be square.");
				if (filtered.GetLength(0) != image.GetLength(0) || filtered.GetLength(1) != image.GetLength(1))
					throw new ArgumentException("The filtered argument must be same size as image argument.");


				int FILTER_2D_SIZE = kernel.filter.GetLength(0);
				// Calculate padding size
				int padding = FILTER_2D_SIZE / 2;
				int image_size = image.GetLength(0);

				// Loop through the image with padding
				for (int i = padding; i < image_size - padding; i++)
					for (int j = padding; j < image_size - padding; j++)
					{
						float sum = 0;

						// Convolve the filter with the image region
						for (int fi = 0; fi < FILTER_2D_SIZE; fi++)
							for (int fj = 0; fj < FILTER_2D_SIZE; fj++)
								sum += image[i + fi - padding, j + fj - padding] * kernel.filter[fi, fj];

						// Store the result in the output matrix
						if (mergeMaximum)
							filtered[i, j] = Math.Max(activation.Function(sum / kernel.divisor), filtered[i, j]);
						else
							filtered[i, j] = activation.Function(sum / kernel.divisor);
					}
			}
			// for 2D CNNs - function to apply 2D convolution with replication padding
			public static void ApplyFilterWithReplicationPadding(float[,] image, Kernel2D kernel, float[,] filtered, CNN.IActivation activation, bool mergeMaximum = false)
			{
				if (image.GetLength(0) != image.GetLength(1))
					throw new ArgumentException("The image argument must be square.");
				if (filtered.GetLength(0) != image.GetLength(0) || filtered.GetLength(1) != image.GetLength(1))
					throw new ArgumentException("The filtered argument must be same size as image argument.");

				int rows = image.GetLength(0), cols = image.GetLength(1);

				int FILTER_2D_SIZE = kernel.filter.GetLength(0);
				int padding = FILTER_2D_SIZE / 2;

				// Apply convolution with replication padding
				for (int i = 0; i < rows; i++)
					for (int j = 0; j < cols; j++)
					{
						double sum = 0.0;

						// Convolve the kernel with the image region
						for (int ki = 0; ki < FILTER_2D_SIZE; ki++)
						{
							for (int kj = 0; kj < FILTER_2D_SIZE; kj++)
							{
								// Calculate the pixel position in the image with padding
								int x = i + ki - padding;
								int y = j + kj - padding;

								// Handle replication padding for out-of-bound pixels
								x = (x < 0) ? 0 : ((x >= rows) ? (rows - 1) : x);
								y = (y < 0) ? 0 : ((y >= cols) ? (cols - 1) : y);

								sum += image[x, y] * kernel.filter[ki, kj];
							}
						}

						// Store the result in the output matrix "filtered"
						if (mergeMaximum)
							filtered[i, j] = Math.Max(activation.Function((float)(sum / kernel.divisor)), filtered[i, j]);
						else
							filtered[i, j] = activation.Function((float)(sum / kernel.divisor));
					}
			}
			// for 2D CNNs
			public static void CropImage(float[,] image, float[,] croppedImage)
			{
				int originalWidth = image.GetLength(0), originalHeight = image.GetLength(1);
				const int padding = 1;

				// Calculate the dimensions of the cropped image
				int croppedWidth = originalWidth - (2 * padding);
				int croppedHeight = originalHeight - (2 * padding);

				if (croppedImage.GetLength(0) != croppedWidth || croppedImage.GetLength(1) != croppedHeight)
					throw new ArgumentException($"The croppedImage argument must have the dimensions [{croppedWidth}, {croppedHeight}]");

				// Copy the relevant part of the image to the cropped image
				for (int i = padding; i < originalWidth - padding; i++)
					for (int j = padding; j < originalHeight - padding; j++)
						croppedImage[i - padding, j - padding] = image[i, j];
			}
			// for 2D CNNs
			public static void ApplyMaxPooling(float[,] filtered, float[,] pooled)
			{
				if (filtered.GetLength(0) != filtered.GetLength(1))
					throw new ArgumentException("The filtered argument must be square.");
				int POOLED_SIZE = filtered.GetLength(0) / POOL_2D_SIZE;
				POOLED_SIZE += POOLED_SIZE & 1;
				if (pooled.GetLength(0) != POOLED_SIZE || pooled.GetLength(1) != POOLED_SIZE)
					throw new ArgumentException($"The dimensions of the pooled argument must be half that of filtered argument ({POOLED_SIZE}).");

				for (int i = 0; i < filtered.GetLength(0); i += POOL_2D_SIZE)
					for (int j = 0; j < filtered.GetLength(1); j += POOL_2D_SIZE)
					{
						float max_val = filtered[i, j];

						// Find the maximum value in the pooling window
						for (int m = 0; m < POOL_2D_SIZE; m++)
							for (int n = 0; n < POOL_2D_SIZE; n++)
								if (filtered[i + m, j + n] > max_val)
									max_val = filtered[i + m, j + n];

						// Store the maximum value in the pooled matrix
						pooled[i / POOL_2D_SIZE, j / POOL_2D_SIZE] = max_val;
					}
			}
			// for 2D CNNs
			public static void FlattenPooled(float[,] pooled, float[] flattened, out float min, out float max)
			{
				if (flattened.Length != pooled.GetLength(0) * pooled.GetLength(1))
					throw new ArgumentException("The flattened argument must have a length equal to the pooled argument's dimensions multiplied.");

				min = max = pooled[0, 0];
				// Loop through the rows and columns of the image
				for (int row = 0; row < pooled.GetLength(0); row++)
					for (int column = 0; column < pooled.GetLength(1); column++)
					{
						var v = pooled[row, column];
						min = Math.Min(min, v);
						max = Math.Max(max, v);
						flattened[row * pooled.GetLength(1) + column] = v;
					}
			}

		}

	}
	namespace CNN
	{
		public interface IActivation
		{
			public float Function(float x);
		}
		public class ActivationLinear : IActivation
		{
			public float Function(float x)
			{
				return CommonFunctions.Linear(x);
			}
		}
		public class ActivationLeakyReLU : IActivation
		{
			private readonly float alpha;
			public ActivationLeakyReLU(float alpha = 0.01f)
			{
				this.alpha = alpha;
			}
			public float Function(float x)
			{
				return CommonFunctions.LeakyReLU(x, alpha);
			}
		}
		public class ActivationReLU : IActivation
		{
			public float Function(float x)
			{
				return CommonFunctions.ReLU(x);
			}
		}
		internal static class CommonFunctions
		{
			public static void NormalizeBetween0And1(float[] flattened, float min, float max)
			{
				float range = max - min;
				if (range == 0.0f)
					range = 1e-15f;
				for (int i = 0; i < flattened.Length; i++)
					flattened[i] = (flattened[i] - min) / range;
			}
			public static void NormalizeBetweenNegative1And1(float[] flattened, float min, float max)
			{
				float range = max - min;
				if (range == 0.0f)
					range = 1e-15f;
				for (int i = 0; i < flattened.Length; i++)
					flattened[i] = (flattened[i] - min) / range * 2 - 1;
			}
			public static void NormalizeBetweenNegativeAlphaAnd1(float[] flattened, float min, float max, float alpha = 0.2f) // this is in beta
			{
				float range = max - min;
				if (range == 0.0f)
					range = 1e-15f;
				alpha = Math.Abs(alpha);
				for (int i = 0; i < flattened.Length; i++)
					flattened[i] = (flattened[i] - min) / range * (1.0f + alpha) - alpha;
			}
			public static void NormalizeJustBelow0And1(float[] flattened, float min, float max)
			{
				float range = max - min;
				if (range == 0.0f)
					range = 1e-15f;
				for (int i = 0; i < flattened.Length; i++)
					flattened[i] = (flattened[i] - min) / range * 1.01f - 0.01f; // values from -0.01 to 1.0 - check for improved results using LeakyReLU with the neural network
			}
			public static float Linear(float x) // use this when applying filters to Word2Vec (or similar) values since it will not modify the values
			{
				return x;
			}
			// alpha might work at 0.01 but can be modified, bigger or smaller
			public static float LeakyReLU(float x, float alpha) // Rectified Linear Unit function ("Leaky" version)
			{
				return x >= 0 ? x : (alpha * x);
			}
			public static float ReLU(float x) // Rectified Linear Unit function
			{
				return x > 0 ? x : 0;
			}

		}
	}
}

Comment