Architecture module

Architecture

Module containing data structures to represent controllers.

AbstractNeuralNetwork

Abstract type for neural networks.

Notes

Subtypes should implement the following method:

• layers(::AbstractNeuralNetwork) - return a list of the layers

The following standard methods are implemented:

• length(::AbstractNeuralNetwork)
• getindex(::AbstractNeuralNetwork, indices)
• lastindex(::AbstractNeuralNetwork)
• ==(::AbstractNeuralNetwork, ::AbstractNeuralNetwork)

layers(N::AbstractNeuralNetwork)

Return a list of the layers of a neural network.

Input

• N – neural network

Output

The list of layers.

dim_in(N::AbstractNeuralNetwork)

Return the input dimension of a neural network.

Input

• N – neural network

Output

The dimension of the input layer of N.

dim_out(N::AbstractNeuralNetwork)

Return the output dimension of a neural network.

Input

• N – neural network

Output

The dimension of the output layer of N.

dim(N::AbstractNeuralNetwork)

Return the input and output dimension of a neural network.

Input

• N – neural network

Output

The pair $(i, o)$ where $i$ is the input dimension and $o$ is the output dimension of N.

Notes

This function is not exported due to name conflicts with other related packages.

FeedforwardNetwork{L} <: AbstractNeuralNetwork

Standard implementation of a feedforward neural network which stores the layer operations.

Fields

• layers – vector of layer operations (see AbstractLayerOp)

Notes

The field layers contains the layer operations, so the number of layers is length(layers) + 1.

Conversion from a Flux.Chain is supported.

AbstractLayerOp

Abstract type for layer operations.

Notes

An AbstractLayerOp represents a layer operation. A classical example is a "dense layer operation" with an affine map followed by an activation function.

dim_in(L::AbstractLayerOp)

Return the input dimension of a layer operation.

Input

• L – layer operation

Output

The input dimension of L.

dim_out(L::AbstractLayerOp)

Return the output dimension of a layer operation.

Input

• L – layer operation

Output

The output dimension of L.

dim(L::AbstractLayerOp)

Return the input and output dimension of a layer operation.

Input

• N – neural network

Output

The pair $(i, o)$ where $i$ is the input dimension and $o$ is the output dimension of N.

Notes

This function is not exported due to name conflicts with other related packages.

AbstractPoolingLayerOp <: AbstractLayerOp

Abstract type for pooling layer operations.

Notes

Pooling is an operation on a three-dimensional tensor that iterates over the first two dimensions in a window and aggregates the values, thus reducing the output dimension.

Implementation

The following (unexported) functions should be implemented:

• window(::AbstractPoolingLayerOp) – return the pair $(p, q)$ representing the window size
• aggregation(::AbstractPoolingLayerOp) – return the aggregation function (applied to a tensor)

DenseLayerOp{F, M, B} <: AbstractLayerOp

A dense layer operation is an affine map followed by an activation function.

Fields

• weights – weight matrix
• bias – bias vector
• activation – activation function

Notes

Conversion from a Flux.Dense is supported.

ConvolutionalLayerOp{F, M, B} <: AbstractLayerOp

A convolutional layer operation is a series of filters, each of which computes a small affine map followed by an activation function.

Fields

• weights – vector with one weight matrix for each filter
• bias – vector with one bias value for each filter
• activation – activation function

Notes

Conversion from a Flux.Conv is supported.

FlattenLayerOp <: AbstractLayerOp

A flattening layer operation converts a multidimensional tensor into a vector.

Notes

The implementation uses row-major ordering for convenience with the machine-learning literature.

julia> T = reshape([1, 3, 2, 4, 5, 7, 6, 8], (2, 2, 2))
2×2×2 Array{Int64, 3}:
[:, :, 1] =
 1  2
 3  4

[:, :, 2] =
 5  6
 7  8

julia> FlattenLayerOp()(T)
8-element Vector{Int64}:
 1
 2
 3
 4
 5
 6
 7
 8

MaxPoolingLayerOp <: AbstractPoolingLayerOp

A max-pooling layer operation. The aggregation function is maximum.

Fields

• p – horizontal window size
• q – vertical window size

MeanPoolingLayerOp <: AbstractPoolingLayerOp

A mean-pooling layer operation. The aggregation function is Statistics.mean.

Fields

• p – horizontal window size
• q – vertical window size

ActivationFunction

Abstract type for activation functions.

Id

Identity activation.

\[ f(x) = x\]

ReLU

Rectified linear unit (ReLU) activation.

\[ f(x) = max(x, 0)\]

Sigmoid

Sigmoid activation.

\[ f(x) = \frac{1}{1 + e^{-x}}\]

Tanh

Hyperbolic tangent activation.

\[ f(x) = tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}}\]

LeakyReLU{N<:Number}

Leaky ReLU activation.

\[ fₐ(x) = x > 0 ? x : a x\]

where $a$ is the parameter.

Fields

• slope – parameter for negative inputs