pymor.reductors.neural_network

Module Contents

class pymor.reductors.neural_network.CustomDataset(training_data)

Bases: torch.utils.data.Dataset

Class that represents the dataset to use in PyTorch.

Parameters

training_data

Set of training parameters and the respective coefficients of the solution in the reduced basis.

class pymor.reductors.neural_network.EarlyStoppingScheduler(size_training_validation_set, patience=10, delta=0.0)

Bases: pymor.core.base.BasicObject

Class for performing early stopping in training of neural networks.

If the validation loss does not decrease over a certain amount of epochs, the training should be aborted to avoid overfitting the training data. This class implements an early stopping scheduler that recommends to stop the training process if the validation loss did not decrease by at least delta over patience epochs.

Parameters

size_training_validation_set

Size of both, training and validation set together.

patience

Number of epochs of non-decreasing validation loss allowed, before early stopping the training process.

delta

Minimal amount of decrease in the validation loss that is required to reset the counter of non-decreasing epochs.

class pymor.reductors.neural_network.NeuralNetworkInstationaryReductor(fom=None, training_set=None, validation_set=None, validation_ratio=0.1, T=None, basis_size=None, rtol=0.0, atol=0.0, l2_err=0.0, pod_params={}, ann_mse='like_basis', scale_inputs=True, scale_outputs=False)

Bases: NeuralNetworkReductor

Reduced Basis reductor for instationary problems relying on artificial neural networks.

This is a reductor that constructs a reduced basis using proper orthogonal decomposition and trains a neural network that approximates the mapping from parameter and time space to coefficients of the full-order solution in the reduced basis. The approach is described in [WHR19].

Parameters

fom

See NeuralNetworkReductor.

training_set

See NeuralNetworkReductor.

validation_set

See NeuralNetworkReductor.

validation_ratio

See NeuralNetworkReductor.

T

The final time T used in case fom is None.

basis_size

See NeuralNetworkReductor.

rtol

See NeuralNetworkReductor.

atol

See NeuralNetworkReductor.

l2_err

See NeuralNetworkReductor.

pod_params

See NeuralNetworkReductor.

ann_mse

See NeuralNetworkReductor.

scale_inputs

See NeuralNetworkReductor.

scale_outputs

See NeuralNetworkReductor.

Methods

compute_training_data

Compute a reduced basis using proper orthogonal decomposition.

compute_training_data()

Compute a reduced basis using proper orthogonal decomposition.

class pymor.reductors.neural_network.NeuralNetworkInstationaryStatefreeOutputReductor(fom=None, nt=1, training_set=None, validation_set=None, validation_ratio=0.1, T=None, validation_loss=None, scale_inputs=True, scale_outputs=False)

Bases: NeuralNetworkStatefreeOutputReductor

Output reductor relying on artificial neural networks.

This is a reductor that trains a neural network that approximates the mapping from parameter space to output space.

Parameters

fom

See NeuralNetworkStatefreeOutputReductor.

nt

See NeuralNetworkInstationaryReductor.

training_set

See NeuralNetworkStatefreeOutputReductor.

validation_set

See NeuralNetworkStatefreeOutputReductor.

T

See NeuralNetworkInstationaryReductor.

validation_ratio

See NeuralNetworkReductor.

validation_loss

See NeuralNetworkStatefreeOutputReductor.

scale_inputs

See NeuralNetworkReductor.

scale_outputs

See NeuralNetworkReductor.

Methods

compute_training_data

Compute the training samples (the outputs to the parameters of the training set).

compute_training_data()

Compute the training samples (the outputs to the parameters of the training set).

class pymor.reductors.neural_network.NeuralNetworkLSTMInstationaryReductor(fom=None, training_set=None, validation_set=None, validation_ratio=0.1, T=None, basis_size=None, rtol=0.0, atol=0.0, l2_err=0.0, pod_params={}, ann_mse='like_basis', scale_inputs=True, scale_outputs=False)

Bases: NeuralNetworkInstationaryReductor

Reduced Basis reductor for instationary problems relying on LSTM neural networks.

This is a reductor that constructs a reduced basis using proper orthogonal decomposition and trains an LSTM neural network that approximates the mapping from parameter to coefficients of the full-order solution in the reduced basis for a fixed number of timesteps.

Methods

reduce

Reduce by LSTM neural networks.

reduce(hidden_dimension='3*N + P', number_layers=1, optimizer=optim.LBFGS, epochs=1000, batch_size=20, learning_rate=1.0, loss_function=None, restarts=10, lr_scheduler=None, lr_scheduler_params={}, es_scheduler_params={'patience': 10, 'delta': 0.0}, weight_decay=0.0, log_loss_frequency=0)

Reduce by LSTM neural networks.

Parameters

hidden_dimension

Number of neurons in the hidden state of the LSTM. Can either be fixed or a Python expression string depending on the reduced basis size respectively output dimension N and the total dimension of the Parameters P.

number_layers

Number of recurred layers, i.e. number of stacked LSTM cells in the neural network.

optimizer

See NeuralNetworkReductor.

epochs

See NeuralNetworkReductor.

batch_size

See NeuralNetworkReductor.

learning_rate

See NeuralNetworkReductor.

loss_function

See NeuralNetworkReductor.

restarts

See NeuralNetworkReductor.

lr_scheduler

See NeuralNetworkReductor.

lr_scheduler_params

See NeuralNetworkReductor.

es_scheduler_params

See NeuralNetworkReductor.

weight_decay

See NeuralNetworkReductor.

log_loss_frequency

See NeuralNetworkReductor.

class pymor.reductors.neural_network.NeuralNetworkLSTMInstationaryStatefreeOutputReductor(fom=None, nt=1, training_set=None, validation_set=None, validation_ratio=0.1, T=None, validation_loss=None, scale_inputs=True, scale_outputs=False)

Bases: NeuralNetworkInstationaryStatefreeOutputReductor, NeuralNetworkLSTMInstationaryReductor

Output reductor relying on LSTM neural networks.

This is a reductor that trains an LSTM neural network that approximates the mapping from parameter space to output space.

reduce

class pymor.reductors.neural_network.NeuralNetworkReductor(fom=None, training_set=None, validation_set=None, validation_ratio=0.1, basis_size=None, rtol=0.0, atol=0.0, l2_err=0.0, pod_params={}, ann_mse='like_basis', scale_inputs=True, scale_outputs=False)

Bases: pymor.core.base.BasicObject

Reduced Basis reductor relying on artificial neural networks.

This is a reductor that constructs a reduced basis using proper orthogonal decomposition and trains a neural network that approximates the mapping from parameter space to coefficients of the full-order solution in the reduced basis. The approach is described in [HU18].

Parameters

fom

The full-order Model to reduce. If None, the training_set has to consist of pairs of parameter values and corresponding solution VectorArrays.

training_set

Set of parameter values to use for POD and training of the neural network. If fom is None, the training_set has to consist of pairs of parameter values and corresponding solution VectorArrays.

validation_set

Set of parameter values to use for validation in the training of the neural network. If fom is None, the validation_set has to consist of pairs of parameter values and corresponding solution VectorArrays.

validation_ratio

Fraction of the training set to use for validation in the training of the neural network (only used if no validation set is provided). Either a validation set or a positive validation ratio is required.

basis_size

Desired size of the reduced basis. If None, rtol, atol or l2_err must be provided.

rtol

Relative tolerance the basis should guarantee on the training set.

atol

Absolute tolerance the basis should guarantee on the training set.

l2_err

L2-approximation error the basis should not exceed on the training set.

pod_params

Dict of additional parameters for the POD-method.

ann_mse

If 'like_basis', the mean squared error of the neural network on the training set should not exceed the error of projecting onto the basis. If None, the neural network with smallest validation error is used to build the ROM. If a tolerance is prescribed, the mean squared error of the neural network on the training set should not exceed this threshold. Training is interrupted if a neural network that undercuts the error tolerance is found.

scale_inputs

Determines whether or not to scale the inputs of the neural networks.

scale_outputs

Determines whether or not to scale the outputs/targets of the neural networks.

Methods

compute_training_data	Compute a reduced basis using proper orthogonal decomposition.
reconstruct	Reconstruct high-dimensional vector from reduced vector u.
reduce	Reduce by training artificial neural networks.

compute_training_data()

Compute a reduced basis using proper orthogonal decomposition.

reconstruct(u)

Reconstruct high-dimensional vector from reduced vector u.

reduce(hidden_layers='[(N+P)*3, (N+P)*3]', activation_function=torch.tanh, optimizer=optim.LBFGS, epochs=1000, batch_size=20, learning_rate=1.0, loss_function=None, restarts=10, lr_scheduler=optim.lr_scheduler.StepLR, lr_scheduler_params={'step_size': 10, 'gamma': 0.7}, es_scheduler_params={'patience': 10, 'delta': 0.0}, weight_decay=0.0, log_loss_frequency=0)

Reduce by training artificial neural networks.

Parameters

hidden_layers

Number of neurons in the hidden layers. Can either be fixed or a Python expression string depending on the reduced basis size respectively output dimension N and the total dimension of the Parameters P.

activation_function

Activation function to use between the hidden layers.

optimizer

Algorithm to use as optimizer during training.

epochs

Maximum number of epochs for training.

batch_size

Batch size to use if optimizer allows mini-batching.

learning_rate

Step size to use in each optimization step.

loss_function

Loss function to use for training. If 'weighted MSE', a weighted mean squared error is used as loss function, where the weights are given as the singular values of the corresponding reduced basis functions. If None, the usual mean squared error is used.

restarts

Number of restarts of the training algorithm. Since the training results highly depend on the initial starting point, i.e. the initial weights and biases, it is advisable to train multiple neural networks by starting with different initial values and choose that one performing best on the validation set.

lr_scheduler

Algorithm to use as learning rate scheduler during training. If None, no learning rate scheduler is used.

lr_scheduler_params

A dictionary of additional parameters passed to the init method of the learning rate scheduler. The possible parameters depend on the chosen learning rate scheduler.

es_scheduler_params

A dictionary of additional parameters passed to the init method of the early stopping scheduler. For the possible parameters, see EarlyStoppingScheduler.

weight_decay

Weighting parameter for the l2-regularization of the weights and biases in the neural network. This regularization is not available for all optimizers; see the PyTorch documentation for more details.

log_loss_frequency

Frequency of epochs in which to log the current validation and training loss during training of the neural networks. If 0, no intermediate logging of losses is done.

Returns

rom

Reduced-order NeuralNetworkModel.

class pymor.reductors.neural_network.NeuralNetworkStatefreeOutputReductor(fom=None, training_set=None, validation_set=None, validation_ratio=0.1, validation_loss=None, scale_inputs=True, scale_outputs=False)

Bases: NeuralNetworkReductor

Output reductor relying on artificial neural networks.

This is a reductor that trains a neural network that approximates the mapping from parameter space to output space.

Parameters

fom

The full-order Model to reduce. If None, the training_set has to consist of pairs of parameter values and corresponding outputs.

training_set

Set of parameter values to use for POD and training of the neural network. If fom is None, the training_set has to consist of pairs of parameter values and corresponding outputs.

validation_set

Set of parameter values to use for validation in the training of the neural network. If fom is None, the validation_set has to consist of pairs of parameter values and corresponding outputs.

validation_ratio

See NeuralNetworkReductor.

validation_loss

The validation loss to reach during training. If None, the neural network with the smallest validation loss is returned.

scale_inputs

See NeuralNetworkReductor.

scale_outputs

See NeuralNetworkReductor.

Methods

compute_training_data

Compute the training samples (the outputs to the parameters of the training set).

compute_training_data()

Compute the training samples (the outputs to the parameters of the training set).

pymor.reductors.neural_network.multiple_restarts_training(training_data, validation_data, neural_network, target_loss=None, max_restarts=10, log_loss_frequency=0, training_parameters={}, scaling_parameters={})

Algorithm that performs multiple restarts of neural network training.

This method either performs a predefined number of restarts and returns the best trained network or tries to reach a given target loss and stops training when the target loss is reached.

See train_neural_network for more information on the parameters.

Parameters

training_data

Data to use during the training phase.

validation_data

Data to use during the validation phase.

neural_network

The neural network to train (parameters will be reset after each restart).

target_loss

Loss to reach during training (if None, the network with the smallest loss is returned).

max_restarts

Maximum number of restarts to perform.

log_loss_frequency

Frequency of epochs in which to log the current validation and training loss. If 0, no intermediate logging of losses is done.

training_parameters

Additional parameters for the training algorithm, see train_neural_network for more information.

scaling_parameters

Additional parameters for scaling inputs respectively outputs, see train_neural_network for more information.

Returns

best_neural_network

The best trained neural network.

losses

The corresponding losses.

Raises

NeuralNetworkTrainingError

Raised if prescribed loss can not be reached within the given number of restarts.

pymor.reductors.neural_network.train_neural_network(training_data, validation_data, neural_network, training_parameters={}, scaling_parameters={}, log_loss_frequency=0)

Training algorithm for artificial neural networks.

Trains a single neural network using the given training and validation data.

Parameters

training_data

Data to use during the training phase. Has to be a list of tuples, where each tuple consists of two elements that are either PyTorch-tensors (torch.DoubleTensor) or NumPy arrays or pyMOR data structures that have to_numpy() implemented. The first element contains the input data, the second element contains the target values.

validation_data

Data to use during the validation phase. Has to be a list of tuples, where each tuple consists of two elements that are either PyTorch-tensors (torch.DoubleTensor) or NumPy arrays or pyMOR data structures that have to_numpy() implemented. The first element contains the input data, the second element contains the target values.

neural_network

The neural network to train (can also be a pre-trained model). Has to be a PyTorch-Module.

training_parameters

Dictionary with additional parameters for the training routine like the type of the optimizer, the (maximum) number of epochs, the batch size, the learning rate or the loss function to use. Possible keys are 'optimizer' (an optimizer from the PyTorch optim package; if not provided, the LBFGS-optimizer is taken as default), 'epochs' (an integer that determines the number of epochs to use for training the neural network (if training is not interrupted prematurely due to early stopping); if not provided, 1000 is taken as default value), 'batch_size' (an integer that determines the number of samples to pass to the optimizer at once; if not provided, 20 is taken as default value; not used in the case of the LBFGS-optimizer since LBFGS does not support mini-batching), 'learning_rate' (a positive real number used as the (initial) step size of the optimizer; if not provided, 1 is taken as default value), 'loss_function' (a loss function from PyTorch; if not provided, the MSE loss is taken as default), 'lr_scheduler' (a learning rate scheduler from the PyTorch optim.lr_scheduler package; if not provided or None, no learning rate scheduler is used), 'lr_scheduler_params' (a dictionary of additional parameters for the learning rate scheduler), 'es_scheduler_params' (a dictionary of additional parameters for the early stopping scheduler), and 'weight_decay' (non-negative real number that determines the strength of the l2-regularization; if not provided or 0., no regularization is applied).

scaling_parameters

Dict of tensors that determine how to scale inputs before passing them through the neural network and outputs after obtaining them from the neural network. If not provided or each entry is None, no scaling is applied. Required keys are 'min_inputs', 'max_inputs', 'min_targets', and 'max_targets'.

log_loss_frequency

Frequency of epochs in which to log the current validation and training loss. If 0, no intermediate logging of losses is done.

Returns

best_neural_network

The best trained neural network with respect to validation loss.

losses

The corresponding losses as a dictionary with keys 'full' (for the full loss containing the training and the validation average loss), 'train' (for the average loss on the training set), and 'val' (for the average loss on the validation set).