pymor.reductors.neural_network

Module Contents

class pymor.reductors.neural_network.CustomDataset(training_data)

Bases: torch.utils.data.Dataset

Class that represents the dataset to use in PyTorch.

Parameters:

training_data – Set of training parameters and the respective coefficients of the solution in the reduced basis.

class pymor.reductors.neural_network.EarlyStoppingScheduler(size_training_validation_parameters, patience=10, delta=0.0)

Bases: pymor.core.base.BasicObject

Class for performing early stopping in training of neural networks.

If the validation loss does not decrease over a certain amount of epochs, the training should be aborted to avoid overfitting the training data. This class implements an early stopping scheduler that recommends to stop the training process if the validation loss did not decrease by at least delta over patience epochs.

Parameters:

• size_training_validation_parameters – Size of both, training and validation parameters together.

• patience – Number of epochs of non-decreasing validation loss allowed, before early stopping the training process.

• delta – Minimal amount of decrease in the validation loss that is required to reset the counter of non-decreasing epochs.

class pymor.reductors.neural_network.NeuralNetworkLSTMReductor(fom=None, reduced_basis=None, training_parameters=None, validation_parameters=None, training_snapshots=None, validation_snapshots=None, validation_ratio=0.1, T=None, nt=1, basis_size=None, rtol=0.0, atol=0.0, l2_err=0.0, pod_params={}, ann_mse='like_basis', scale_inputs=True, scale_outputs=False)

Bases: NeuralNetworkReductor

Reduced Basis reductor for instationary problems relying on LSTM neural networks.

This is a reductor that constructs a reduced basis using proper orthogonal decomposition and trains an LSTM neural network that approximates the mapping from parameter to coefficients of the full-order solution in the reduced basis for a fixed number of timesteps.

Methods

reduce

Reduce by LSTM neural networks.

reduce(hidden_dimension='3*N + P', number_layers=1, optimizer=optim.LBFGS, epochs=1000, batch_size=20, learning_rate=1.0, loss_function=None, restarts=10, lr_scheduler=None, lr_scheduler_params={}, es_scheduler_params={'patience': 10, 'delta': 0.0}, weight_decay=0.0, log_loss_frequency=0)

Reduce by LSTM neural networks.

Parameters:

• hidden_dimension – Number of neurons in the hidden state of the LSTM. Can either be fixed or a Python expression string depending on the reduced basis size respectively output dimension N and the total dimension of the Parameters P.

• number_layers – Number of recurred layers, i.e. number of stacked LSTM cells in the neural network.

• optimizer – See NeuralNetworkReductor.

• epochs – See NeuralNetworkReductor.

• batch_size – See NeuralNetworkReductor.

• learning_rate – See NeuralNetworkReductor.

• loss_function – See NeuralNetworkReductor.

• restarts – See NeuralNetworkReductor.

• lr_scheduler – See NeuralNetworkReductor.

• lr_scheduler_params – See NeuralNetworkReductor.

• es_scheduler_params – See NeuralNetworkReductor.

• weight_decay – See NeuralNetworkReductor.

• log_loss_frequency – See NeuralNetworkReductor.

class pymor.reductors.neural_network.NeuralNetworkLSTMStatefreeOutputReductor(fom=None, training_parameters=None, validation_parameters=None, training_outputs=None, validation_outputs=None, validation_ratio=0.1, T=None, nt=1, validation_loss=None, scale_inputs=True, scale_outputs=False)

Bases: NeuralNetworkStatefreeOutputReductor, NeuralNetworkLSTMReductor

Output reductor relying on LSTM neural networks.

This is a reductor that trains an LSTM neural network that approximates the mapping from parameter space to output space.

reduce

class pymor.reductors.neural_network.NeuralNetworkReductor(fom=None, reduced_basis=None, training_parameters=None, validation_parameters=None, training_snapshots=None, validation_snapshots=None, validation_ratio=0.1, T=None, nt=1, basis_size=None, rtol=0.0, atol=0.0, l2_err=0.0, pod_params={}, ann_mse='like_basis', scale_inputs=True, scale_outputs=False)

Bases: pymor.core.base.BasicObject

Reduced Basis reductor relying on artificial neural networks.

This reductor either takes a precomputed reduced basis or constructs a reduced basis using proper orthogonal decomposition. It then trains a neural network that approximates the mapping from parameter space to coefficients of the full-order solution in the reduced basis. Moreover, the reductor also works without providing a full-order model, in which case it requires a set of training parameters and corresponding solution snapshots. This way, the reductor can be used in a completely data-driven manner. The approach is described in [HU18].

Parameters:

• fom – The full-order Model to reduce. If None, the training_parameters with parameter values and the training_snapshots with corresponding solution VectorArrays have to be set.

• reduced_basis – VectorArray of basis vectors of the reduced space onto which to project. If None, the reduced basis is computed using the pod method.

• training_parameters – Parameter values to use for POD (in case no reduced_basis is provided) and training of the neural network.

• training_snapshots – VectorArray to use for POD and training of the neural network. Contains the solutions to the parameters of the training_parameters and can be None when fom is not None. In the case of a time-dependent problem, the snapshots are assumed to be equidistant in time.

• validation_parameters – Parameter values to use for validation in the training of the neural network.

• validation_snapshots – VectorArray to use for validation in the training of the neural network. Contains the solutions to the parameters of the validation_parameters and can be None when fom is not None. In the case of a time-dependent problem, the snapshots are assumed to be equidistant in time.

• validation_ratio – Fraction of the training parameters to use for validation in the training of the neural network (only used if no validation parameters are provided). Either validation parameters or a positive validation ratio is required.

• basis_size – Desired size of the reduced basis. If None, rtol, atol or l2_err must be provided.

• rtol – Relative tolerance the basis should guarantee on the training parameters.

• atol – Absolute tolerance the basis should guarantee on the training parameters.

• l2_err – L2-approximation error the basis should not exceed on the training parameters.

• pod_params – Dict of additional parameters for the POD-method.

• ann_mse – If 'like_basis', the mean squared error of the neural network on the training parameters should not exceed the error of projecting onto the basis. If None, the neural network with smallest validation error is used to build the ROM. If a tolerance is prescribed, the mean squared error of the neural network on the training parameters should not exceed this threshold. Training is interrupted if a neural network that undercuts the error tolerance is found.

• scale_inputs – Determines whether or not to scale the inputs of the neural networks.

• scale_outputs – Determines whether or not to scale the outputs/targets of the neural networks.

Methods

compute_reduced_basis	Compute a reduced basis using proper orthogonal decomposition.
compute_training_data	Compute training data for the neural network using the reduced basis.
compute_training_snapshots	Compute training snapshots for the neural network.
compute_validation_data	Compute validation data for the neural network using the reduced basis.
compute_validation_snapshots	Compute validation data for the neural network.
reconstruct	Reconstruct high-dimensional vector from reduced vector u.
reduce	Reduce by training artificial neural networks.

compute_reduced_basis()

Compute a reduced basis using proper orthogonal decomposition.

compute_training_data()

Compute training data for the neural network using the reduced basis.

compute_training_snapshots()

Compute training snapshots for the neural network.

compute_validation_data()

Compute validation data for the neural network using the reduced basis.

compute_validation_snapshots()

Compute validation data for the neural network.

reconstruct(u)

Reconstruct high-dimensional vector from reduced vector u.

reduce(hidden_layers='[(N+P)*3, (N+P)*3]', activation_function=torch.tanh, optimizer=optim.LBFGS, epochs=1000, batch_size=20, learning_rate=1.0, loss_function=None, restarts=10, lr_scheduler=optim.lr_scheduler.StepLR, lr_scheduler_params={'step_size': 10, 'gamma': 0.7}, es_scheduler_params={'patience': 10, 'delta': 0.0}, weight_decay=0.0, log_loss_frequency=0)

Reduce by training artificial neural networks.

Parameters:

• hidden_layers – Number of neurons in the hidden layers. Can either be fixed or a Python expression string depending on the reduced basis size respectively output dimension N and the total dimension of the Parameters P.

• activation_function – Activation function to use between the hidden layers.

• optimizer – Algorithm to use as optimizer during training.

• epochs – Maximum number of epochs for training.

• batch_size – Batch size to use if optimizer allows mini-batching.

• learning_rate – Step size to use in each optimization step.

• loss_function – Loss function to use for training. If 'weighted MSE', a weighted mean squared error is used as loss function, where the weights are given as the singular values of the corresponding reduced basis functions. If None, the usual mean squared error is used.

• restarts – Number of restarts of the training algorithm. Since the training results highly depend on the initial starting point, i.e. the initial weights and biases, it is advisable to train multiple neural networks by starting with different initial values and choose that one performing best on the validation parameters.

• lr_scheduler – Algorithm to use as learning rate scheduler during training. If None, no learning rate scheduler is used.

• lr_scheduler_params – A dictionary of additional parameters passed to the init method of the learning rate scheduler. The possible parameters depend on the chosen learning rate scheduler.

• es_scheduler_params – A dictionary of additional parameters passed to the init method of the early stopping scheduler. For the possible parameters, see EarlyStoppingScheduler.

• weight_decay – Weighting parameter for the l2-regularization of the weights and biases in the neural network. This regularization is not available for all optimizers; see the PyTorch documentation for more details.

• log_loss_frequency – Frequency of epochs in which to log the current validation and training loss during training of the neural networks. If 0, no intermediate logging of losses is done.

Returns:

rom – Reduced-order NeuralNetworkModel.

class pymor.reductors.neural_network.NeuralNetworkStatefreeOutputReductor(fom=None, training_parameters=None, validation_parameters=None, training_outputs=None, validation_outputs=None, validation_ratio=0.1, T=None, nt=1, validation_loss=None, scale_inputs=True, scale_outputs=False)

Bases: NeuralNetworkReductor

Output reductor relying on artificial neural networks.

This is a reductor that trains a neural network that approximates the mapping from parameter space to output space.

Parameters:

• fom – The full-order Model to reduce. If None, both the training_parameters with parameter values and the training_outputs with corresponding outputs have to be set.

• training_parameters – List of Parameter values to use for training of the neural network.

• training_outputs – 2D NumPy array of outputs corresponding the Parameter values given by training_parameters. Axis 0 corresponds to the output index, and axis 1 corresponds to the parameter sample index. Can be None when fom is not None.

• validation_parameters – List of Parameter values to use for validation in the training of the neural network.

• validation_outputs – 2D NumPy array of outputs corresponding the Parameter values given by validation_parameters. Axis 0 corresponds to the output index, and axis 1 corresponds to the parameter sample index. Can be None when fom is not None.

• validation_ratio – See NeuralNetworkReductor.

• validation_loss – The validation loss to reach during training. If None, the neural network with the smallest validation loss is returned.

• scale_inputs – See NeuralNetworkReductor.

• scale_outputs – See NeuralNetworkReductor.

Methods

compute_reduced_basis	Empty function to avoid computing a reduced basis.
compute_training_data	Compute the training samples (the outputs to the training parameters).
compute_training_snapshots	Empty function to avoid computing training_snapshots.
compute_validation_data	Compute the validation samples (the outputs to the validation parameters).
compute_validation_snapshots	Empty function to avoid computing validation_snapshots.

compute_reduced_basis()

Empty function to avoid computing a reduced basis.

compute_training_data()

Compute the training samples (the outputs to the training parameters).

compute_training_snapshots()

Empty function to avoid computing training_snapshots.

compute_validation_data()

Compute the validation samples (the outputs to the validation parameters).

compute_validation_snapshots()

Empty function to avoid computing validation_snapshots.

pymor.reductors.neural_network.multiple_restarts_training(training_data, validation_data, neural_network, target_loss=None, max_restarts=10, log_loss_frequency=0, training_parameters={}, scaling_parameters={})

Algorithm that performs multiple restarts of neural network training.

This method either performs a predefined number of restarts and returns the best trained network or tries to reach a given target loss and stops training when the target loss is reached.

See train_neural_network for more information on the parameters.

Parameters:

• training_data – Data to use during the training phase.

• validation_data – Data to use during the validation phase.

• neural_network – The neural network to train (parameters will be reset after each restart).

• target_loss – Loss to reach during training (if None, the network with the smallest loss is returned).

• max_restarts – Maximum number of restarts to perform.

• log_loss_frequency – Frequency of epochs in which to log the current validation and training loss. If 0, no intermediate logging of losses is done.

• training_parameters – Additional parameters for the training algorithm, see train_neural_network for more information.

• scaling_parameters – Additional parameters for scaling inputs respectively outputs, see train_neural_network for more information.

Returns:

• best_neural_network – The best trained neural network.

• losses – The corresponding losses.

Raises:

NeuralNetworkTrainingError – Raised if prescribed loss can not be reached within the given number of restarts.

pymor.reductors.neural_network.train_neural_network(training_data, validation_data, neural_network, training_parameters={}, scaling_parameters={}, log_loss_frequency=0)

Training algorithm for artificial neural networks.

Trains a single neural network using the given training and validation data.

Parameters:

• training_data – Data to use during the training phase. Has to be a list of tuples, where each tuple consists of two elements that are either PyTorch-tensors (torch.DoubleTensor) or NumPy arrays or pyMOR data structures that have to_numpy() implemented. The first element contains the input data, the second element contains the target values.

• validation_data – Data to use during the validation phase. Has to be a list of tuples, where each tuple consists of two elements that are either PyTorch-tensors (torch.DoubleTensor) or NumPy arrays or pyMOR data structures that have to_numpy() implemented. The first element contains the input data, the second element contains the target values.

• neural_network – The neural network to train (can also be a pre-trained model). Has to be a PyTorch-Module.

• training_parameters – Dictionary with additional parameters for the training routine like the type of the optimizer, the (maximum) number of epochs, the batch size, the learning rate or the loss function to use. Possible keys are 'optimizer' (an optimizer from the PyTorch optim package; if not provided, the LBFGS-optimizer is taken as default), 'epochs' (an integer that determines the number of epochs to use for training the neural network (if training is not interrupted prematurely due to early stopping); if not provided, 1000 is taken as default value), 'batch_size' (an integer that determines the number of samples to pass to the optimizer at once; if not provided, 20 is taken as default value; not used in the case of the LBFGS-optimizer since LBFGS does not support mini-batching), 'learning_rate' (a positive real number used as the (initial) step size of the optimizer; if not provided, 1 is taken as default value), 'loss_function' (a loss function from PyTorch; if not provided, the MSE loss is taken as default), 'lr_scheduler' (a learning rate scheduler from the PyTorch optim.lr_scheduler package; if not provided or None, no learning rate scheduler is used), 'lr_scheduler_params' (a dictionary of additional parameters for the learning rate scheduler), 'es_scheduler_params' (a dictionary of additional parameters for the early stopping scheduler), and 'weight_decay' (non-negative real number that determines the strength of the l2-regularization; if not provided or 0., no regularization is applied).

• scaling_parameters – Dict of tensors that determine how to scale inputs before passing them through the neural network and outputs after obtaining them from the neural network. If not provided or each entry is None, no scaling is applied. Required keys are 'min_inputs', 'max_inputs', 'min_targets', and 'max_targets'.

• log_loss_frequency – Frequency of epochs in which to log the current validation and training loss. If 0, no intermediate logging of losses is done.

Returns:

• best_neural_network – The best trained neural network with respect to validation loss.

• losses – The corresponding losses as a dictionary with keys 'full' (for the full loss containing the training and the validation average loss), 'train' (for the average loss on the training parameters), and 'val' (for the average loss on the validation parameters).