pymor.reductors.neural_network
¶
Module Contents¶
- class pymor.reductors.neural_network.CustomDataset(training_data)[source]¶
Bases:
torch.utils.data.Dataset
Class that represents the dataset to use in PyTorch.
Parameters
- training_data
Set of training parameters and the respective coefficients of the solution in the reduced basis.
- class pymor.reductors.neural_network.EarlyStoppingScheduler(size_training_validation_set, patience=10, delta=0.0)[source]¶
Bases:
pymor.core.base.BasicObject
Class for performing early stopping in training of neural networks.
If the validation loss does not decrease over a certain amount of epochs, the training should be aborted to avoid overfitting the training data. This class implements an early stopping scheduler that recommends to stop the training process if the validation loss did not decrease by at least
delta
overpatience
epochs.Parameters
- size_training_validation_set
Size of both, training and validation set together.
- patience
Number of epochs of non-decreasing validation loss allowed, before early stopping the training process.
- delta
Minimal amount of decrease in the validation loss that is required to reset the counter of non-decreasing epochs.
- class pymor.reductors.neural_network.NeuralNetworkInstationaryReductor(fom=None, training_set=None, validation_set=None, validation_ratio=0.1, T=None, basis_size=None, rtol=0.0, atol=0.0, l2_err=0.0, pod_params={}, ann_mse='like_basis', scale_inputs=True, scale_outputs=False)[source]¶
Bases:
NeuralNetworkReductor
Reduced Basis reductor for instationary problems relying on artificial neural networks.
This is a reductor that constructs a reduced basis using proper orthogonal decomposition and trains a neural network that approximates the mapping from parameter and time space to coefficients of the full-order solution in the reduced basis. The approach is described in [WHR19].
Parameters
- fom
- training_set
- validation_set
- validation_ratio
- T
The final time T used in case
fom
isNone
.- basis_size
- rtol
- atol
- l2_err
- pod_params
- ann_mse
- scale_inputs
- scale_outputs
Methods
Compute a reduced basis using proper orthogonal decomposition.
- class pymor.reductors.neural_network.NeuralNetworkInstationaryStatefreeOutputReductor(fom=None, nt=1, training_set=None, validation_set=None, validation_ratio=0.1, T=None, validation_loss=None, scale_inputs=True, scale_outputs=False)[source]¶
Bases:
NeuralNetworkStatefreeOutputReductor
Output reductor relying on artificial neural networks.
This is a reductor that trains a neural network that approximates the mapping from parameter space to output space.
Parameters
- fom
- nt
- training_set
- validation_set
- T
- validation_ratio
- validation_loss
- scale_inputs
- scale_outputs
Methods
Compute the training samples (the outputs to the parameters of the training set).
- class pymor.reductors.neural_network.NeuralNetworkLSTMInstationaryReductor(fom=None, training_set=None, validation_set=None, validation_ratio=0.1, T=None, basis_size=None, rtol=0.0, atol=0.0, l2_err=0.0, pod_params={}, ann_mse='like_basis', scale_inputs=True, scale_outputs=False)[source]¶
Bases:
NeuralNetworkInstationaryReductor
Reduced Basis reductor for instationary problems relying on LSTM neural networks.
This is a reductor that constructs a reduced basis using proper orthogonal decomposition and trains an LSTM neural network that approximates the mapping from parameter to coefficients of the full-order solution in the reduced basis for a fixed number of timesteps.
Methods
Reduce by LSTM neural networks.
- reduce(hidden_dimension='3*N + P', number_layers=1, optimizer=optim.LBFGS, epochs=1000, batch_size=20, learning_rate=1.0, loss_function=None, restarts=10, lr_scheduler=None, lr_scheduler_params={}, es_scheduler_params={'patience': 10, 'delta': 0.0}, weight_decay=0.0, log_loss_frequency=0)[source]¶
Reduce by LSTM neural networks.
Parameters
- hidden_dimension
Number of neurons in the hidden state of the LSTM. Can either be fixed or a Python expression string depending on the reduced basis size respectively output dimension
N
and the total dimension of theParameters
P
.- number_layers
Number of recurred layers, i.e. number of stacked LSTM cells in the neural network.
- optimizer
- epochs
- batch_size
- learning_rate
- loss_function
- restarts
- lr_scheduler
- lr_scheduler_params
- es_scheduler_params
- weight_decay
- log_loss_frequency
- class pymor.reductors.neural_network.NeuralNetworkLSTMInstationaryStatefreeOutputReductor(fom=None, nt=1, training_set=None, validation_set=None, validation_ratio=0.1, T=None, validation_loss=None, scale_inputs=True, scale_outputs=False)[source]¶
Bases:
NeuralNetworkInstationaryStatefreeOutputReductor
,NeuralNetworkLSTMInstationaryReductor
Output reductor relying on LSTM neural networks.
This is a reductor that trains an LSTM neural network that approximates the mapping from parameter space to output space.
- class pymor.reductors.neural_network.NeuralNetworkReductor(fom=None, training_set=None, validation_set=None, validation_ratio=0.1, basis_size=None, rtol=0.0, atol=0.0, l2_err=0.0, pod_params={}, ann_mse='like_basis', scale_inputs=True, scale_outputs=False)[source]¶
Bases:
pymor.core.base.BasicObject
Reduced Basis reductor relying on artificial neural networks.
This is a reductor that constructs a reduced basis using proper orthogonal decomposition and trains a neural network that approximates the mapping from parameter space to coefficients of the full-order solution in the reduced basis. The approach is described in [HU18].
Parameters
- fom
The full-order
Model
to reduce. IfNone
, thetraining_set
has to consist of pairs ofparameter values
and corresponding solutionVectorArrays
.- training_set
Set of
parameter values
to use for POD and training of the neural network. Iffom
isNone
, thetraining_set
has to consist of pairs ofparameter values
and corresponding solutionVectorArrays
.- validation_set
Set of
parameter values
to use for validation in the training of the neural network. Iffom
isNone
, thevalidation_set
has to consist of pairs ofparameter values
and corresponding solutionVectorArrays
.- validation_ratio
Fraction of the training set to use for validation in the training of the neural network (only used if no validation set is provided). Either a validation set or a positive validation ratio is required.
- basis_size
Desired size of the reduced basis. If
None
, rtol, atol or l2_err must be provided.- rtol
Relative tolerance the basis should guarantee on the training set.
- atol
Absolute tolerance the basis should guarantee on the training set.
- l2_err
L2-approximation error the basis should not exceed on the training set.
- pod_params
Dict of additional parameters for the POD-method.
- ann_mse
If
'like_basis'
, the mean squared error of the neural network on the training set should not exceed the error of projecting onto the basis. IfNone
, the neural network with smallest validation error is used to build the ROM. If a tolerance is prescribed, the mean squared error of the neural network on the training set should not exceed this threshold. Training is interrupted if a neural network that undercuts the error tolerance is found.- scale_inputs
Determines whether or not to scale the inputs of the neural networks.
- scale_outputs
Determines whether or not to scale the outputs/targets of the neural networks.
Methods
Compute a reduced basis using proper orthogonal decomposition.
Reconstruct high-dimensional vector from reduced vector
u
.Reduce by training artificial neural networks.
- reduce(hidden_layers='[(N+P)*3, (N+P)*3]', activation_function=torch.tanh, optimizer=optim.LBFGS, epochs=1000, batch_size=20, learning_rate=1.0, loss_function=None, restarts=10, lr_scheduler=optim.lr_scheduler.StepLR, lr_scheduler_params={'step_size': 10, 'gamma': 0.7}, es_scheduler_params={'patience': 10, 'delta': 0.0}, weight_decay=0.0, log_loss_frequency=0)[source]¶
Reduce by training artificial neural networks.
Parameters
- hidden_layers
Number of neurons in the hidden layers. Can either be fixed or a Python expression string depending on the reduced basis size respectively output dimension
N
and the total dimension of theParameters
P
.- activation_function
Activation function to use between the hidden layers.
- optimizer
Algorithm to use as optimizer during training.
- epochs
Maximum number of epochs for training.
- batch_size
Batch size to use if optimizer allows mini-batching.
- learning_rate
Step size to use in each optimization step.
- loss_function
Loss function to use for training. If
'weighted MSE'
, a weighted mean squared error is used as loss function, where the weights are given as the singular values of the corresponding reduced basis functions. IfNone
, the usual mean squared error is used.- restarts
Number of restarts of the training algorithm. Since the training results highly depend on the initial starting point, i.e. the initial weights and biases, it is advisable to train multiple neural networks by starting with different initial values and choose that one performing best on the validation set.
- lr_scheduler
Algorithm to use as learning rate scheduler during training. If
None
, no learning rate scheduler is used.- lr_scheduler_params
A dictionary of additional parameters passed to the init method of the learning rate scheduler. The possible parameters depend on the chosen learning rate scheduler.
- es_scheduler_params
A dictionary of additional parameters passed to the init method of the early stopping scheduler. For the possible parameters, see
EarlyStoppingScheduler
.- weight_decay
Weighting parameter for the l2-regularization of the weights and biases in the neural network. This regularization is not available for all optimizers; see the PyTorch documentation for more details.
- log_loss_frequency
Frequency of epochs in which to log the current validation and training loss during training of the neural networks. If
0
, no intermediate logging of losses is done.
Returns
- rom
Reduced-order
NeuralNetworkModel
.
- class pymor.reductors.neural_network.NeuralNetworkStatefreeOutputReductor(fom=None, training_set=None, validation_set=None, validation_ratio=0.1, validation_loss=None, scale_inputs=True, scale_outputs=False)[source]¶
Bases:
NeuralNetworkReductor
Output reductor relying on artificial neural networks.
This is a reductor that trains a neural network that approximates the mapping from parameter space to output space.
Parameters
- fom
The full-order
Model
to reduce. IfNone
, thetraining_set
has to consist of pairs ofparameter values
and corresponding outputs.- training_set
Set of
parameter values
to use for POD and training of the neural network. Iffom
isNone
, thetraining_set
has to consist of pairs ofparameter values
and corresponding outputs.- validation_set
Set of
parameter values
to use for validation in the training of the neural network. Iffom
isNone
, thevalidation_set
has to consist of pairs ofparameter values
and corresponding outputs.- validation_ratio
- validation_loss
The validation loss to reach during training. If
None
, the neural network with the smallest validation loss is returned.- scale_inputs
- scale_outputs
Methods
Compute the training samples (the outputs to the parameters of the training set).
- pymor.reductors.neural_network.multiple_restarts_training(training_data, validation_data, neural_network, target_loss=None, max_restarts=10, log_loss_frequency=0, training_parameters={}, scaling_parameters={})[source]¶
Algorithm that performs multiple restarts of neural network training.
This method either performs a predefined number of restarts and returns the best trained network or tries to reach a given target loss and stops training when the target loss is reached.
See
train_neural_network
for more information on the parameters.Parameters
- training_data
Data to use during the training phase.
- validation_data
Data to use during the validation phase.
- neural_network
The neural network to train (parameters will be reset after each restart).
- target_loss
Loss to reach during training (if
None
, the network with the smallest loss is returned).- max_restarts
Maximum number of restarts to perform.
- log_loss_frequency
Frequency of epochs in which to log the current validation and training loss. If
0
, no intermediate logging of losses is done.- training_parameters
Additional parameters for the training algorithm, see
train_neural_network
for more information.- scaling_parameters
Additional parameters for scaling inputs respectively outputs, see
train_neural_network
for more information.
Returns
- best_neural_network
The best trained neural network.
- losses
The corresponding losses.
Raises
- NeuralNetworkTrainingError
Raised if prescribed loss can not be reached within the given number of restarts.
- pymor.reductors.neural_network.train_neural_network(training_data, validation_data, neural_network, training_parameters={}, scaling_parameters={}, log_loss_frequency=0)[source]¶
Training algorithm for artificial neural networks.
Trains a single neural network using the given training and validation data.
Parameters
- training_data
Data to use during the training phase. Has to be a list of tuples, where each tuple consists of two elements that are either PyTorch-tensors (
torch.DoubleTensor
) orNumPy arrays
or pyMOR data structures that haveto_numpy()
implemented. The first element contains the input data, the second element contains the target values.- validation_data
Data to use during the validation phase. Has to be a list of tuples, where each tuple consists of two elements that are either PyTorch-tensors (
torch.DoubleTensor
) orNumPy arrays
or pyMOR data structures that haveto_numpy()
implemented. The first element contains the input data, the second element contains the target values.- neural_network
The neural network to train (can also be a pre-trained model). Has to be a PyTorch-Module.
- training_parameters
Dictionary with additional parameters for the training routine like the type of the optimizer, the (maximum) number of epochs, the batch size, the learning rate or the loss function to use. Possible keys are
'optimizer'
(an optimizer from the PyTorchoptim
package; if not provided, the LBFGS-optimizer is taken as default),'epochs'
(an integer that determines the number of epochs to use for training the neural network (if training is not interrupted prematurely due to early stopping); if not provided, 1000 is taken as default value),'batch_size'
(an integer that determines the number of samples to pass to the optimizer at once; if not provided, 20 is taken as default value; not used in the case of the LBFGS-optimizer since LBFGS does not support mini-batching),'learning_rate'
(a positive real number used as the (initial) step size of the optimizer; if not provided, 1 is taken as default value),'loss_function'
(a loss function from PyTorch; if not provided, the MSE loss is taken as default),'lr_scheduler'
(a learning rate scheduler from the PyTorchoptim.lr_scheduler
package; if not provided orNone
, no learning rate scheduler is used),'lr_scheduler_params'
(a dictionary of additional parameters for the learning rate scheduler),'es_scheduler_params'
(a dictionary of additional parameters for the early stopping scheduler), and'weight_decay'
(non-negative real number that determines the strength of the l2-regularization; if not provided or 0., no regularization is applied).- scaling_parameters
Dict of tensors that determine how to scale inputs before passing them through the neural network and outputs after obtaining them from the neural network. If not provided or each entry is
None
, no scaling is applied. Required keys are'min_inputs'
,'max_inputs'
,'min_targets'
, and'max_targets'
.- log_loss_frequency
Frequency of epochs in which to log the current validation and training loss. If
0
, no intermediate logging of losses is done.
Returns
- best_neural_network
The best trained neural network with respect to validation loss.
- losses
The corresponding losses as a dictionary with keys
'full'
(for the full loss containing the training and the validation average loss),'train'
(for the average loss on the training set), and'val'
(for the average loss on the validation set).