PositionalEncoding

class neuralhydrology.modelzoo.positional_encoding.PositionalEncoding(embedding_dim, position_type, dropout, max_len=5000)

Bases: Module

Class to create a positional encoding vector for timeseries inputs to a model without an explicit time dimension.

This class implements a sin/cos type embedding vector with a specified maximum length. Adapted from the PyTorch example here: https://pytorch.org/tutorials/beginner/transformer_tutorial.html

Parameters:

• embedding_dim (int) – Dimension of the model input, which is typically output of an embedding layer.

• dropout (float) – Dropout rate [0, 1) applied to the embedding vector.

• max_len (int, optional) – Maximum length of positional encoding. This must be larger than the largest sequence length in the sample.

forward(x: Tensor) → Tensor

Forward pass for positional encoding. Either concatenates or adds positional encoding to encoder input data.

Parameters:

x (torch.Tensor) – Dimension is [sequence length, batch size, embedding output dimension]. Data that is to be the input to a transformer encoder after including positional encoding. Typically this will be output from an embedding layer.

Returns:

Dimension is [sequence length, batch size, encoder input dimension]. The encoder input dimension is either equal to the embedding output dimension (if position_type == sum) or twice the embedding output dimension (if position_type == concatenate). Encoder input augmented with positional encoding.

Return type:

torch.Tensor