Python v0.4.0a0+62b766a

TransformerEncoderLayer

class dragon.vm.torch.nn.TransformerEncoderLayer(
  d_model,
  nhead,
  dim_feedforward=2048,
  dropout=0.1,
  activation='relu',
  norm_first=False
)[source]

Layer for a standard transformer encoder . [Vaswani et.al, 2017].

Examples:

src = torch.ones(4, 2, 8)
encoder_layer = torch.nn.TransformerEncoderLayer(d_model=8, nhead=2)
out = encoder_layer(src)

__init__

TransformerEncoderLayer.__init__(
  d_model,
  nhead,
  dim_feedforward=2048,
  dropout=0.1,
  activation='relu',
  norm_first=False
)[source]

Create a TransformerEncoderLayer.

Parameters:
  • d_model (int) – The dimension of features.
  • nhead (int) – The number of parallel heads.
  • dim_feedforward (int, optional, default=2048) – The dimension of feedforward network.
  • dropout (float, optional, default=0.1) – The dropout ratio.
  • activation (str, optional, default='relu') – The activation function.
  • norm_first (bool, optional, default=False) – Apply layer form before attention and feedforward.