GELU¶

class dragon.vm.torch.nn.GELU(approximate='none')[source]¶

Apply the gaussian error linear unit. [Hendrycks & Gimpel, 2016].

The GELU function is defined as:

\[\text{GELU}(x) = x\cdot\frac{1}{2}[1 + \text{erf}(x / \sqrt{2})] \]

Examples:

m = torch.nn.GELU()
x = torch.randn(2, 3)
y = m(x)

See also

torch.nn.functional.gelu(…)