Gaussian Error Linear Unit (GELU)

https://paperswithcode.com/method/gelu $GELU(x)=0.5x\left(1+\tanh \left(\sqrt{\frac{2}{\pi }}\left(x+0.044715x^3\right)\right)\right)$

• Similar to ELU (Exponential Linear Unit).
• It looks VERY similar to Sigmoid Linear Unit (Swish)
  • https://towardsdatascience.com/on-the-disparity-between-swish-and-gelu-1ddde902d64b
  • 
    • Swish’s bump (on the right) is a bit lower (and lasts longer)