What is the ReLU activation function?
ReLU, short for Rectified Linear Unit, is one of the most widely used activation functions in modern deep learning and neural networks. It is defined as \(f(x) = \max(0,\ x)\), meaning it passes positive inputs through unchanged and clamps negative inputs (and zero) to 0. This simple rule introduces non-linearity into a network while remaining extremely cheap to compute, which is why it powers most convolutional and fully connected layers today.
How to use this calculator
Enter any real number in the x field and the calculator returns \(f(x) = \text{ReLU}(x)\). Negative, zero, and positive values are all valid. The result equals x when x is greater than 0, and equals 0 when x is 0 or negative. The calculator also reports the conventional derivative: 1 for positive inputs and 0 otherwise.
The formula explained
The ReLU function is defined piecewise: \(f(x) = x\) if \(x > 0\), and \(f(x) = 0\) if \(x \le 0\). Its domain is all real numbers \((-\infty,\ +\infty)\) and its range is \([0,\ +\infty)\). ReLU is continuous everywhere, but its derivative is technically undefined exactly at \(x = 0\); by convention it is set to 0 there, giving \(f'(x) = 0\) for \(x \le 0\) and \(f'(x) = 1\) for \(x > 0\). Because there is no division involved, there are no edge cases to guard against.
Worked example
Suppose \(x = -3.2\). Then $$f(x) = \max(0,\ -3.2) = 0$$ since -3.2 is negative. If instead \(x = 7\), then $$f(x) = \max(0,\ 7) = 7$$ For the default input \(x = 0.5\), $$f(x) = \max(0,\ 0.5) = 0.5$$
FAQ
Why is ReLU so popular? It avoids the vanishing-gradient problem that plagues sigmoid and tanh for large inputs, and it is computationally trivial — just a comparison with zero.
What happens at \(x = 0\)? The function value is 0, and the derivative is conventionally taken as 0.
What is the difference between ReLU, Sigmoid and Softmax? Sigmoid squashes values into \((0,\ 1)\) and Softmax produces a probability distribution over a vector, while ReLU simply rectifies a single value to be non-negative.
