#ResNet: Deep Residual Learning

An examination of the training curves of very deep neural networks reveals a phenomenon documented by the authors of the 2015 ResNet paper. While it is intuitive to assume that adding more layers to a model should always improve performance—or at least maintain it—researchers found that after a certain point, accuracy begins to drop rapidly. This is not the result of overfitting, as the training error itself increases. The paper identified this as the 'degradation problem,' suggesting that as networks grow deeper, the optimization landscape becomes so complex that standard algorithms struggle to find even a simple identity mapping. This discovery shifted the focus of deep learning from raw model capacity to the fundamental challenge of information flow.

#Residual Learning and Identity Mapping

A residual building block showing the identity shortcut connection that enables H(x) = F(x) + x.

The primary technical shift in ResNet was the introduction of residual learning, which reformulates the task of each layer. Instead of attempting to learn a direct mapping H(x), the network is designed to learn the 'residual' function F(x) = H(x) - x. This is achieved through 'shortcut connections' that simply add the input x back to the output of the weight layers. Theoretically, it should be easy for a network to learn to do nothing—an identity mapping—by setting its weights to zero. However, in standard architectures, approximating an identity mapping through multiple non-linear layers is surprisingly difficult. By making the identity mapping the default state of the network, ResNet proved that deep models are more efficient when they only need to learn the subtle deviations from their inputs rather than reconstructing the entire signal from scratch at every stage.