In the realm of deep learning, computational efficiency is paramount, especially for deployment on resource-constrained devices like mobile phones. MobileNets represent a family of convolutional neural networks (CNNs) designed to achieve high accuracy while significantly reducing computational cost. A key innovation enabling this efficiency is the depthwise separable convolution.

Before diving into depthwise separable convolutions, let's briefly recap standard convolutions. A standard convolutional layer performs two operations simultaneously: spatial filtering (across width and height) and depthwise filtering (across channels). It applies a set of learnable filters to an input volume, producing an output volume where each output channel is a weighted sum of the input channels, filtered spatially.

Standard convolutions are computationally expensive because they combine spatial and channel-wise operations. For an input volume of size $D_K \times D_K \times D_M$ and an output volume of size $D_K \times D_K \times D_N$, where $D_K$ is the kernel size and $D_M$ and $D_N$ are the input and output channel dimensions respectively, the computational cost is proportional to $D_K^2 \times D_M \times D_N$. This cost grows rapidly with the number of channels.

Depthwise separable convolutions decouple the standard convolution into two distinct steps: a depthwise convolution and a pointwise convolution.

The computational cost of a depthwise separable convolution is significantly lower than that of a standard convolution. For the same input and output dimensions, the cost is approximately $D_K^2 \times D_M + D_M \times D_N$. This is a reduction by a factor of roughly $1/D_N + 1/D_K^2$. For typical kernel sizes ($D_K=3$) and a reasonable number of channels, this leads to substantial computational savings, often an order of magnitude.

MobileNets leverage depthwise separable convolutions as their building blocks. Different versions of MobileNets (v1, v2, v3) introduce further optimizations such as width multipliers, resolution multipliers, inverted residuals, and linear bottlenecks to further enhance efficiency and accuracy. These architectures are crucial for enabling complex deep learning models on mobile devices and in real-time applications.

The development of MobileNets and the concept of depthwise separable convolutions have had a profound impact on the field of computer vision. They have democratized the use of deep learning models by making them accessible on a wide range of devices, enabling applications like real-time object detection, image classification on smartphones, and on-device natural language processing.