MMPrune4U: Regularizing Multimodal Feature Distortion in Weight Pruning for Deep Neural Network Compression

Despite the remarkable success of multimodal models in automotive applications, their practical benefits are often accompanied by a large number of parameters, including redundant and excessive weights. This poses hurdles to their deployment on embedded devices due to the substantial computational costs compared to unimodal models. Model sparsification is among the common solutions to reduce the resources required for computation and increase throughput of the system. Although many recent studies in model sparsification and pruning achieve remarkable performance for unimodal models, they overlook capturing the layer-wise sensitivity towards accuracy and behaviors for distinct modalities in response to the pruning, leading to information loss in the downstream tasks of the pruned model. We introduce MMPrune4U, a layer-adaptive weight pruning method explicitly designed to support multimodal 3D scene understanding that incorporates a regularizer based on log-Sobolev inequality. This approach uncovers a crucial property related to the distortion of features resulting from pruning weights across multiple layers while keeping a predefined pruning ratio. As per the changes in the output distribution of the each layer during pruning compared to unpruned model, we regularize the distortion through the functional Fisher information. We formulate our layer-adaptive pruning by considering the layerwise impact to the downstream tasks and optimise the objective function through combinatorial optimization challenge, which we effectively address using dynamic programming techniques. The proposed MMPrune4U method demonstrates superior performance in comparison to the existing state-of-the-art methods, as shown by experimental results on both nuScenes and SemanticKITTI datasets.