Equi-Tuning: Group Equivariant Fine-Tuning of Pretrained Models

Wish that your foundation/pretrained model were group equivariant?

At AAAI’23, we will present “Equi-Tuning: Group Equivariant Fine-Tuning of Pretrained Models” that “equivariantly finetunes’’ pretrained models into equivariant ones while leveraging their pretrained knowledge.

The gif above shows a rotating airplane in the middle, used as an input to a pretrained Alexnet. To the right, we see the features of the Alexnet that changes abruptly with rotation. And to the left, we see the features of the equituned Alexnet.

Implementation

Our idea is to minimize the feature distance between pretrained models and the equituned model. The solution obtained is a parameter sharing scheme where instead of sharing parameters, we share the entire pretrained model as shown below for the c4 group of 90 deg rotations

Our implementation shows that equituning is computationally as efficient as parameter sharing with the advantage of leveraging pretrained knowledge.

Experiments

Our method is simple, general, and easy to use. Hence, we apply it on a variety of tasks, such as image classification, fairness in NLG, and compositional generalization in languages.

For image classification, we find equituning outperforms finetuning

For fairness in NLG, we first define the notion of group-theoretic fairness. Using this new definition, we obtain EquiGPT2 and R-EquiGPT2, which generate debiased texts without losing the quality of generation. Fig.3 below compares bias in GPT2 with EquiGPT2 and R-EquiGPT2.

In compositional generalization, we find that equituning excels at compositional generalization tasks (SCAN dataset) with only a few equituning steps on a pretrained model that has negligible compositional capability.

In conclusion, we provide a simple yet powerful method to convert pretrained models equivariant. These models leverage both pretrained knowledge as well as group equivariance. The simplicity of the algorithm implies its usefulness across a variety of tasks and models. We only show its applications in three downstream tasks: image classification, natural language generation, and compositional generalization. But we believe, this algorithm can be extended to several other domains in the future.

This blog post only provides an overview of equituning with some of its results, for further details, please refer to the preprint available here: https://arxiv.org/abs/2210.06475

This work was done in part during an internship at IBM Research.