BDC-Adapter: Brownian Distance Covariance for Better Vision-Language Reasoning
Yi Zhang (Southern University of Science and Technology), Ce Zhang (Carnegie Mellon University), Zihan Liao (Southern University of Science and Technology), Yushun Tang (Southern University of Science and Technology), Zhihai He (Southern University of Science and Technology)*
The 34th British Machine Vision Conference
Abstract
Large-scale pre-trained Vision-Language Models (VLMs), such as CLIP and ALIGN, have introduced a new paradigm for learning transferable visual representations. Recently, there has been a surge of interest among researchers in developing lightweight fine-tuning techniques to adapt these models to downstream visual tasks. We recognize that current state-of-the-art fine-tuning methods, such as Tip-Adapter, simply consider the covariance between the query image feature and features of support few-shot training samples, which only captures linear relations and potentially instigates a deceptive perception of independence. To address this issue, in this work, we innovatively introduce Brownian Distance Covariance (BDC) to the field of vision-language reasoning. The BDC metric can model all possible relations, providing a robust metric for measuring feature dependence. Based on this, we present a novel method called BDC-Adapter, which integrates BDC prototype similarity reasoning and multi-modal reasoning network prediction to perform classification tasks. Our extensive experimental results show that the proposed BDC-Adapter can freely handle non-linear relations and fully characterize independence, outperforming the current state-of-the-art methods by large margins.Video
Citation
@inproceedings{Zhang_2023_BMVC,
author = {Yi Zhang and Ce Zhang and Zihan Liao and Yushun Tang and Zhihai He},
title = {BDC-Adapter: Brownian Distance Covariance for Better Vision-Language Reasoning},
booktitle = {34th British Machine Vision Conference 2023, {BMVC} 2023, Aberdeen, UK, November 20-24, 2023},
publisher = {BMVA},
year = {2023},
url = {https://papers.bmvc2023.org/0182.pdf}
}
Copyright © 2023
The British Machine Vision Conference is organised by