The cost of vision-and-language pre-training has become increasingly prohibitive due to end-to-end training of large-scale models. This paper proposes BLIP-2, a generic and efficient pre-training strategy that bootstraps vision-language pre-training from off-the-shelf frozen pre-trained image encoders and frozen large language models. BLIP-2 bridges the modality gap with a lightweight Querying Transformer, which is pre-trained in two stages. The first stage bootstraps vision-language representation learning from a frozen image encoder. The second stage bootstraps vision-to-language generative learning from a frozen language model. BLIP-2 achieves state-of-the-art performance on various vision-language tasks, despite having significantly fewer trainable parameters than existing methods. For example, our model outperforms Flamingo80B by 8.7% on zero-shot VQAv2 with 54x fewer trainable parameters. We also demonstrate the model's emerging capabilities of zero-shot image-to-text generation that can follow natural language instructions.
来源出处
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders an…
http://arxiv.org/abs/2301.12597
相关内容
发布日期
12/12/2023 - 01:18
发布日期
01/21/2024 - 12:12
发布日期
06/17/2022 - 10:21
发布日期
06/17/2022 - 10:21
发布日期
01/10/2022 - 19:31
发布日期
01/22/2024 - 01:44
发布日期
06/17/2022 - 10:21
发布日期
08/23/2024 - 19:21
发布日期
06/17/2022 - 10:21
发布日期
08/18/2024 - 19:19
发布日期
08/04/2020 - 01:35
发布日期
06/17/2022 - 10:21
发布日期
08/04/2020 - 01:35
发布日期
06/17/2022 - 10:21
发布日期
09/10/2023 - 22:37
发布日期
10/31/2021 - 01:12
发布日期
10/31/2021 - 01:48
发布日期
06/17/2022 - 10:21
发布日期
10/12/2023 - 23:10
发布日期
08/04/2020 - 01:35