Vis-IR: Unifying Search With Visualized Information Retrieval

</a>
<a href="https://github.com/VectorSpaceLab/Vis-IR">
    
</a>
<a href="https://huggingface.co/datasets/marsh123/VIRA/">
    
</a>  
<a href="https://huggingface.co/datasets/marsh123/MVRB">
    
</a>  
<!-- 
    
  -->
<a href="https://huggingface.co/marsh123/UniSE">
    
</a>

2025-04-02 🚀🚀 VIRA Dataset are released on Huggingface: VIRA

2025-04-01 🚀🚀 UniSE models are released on Huggingface: UniSE-MLMM

2025-02-17 🎉🎉 Release our paper: Any Information Is Just Worth One Single Screenshot: Unifying Search With Visualized Information Retrieval.

Release Plan

Paper
UniSE models
VIRA Dataset
MVRB benchmark
Evaluation code
Fine-tuning code

Overview

In this work, we formally define an emerging IR paradigm called Visualized Information Retrieval, or VisIR, where multimodal information, such as texts, images, tables and charts, is jointly represented by a unified visual format called Screenshots, for various retrieval applications. We further make three key contributions for VisIR. First, we create VIRA (Vis-IR Aggregation), a large-scale dataset comprising a vast collection of screenshots from diverse sources, carefully curated into captioned and questionanswer formats. Second, we develop UniSE (Universal Screenshot Embeddings), a family of retrieval models that enable screenshots to query or be queried across arbitrary data modalities. Finally, we construct MVRB (Massive Visualized IR Benchmark), a comprehensive benchmark covering a variety of task forms and application scenarios. Through extensive evaluations on MVRB, we highlight the deficiency from existing multimodal retrievers and the substantial improvements made by UniSE.

Model Usage

Using Sentence Transformers

Install Sentence Transformers:

pip install "sentence_transformers[image]"

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("BAAI/BGE-VL-Screenshot")

# Queries: composed image + text inputs (prefix text with "Query:")
query_inputs = [
    {"text": "Query:After a 17% drop, what is Nvidia's closing stock price?", "image": "https://huggingface.co/BAAI/BGE-VL-Screenshot/resolve/main/assets/query_1.png"},
    {"text": "Query:I would like to see a detailed and intuitive performance comparison between the two models.", "image": "https://huggingface.co/BAAI/BGE-VL-Screenshot/resolve/main/assets/query_2.png"},
]
query_embeddings = model.encode_query(query_inputs)
print(query_embeddings.shape)
# (2, 2048)

# Candidates: screenshot images
candidate_inputs = [
    "https://huggingface

26 个文件

浏览文件

数据集版权信息

本数据集的许可证为 MIT License。如有违反相关条款，请联系 WEHUB，我们将及时处理。查看许可证

BAAI/BGE-VL-Screenshot 已完整同步