A new benchmark called MMLongBench-Doc, jointly developed by researchers from Nanyang Technological University and other institutions, comprehensively evaluates the long document understanding capabilities of large visual language models (LVLMs) using 135 PDF documents spanning multiple domains. The results indicate that existing models still face significant challenges in cross-page understanding, multimodal information fusion, and other aspects, highlighting the urgent need for further improvements.
Introduction to Document Understanding
Document understanding (DU) focuses on the automatic interpretation and processing of documents, encompassing complex layout structures and multimodal elements such as text, tables, charts, and images. This task is crucial for extracting and utilizing the information contained in the vast amounts of documents generated each year.
One key challenge lies in understanding long-context documents that span multiple pages and require comprehension across various modalities and pages. Traditional single-page DU models struggle to address this issue, making it essential to develop benchmarks for evaluating model performance on long documents. Researchers have found that these long-context documents require specific capabilities, such as locating and cross-page understanding, which current single-page DU datasets do not adequately address.
Paper Link：https://arxiv.org/abs/2407.01523
The Importance of Document Understanding
As the volume of unstructured document data continues to grow exponentially, effective DU solutions have become increasingly critical for unlocking the valuable insights hidden within these documents.
According to a recent report by Gartner, 80% of enterprise data is unstructured, and this proportion is expected to rise to 90% by 2025. The ability to efficiently process and extract meaningful information from this vast trove of data could have transformative impacts across industries, from healthcare and finance to legal services and beyond.
However, traditional single-page DU models struggle to handle long-context documents that require understanding across multiple pages and modalities. This limitation has hindered the development of truly comprehensive DU solutions capable of tackling real-world document understanding challenges.
Current Approaches and Limitations
Recent advancements in large visual language models (LVLMs) have shown promise in addressing the complexities of document understanding. LVLMs, such as GPT-4o, Gemini-1.5, and Claude-3, developed by leading AI research organizations like OpenAI and Anthropic, have demonstrated impressive performance on single-page DU tasks.
However, these models still face significant challenges when it comes to long-context document understanding. The need to comprehend and integrate information across multiple pages and modalities poses a formidable obstacle for current LVLM architectures. As we discussed in our previous article on multimodal learning, the integration of visual and textual information is crucial for advanced AI systems.
At a fundamental level, LVLMs differ from traditional language models in their ability to process both visual and textual inputs. While this multimodal capability is essential for document understanding, the increased complexity of the input data and the need to maintain long-range dependencies across pages strain the models’ ability to effectively capture and utilize the relevant information.
Moreover, the scarcity of comprehensive benchmarks specifically designed to evaluate long-context DU performance has hindered the development of more advanced LVLM architectures. Without rigorous and representative test beds, researchers have lacked the necessary tools to identify and address the limitations of existing models.
MMLongBench-Doc: A Comprehensive Benchmark
To bridge this critical gap, the team behind MMLongBench-Doc has created a benchmark that consists of 135 PDF documents from various domains, averaging 47.5 pages and 21,214.1 text tokens each. The benchmark includes 1,091 questions that require evidence from text, images, charts, tables, and layout structures, with a significant portion necessitating cross-page understanding.
As Dr. Xiang Yao, one of the lead researchers involved in developing MMLongBench-Doc, explains, “Our goal was to create a benchmark that truly pushes the boundaries of what current DU models can achieve. By incorporating a diverse range of document types and question formats, we aim to provide a comprehensive assessment of LVLM capabilities and identify areas for further improvement.”
The benchmark’s construction process was rigorous and meticulous, involving ten expert annotators who edited questions from existing datasets and created new ones to ensure comprehensive coverage. A three-round semi-automatic review process was employed to maintain high quality and consistency across the benchmark.
One of the key innovations of MMLongBench-Doc is its use of screenshots of document pages as input to the LVLMs, rather than relying solely on OCR-parsed text. This approach allows for a more realistic evaluation of the models’ ability to handle the visual complexity of real-world documents.
Performance Evaluation and Challenges Revealed
The performance evaluation of LVLMs on MMLongBench-Doc reveals significant challenges in long-context document understanding. The best-performing model, GPT-4o, achieves an F1 score of 44.9%, while the second-ranked GPT-4V has an F1 score of 30.5%. Other models, such as Gemini-1.5 and Claude-3, exhibit even lower performance.
These results underscore the substantial room for improvement in LVLM capabilities when it comes to handling lengthy, multimodal documents. The study also compared these results to OCR-based models, noting that some LVLMs perform worse than unimodal language models when given imperfect OCR-parsed text as input.
A deeper analysis of the results shows that while LVLMs can handle multimodal inputs to some extent, their abilities still fall short in several key areas. For instance, 33.0% of the questions in the benchmark are cross-page questions requiring multi-page understanding, and 22.5% are designed to be unanswerable to detect potential hallucinations.
Dr. Yao comments on the implications of these findings: “The performance gaps we observed between different models and question types highlight the need for more advanced LVLM architectures that can effectively integrate information across pages and modalities. This is an area where we expect to see significant research and development efforts in the coming years.”
Conclusion and Future Directions
The MMLongBench-Doc benchmark has shed light on the complexities and challenges of long-context document understanding, emphasizing the need for advanced models capable of processing and comprehending lengthy, multimodal documents effectively. By providing a rigorous and comprehensive evaluation tool, this benchmark serves as a valuable resource for researchers and developers working to push the boundaries of document understanding.
As Dr. Yao concludes, “Our hope is that MMLongBench-Doc will inspire and guide the development of more sophisticated LVLMs that can tackle the real-world challenges of long-context document understanding. By advancing the state of the art in this critical area, we can unlock the vast potential of unstructured document data and drive transformative impacts across industries.”
The findings of this study underscore the significant challenges that remain in long-context document understanding and the need for continued research and innovation in this field. As the volume of unstructured document data continues to grow, the development of effective and comprehensive DU solutions will be essential for harnessing the full value of this information.