Microsoft’s Florence-2: Ultimate AI Vision Model

Florence-2 is an innovative vision foundation model capable of understanding text prompts and performing a variety of tasks including image captioning, object detection, and segmentation. It was trained on a large dataset called FLD-5B, which contains over 126 million images and 5.4 billion annotations, enabling the model’s multi-task learning.

Florence-2 boasts exceptional OCR capabilities, particularly in recognizing handwritten text.

Florence-2 Usage Scenario

The Florence-2 vision model supports multiple tasks such as image captioning, object detection, image segmentation, OCR, and more. The list of supported tasks is shown in the image below:

OCR

OCR with Region

Object Detection

Detailed Caption

Online demo: https://huggingface.co/spaces/gokaygokay/Florence-2

Florence-2 Model Information:

• Florence-2-base
• Florence-2-large
• Florence-2-base-ft
• Florence-2-large-ft

Getting Started with Florence-2

The model can perform different tasks by modifying the prompt. First, let’s define a function to run prompts.

import requests

from PIL import Image  
from transformers import AutoProcessor, AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("microsoft/Florence-2-large", trust_remote_code=True)
processor = AutoProcessor.from_pretrained("microsoft/Florence-2-large", trust_remote_code=True)

url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg?download=true"
image = Image.open(requests.get(url, stream=True).raw)

def run_example(task_prompt, text_input=None):
    if text_input is None:
        prompt = task_prompt
    else:
        prompt = task_prompt + text_input

    inputs = processor(text=prompt, images=image, return_tensors="pt")
    generated_ids = model.generate(
      input_ids=inputs["input_ids"], 
      pixel_values=inputs["pixel_values"],
      max_new_tokens=1024,
      num_beams=3
    )
    generated_text = processor.batch_decode(generated_ids, skip_special_tokens=False)[0]

    parsed_answer = processor.post_process_generation(generated_text, task=task_prompt, image_size=(image.width, image.height))

    print(parsed_answer)

Then set the prompt to perform the corresponding task:

python prompt = "<CAPTION>" 
run_example(prompt)

Paper: https://arxiv.org/abs/2311.06242