Google research introduces agile deliberation framework for subjective image classification

Google Research has unveiled Agile Deliberation, a novel human-in-the-loop framework designed to build image classifiers for concepts lacking clear boundaries, such as identifying 'healthy food' or 'inspiring artwork'. Unlike existing tools that assume concept definitions are known, this framework assists users in establishing those subjective boundaries by transforming vague ideas into structured, written policies that also serve as prompts for Vision-Language Model (VLM) classifiers. The methodology, developed following interviews with content moderation specialists, involves a two-stage process. Stage 1 uses prompt chaining with Gemini 2.5 Pro to break down a concept into up to three unit concepts, presenting users with candidate positive and negative subconcepts alongside retrieved images for refinement. Stage 2 bypasses traditional uncertainty sampling, instead focusing on finding images that are semantically borderline, given the uncalibrated confidence scores of the generative VLM. The research team conducted 18 live sessions, each lasting approximately 90 minutes, involving nine participants without prior professional moderation experience. These sessions compared Agile Deliberation against manual deliberation for the concepts of 'paid to play' clickbait and 'healthy food'. The framework is available via a Demo.ipynb, which requires a Gemini API key and utilizes Gemini-2.5-Pro and Gemini-2.5-Flash, and includes a setup script to build a FAISS index over LAION-400M CLIP embeddings.