--- title: QB-Highlights canonical_url: "https://www.modelscope.ai/models/gclonghorn/QB-Highlights" md_url: "https://www.modelscope.ai/models/gclonghorn/QB-Highlights.md" repository: gclonghorn/QB-Highlights last_updated: 2026-10-13 license: apache-2.0 model_type: - qwen3_vl architectures: - Qwen3VLForConditionalGeneration base_model: - Qwen/Qwen3-VL-8B-Instruct base_model_relation: finetune parameters: 8.8B tensor_type: - BF16 library_name: - pytorch - transformer - safetensors language: - en downloads: 14 stars: 0 tags: - video-understanding - highlight-detection - temporal-grounding - qwen3-vl - grpo --- # QB-Highlights > QB-Highlights - An open-source model by gclonghorn on ModelScope. Quality-guided budgeted highlight detection (QB-Highlights, QCR checkpoint): Qwen3-VL-8B-Instruct + SFT (17.5K CoT) + GRPO (49.1K). gclonghorn/QB-Highlights is a 8.8B-parameter machine learning model on ModelScope. licensed under apache-2.0. derived from Qwen/Qwen3-VL-8B-Instruct. - **Repository**: gclonghorn/QB-Highlights - **License**: apache-2.0 - **Parameters**: 8.8B - **Base model**: Qwen/Qwen3-VL-8B-Instruct - **Tags**: video-understanding, highlight-detection, temporal-grounding, qwen3-vl, grpo - **Downloads**: 14 - **Stars**: 0 - **Last updated**: 2026-10-13 Source: https://www.modelscope.ai/models/gclonghorn/QB-Highlights --- # QB-Highlights (QCR) Model checkpoint for **QB-Highlights: Quality-Guided Budgeted Highlight Detection in Videos with Dense Query-Relevant Moments**. Given a full-length video and a text query, the model produces a **10-second highlight** as a small set of high-quality, query-relevant segments. It reasons in two steps — a *quality-guided global scan* over candidate segments, followed by a *budgeted set composition* — then emits structured JSON with `MM:SS.xx` timestamps. This checkpoint is the final **QCR** model: [Qwen3-VL-8B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct) fine-tuned with Stage-1 SFT (chain-of-thought annotations) and Stage-2 GRPO (set-level rewards), with the LoRA adapter **merged** into the base weights — it loads directly with `transformers`, no PEFT required. ## Model details | | | |---|---| | Base model | Qwen3-VL-8B-Instruct (`Qwen3VLForConditionalGeneration`) | | Parameters | ~8.7B, bfloat16 | | Video sampling | 1 fps, up to 180 frames | | Max context / completion | 32,768 / 4,096 tokens | | Transformers | >= 4.57.0 (developed with 4.57.0) | ## Training **Stage 1 — SFT (17.5K chain-of-thought samples).** Full fine-tuning with the vision encoder frozen; learning rate 1e-5 (aligner 5e-5); maximum sequence length 49,152; 3 epochs; effective batch size 128; cosine schedule; timestamp-aware loss scaling. **Stage 2 — GRPO (49.1K samples).** LoRA (r=64, alpha=128, all-linear) on top of the Stage-1 model with the vision encoder frozen; learning rate 5e-6; 8 generations per prompt; temperature 1.0; KL coefficient 0.1; effective batch size 96; vLLM-colocated rollout. Reward: ``` R = 0.6 * R_qual + 0.2 * R_comp + 0.2 * R_ground, R_qual = 0.5 * R_global + 0.5 * R_local ``` - `R_global` — union IoU between the merged predicted clips and the ground-truth clip set. - `R_local` — pairwise nearby-GIoU between predicted and ground-truth clips, weighted by temporal distance (exponential decay, k=1.0, minimum GT duration 5 s). - `R_ground` — fraction of each answer clip covered by the model's scanned candidates (answer segments must be grounded in the `` scan). - `R_comp` — budgeted composition quality: 0.6 × gap regularity (minimum adjacent gap, capped at 5 s) + 0.4 × duration-bin coverage of the selected clips. The LoRA adapter is merged into the base weights via `Swift.from_pretrained(...).merge_and_unload()` (see `training/merge_lora.py` in the code release). ## Usage ```python import re import json import torch from transformers import Qwen3VLForConditionalGeneration, AutoProcessor model_path = "" # local path or hub id model = Qwen3VLForConditionalGeneration.from_pretrained( model_path, torch_dtype=torch.bfloat16, attn_implementation="flash_attention_2", device_map="cuda:0", ).eval() processor = AutoProcessor.from_pretrained(model_path) query = "present the wireless earbuds and its key features" messages = [ {"role": "system", "content": [{"type": "text", "text": SYSTEM_PROMPT}]}, {"role": "user", "content": [ {"type": "video", "video": "video.mp4", "fps": 1.0}, {"type": "text", "text": PROMPT_TEMPLATE.format(query=query)}, ]}, ] inputs = processor.apply_chat_template( messages, tokenize=True, add_generation_prompt=True, return_dict=True, return_tensors="pt", ).to("cuda:0") with torch.no_grad(): generated_ids = model.generate(**inputs, max_new_tokens=4096) trimmed = [out[len(inp):] for inp, out in zip(inputs.input_ids, generated_ids)] text = processor.batch_decode(trimmed, skip_special_tokens=True)[0] # Parse the JSON answer = re.search(r"(.*?)", text, re.DOTALL).group(1).strip() answer = re.sub(r"^```(?:json)?\s*|\s*```$", "", answer) clips = json.loads(answer)["clips"] # [{"clip_id": ..., "timestamp": "MM:SS.xx-MM:SS.xx", "description": ..., "core_product": ...}, ...] ``` `SYSTEM_PROMPT` and `PROMPT_TEMPLATE` are reproduced verbatim below (identical to `inference/prompts/temporal_grounding_prompts.py` in the code release). ## Prompt template System prompt: ```text You are an expert video analyst specializing in identifying highlight segments from live-streaming e-commerce videos. When analyzing videos, think deeply like a human, using natural inner-monologue expressions such as 'let me think', 'wait', 'Hmm', 'oh I see', 'let me take a closer look', etc. After examining visual clues, connect your observations to decisions through deeper reasoning. When you spot a potential issue with your initial selection—such as a transition, redundant angles, or duration violation—pause, reflect, and revise your choice before finalizing. Self-reflection and correction during reasoning are encouraged. ``` User prompt (`{query}` is substituted): ```text Your task is to create a 10-second highlight video clip for {query} from the video. High-quality segments must simultaneously satisfy:① The target query content appears clearly, fully, and without obstruction;② The shot is stable and continuous with no cuts or transitions;③ The segment carries substantive query-relevant visual content with high information density.When combining multiple segments, ensure they cover **different angles or states** to avoid redundancy; prioritize the highest-quality, cover-worthy segment as the opening. Reason step by step inside the tag using the following structure: [Quality-guided Global Scan] Briefly describe the overall video pattern in 1–2 sentences. Then scan for candidate segments and evaluate each one: C1: One concise sentence on clip quality. C2: One concise sentence on clip quality. (add more Ci as needed) [Budgeted Set Composition] In 1–3 sentences, explain which clips are selected to form the final 10-second highlight and why unselected clips are excluded. Then output a JSON inside the tag. Each segment in the clips array must include: clip_id, timestamp (format MM:SS.xx-MM:SS.xx), description, and core_product. ``` ## Output format The model first produces a `` block containing the quality-guided scan (`` and `...` pairs) and the composition rationale, then an `` block with JSON: ```json {"clips": [{"clip_id": 1, "timestamp": "00:15.00-00:20.00", "description": "...", "core_product": "..."}]} ``` ## Evaluation Evaluated on the QB-Highlights test set (0.9K videos with dense per-clip human annotations; 6 metrics) — see the paper for details and results. ## Limitations - Trained on live-streaming e-commerce video data; other domains may need adaptation. - The output budget is fixed at 10 seconds per query. - All timestamps follow the 1 fps sampling grid (`MM:SS.xx`). ## License Apache-2.0, inherited from the base model [Qwen3-VL-8B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct). ## Citation ```bibtex @article{qb-highlights, title = {QB-Highlights: Quality-Guided Budgeted Highlight Detection in Videos with Dense Query-Relevant Moments}, year = {2026} } ```