Document Type

Dissertation - Open Access

Award Date

2026

Degree Name

Doctor of Philosophy (PhD)

Department / School

Electrical Engineering and Computer Science

First Advisor

Kwanghee Won

Abstract

Pretrained vision–language models (VLMs) encode broad visual and semantic knowledge, but this knowledge does not always translate into reliable performance on specialized visual tasks. Multi-object region understanding requires fine-grained relational alignment, scientific document question answering requires structured reasoning over tables and charts, and robotic manipulation requires task-relevant perception to influence action generation. This dissertation studies these gaps as a problem of knowledge utilization: helping a pretrained model use available knowledge in a form aligned with the target task. Its central claim is that effective adaptation depends not only on model scale, data volume, or full-backbone fine-tuning, but also on three design choices: what task-relevant information is provided, how it is learned, and where it is introduced. The first study investigates region–text alignment when one region contains multiple objects and supports heterogeneous descriptions. Object-centric descriptions enumerate visible entities, whereas holistic descriptions summarize the scene at a broader semantic level. Standard contrastive learning does not represent the directional differences in informational scope between these descriptions. The study therefore introduces CLIP-MORE, a unified region-representation framework with an asymmetric soft-contrastive objective. Using large-language-model-derived relations, the objective models graded sufficiency and non-necessity between descriptions. On the human-annotated Flickr Multi-Object benchmark, CLIP-MORE improves Region-to-Text Recall@1 from 0.379 to 0.561, a 48.0% relative gain. The results show that directional supervision enables a region representation to accommodate multiple valid descriptions of the same visual content while preserving their partial and asymmetric semantic relationships. The magnitude of the improvement nevertheless varies across encoder architectures, with the strongest gains observed for the self-distilled dense-feature encoder. The second study addresses question answering over structured scientific visuals, including tables and charts. It proposes quality-aware cascade description distillation to transfer figure understanding from a 7B teacher VLM to a 3B student. The teacher generates structural and semantic descriptions, and a frozen natural language inference model estimates the reliability of each description against the gold answer. These quality scores weight the description losses during training. The target sequence places the descriptions before the answer, allowing the autoregressive student to condition its answer on its own intermediate descriptions. On in-domain short-answer scientific figure QA, adaptive quality weighting and cascade ordering raise the 3B student to 30.61% ANLS, within 0.28 points of the 7B teacher at 30.89%. At inference, the student operates without the teacher or quality scorer and self-generates the descriptions before producing the answer. Cross-domain results reveal a complementary trade-off: cascade learning provides the strongest in-domain gains, while an answer-first multi-task arrangement better preserves general zero-shot capability. The third study examines robotic manipulation with a fully frozen vision–language–action backbone. It introduces explicit Where, How, and When tokens that represent target location, grasp-relevant geometry, and task progress, while training only a compact flow-matching action expert. Using the original demonstrations without updating the backbone or collecting perturbed data, action-expert adaptation increases instruction-paraphrase success from 9% to 43%. The factored Where/How/When interface achieves the strongest novel-object transfer, reaching 50.0% success on the mug scene and 30.0% on the apple scene under late fusion. Comparing early fusion at the frozen backbone input with late fusion in the action expert shows that the injection site affects which form of generalization is recovered. Spatial-layout perturbations remain difficult even when the correct target location reaches the policy, providing evidence that the residual failure is tied to learned action behavior rather than perception alone. Taken together, these studies show that effective knowledge transfer depends on structuring task-relevant information so that pretrained models can use it at the level required by the downstream task. In region-level alignment, directional supervision enables the model to learn multiple valid descriptions of the same visual content while preserving their partial and asymmetric semantic relationships. In scientific document question answering, quality-aware cascade distillation makes structural and semantic descriptions usable by a compact student during answer generation. In robotic manipulation, explicit Where/How/When cues connect task-relevant perceptual information to the action-generation process. These results recast knowledge transfer as the design of task-aligned interfaces between pretrained representations and downstream outputs. Although the effectiveness of each interface depends on the task and model architecture, the overall findings demonstrate that carefully structured supervision and conditioning can make pretrained vision–language models more useful for specialized visual tasks without relying solely on larger backbones or full-model fine-tuning.

Publisher

South Dakota State University

Download

Share

COinS
 

Rights Statement

In Copyright