MULTIMODAL FOUNDATION MODELS
Multimodal foundation-model systems at Moonvalley
Text conditioning, a 70B prompt rewriter, and dataset pipelines for customer-facing video generation.
The challenge
Improve rendered text and branding while closing the mismatch between short user requests and the detailed captions used to train video models.
My contribution
Tested adapters against conditioning integrated into from-scratch training and co-developed multi-encoder text conditioning, text-to-render extraction, and targeted attention weighting. Owned a 70B Llama-family prompt rewriter from hundreds of thousands of training pairs through SFT, GRPO, and vLLM serving. Built a pipeline that annotated hundreds of millions of licensed videos.
The outcome
The conditioning approach shipped in the customer-facing model, the prompt rewriter entered the public generation product, and the video labels shaped a later production training curriculum while helping avoid unsuitable supplier purchases worth hundreds of thousands of dollars.
Areas of work
- Multimodal conditioning
- LLM post-training
- Data curation