Research Scientist - Vision Language Model
IFM is hiring a Research Scientist - Vision Language Model in Sunnyvale, United States. It pays $150k-$450k a year and Level rates it ; you can apply on IFM.
Level's read on this job
- On Level's scale, this research scientist role is Builds AI ●●●●: the job is building AI systems.
- The posting mentions PyTorch, fine tuning and AI evaluation among its tools and skills.
- IFM has 43 other open roles on Level.
AI in this role
About the Institute of Foundation Models
We are a dedicated research lab for building, understanding, using, and risk-managing foundation models. Our mandate is to advance research, nurture the next generation of AI builders, and drive transformative contributions to a knowledge-driven economy.
As part of our team, you’ll have the opportunity to work on the core of cutting-edge foundation model training, alongside world-class researchers, data scientists, and engineers, tackling the most fundamental and impactful challenges in AI development. You will participate in the development of groundbreaking AI solutions that have the potential to reshape entire industries. Strategic and innovative problem-solving skills will be instrumental in establishing MBZUAI as a global hub for high-performance computing in deep learning, driving impactful discoveries that inspire the next generation of AI pioneers.
Position Summary
- As a Research Scientist in the Vision Language Model (VLM) team, your role will be central to advancing state-of-the-art multimodal foundation models that integrate visual understanding, reasoning, and agentic capabilities. You will work on the research and development of large-scale VLM systems, spanning model architectures, data recipes for pre-training and post-training, and evaluation benchmarks. The role combines cutting-edge research with practical engineering, emphasizing large-scale data processing, filtering, and weighting pipelines, distributed training systems, and reinforcement learning algorithms and systems for multimodal reasoning and agent development.
Key Responsibilities
-
Research and development of next-generation Vision Language Models across pre-training, instruction tuning, reasoning, and agents.
-
Develop novel architectures and training methodologies for integrating visual understanding, language reasoning, and tool-use capabilities.
-
Research efficient multimodal learning techniques, including data-efficient training, long-context modeling, model modularity, and inference optimization.
-
Build and improve large-scale multimodal datasets, synthetic data generation pipelines, and evaluation benchmarks for VLM capabilities.
-
Investigate multimodal reasoning, agentic behavior, OCR, grounding, document understanding, chart understanding, and visual question answering capabilities.
-
Contribute to technical reports, research publications, and open-source software.
-
Represent MBZUAI at research conferences and industry events, showcasing advancements in multimodal foundation models and large-scale AI systems.
-
Mentor junior researchers and collaborate across teams to drive impactful research initiatives.
Academic Qualifications
- PhD or equivalent research experience in Machine Learning, Computer Vision, Natural Language Processing, or Multimodal AI.
Compensation
$150,000 – $450,000/year + discretionary performance bonus + benefits
The posted salary range represents the company’s good faith estimate of the base compensation for this position upon hire. The actual base compensation offered may vary within this range depending on individual qualifications, including but not limited to relevant skills, experience, education, certifications, and specific business needs. In addition to base salary, total compensation includes an annual discretionary performance bonus and comprehensive benefits.
Benefits Include
- Comprehensive medical, dental, and vision benefits
- Bonus
- 401K Plan
- Generous paid time off, sick leave and holidays
- Paid Parental Leave
- Employee Assistance Program
- Life insurance and disability
Professional Experience
Minimum
-
Experience working with large language models and/or vision-language models, including pre-training, fine-tuning, evaluation, or inference.
-
Strong Python and PyTorch development skills for large-scale machine learning research.
-
Experience with distributed training systems and large-scale model optimization.
-
Familiarity with multimodal datasets and data processing pipelines involving images, text, and video.
-
Understanding of modern deep learning architectures, including Transformers, attention mechanisms, and multimodal fusion techniques.
-
Experience with ML infrastructure, including model evaluation, debugging, optimization, and large-scale experimentation.
-
Problem-solving and research skills with the ability to independently drive research/engineering projects.
-
Effective communication and collaboration skills for working across research and engineering teams.
Preferred Skills
-
Hands-on experience training or fine-tuning large Vision Language Models or multimodal foundation models at scale.
-
Experience with distributed learning frameworks and infrastructure such as PyTorch Distributed, Megatron, Triton, or CUDA.
-
Research experience in multimodal reasoning, agentic systems, tool use, OCR, grounding, document understanding, or multimodal retrieval.
-
Experience with synthetic data generation, multimodal data curation, or automated evaluation frameworks for VLMs.
-
Familiarity with efficient training and inference techniques such as FlashAttention, quantization, tensor parallelism, pipeline parallelism, or memory optimization.
-
Experience contributing to open-source ML software and large-scale research codebases.
-
Strong publication record in leading AI conferences such as NeurIPS, ICML, ICLR, CVPR, ICCV, ECCV, ACL, EMNLP, or related venues.
-
Experience collaborating across research, infrastructure, and product-oriented teams to deliver state-of-the-art multimodal systems.
How we rate this
Research Scientist - Vision Language Model at IFM rates 100 out of 100 for how much of the daily work is AI. That makes it Builds AI (AI Level 4 of 4). The level is about AI in the job, not seniority.
Builds AI. The job is building AI systems.
- ●●●● Builds AI80 to 100
- ●●●○ Works on AI60 to 79
- ●●○○ Uses AI40 to 59
- ●○○○ Little AI0 to 39
Levels come from how often the tools, models and workflows of the role are named in the posting itself. Read the description and count the AI tools, models and workflows it names.
Prepare for this job
Built from this posting and from similar jobs on Level.
Skills and AI tools this role asks for
Also common in research scientist jobs at this level
Pay
- This job starts at
- $150k
- Typical research scientist job
- $192k
This posting starts 22% below the typical research scientist job. See research scientist jobs
Questions you could be asked
- Walk me through fine-tuning a model: what data did you use, and how did you check the result?
- How do you decide that one model's output is better than another's for a given task?
- Walk me through a computer vision problem you solved, from raw data to a deployed model.
- What NLP problem have you worked on, and how did you measure whether it actually worked?
- Tell me about a research question you investigated. What did you find?
Adapt your resume
- List these exact terms on your resume: Fine-tuning, AI Evaluation, Computer vision, NLP, and AI Research. An applicant tracking system matches the wording, not the idea.
- Attach one line of real, concrete experience to at least one of them. A tool named with nothing behind it rarely survives a human read.
- Lead with what you built, trained or shipped. This role is judged on the AI system itself, not the tools around it.
Get your CV reviewed before you apply
Send your CV and the role you want next. Your review arrives by email within 3 days.
Get new research scientist jobs (Builds AI ●●●●) by email
One email a week, each job rated for how much AI is in the work. No recruiter spam, unsubscribe in one click.
Free.
Similar roles
Research roles that build AI, at other companies.
What kind of AI work fits you?
Answer 12 practical questions in about three minutes. Get a simple profile, the work it points to, and live roles to explore next.
Find my next step