◈ Homepage — https://snorkel.ai/Capabilities Research Resources Company Get started New Continual Learning Bench by Berkeley & Snorkel We build the data that pushes the frontier Snorkel helps frontier labs and AI teams develop specialized training data and environments that set their models and agents apart. Request dataset samples Explore research PROUD TO PARTNER WITH TOP FRONTIER AI AND RESEARCH TEAMS Frontier models break at the edges. We build for that. Most data pipelines are built for volume, not difficulty. Frontier models fail on distributional gaps in specialized domains, benchmark blind spots, and tasks where correctness is hard to define. Snorkel is built specifically for these problems. Founded out of Stanford AI Lab, we've been shaping and benchmarking frontier AI for nearly a decade. See our research Open Benchmarks Grants WHAT WE'VE FOUND New RL Research RLVR in Low Data & Compute Regimes Better data beats more compute — measured across low-resource settings. Evaluation Research RIFT: Rubric Failure Mode Taxonomy A diagnostic framework for when AI evaluation rubrics break down. Domain Agents Benchmarking Agents in Insurance Underwriting Environment-first benchmarking for agents in a genuinely high-stakes domain. COLLABORATIONS New Research Collaboration Continual Learning Bench Expert-validated tasks for agents that learn across task sequences, not isolated prompts. Code Quality SlopCodeBench Generic code evals miss sloppy code. This measures what they ignore. Agentic Coding Terminal-Bench 2.0 Real terminal tasks — exposing where today's coding agents fail. Legal AI Harvey BigLaw Bench Expert data for the hardest agentic legal research benchmark. Built with Harvey AI. THE FRONTIER AI DATA LAB Data development for the frontier Snorkel partners with frontier AI teams to build research-grade datasets, evaluation systems, and runnable environments where generic coverage runs out. Explore capabilities Snorkel Data Series Curriculum-structured datasets for the task areas frontier models are pushing hardest, with rubrics, reviewer guidance, difficulty tiers, and eval slices built in. Custom data development Specialized agents DATA Expert Demonstrations & Reasoning Human solution traces Reasoning traces SME Q&A rationales Workflow demos and decision workflows Tool-use demos Preference Labels & Rankings Patch/draft/report quality ranking Trajectory QA Risk/safety/style calibration Helpful/harmless ranking Grounding & style Rubrics & Verifiable Outcomes Unit tests / compile Deterministic graders Citation correctness Numerical consistency/scorable math/science Long-horizon tasks ENVIRONMENTS Standard & Custom Environments Repo + CLI tools Browser/GUI harness Multi-step/stateful workflows Simulated environments Your tools, codebase, corpus, data & permissions DATA DEVELOPMENT Good data is a set of design choices Request dataset samples Most data quality problems are design problems. Ambiguous task definitions produce inconsistent labels. Uncalibrated reviewers introduce systematic bias. Missing provenance makes failure analysis guesswork. Snorkel's proprietary process is built around the decisions that determine whether training data actually drives model improvement: Well-specified expert-level tasks Calibrated expert review Rubrics and programmatic checks Adjudication and provenance Edge-case coverage Benchmarks and evals CUSTOM AGENTS Specialized agents grounded in expert data Explore agent solutions The same data development system we use to improve frontier models powers our specialized agents. That means agents evaluated against task-specific rubrics and programmatic checks – not generic benchmarks – and refined through the same adjudication and provenance practices used in production model development. Built for specialized workflows and high-consequence decisions, not generic copilots Evaluation on environment-grounded tasks with programmatic pass/fail criteria Same rigor used to train frontier-class models, applied to your enterprise deployment PUBLISHED RESEARCH Research that shapes the work Every dataset, benchmark, and environment we create is the output of active research co-developed and peer-reviewed with leading academic teams and frontier labs. Explore research Benchmark Agentic coding benchmark: Evaluating AI models on complex, real-world coding tasks Blog SlopCodeBench: Measuring Code Erosion as Agents Iterate Blog Terminal-Bench 2.0: Raising the bar for AI agent evaluation Research Paper Learning from Less: Measuring the Effectiveness of RLVR in Low Data and Compute Regimes Research Paper RIFT: A Rubric Failure Mode Taxonomy and Automated Diagnostics Benchmark Agentic coding benchmark: Evaluating AI models on complex, real-world coding tasks Blog SlopCodeBench: Measuring Code Erosion as Agents Iterate Blog Terminal-Bench 2.0: Raising the bar for AI agent evaluation Research Paper Learning from Less: Measuring the Effectiveness of RLVR in Low Data and Compute Regimes Research Paper RIFT: A Rubric Failure Mode Taxonomy and Automated Diagnostics Benchmark Agentic coding benchmark: Evaluating AI models on complex, real-world coding tasks Blog SlopCodeBench: Measuring Code Erosion as Agents Iterate Blog Terminal-Bench 2.0: Raising the bar for AI agent evaluation Research Paper Learning from Less: Measuring the Effectiveness of RLVR in Low Data and Compute Regimes Research Paper RIFT: A Rubric Failure Mode Taxonomy and Automated Diagnostics 1 of 5 For models that need to be right. Not just good enough. Request dataset samples Explore research Capabilities How it works Data development Specialized agents Use cases Customer stories Research Research hub Leaderboards Open Benchmarks Grants Resources Resource library Events Reading Group Blog Company About Careers Press Partners Security Contact us Contact Get started Join expert network Compliance Copyright © 2026 Snorkel AI, Inc. All rights reserved. Terms of Use Privacy Cookie Policy ◈ Interior Pages — 35 pages crawledTerms | Snorkel AI Capabilities Capabilities How it works Research-led data and environment development for the frontier's hardest problems Learn more Data development Overview Expert-curated datasets for frontier AI Use cases See how our data improves frontier models Specialized agents Overview Custom AI systems built to unlock ROI fast Customer stories Real-world results from enterprise deployments Research Research Research Research hub Our latest papers and data-centric AI findings Leaderboards Compare model performance across benchmarks Open Benchmarks Grants Funding for open-source AI research Featured BENCHMARK Agentic Coding benchmark A benchmark for evaluating AI models on complex, real-world coding tasks that require multi-step reasoning, tool use, and autonomous problem-solving. Resources Resources Resources Resource library Guides, papers, and tools for data-centric AI Events Upcoming talks, workshops, and conferences Reading Group AI discussions for researchers and practitioners Blog News, updates, and perspectives from our team featured BLOG Terminal-Bench 2.0 Developed by Stanford and Laude Institute with contributions from Snorkel AI, it’s a major leap forward in evaluating AI coding agents. Company Company Company About Our mission, story, and values Careers Open roles and life at our company Press Media resources and announcements Partners Organizations we work with Security How we keep data safe Contact us Get in touch with our team Join our expert community Get paid to shape safer, smarter AI Learn more Get started Get started WEBSITE TERMS OF USE VERSION 1.0 Last Updated: July 11, 2020 The website located at www.snorkel.ai (the “ Site ”) is a copyrighted work belonging to Snorkel AI, Inc. (“ Company ”, “ us ”, “ our ”, and “ we ”). Certain features of the Site may be subject to additional guidelines, terms, or rules, which will be posted on the Site in connection with such features. All such additional terms, guidelines, and rules are incorporated by reference into these Terms. THESE TERMS OF USE (THESE “ TERMS ”) SET FORTH THE LEGALLY BINDING TERMS AND CONDITIONS THAT GOVERN YOUR USE OF THE SITE. BY ACCESSING OR USING THE SITE, YOU ARE ACCEPTING THESE TERMS (ON BEHALF OF YOURSELF OR THE ENTITY THAT YOU REPRESENT), AND YOU REPRESENT AND WARRANT THAT YOU HAVE THE RIGHT, AUTHORITY, AND CAPACITY TO ENTER INTO THESE TERMS (ON BEHALF OF YOURSELF OR THE ENTITY THAT YOU REPRESENT). YOU MAY NOT ACCESS OR USE THE SITE OR ACCEPT THE TERMS IF YOU ARE NOT AT LEAST 18 YEARS OLD. IF YOU DO NOT AGREE WITH ALL OF THE PROVISIONS OF THESE TERMS, DO NOT ACCESS AND/OR USE THE SITE. THESE TERMS REQUIRE THE USE OF ARBITRATION (SECTION 8.2) ON AN INDIVIDUAL BASIS TO RESOLVE DISPUTES, RATHER THAN JURY TRIALS OR CLASS ACTIONS, AND ALSO LIMIT THE REMEDIES AVAILABLE TO YOU IN THE EVENT OF A DISPUTE. 1. ACCOUNTS 1.1.  Account Creation . In order to use certain features of the Site, you may be required to register for an account (“Account”) and provide certain information about yourself as prompted by the account registration form. You represent and warrant that: (a) all required registration information you submit is truthful and accurate; (b) you will maintain the accuracy of such information. You may delete your Account at any time, for any reason, by following the instructions on the Site. Company may suspend or terminate your Account in accordance with Section 7. 1.2.  Account Responsibilities.  You are responsible for maintaining the confidentiality of your Account login information and are fully responsible for all activities that occur under your Account. You agree to immediately notify Company of any unauthorized use, or suspected unauthorized use of your Account or any other breach of security. Company cannot and will not be liable for any loss or damage arising from your failure to comply with the above requirements. 2. ACCESS TO THE SITE 2.1.  License . Subject to these Terms, Company grants you a non-transferable, non-exclusive, revocable, limited license to use and access the Site solely for your own personal, noncommercial use. 2.2.  Certain Restrictions . The rights granted to you in these Terms are subject to the following restrictions: (a) you shall not license, sell, rent, lease, transfer, assign, distribute, host, or otherwise commercially exploit the Site, whether in whole or in part, or any content displayed on the Site; (b) you shall not modify, make derivative works of, disassemble, reverse compile or reverse engineer any part of the Site; (c) you shall not access the Site in order to build a similar or competitive website, product, or service; and (d) except as expressly stated herein, no part of the Site may be copied, reproduced, distributed, republished, downloaded, displayed, posted or transmitted in any form or by any means. Unless otherwise indicated, any future release, update, or other addition to functionality of the Site shall be subject to these Terms. All copyright and other proprietary notices on the Site (or on any content displayed on the Site) must be retained on all copies thereof. 2.3.  Modification . Company reserves the right, at any time, to modify, suspend, or discontinue the Site (in whole or in part) with or without notice to you. You agree that Company will not be liable to you or to any third party for any modification, suspension, or discontinuation of the Site or any part thereof. 2.4.  No Support or Maintenance . You acknowledge and agree that Company will have no obligation to provide you with any support or maintenance in connection with the Site. 2.5.  Ownership . You acknowledge that all the intellectual property rights, including copyrights, patents, trade marks, and trade secrets, in the Site and its content are owned by Company or Company’s suppliers. Neither these Terms (nor your access to the Site) transfers to you or any third party any rights, title or interest in or to such intellectual property rights, except for the limited access rights expressly set forth in Section 2.1. Company and its suppliers reserve all rights not granted in these Terms. There are no implied licenses granted under these Terms. 3. INDEMNIFICATION You agree to indemnify and hold Company (and its officers, employees, and agents) harmless, including costs and attorneys’ fees, from any claim or demand made by any third party due to or arising out of (a) your use of the Site, (b) your violation of these Terms or (c) your violation of applicable laws or regulations. Company reserves the right, at your expense, to assume the exclusive defense and control of any matter for which you are required to indemnify us, and you agree to cooperate with our defense of these claims. You agree not to settle any matter without the prior written consent of Company. Company will use reasonable efforts to notify you of any such claim, action or proceeding upon becoming aware of it. 4. THIRD-PARTY LINKS & ADS; OTHER USERS 4.1.  Third-Party Links & Ads . The Site may contain links to third-party websites and services, and/or display advertisements for third parties (collectively, “Third-Party Links & Ads”). Such Third-Party Links & Ads are not under the control of Company, and Company is not responsible for any Third-Party Links & Ads. Company provides access to these Third-Party Links & Ads only as a convenience to you, and does not review, approve, monitor, endorse, warrant, or make any representations with respect to Third-Party Links & Ads. You use all Third-Party Links & Ads at your own risk, and should apply a suitable level of caution and discretion in doing so. When you click on any of the Third-Party Links & Ads, the applicable third party’s terms and policies apply, including the third party’s privacy and data gathering practices. You Use Cases | Snorkel AI Capabilities Capabilities How it works Research-led data and environment development for the frontier's hardest problems Learn more Data development Overview Expert-curated datasets for frontier AI Use cases See how our data improves frontier models Specialized agents Overview Custom AI systems built to unlock ROI fast Customer stories Real-world results from enterprise deployments Research Research Research Research hub Our latest papers and data-centric AI findings Leaderboards Compare model performance across benchmarks Open Benchmarks Grants Funding for open-source AI research Featured BENCHMARK Agentic Coding benchmark A benchmark for evaluating AI models on complex, real-world coding tasks that require multi-step reasoning, tool use, and autonomous problem-solving. Resources Resources Resources Resource library Guides, papers, and tools for data-centric AI Events Upcoming talks, workshops, and conferences Reading Group AI discussions for researchers and practitioners Blog News, updates, and perspectives from our team featured BLOG Terminal-Bench 2.0 Developed by Stanford and Laude Institute with contributions from Snorkel AI, it’s a major leap forward in evaluating AI coding agents. Company Company Company About Our mission, story, and values Careers Open roles and life at our company Press Media resources and announcements Partners Organizations we work with Security How we keep data safe Contact us Get in touch with our team Join our expert community Get paid to shape safer, smarter AI Learn more Get started Get started Snorkel expert data-as-a-service Use cases From coding and agentic reasoning to text generation and more, discover how Snorkel enables AI teams to build the next generation of models with unparalleled speed and accuracy. Speak with a Snorkel expert Agentic Annotation Coding Multi-modal Text generation Agentic Agentic The frontiers of multi-turn math reasoning Snorkel provided a frontier LLM team with a dataset to assess LLM math reasoning skills on high school to graduate-level challenges. Our data development approach saw experts correct responses and reasoning traces and allowed the customer to control distribution across topics, skills, and complexity. 0% Pass rate for frontier LLMs 900 Mathematical skills Agentic AI voice assistant training data for a tech industry giant A tech industry giant aimed to build better, more usable voice assistants for its customers. We collaborated with them to build a deep, expert-crafted dataset of realistic multi-turn, multi-agent conversations, including simulated tool use. 3+ Tool calls per conversation, ~9+ turns 15+ Reasoning scenarios represented Agentic Robust agentic evaluation benchmarks A Global 2000 telecom partnered with Snorkel to curate a gold-standard set of prompts, responses, and tool calls targeting reasoning and multi-step planning. This custom benchmark revealed critical model failures, enabling the team to target training and correction and progress to production faster than manual reviews. 10+ Tools +35 Points in function calling (via MMAU) Agentic Text generation Multi-step, multi-turn, and multi-tool deep research data A leading LLM provider hired Snorkel AI to create a dataset to enhance its models’ deep research capabilities. Snorkel researchers assembled a dataset where each data point included a complex user query, a high-quality research plan, and a fine-grained response quality evaluation rubric. 10+ Average interactions between model and user 30+ Evaluation criteria developed per task on average Annotation Annotation Grading LLM information retrieval and synthesis An open-source LLM developer sought to improve its models’ ability to extract questions and answers from technical documents like textbooks and research papers. Snorkel experts graded and corrected model attempts to cite sources and answer questions from these documents, creating a golden set of retrievals. 30+ Grading dimensions 10+ Domains Annotation Multi-modal Enabling FMs to understand charts A leading LLM developer sought high-quality annotations of graphs, maps, and other visuals used to solve middle-school and high-school math problems. Snorkel experts reviewed documents and curated annotations (including chart elements, data points, and implied relationships) for training and evaluation purposes. 22+ Average data points labeled per graph 15+ Visual attributes labeled Coding Coding Alignment for better code generation A frontier model developer sought to improve code generation outputs using human feedback. Snorkel rapidly assembled a team of qualified engineers to assess, review, and grade multiple candidate code responses to user queries, resulting in a rich training set to better align the model. 8 Assessment criteria per code generation 21 Coding languages assessed Coding Training and evaluation data for code generation A tech industry giant sought unique prompts and answers to train and evaluate its frontier LLMs’ code generation capabilities. Snorkel experts curated unique competition-style coding prompts with verifiable solutions and accompanying unit tests to validate samples automatically. 20+ Problem classes 4 Factors in quality rubric Multi-modal Annotation Multi-modal Enabling FMs to understand charts A leading LLM developer sought high-quality annotations of graphs, maps, and other visuals used to solve middle-school and high-school math problems. Snorkel experts reviewed documents and curated annotations (including chart elements, data points, and implied relationships) for training and evaluation purposes. 22+ Average data points labeled per graph 15+ Visual attributes labeled Multi-modal Image-based search for retail An e-commerce giant aimed to let customers search products by image and feeling (such as “summer vibes.”) Snorkel researchers generated pairs of user queries and associated results that boosted downstream search mode performance. 10,000+ Products +37 Point recall on image + text search Text generation Text generation A PhD-level benchmark for frontier LLMs A leading LLM developer sought a dataset of multiple-choice Q&A questions that stretched beyond the limits of frontier LLMs. Snorkel AI developed a dataset that probed for PhD-level understanding, covering thousands of topics across humanities, STEM, and professional domains. Pass rate by two frontier LLMs 1,000+ PhD-level sub-domains Text generation Q&A training data for customer billing SLM A Fortune 500 telecom wanted an SLM to automatically answer customer billing questions. Using expert input and programmatic acceleration, Snorkel curated data that covered all expected question types and improved the model’s performance, enabling the team to deploy 10+ supported use cases to production. +41 Point improvement in SLM answer accuracy 93% Alignment between SMEs and AI evaluators Agentic Text generation Multi-step, multi-turn, and multi-tool deep research data A leading LLM provider hired Snorkel AI to create a dataset to enhance its models’ deep research capabilities. Snorkel researchers assembled a dataset where each data point included a complex user query, a high-quality research plan, and a fine-grained response quality evaluation rubric. 10+ Average interactions between model and user 30+ Evaluation criteria developed per task on average See how Snorkel can help you get up to: 100x Faster data curation 40x Faster model delivery 99% Model accuracy Let’s talk Get started How do you want to work with Snorkel? AI data development services Accelerate the development of frontier AI models with expert-curated, enterprise-grade data. Learn how Snorkel’s Data-as-a-Service helps teams label, refine, and evaluate high-quality, domain-specific datasets for your projects. Talk to a data researcher Build specialized agents Explore how Snorkel can collaborate with your product and development teams to build and deploy custom AI and agentic systems. Solutions undergo rigorous testing based Events & Conferences | Snorkel AI Capabilities Capabilities How it works Research-led data and environment development for the frontier's hardest problems Learn more Data development Overview Expert-curated datasets for frontier AI Use cases See how our data improves frontier models Specialized agents Overview Custom AI systems built to unlock ROI fast Customer stories Real-world results from enterprise deployments Research Research Research Research hub Our latest papers and data-centric AI findings Leaderboards Compare model performance across benchmarks Open Benchmarks Grants Funding for open-source AI research Featured BENCHMARK Agentic Coding benchmark A benchmark for evaluating AI models on complex, real-world coding tasks that require multi-step reasoning, tool use, and autonomous problem-solving. Resources Resources Resources Resource library Guides, papers, and tools for data-centric AI Events Upcoming talks, workshops, and conferences Reading Group AI discussions for researchers and practitioners Blog News, updates, and perspectives from our team featured BLOG Terminal-Bench 2.0 Developed by Stanford and Laude Institute with contributions from Snorkel AI, it’s a major leap forward in evaluating AI coding agents. Company Company Company About Our mission, story, and values Careers Open roles and life at our company Press Media resources and announcements Partners Organizations we work with Security How we keep data safe Contact us Get in touch with our team Join our expert community Get paid to shape safer, smarter AI Learn more Get started Get started RESOURCES Events & conferences Connect with Snorkel researchers and partners at upcoming events. Featured events In-person event AI Council A technical conference for senior AI infrastructure leaders featuring in-depth sessions and peer collaboration, with Snorkel joining as a Silver Sponsor. May 12 - 14, 2026 San Francisco, CA Learn more Learn More In-person event Reading Group with Google Deepmind’s Carter Wendelken He’ll dive into two papers just presented at ICLR: “AutoHarness: Improving LLM Agents by Automatically Synthesizing a Code Harness” and “Code World Models for General Game Playing.” May 13, 2026 San Francisco, CA Learn more Learn More In-person event MLSys 2026 Connect with us at MLSys, where machine learning meets scalable systems. May 18 - 22, 2026 Bellevue, WA Learn more Learn More In-person event MLSys: Swing Suite Rewind Join the Snorkel AI research team at MLSys for drinks, ideas, and off-the-record conversations in private swing suites with AI researchers and builders. May 21, 2026 Forum Social House (Bellevue, Washington) Learn more Learn More In-person event CAIS Join Snorkel at CAIS, connecting leaders building safe, reliable AI systems. May 26 - 29, 2026 San Jose, CA Learn more Learn More In-person event Mistral AI Now Summit Join Snorkel at Mistral’s AI Now Summit in Paris, the flagship event for AI leaders, builders, and innovators. May 28, 2026 Paris, France Learn more Learn More In-person event Databricks Data + AI Summit 2026 An industry conference focused on data, analytics, and AI, where Snorkel will participate as an Innovator, with more details coming soon. June 15 - 18, 2026 San Francisco, CA Learn more Learn More In-person event AI Engineer World’s Fair 2026 A global gathering of the AI engineering community focused on practical systems and workflows, where Snorkel will participate as a Gold Sponsor. June 29 - July 2, 2026 San Francisco, CA Learn more Learn More In-person event ACL Meet our team at ACL, the leading conference for NLP and language AI research. July 2 - 7, 2026 San Diego, CA Learn more Learn More In-person event ICML Connect with Snorkel at ICML, where we’ll presenting research and joining as a Silver sponsor. July 6th-11th, 2026 South Korea Learn more Learn More In-person event Agentic AI Summit We’ll be at Agentic AI Summit to explore, discuss, and collaboratively shape the exciting future of AI and Agentic AI. August 1 - 2, 2026 UC Berkley Learn more Learn More Past In-person event ICLR 2026 The Fourteenth International Conference on Learning Representations, bringing together researchers and practitioners advancing representation learning, with Snorkel participating in the research community. Learn more Learn More Past In-person event AI Engineer Europe Join Snorkel at this three-day conference for AI engineers and builders focused on production-ready AI systems – and come meet the team. Learn more Learn More Past In-person event NeurIPS Conference 2025 Learn more about Snorkel’s presence at the 39th Annual Conference on neural information processing systems. Learn more Learn More Coming Fall 2026 Frontier Data Summit is on the horizon Sign up for updates on the Benchmark Grants Program and the upcoming Frontier Data Summit. Get started How do you want to work with Snorkel? AI data development services Accelerate the development of frontier AI models with expert-curated, enterprise-grade data. Learn how Snorkel’s Data-as-a-Service helps teams label, refine, and evaluate high-quality, domain-specific datasets for your projects. Talk to a data researcher Build specialized agents Explore how Snorkel can collaborate with your product and development teams to build and deploy custom AI and agentic systems. Solutions undergo rigorous testing based on your business criteria to ensure positive ROI, faster. Talk to a strategist Become an expert contributor Join the Snorkel Expert Contributor community and help shape the future of AI with your expertise. Contribute to groundbreaking projects, share domain-specific insights, and get rewarded for your impact. Join now Capabilities How it works Data development Specialized agents Use cases Customer stories Research Research hub Leaderboards Open Benchmarks Grants Resources Resource library Events Reading Group Blog Company About Careers Press Partners Security Contact us Contact Get started Join expert network Compliance Copyright © 2026 Snorkel AI, Inc. All rights reserved. Terms of Use Privacy Cookie Policy Expert Data Development for Frontier AI | Snorkel AI Capabilities Capabilities How it works Research-led data and environment development for the frontier's hardest problems Learn more Data development Overview Expert-curated datasets for frontier AI Use cases See how our data improves frontier models Specialized agents Overview Custom AI systems built to unlock ROI fast Customer stories Real-world results from enterprise deployments Research Research Research Research hub Our latest papers and data-centric AI findings Leaderboards Compare model performance across benchmarks Open Benchmarks Grants Funding for open-source AI research Featured BENCHMARK Agentic Coding benchmark A benchmark for evaluating AI models on complex, real-world coding tasks that require multi-step reasoning, tool use, and autonomous problem-solving. Resources Resources Resources Resource library Guides, papers, and tools for data-centric AI Events Upcoming talks, workshops, and conferences Reading Group AI discussions for researchers and practitioners Blog News, updates, and perspectives from our team featured BLOG Terminal-Bench 2.0 Developed by Stanford and Laude Institute with contributions from Snorkel AI, it’s a major leap forward in evaluating AI coding agents. Company Company Company About Our mission, story, and values Careers Open roles and life at our company Press Media resources and announcements Partners Organizations we work with Security How we keep data safe Contact us Get in touch with our team Join our expert community Get paid to shape safer, smarter AI Learn more Get started Get started New Continual Learning Bench by Berkeley & Snorkel We build the data that pushes the frontier Snorkel helps frontier labs and AI teams develop specialized training data and environments that set their models and agents apart. Request dataset samples Explore research Proud to partner with top frontier AI and research teams Frontier models break at the edges. We build for that. Most data pipelines are built for volume, not difficulty. Frontier models fail on distributional gaps in specialized domains, benchmark blind spots, and tasks where correctness is hard to define. Snorkel is built specifically for these problems. Founded out of Stanford AI Lab, we've been shaping and benchmarking frontier AI for nearly a decade. See our research Open Benchmarks Grants What we've found New RL Research RLVR in Low Data & Compute Regimes Better data beats more compute — measured across low-resource settings. Evaluation Research RIFT: Rubric Failure Mode Taxonomy A diagnostic framework for when AI evaluation rubrics break down. Domain Agents Benchmarking Agents in Insurance Underwriting Environment-first benchmarking for agents in a genuinely high-stakes domain. Collaborations New Research Collaboration Continual Learning Bench Expert-validated tasks for agents that learn across task sequences, not isolated prompts. Code Quality SlopCodeBench Generic code evals miss sloppy code. This measures what they ignore. Agentic Coding Terminal-Bench 2.0 Real terminal tasks — exposing where today's coding agents fail. Legal AI Harvey BigLaw Bench Expert data for the hardest agentic legal research benchmark. Built with Harvey AI. The Frontier AI Data Lab Data development for the frontier Snorkel partners with frontier AI teams to build research-grade datasets, evaluation systems, and runnable environments where generic coverage runs out. Explore capabilities Snorkel Data Series Curriculum-structured datasets for the task areas frontier models are pushing hardest, with rubrics, reviewer guidance, difficulty tiers, and eval slices built in. Custom data development When off-the-shelf coverage runs out, we build bespoke datasets, evals, and benchmark expansions for the exact failure surface you need to close. Specialized agents Custom agents built on specialized data and evaluated in real workflows, with pass/fail criteria tied to the performance standards that moves ROI. Data Expert Demonstrations & Reasoning Human solution traces Reasoning traces SME Q&A rationales Workflow demos and decision workflows Tool-use demos Preference Labels & Rankings Patch/draft/report quality ranking Trajectory QA Risk/safety/style calibration Helpful/harmless ranking Grounding & style Rubrics & Verifiable Outcomes Unit tests / compile Deterministic graders Citation correctness Numerical consistency/scorable math/science Long-horizon tasks Environments Standard & Custom Environments Repo + CLI tools Browser/GUI harness Multi-step/stateful workflows Simulated environments Your tools, codebase, corpus, data & permissions DATA DEVELOPMENT Good data is a set of design choices Request dataset samples Most data quality problems are design problems. Ambiguous task definitions produce inconsistent labels. Uncalibrated reviewers introduce systematic bias. Missing provenance makes failure analysis guesswork. Snorkel's proprietary process is built around the decisions that determine whether training data actually drives model improvement: Well-specified expert-level tasks Tasks scoped to actual model failure modes, with target distributions, acceptance criteria, and verifier definitions. Each spec is a research artifact, written before data work begins. Calibrated expert review Calibration is a research workflow, not a staffing layer. Reviewers are trained against gold sets authored by Snorkel researchers, scored for agreement and bias, and re-calibrated per task. Rubrics and programmatic checks Fine-tuned evaluator models, not just human spot-checks. Snorkel researchers co-design rubrics with domain experts, then distill them into programmatic graders and specialized evaluator models. Adjudication and provenance Author, multi-reviewer, and final-adjudicator pipeline with full audit trails. Every label is traceable to who decided what, when, and on what evidence. Edge-case coverage Distributional precision, not volume. Expert-authored seeds expand into controlled coverage across difficulty bands, edge cases, and failure modes through templated generation. Benchmarks and evals Eval harnesses built alongside the data: task-specific rubrics, deterministic graders, and runnable environments that produce reproducible scores across model versions. Custom AGENTS Specialized agents grounded in expert data Explore agent solutions The same data development system we use to improve frontier models powers our specialized agents. That means agents evaluated against task-specific rubrics and programmatic checks – not generic benchmarks – and refined through the same adjudication and provenance practices used in production model development. Built for specialized workflows and high-consequence decisions, not generic copilots Evaluation on environment-grounded tasks with programmatic pass/fail criteria Same rigor used to train frontier-class models, applied to your enterprise deployment PUBLISHED RESEARCH Research that shapes the work Every dataset, benchmark, and environment we create is the output of active research co-developed and peer-reviewed with leading academic teams and frontier labs. Explore research Benchmark Agentic coding benchmark: Evaluating AI models on complex, real-world coding tasks Blog SlopCodeBench: Measuring Code Erosion as Agents Iterate Blog Terminal-Bench 2.0: Raising the bar for AI agent evaluation Research Paper Learning from Less: Measuring the Effectiveness of RLVR in Low Data and Compute Regimes Research Paper RIFT: A Rubric Failure Mode Taxonomy and Automated Diagnostics of For models that need to be right. Not just good enough. Request dataset samples Explore research Get started How do you want to work with Snorkel? AI data development services Accelerate the development of frontier AI models with expert-curated, enterprise-grade data. Learn how Snorkel’s Data-as-a-Service helps teams label, refine, and evaluate high-quality, domain-specific datasets for your projects. Talk to a da AI Solutions Contact | Snorkel AI Capabilities Capabilities How it works Research-led data and environment development for the frontier's hardest problems Learn more Data development Overview Expert-curated datasets for frontier AI Use cases See how our data improves frontier models Specialized agents Overview Custom AI systems built to unlock ROI fast Customer stories Real-world results from enterprise deployments Research Research Research Research hub Our latest papers and data-centric AI findings Leaderboards Compare model performance across benchmarks Open Benchmarks Grants Funding for open-source AI research Featured BENCHMARK Agentic Coding benchmark A benchmark for evaluating AI models on complex, real-world coding tasks that require multi-step reasoning, tool use, and autonomous problem-solving. Resources Resources Resources Resource library Guides, papers, and tools for data-centric AI Events Upcoming talks, workshops, and conferences Reading Group AI discussions for researchers and practitioners Blog News, updates, and perspectives from our team featured BLOG Terminal-Bench 2.0 Developed by Stanford and Laude Institute with contributions from Snorkel AI, it’s a major leap forward in evaluating AI coding agents. Company Company Company About Our mission, story, and values Careers Open roles and life at our company Press Media resources and announcements Partners Organizations we work with Security How we keep data safe Contact us Get in touch with our team Join our expert community Get paid to shape safer, smarter AI Learn more Get started Get started ai solutions For AI agents that need to be right, not just good enough Snorkel solutions helps enterprises build custom agents for specialized, high-impact workflows where off-the-shelf LLMs and vertical tools fall short. Talk to a strategist about identifying high-value use cases and deploying production-ready agents grounded in your data, context, and operating environment. Proud to partner with top frontier AI and research teams What we help with Benchmark & evaluation design Custom agent development Enterprise-specific data & context Model behavior tuning Production deployment Monitoring & continuous iteration FDE embedded collaboration Talk to a strategist " * " indicates required fields First name * Last name * Work email * Phone number * Job title * Company * What is your timeline for deployment? * Active in production Targeting production in 6 months Targeting production in 12 months Starting to explore Academic/personal interest Partnership opportunity What is your primary area of interest? * Expert data services Custom enterprise AI solutions Self-serve data development platform Research collaboration Not sure yet By submitting this form, I agree to the Terms of Use and acknowledge that my information will be used in accordance with the Privacy Policy . Δ For workflows generic copilots can't handle Talk to a strategist Explore research Get started How do you want to work with Snorkel? AI data development services Accelerate the development of frontier AI models with expert-curated, enterprise-grade data. Learn how Snorkel’s Data-as-a-Service helps teams label, refine, and evaluate high-quality, domain-specific datasets for your projects. Talk to a data researcher Build specialized agents Explore how Snorkel can collaborate with your product and development teams to build and deploy custom AI and agentic systems. Solutions undergo rigorous testing based on your business criteria to ensure positive ROI, faster. Talk to a strategist Become an expert contributor Join the Snorkel Expert Contributor community and help shape the future of AI with your expertise. Contribute to groundbreaking projects, share domain-specific insights, and get rewarded for your impact. Join now Capabilities How it works Data development Specialized agents Use cases Customer stories Research Research hub Leaderboards Open Benchmarks Grants Resources Resource library Events Reading Group Blog Company About Careers Press Partners Security Contact us Contact Get started Join expert network Compliance Copyright © 2026 Snorkel AI, Inc. All rights reserved. Terms of Use Privacy Cookie Policy Specialized agents | Snorkel AI Capabilities Capabilities How it works Research-led data and environment development for the frontier's hardest problems Learn more Data development Overview Expert-curated datasets for frontier AI Use cases See how our data improves frontier models Specialized agents Overview Custom AI systems built to unlock ROI fast Customer stories Real-world results from enterprise deployments Research Research Research Research hub Our latest papers and data-centric AI findings Leaderboards Compare model performance across benchmarks Open Benchmarks Grants Funding for open-source AI research Featured BENCHMARK Agentic Coding benchmark A benchmark for evaluating AI models on complex, real-world coding tasks that require multi-step reasoning, tool use, and autonomous problem-solving. Resources Resources Resources Resource library Guides, papers, and tools for data-centric AI Events Upcoming talks, workshops, and conferences Reading Group AI discussions for researchers and practitioners Blog News, updates, and perspectives from our team featured BLOG Terminal-Bench 2.0 Developed by Stanford and Laude Institute with contributions from Snorkel AI, it’s a major leap forward in evaluating AI coding agents. Company Company Company About Our mission, story, and values Careers Open roles and life at our company Press Media resources and announcements Partners Organizations we work with Security How we keep data safe Contact us Get in touch with our team Join our expert community Get paid to shape safer, smarter AI Learn more Get started Get started Specialized agents built on frontier data Agents for workflows generic copilots can’t handle Generic copilots weren't built for your workflows, your data, or your performance standards. Snorkel builds custom agents grounded in enterprise-specific data and evaluated against your real-world criteria. Talk to our team Off-the-shelf agents fall short in enterprise ROI Most enterprise agents fail for the same reasons: they weren't trained on data that reflects the actual workflow, they were evaluated against benchmarks that don't map to real performance, and there's no systematic way to improve them when they underperform. How we build Custom agents for specialized workflows For workflows where enterprise-specific data, context, and operating knowledge create an advantage that off-the-shelf solutions can't match. 01 Use case scoping Identifying the workflows where a custom agent creates measurable, defensible value over generic alternatives. 02 Specialized dataset development Building the training and evaluation data that reflects your actual domain, edge cases, and operating requirements. 03 Environment-grounded evaluation Agents tested against task-specific rubrics and programmatic pass/fail criteria. 04 Production deployment Systems you can run, monitor, and own in your environment. 05 Continuous improvement The same evaluate → curate → refine loop used in frontier model development, applied to your agent over time. Use cases Where AI needs to be right, not just good enough Snorkel helps teams deploy agents for decisions that carry real consequences,   where domain-specific data, expert judgment, and auditable evaluation criteria are the difference between a system you can trust and one you can't. Discuss your use case Credit decisioning Agents that analyze financial documents and proprietary data, measured against institution-specific accuracy criteria and regulatory requirements. Insurance underwriting Agents that evaluate complex risk submissions against expert-specific guidelines and evaluation criteria grounded in underwriter judgment. Clinical diagnostics Agents that process unstructured medical records, evaluated against clinician-defined criteria with application-specific priorities like diagnostic sensitivity. OUR APPROACH Reliable agents aren't a prompting problem. They're a data problem. See how the lab works The same data development system Snorkel uses to improve frontier models is what powers our specialized agents. Evaluated against task-specific rubrics and programmatic checks, refined through adjudication and provenance practices that make improvement systematic rather than intuitive. When an agent underperforms, you know exactly where, why, and what data to build to fix it. PUBLISHED RESEARCH Research-backed. Production-ready. Snorkel's approach to agent development is grounded in the same research methodology used with leading frontier AI labs. Explore research and benchmarks Benchmark Agentic coding benchmark: Evaluating AI models on complex, real-world coding tasks Blog SlopCodeBench: Measuring Code Erosion as Agents Iterate Blog Terminal-Bench 2.0: Raising the bar for AI agent evaluation Research Paper Learning from Less: Measuring the Effectiveness of RLVR in Low Data and Compute Regimes Research Paper RIFT: A Rubric Failure Mode Taxonomy and Automated Diagnostics of For models that need to be right. Not just good enough. Request dataset samples Explore research Get started How do you want to work with Snorkel? AI data development services Accelerate the development of frontier AI models with expert-curated, enterprise-grade data. Learn how Snorkel’s Data-as-a-Service helps teams label, refine, and evaluate high-quality, domain-specific datasets for your projects. Talk to a data researcher Build specialized agents Explore how Snorkel can collaborate with your product and development teams to build and deploy custom AI and agentic systems. Solutions undergo rigorous testing based on your business criteria to ensure positive ROI, faster. Talk to a strategist Become an expert contributor Join the Snorkel Expert Contributor community and help shape the future of AI with your expertise. Contribute to groundbreaking projects, share domain-specific insights, and get rewarded for your impact. Join now Capabilities How it works Data development Specialized agents Use cases Customer stories Research Research hub Leaderboards Open Benchmarks Grants Resources Resource library Events Reading Group Blog Company About Careers Press Partners Security Contact us Contact Get started Join expert network Compliance Copyright © 2026 Snorkel AI, Inc. All rights reserved. Terms of Use Privacy Cookie Policy Terminal-lBench 2.0: The Frontier Agentic Coding Benchmark Capabilities Capabilities How it works Research-led data and environment development for the frontier's hardest problems Learn more Data development Overview Expert-curated datasets for frontier AI Use cases See how our data improves frontier models Specialized agents Overview Custom AI systems built to unlock ROI fast Customer stories Real-world results from enterprise deployments Research Research Research Research hub Our latest papers and data-centric AI findings Leaderboards Compare model performance across benchmarks Open Benchmarks Grants Funding for open-source AI research Featured BENCHMARK Agentic Coding benchmark A benchmark for evaluating AI models on complex, real-world coding tasks that require multi-step reasoning, tool use, and autonomous problem-solving. Resources Resources Resources Resource library Guides, papers, and tools for data-centric AI Events Upcoming talks, workshops, and conferences Reading Group AI discussions for researchers and practitioners Blog News, updates, and perspectives from our team featured BLOG Terminal-Bench 2.0 Developed by Stanford and Laude Institute with contributions from Snorkel AI, it’s a major leap forward in evaluating AI coding agents. Company Company Company About Our mission, story, and values Careers Open roles and life at our company Press Media resources and announcements Partners Organizations we work with Security How we keep data safe Contact us Get in touch with our team Join our expert community Get paid to shape safer, smarter AI Learn more Get started Get started See all articles Data development , Research Terminal-Bench 2.0: Raising the bar for AI agent evaluation Kobie Crawford Published: November 07, 2025 Terminal-Bench , a joint project between Stanford University and Laude Institute, has quickly become the gold standard for evaluating AI coding agents’ capabilities in command-line environments. Since its launch earlier this year, the benchmark has garnered over 1,000 GitHub stars and attracted contributions from nearly 100 developers worldwide. At Snorkel AI, we’re proud to be one of the top external contributors to this project, and we’re thrilled to see the release of Terminal-Bench 2.0 –a significant leap forward in both difficulty and quality. Why Terminal-Bench 2.0 matters From the beginning, Terminal-Bench was designed as a living benchmark that would evolve alongside AI capabilities. As frontier models improved, with performance climbing over 50% on the original benchmark, the team knew it was time to raise the stakes. Terminal-Bench 2.0 delivers on this vision with 89 carefully curated tasks that push the boundaries of what AI agents can accomplish in terminal environments. Each task underwent rigorous verification, with the development team meticulously reviewing every challenge to ensure it meets the highest standards. What’s new in 2.0 Increased difficulty: Terminal-Bench 2.0 is substantially more challenging than its predecessor. Tasks now better represent the frontier challenges that distinguish truly capable agents from those that can only handle routine operations. This ensures the benchmark remains relevant as AI capabilities advance—maintaining the crucial 50% performance ceiling where there’s clear room for improvement while still delivering meaningful signal for evaluation. Enhanced verification: One of the most significant improvements in 2.0 is the dramatic increase in task quality and reproducibility. The original benchmark included several problematic tasks—some were unsolvable for artificial reasons, others set arbitrary thresholds, and a few lacked robustness (like the YouTube download task affected by changing anti-bot protections). Terminal-Bench 2.0 eliminates these problems. Every task is now reproducible, properly specified, and genuinely solvable, with the team confident that near-100% performance is attainable for sufficiently capable agents. Real-world impact: Terminal-Bench 2.0 removes tasks that don’t reflect valuable real-world work. The easy “Hello World” debugging task is gone, along with other challenges that, while interesting academically, don’t provide meaningful details about an agent’s economic impact or ability to perform valuable tasks that engineers actually get paid to do. Snorkel AI’s contributions Snorkel’s researchers joined other contributors to Terminal-Bench to help the project achieve its goals of delivering a thoroughly vetted, challenging dataset. The team’s main contributions to the project centered on three areas: Reliable difficulty assessment: Our research team leveraged its expertise to develop a systematic assessment of Terminal-Bench 2.0 task difficulty, providing consistent criteria that could be applied across all contributed tasks. Extended failure mode analysis: We developed a failure taxonomy and collected traces to better understand where LLMs fail when executing Terminal-Bench tasks. These insights can help with better task design and inform how to improve agents and models under test. Tasks: The Snorkel team is a top contributor to the Terminal-Bench registry, and we’re pleased that we have been able to provide tasks to the Terminal-Bench 2.0 dataset. The next level: Introducing Harbor In addition to Terminal-Bench 2.0, the team has just announced an exciting new project named Harbor ( site | repo ). Harbor represents a significant evolution in how developers can scale up containerized AI agent environments. Born from observing how the community actually used Terminal-Bench, Harbor abstracts away the complexities of container-based rollouts—a challenge that seems simple but quickly grows complex at scale. The framework emerged after the Terminal-Bench team noticed users deploying the benchmark in unexpected ways—for example, as CI/CD tests for agents, for reinforcement learning with synthetic tasks, and for prompt optimization. All these use cases had the same abstraction in common: containerized environments performing rollouts that return tokens and rewards. Harbor makes this pattern accessible with minimal code, allowing developers to scale from local testing to thousands of parallel containers across neocloud providers like Daytona , E2B , and Modal , or on self-managed Kubernetes clusters. Better together: the benchmark and the framework Terminal-Bench 2.0 represents more than just a version update—it’s a commitment to maintaining the highest quality evaluation infrastructure as AI agent capabilities increase. By prioritizing rigorous verification and meaningful difficulty over arbitrary metrics, the benchmark ensures that progress on Terminal-Bench translates to genuine improvements in AI agents’ ability to perform complex, real-world tasks. And now, with Harbor, practitioners can parallelize the execution of entire datasets for faster iteration and greater efficiency. For teams building the next generation of AI coding assistants, Terminal-Bench 2.0 and Harbor provide an evaluation framework with the necessary robustness and reproducibility to measure true progress in this rapidly evolving space. At Snorkel, we enthusiastically support the Terminal-Bench community and are confident that reproducible, containerized environments will accelerate the development of accurate and reliable agentic AI systems. If you need to build or improve your agents with expert-verified data in an RL environment, come talk to us ! Terminal-Bench is an open-source project led by Stanford University and Laude Institute, with contributions from a vibrant community of individuals and organizations, including Snorkel AI. To learn more about the benchmark, contribute tasks, or evaluate your own agents, visit tbench.ai or join the project’s Discord community . To find out more about Snorkel’s data development platform and our work with frontier AI labs, visit us at snorkel.ai and connect with our team. Sh Open Benchmarks Grant for Agentic AI | Snorkel AI A Frontier Data Summit is on the horizon. Stay in the loop Blog Apply now $3M in research grants // Rolling acceptances open Funding the next wave of frontier benchmarks Apply for a grant In partnership with Benchmarks that define and advance the frontier Our ability to measure AI has been outpaced by our ability to develop it, and this evaluation gap is one of the most important problems in AI. Looking ahead, Benchmarks must close the gap between what we measure and actually encounter, falling along three core dimensions: environment complexity, autonomy horizon, and output complexity. Backed by a $3M commitment, the Open Benchmarks Grants program funds open-source datasets, benchmarks, and evaluation artifacts that shape how frontier AI systems are built and evaluated. Read the blog Call for proposals We are seeking applications from researchers, labs, and engineers building benchmarks for the next wave of AI capabilities. We’re looking for benchmarks that drive the fundamental axes for AI agency ( read more on our blog ) and welcome independent directions from the research community as well. Apply for a grant How to apply 01 Apply Submit a brief application for the Open Benchmarks Grant. Applications are reviewed on a rolling basis. 02 Selection Proposals are reviewed with input from a steering committee of academic and industry leaders. 03 Launch Selected teams receive expert data credits and begin collaboration with Snorkel, partner research teams, and platform partners where applicable. 04 Publication Recipients create and publish the resulting open-source dataset or paper, with acknowledgment of Snorkel’s support. Start your application Terms and conditions Rolling acceptances — no fixed deadline. Apply when you're ready. Steering committee Proposals are reviewed by a committee of researchers and engineers at the frontier of AI evaluation. They bring independent academic judgment — Snorkel does not direct their decisions. Karthik Narasimhan Princeton University Professor of Computer Science at Princeton. Research focuses on reinforcement learning for language and agentic systems. Chris Ré Stanford University Associate Professor at Stanford and co-founder of Snorkel AI. Pioneered programmatic data development for machine learning. Ludwig Schmidt Stanford University · LAION Stanford researcher and LAION collaborator. Co-creator of CIFAR-10.1, WILDS, and several foundational evaluation benchmarks. Yu Su Ohio State University Professor at Ohio State. Research spans conversational AI, question answering, and evaluation methodology for language models. Lewis Tunstall Hugging Face Machine learning engineer at Hugging Face and co-author of Natural Language Processing with Transformers. Fred Sala Univ. of Wisconsin–Madison Assistant Professor at Wisconsin–Madison. Research focuses on data-centric AI, weak supervision, and programmable training pipelines. OPEN FOR APPLICATIONS Shaping the future of AI starts with better data. Rolling acceptances begin March 1. Apply for a grant Contact us Coming fall 2026 Frontier Data Summit is on the horizon Sign up to hear about new grants, funded projects, and the upcoming Frontier Data Summit — no marketing, just signal. One email per major announcement. Unsubscribe anytime. Copyright © 2026 Snorkel AI, Inc. All rights reserved. Blog Terms of Use Privacy Cookie Policy Blog Apply now Contact Us | Snorkel AI Address | Snorkel AI Capabilities Capabilities How it works Research-led data and environment development for the frontier's hardest problems Learn more Data development Overview Expert-curated datasets for frontier AI Use cases See how our data improves frontier models Specialized agents Overview Custom AI systems built to unlock ROI fast Customer stories Real-world results from enterprise deployments Research Research Research Research hub Our latest papers and data-centric AI findings Leaderboards Compare model performance across benchmarks Open Benchmarks Grants Funding for open-source AI research Featured BENCHMARK Agentic Coding benchmark A benchmark for evaluating AI models on complex, real-world coding tasks that require multi-step reasoning, tool use, and autonomous problem-solving. Resources Resources Resources Resource library Guides, papers, and tools for data-centric AI Events Upcoming talks, workshops, and conferences Reading Group AI discussions for researchers and practitioners Blog News, updates, and perspectives from our team featured BLOG Terminal-Bench 2.0 Developed by Stanford and Laude Institute with contributions from Snorkel AI, it’s a major leap forward in evaluating AI coding agents. Company Company Company About Our mission, story, and values Careers Open roles and life at our company Press Media resources and announcements Partners Organizations we work with Security How we keep data safe Contact us Get in touch with our team Join our expert community Get paid to shape safer, smarter AI Learn more Get started Get started Company Contact us Looking to get in touch with us? We would love to connect. Sales Looking to evalute if Snorkel is right for your AI needs? Request a meeting. Support Looking for help using Snorkel or AI solutions? Get in touch with [email protected]. Media For all press and media inquiries reach out to us at [email protected]. General For general inquiries reach out to us at [email protected]. Expert Community Want to join? Apply here. Already a member and have a question? Reach out to [email protected] . Careers Want to learn more about working at Snorkel AI? Contact  [email protected]. Partners If you are a BPO looking to become a Snorkel Data Development partner, please fill out this form. Open Benchmarks Grants Have a question about Open Benchmarks Grants, powered by Snorkel? Email [email protected]. Sales Want to evaluate if Snorkel is right for your AI needs? Use the form. " * " indicates required fields First Name * Last Name * Work Email * Company * Phone Job Title * Country Select... United States Canada Afghanistan Albania Algeria American Samoa Andorra Angola Anguilla Antarctica Antigua and Barbuda Argentina Armenia Aruba Australia Austria Azerbaijan Bahamas Bahrain Bangladesh Barbados Belarus Belgium Belize Benin Bermuda Bhutan Bolivia Bonaire, Sint Eustatius and Saba Bosnia and Herzegovina Botswana Bouvet Island Brazil British Indian Ocean Territory Brunei Darussalam Bulgaria Burkina Faso Burundi Cabo Verde Cambodia Cameroon Cayman Islands Central African Republic Chad Chile China Christmas Island Cocos Islands Colombia Comoros Congo Congo, Democratic Republic of the Cook Islands Costa Rica Croatia Cuba Curaçao Cyprus Czechia Côte d'Ivoire Denmark Djibouti Dominica Dominican Republic Ecuador Egypt El Salvador Equatorial Guinea Eritrea Estonia Eswatini Ethiopia Falkland Islands Faroe Islands Fiji Finland France French Guiana French Polynesia French Southern Territories Gabon Gambia Georgia Germany Ghana Gibraltar Greece Greenland Grenada Guadeloupe Guam Guatemala Guernsey Guinea Guinea-Bissau Guyana Haiti Heard Island and McDonald Islands Holy See Honduras Hong Kong Hungary Iceland India Indonesia Iran Iraq Ireland Isle of Man Israel Italy Jamaica Japan Jersey Jordan Kazakhstan Kenya Kiribati Korea, Democratic People's Republic of Korea, Republic of Kuwait Kyrgyzstan Lao People's Democratic Republic Latvia Lebanon Lesotho Liberia Libya Liechtenstein Lithuania Luxembourg Macao Madagascar Malawi Malaysia Maldives Mali Malta Marshall Islands Martinique Mauritania Mauritius Mayotte Mexico Micronesia Moldova Monaco Mongolia Montenegro Montserrat Morocco Mozambique Myanmar Namibia Nauru Nepal Netherlands New Caledonia New Zealand Nicaragua Niger Nigeria Niue Norfolk Island North Macedonia Northern Mariana Islands Norway Oman Pakistan Palau Palestine, State of Panama Papua New Guinea Paraguay Peru Philippines Pitcairn Poland Portugal Puerto Rico Qatar Romania Russian Federation Rwanda Réunion Saint Barthélemy Saint Helena, Ascension and Tristan da Cunha Saint Kitts and Nevis Saint Lucia Saint Martin Saint Pierre and Miquelon Saint Vincent and the Grenadines Samoa San Marino Sao Tome and Principe Saudi Arabia Senegal Serbia Seychelles Sierra Leone Singapore Sint Maarten Slovakia Slovenia Solomon Islands Somalia South Africa South Georgia and the South Sandwich Islands South Sudan Spain Sri Lanka Sudan Suriname Svalbard and Jan Mayen Sweden Switzerland Syria Arab Republic Taiwan Tajikistan Tanzania, the United Republic of Thailand Timor-Leste Togo Tokelau Tonga Trinidad and Tobago Tunisia Turkmenistan Turks and Caicos Islands Tuvalu Türkiye US Minor Outlying Islands Uganda Ukraine United Arab Emirates United Kingdom Uruguay Uzbekistan Vanuatu Venezuela Viet Nam Virgin Islands, British Virgin Islands, U.S. Wallis and Futuna Western Sahara Yemen Zambia Zimbabwe Åland Islands State * Select... AK AL AR AZ CA CO CT DE FL GA HI IA ID IL IN KS KY LA MA MD ME MI MN MO MS MT NC ND NE NH NJ NM NV NY OH OK OR PA RI SC SD TN TX UT VA VT WA WI WV WY What would you like to discuss on your call? Email consent checkbox * Yes, I would like to receive marketing communications regarding Snorkel AI products, services, and events. I can unsubscribe at any time later. By selecting this box, I confirm that I agree to the storing and processing of my personal data by Snorkel AI as described in the privacy policy. Email consent checkbox * By checking this box, I agree to receive SMS messages from Snorkel at the phone number provided above, which can contain reminders and notifications. Message and data rates may apply. Reply STOP to opt out. Please view our Privacy Policy here . Δ By submitting this form, I agree to the Terms of Use and acknowledge that my information will be used in accordance with the Privacy Policy . Get started How do you want to work with Snorkel? AI data development services Accelerate the development of frontier AI models with expert-curated, enterprise-grade data. Learn how Snorkel’s Data-as-a-Service helps teams label, refine, and evaluate high-quality, domain-specific datasets for your projects. Talk to a data researcher Build specialized agents Explore how Snorkel can collaborate with your product and development teams to build and deploy custom AI and agentic systems. Solutions undergo rigorous testing based on your business criteria to ensure positive ROI, faster. Talk to a strategist Become an expert contributor Join the Snorkel Expert Contributor community and help shape the future of AI with your expertise. Contribute to groundbreaking projects, share domain-specific insights, and get rewarded for your impact. Join now Capabilities How it works Data development Specialized agents Use cases Customer stories Research Research hub Leaderboards Open Benchmarks Grants Resources Resource library Events Reading Group Blog Company About Careers Press Partners Security Contact us Contact Get started Join expert network Compliance Copyright © 2026 Snorkel AI, Inc. All rights reserved. Terms of Use Privacy Cookie Policy Learning from Less: Measuring the Effectiveness of RLVR in Low Data and Compute Regimes | Snorkel AI Capabilities Capabilities How it works Research-led data and environment development for the frontier's hardest problems Learn more Data development Overview Expert-curated datasets for frontier AI Use cases See how our data improves frontier models Specialized agents Overview Custom AI systems built to unlock ROI fast Customer stories Real-world results from enterprise deployments Research Research Research Research hub Our latest papers and data-centric AI findings Leaderboards Compare model performance across benchmarks Open Benchmarks Grants Funding for open-source AI research Featured BENCHMARK Agentic Coding benchmark A benchmark for evaluating AI models on complex, real-world coding tasks that require multi-step reasoning, tool use, and autonomous problem-solving. Resources Resources Resources Resource library Guides, papers, and tools for data-centric AI Events Upcoming talks, workshops, and conferences Reading Group AI discussions for researchers and practitioners Blog News, updates, and perspectives from our team featured BLOG Terminal-Bench 2.0 Developed by Stanford and Laude Institute with contributions from Snorkel AI, it’s a major leap forward in evaluating AI coding agents. Company Company Company About Our mission, story, and values Careers Open roles and life at our company Press Media resources and announcements Partners Organizations we work with Security How we keep data safe Contact us Get in touch with our team Join our expert community Get paid to shape safer, smarter AI Learn more Get started Get started Back to library arXiv Preprint/ MLSys | 2026 Learning from Less: Measuring the Effectiveness of RLVR in Low Data and Compute Regimes Justin Bauer , Thomas Walsh , Derek Pham , Harit Vishwakarma , Armin Parchami , Fred Sala , Paroma Varma Abstract Fine-tuning Large Language Models (LLMs) typically relies on large quantities of high-quality annotated data, or questions with well-defined ground truth answers in the case of Reinforcement Learning with Verifiable Rewards (RLVR). While previous work has explored the benefits to model reasoning capabilities by scaling both data and compute used for RLVR, these results lack applicability in many real-world settings where annotated data and accessible compute may be scarce. In this work, we present a comprehensive empirical study of open-source Small Language Model (SLM) performance after RLVR in low data regimes. Across three novel datasets covering number counting problems, graph reasoning, and spatial reasoning, we characterize how model performance scales with dataset size, diversity, and complexity. We demonstrate that (1) procedural datasets allow for fine-grained evaluation and training dataset development with controllable properties (size, diversity, and complexity), (2) under RLVR, models trained on lower complexity tasks can generalize to higher complexity tasks, and (3) training on mixed complexity datasets is associated with the greatest benefits in low data regimes, providing up to 5x sample efficiency versus training on easy tasks. These findings inspire future work on the development of data scaling laws for RLVR and the use of procedural data generators to further understand effective data development for efficient LLM fine-tuning. Read the paper Get started How do you want to work with Snorkel? AI data development services Accelerate the development of frontier AI models with expert-curated, enterprise-grade data. Learn how Snorkel’s Data-as-a-Service helps teams label, refine, and evaluate high-quality, domain-specific datasets for your projects. Talk to a data researcher Build specialized agents Explore how Snorkel can collaborate with your product and development teams to build and deploy custom AI and agentic systems. Solutions undergo rigorous testing based on your business criteria to ensure positive ROI, faster. Talk to a strategist Become an expert contributor Join the Snorkel Expert Contributor community and help shape the future of AI with your expertise. Contribute to groundbreaking projects, share domain-specific insights, and get rewarded for your impact. Join now Capabilities How it works Data development Specialized agents Use cases Customer stories Research Research hub Leaderboards Open Benchmarks Grants Resources Resource library Events Reading Group Blog Company About Careers Press Partners Security Contact us Contact Get started Join expert network Compliance Copyright © 2026 Snorkel AI, Inc. All rights reserved. Terms of Use Privacy Cookie Policy The Snorkel AI Blog | Home Capabilities Capabilities How it works Research-led data and environment development for the frontier's hardest problems Learn more Data development Overview Expert-curated datasets for frontier AI Use cases See how our data improves frontier models Specialized agents Overview Custom AI systems built to unlock ROI fast Customer stories Real-world results from enterprise deployments Research Research Research Research hub Our latest papers and data-centric AI findings Leaderboards Compare model performance across benchmarks Open Benchmarks Grants Funding for open-source AI research Featured BENCHMARK Agentic Coding benchmark A benchmark for evaluating AI models on complex, real-world coding tasks that require multi-step reasoning, tool use, and autonomous problem-solving. Resources Resources Resources Resource library Guides, papers, and tools for data-centric AI Events Upcoming talks, workshops, and conferences Reading Group AI discussions for researchers and practitioners Blog News, updates, and perspectives from our team featured BLOG Terminal-Bench 2.0 Developed by Stanford and Laude Institute with contributions from Snorkel AI, it’s a major leap forward in evaluating AI coding agents. Company Company Company About Our mission, story, and values Careers Open roles and life at our company Press Media resources and announcements Partners Organizations we work with Security How we keep data safe Contact us Get in touch with our team Join our expert community Get paid to shape safer, smarter AI Learn more Get started Get started NEW Continual Learning Bench by Berkeley & Snorkel RESOURCES Blog Ideas, updates, and practical guidance from the Snorkel team. Research Closing the Evaluation Gap in Agentic AI Announcing a $3M commitment to launch Open Benchmarks Grants Vincent Sunn Chen February 11, 2026 All articles Sort: Newest Newest Oldest NEW Why coding agents need better data, evals, and environments Coding agents have moved from tab-complete to teammate. They autonomously inspect repositories, edit files, run commands, diagnose failures, and work through multi-step engineering tasks. That creates a harder reliability problem. A model that only suggests code is easy for a human to evaluate. A coding agent refactoring your repository and testing its own changes is much harder to supervise –… May 11, 2026 • Justin Bauer Learn more about Why coding agents need better data, evals, and environments Understanding Olmix: A Framework for Data Mixing Throughout Language Model Development At our latest Snorkel AI Reading Group, Mayee Chen (Stanford, Hazy Research) stopped by our San Francisco office to walk us through Olmix: A Framework for Data Mixing Throughout LM Development — work she contributed to during her internship at Ai2 on OLMo 3. Olmix tackles one of the messiest, least-documented levers in LLM pre-training: how to set the ratios… May 01, 2026 • David Burch Learn more about Understanding Olmix: A Framework for Data Mixing Throughout Language Model Development Benchmarks should shape the frontier, not just measure it Since launching the Open Benchmarks Grants, we’ve received more than 100 applications from academic groups and industry labs spanning a wide range of domains and capabilities. As the best benchmarks drive how the field allocates research effort, the bar for benchmarks has risen as well. Here, we share what’s now table stakes for useful benchmarks, and what separates the ones… Apr 07, 2026 • Vincent Sunn Chen Learn more about Benchmarks should shape the frontier, not just measure it Benchtalks #1: Alex Shaw (Terminal-Bench, Harbor) – Building the Benchmark Factory To kick off our inaugural Benchtalks, a series dedicated to the researchers building these measurement toolkits, Snorkel AI co-founder Vincent Sunn Chen sat down with Alex Shaw, Founding MTS at Laude Institute and co-creator of Terminal-Bench and Harbor. Highlights More on Terminal-Bench: See the leaderboard and the catalog of tasks at tbench.ai. Explore Harbor: Learn how to scale your agent… Mar 31, 2026 • Vincent Sunn Chen Learn more about Benchtalks #1: Alex Shaw (Terminal-Bench, Harbor) – Building the Benchmark Factory Building FinQA: An Open RL Environment for Financial Reasoning Agents TL;DR: We built FinQA — a financial question-answering environment with 290 expert-curated questions across 22 public companies, now available on OpenEnv. Agents use MCP tools to discover schemas, write constrained SQL queries, and answer multi-step questions from real SEC 10-K filings. Most open-source models struggle with this kind of multi-step tool use, and even frontier closed-source models, while more accurate,… Mar 30, 2026 • Bhavishya Pohani Learn more about Building FinQA: An Open RL Environment for Financial Reasoning Agents How Tool Discipline Let a 4B Model Outsmart a 235B Giant on Financial Tasks The Snorkel research team collaborated with the rLLM team at UC Berkeley on the Agentica project, using their open-source rLLM framework to fine-tune Qwen3-4B-Instruct-2507, delivering a model that beats Qwen3-235B-A22B on Snorkel AI’s expert-curated financial benchmarks – at 1/60th the size. A full breakdown of the results are published in the rLLM blog here. The key insight? Just focus on… Feb 18, 2026 • Chris Glaze Learn more about How Tool Discipline Let a 4B Model Outsmart a 235B Giant on Financial Tasks Coding agents don’t need to be perfect, they need to recover Error analysis of 8 models on Agentic Coding tasks Successful completion of complex tasks doesn’t come from models being always right. It comes from models being resilient when things go wrong. To get a deeper understanding of model behavior in agentic environments, our team analyzed all of the errors found in the full traces of tasks from our Agentic Coding… Feb 13, 2026 • Ramya Ramakrishnan Learn more about Coding agents don’t need to be perfect, they need to recover Closing the Evaluation Gap in Agentic AI Today, AI is marked by a growing asymmetry: the excitement around agentic AI is real — backed by quantitative progress on model cards and genuine leaps forward, especially in coding. But ask individuals or enterprises where they feel ready to deploy agentic automation in high-stakes, domain-specific settings outside of coding… and you will find hesitation. The reason: our ability to… Feb 11, 2026 • Vincent Sunn Chen Learn more about Closing the Evaluation Gap in Agentic AI SlopCodeBench: Measuring Code Erosion as Agents Iterate SlopCodeBench reveals how AI coding agents degrade code quality over time—measuring “slop,” technical debt, and architectural erosion across iterations. Jan 20, 2026 • Kobie Crawford Learn more about SlopCodeBench: Measuring Code Erosion as Agents Iterate 1 2 … 35 36 Join our newsletter For expert advice, the latest research, and exclusive events. By submitting this form, I acknowledge I will receive email updates from Snorkel AI, and I agree to the Terms of Use and acknowledge that my information will be used in accordance with the Privacy Policy . Email address (Required) Δ Get started How do you want to work with Snorkel? AI data development services Accelerate the development of frontier AI models with expert-curated, enterprise-grade data. Learn how Snorkel’s Data-as-a-Service helps teams label, refine, and evaluate high-quality, domain-specific datasets for your projects. Talk to a data researcher Build specialized agents Explore how Snorkel can collaborate with your product and development teams to build and deploy custom AI and agentic systems. Solutions undergo rigorous testing based on your business criteria to ensure positive ROI, faster. Talk to a strategist Become an expert contributor Join the Snorkel Expert Contributor community and help shape the future of AI with your expertise. Contribute to groundbreaking projects, share domain-specific insights, and get rewarded Customer Stories | Snorkel AI Capabilities Capabilities How it works Research-led data and environment development for the frontier's hardest problems Learn more Data development Overview Expert-curated datasets for frontier AI Use cases See how our data improves frontier models Specialized agents Overview Custom AI systems built to unlock ROI fast Customer stories Real-world results from enterprise deployments Research Research Research Research hub Our latest papers and data-centric AI findings Leaderboards Compare model performance across benchmarks Open Benchmarks Grants Funding for open-source AI research Featured BENCHMARK Agentic Coding benchmark A benchmark for evaluating AI models on complex, real-world coding tasks that require multi-step reasoning, tool use, and autonomous problem-solving. Resources Resources Resources Resource library Guides, papers, and tools for data-centric AI Events Upcoming talks, workshops, and conferences Reading Group AI discussions for researchers and practitioners Blog News, updates, and perspectives from our team featured BLOG Terminal-Bench 2.0 Developed by Stanford and Laude Institute with contributions from Snorkel AI, it’s a major leap forward in evaluating AI coding agents. Company Company Company About Our mission, story, and values Careers Open roles and life at our company Press Media resources and announcements Partners Organizations we work with Security How we keep data safe Contact us Get in touch with our team Join our expert community Get paid to shape safer, smarter AI Learn more Get started Get started Capabilities Customer stories See how leading teams use Snorkel to build better AI systems. Customer story DIU Enhances Decision-Making Resilience with Snorkel AI This Top 5 Global Telco aimed to evolve its internal billing co-pilot into a customer-facing chatbot capable of serving its global customer base. However, the project stalled at 54% accuracy due to data blind spots and reasoning errors that frustrated efforts to launch. Customer story Experian improved agent response times under 3 seconds with Snorkel Experian’s customer support team used to require agents to manually find and customize templates to respond to common questions. Customer story How Rox achieved 99% accuracy with Snorkel Evaluate Enterprises facing aggressive revenue targets without more headcount are turning to agentic AI innovator Rox. Rox is redefining the revenue stack with it’s AI-powered sales productivity platform, starting with the Rox sales agent swarm which provides agents that can perform at the level of top sales reps. Customer story Deploying Production AI in All customer stories Sort: Newest Newest Oldest Deploying Production AI in Learn more about Deploying Production AI in DIU Enhances Decision-Making Resilience with Snorkel AI This Top 5 Global Telco aimed to evolve its internal billing co-pilot into a customer-facing chatbot capable of serving its global customer base. However, the project stalled at 54% accuracy due to data blind spots and reasoning errors that frustrated efforts to launch. Learn more about DIU Enhances Decision-Making Resilience with Snorkel AI From stalled pilot to $43M annual ROI: Top 5 global telco achieves 95% accuracy with Snorkel AI This Top 5 Global Telco aimed to evolve its internal billing co-pilot into a customer-facing chatbot capable of serving its global customer base. Learn more about From stalled pilot to $43M annual ROI: Top 5 global telco achieves 95% accuracy with Snorkel AI How a leading international telecom provider scaled agentic AI with high-quality synthetic data An Asian telecom leader aimed to expand its offerings with a flagship AI personal assistant. However, the team faced critical roadblocks: Learn more about How a leading international telecom provider scaled agentic AI with high-quality synthetic data Experian improved agent response times under 3 seconds with Snorkel Experian’s customer support team used to require agents to manually find and customize templates to respond to common questions. Learn more about Experian improved agent response times under 3 seconds with Snorkel How Rox achieved 99% accuracy with Snorkel Evaluate Enterprises facing aggressive revenue targets without more headcount are turning to agentic AI innovator Rox. Rox is redefining the revenue stack with it’s AI-powered sales productivity platform, starting with the Rox sales agent swarm which provides agents that can perform at the level of top sales reps. Learn more about How Rox achieved 99% accuracy with Snorkel Evaluate How an F500 telecom uses Snorkel AI to measure and improve virtual assistant CX Our client values customer satisfaction as a core business goal, but their ability to track it with their virtual assistant was limited. The company typically tried to collect experience ratings at the end of interactions. Customers answered these surveys infrequently—and typically when dissatisfied—making the data unreliable in its raw form. Learn more about How an F500 telecom uses Snorkel AI to measure and improve virtual assistant CX Wayfair achieves 99% category win rate and 7-point clickthrough lift with data-centric AI Wayfair is a Boston-based e-commerce company specializing in home goods and furniture, serving ~22M customers and partnering with ~20K suppliers. Learn more about Wayfair achieves 99% category win rate and 7-point clickthrough lift with data-centric AI Consulting giant eliminates a year of labeling time with Snorkel Using AI to support audit relevance and improve operational efficiency, this global… Learn more about Consulting giant eliminates a year of labeling time with Snorkel 1 2 For models that need to be right. Not just good enough. Request dataset samples Talk to our team Get started How do you want to work with Snorkel? AI data development services Accelerate the development of frontier AI models with expert-curated, enterprise-grade data. Learn how Snorkel’s Data-as-a-Service helps teams label, refine, and evaluate high-quality, domain-specific datasets for your projects. Talk to a data researcher Build specialized agents Explore how Snorkel can collaborate with your product and development teams to build and deploy custom AI and agentic systems. Solutions undergo rigorous testing based on your business criteria to ensure positive ROI, faster. Talk to a strategist Become an expert contributor Join the Snorkel Expert Contributor community and help shape the future of AI with your expertise. Contribute to groundbreaking projects, share domain-specific insights, and get rewarded for your impact. Join now Capabilities How it works Data development Specialized agents Use cases Customer stories Research Research hub Leaderboards Open Benchmarks Grants Resources Resource library Events Reading Group Blog Company About Careers Press Partners Security Contact us Contact Get started Join expert network Compliance Copyright © 2026 Snorkel AI, Inc. All rights reserved. Terms of Use Privacy Cookie Policy Snorkel Expert Data-as-a-Service | Snorkel AI Capabilities Capabilities How it works Research-led data and environment development for the frontier's hardest problems Learn more Data development Overview Expert-curated datasets for frontier AI Use cases See how our data improves frontier models Specialized agents Overview Custom AI systems built to unlock ROI fast Customer stories Real-world results from enterprise deployments Research Research Research Research hub Our latest papers and data-centric AI findings Leaderboards Compare model performance across benchmarks Open Benchmarks Grants Funding for open-source AI research Featured BENCHMARK Agentic Coding benchmark A benchmark for evaluating AI models on complex, real-world coding tasks that require multi-step reasoning, tool use, and autonomous problem-solving. Resources Resources Resources Resource library Guides, papers, and tools for data-centric AI Events Upcoming talks, workshops, and conferences Reading Group AI discussions for researchers and practitioners Blog News, updates, and perspectives from our team featured BLOG Terminal-Bench 2.0 Developed by Stanford and Laude Institute with contributions from Snorkel AI, it’s a major leap forward in evaluating AI coding agents. Company Company Company About Our mission, story, and values Careers Open roles and life at our company Press Media resources and announcements Partners Organizations we work with Security How we keep data safe Contact us Get in touch with our team Join our expert community Get paid to shape safer, smarter AI Learn more Get started Get started Join our expert network Get paid to put your expertise to work on frontier AI Join Snorkel’s Expert Community to shape safer, smarter AI across 1,000+ domains. Join now How it works Snorkel’s Expert Community Our Expert Community is a network of professionals and academics who use their knowledge to shape real-world AI projects. You’ll work on opportunities that match your background from coding, STEM and medicine to law, the humanities, and even lifestyle areas like travel and fitness. As a member, you’ll connect with peers, share best practices, and contribute expertise that top AI developers rely on. These projects require true subject matter knowledge, not just surface-level skills, and your work helps ensure AI is accurate and reliable. Expert‑driven, not generic crowd work Meaningful projects aligned to your background Flexible, paid contributions with clear quality standards Compensation Expert Community members are paid per project, with rates based on expertise and complexity. Top experts earn $3,000+ per week. Every contribution goes through a quality review, combining automated checks and peer feedback. Many projects pay for attempts and include bonuses for high-quality submissions. Snorkel provides feedback so you can improve and maximize your earnings. Join now Put your knowledge to work Get paid for remote, project-based work aligned to your skills. Coding & software Software engineers, developers, code reviewers, and technical writers STEM Computer scientists, engineers across technical domains, and applied or pure mathematicians Health & medicine Medical practitioners across specialties, surgeons, clinicians, psychologists, internal medicine physicians, and biomedical scientists Finance & business Finance and accounting professionals, economists, HR and compliance specialists, and legal practitioners Humanities & arts Linguists, cultural scholars, philosophers, writers, and visual, performing, or applied artists Social sciences Psychologists, political scientists, economists, and public policy experts Next steps Getting started is easy: complete a quick screening or assessment, tell us about your expertise, sign up to receive payments, and start contributing on eligible projects. 1 Create a profile Simply sign up and create an account. 2 Complete screenings Complete a short assessment or sample task. 3 Onboard Provide your information to receive payments. 4 Join projects Opt into eligible work and start contributing. Qualifications We partner with skilled professionals from academia and industry. Some projects require advanced degrees (PhD, MD, JD, CPA) and active licenses, while others accept equivalent experience. Experts must be able to work as independent contractors, and we cannot support visa sponsorship or restricted regions at this time. What we look for: Deep domain expertise and up-to-date knowledge Clear written reasoning and attention to detail Ability to follow detailed rubrics and maintain consistent quality What you’ll do Most work is asynchronous with clear instructions and rubrics. Projects range from one-off small projects to multi-week engagements—you choose when and how much you contribute. Create Author data in your domain of expertise, such as creating questions that stump frontier LLMs. Review Check data authored by peer domain experts for accuracy and clarity, and recommend fixes or adjustments. Evaluate & improve Compare outputs, apply provided criteria, and flag safety or compliance issues. What they said I started contributing to Snorkel as a task submitter and quickly earned the opportunity to support the platform as a reviewer, which gave me a deeper view into the rigor behind projects. The work consistently operates at a research-grade standard, emphasizing clarity, fairness, and real-world impact in how AI systems are evaluated. Nischal P. Coding What stands out to me most about working with Snorkel is the consistency and clarity of the communication. The team is always responsive, thoughtful, and approachable, which makes it easy to ask questions and feel supported throughout the work. I also really value how strongly Snorkel prioritizes quality, setting clear expectations while ensuring contributors truly understand the intent behind each task. That balance makes the work both meaningful and rewarding. Beverly B. Generalist As an expert contributor, I have the opportunity to apply my clinical pharmacy knowledge and drug information expertise in a unique and impactful way. The work is challenging, and it's rewarding to contribute to the advancement of AI in healthcare. The collaborative nature of the Snorkel team makes the experience even more enriching. Pooravi G., Pharm.D. Drug Information Specialist & Clinical Pharmacist Snorkel's Expert Platform allowed me connect and collaborate with brilliant and accomplished experts from a wide range of fields and industries. The communication, care, and effort from the Snorkel team has been nothing short of excellent. The projects are extremely engaging, interesting, and thought provoking. I am looking forward to the continued future collaborations with such an amazing and outstanding company! Peter P. Generalist FAQs Who is eligible to apply? Projects with Snorkel require a wide variety of domain expertise. If you are an expert in one of the broad fields above (projects typically require an advanced degree in one of these areas), are based in an approved geographic region for that particular project, are open to work as an independent contractor, and are interested in contributing to the future of AI in your speciality area, please apply! I’m interested. How do I apply? Click join now to get started. If you’re new to Snorkel, you’ll create an account and complete a short profile about your background and expertise. Once your information is reviewed and you meet our eligibility criteria, we’ll provide next steps to complete your onboarding and start working on a project. How long does onboarding take? Approved experts can go from applying to contributing within the same day! Once you've completed your onboarding, if your expertise matches with an eligible project, you’ll receive project-specific instructions to get started. When do I get paid? Payments are processed every week for accepted work from roughly the week prior. Snorkel aims to process all payments within two weeks of contribution How Tool Discipline Let a 4B Model Outsmart a 235B Giant on Financial Tasks | Snorkel AI Capabilities Capabilities How it works Research-led data and environment development for the frontier's hardest problems Learn more Data development Overview Expert-curated datasets for frontier AI Use cases See how our data improves frontier models Specialized agents Overview Custom AI systems built to unlock ROI fast Customer stories Real-world results from enterprise deployments Research Research Research Research hub Our latest papers and data-centric AI findings Leaderboards Compare model performance across benchmarks Open Benchmarks Grants Funding for open-source AI research Featured BENCHMARK Agentic Coding benchmark A benchmark for evaluating AI models on complex, real-world coding tasks that require multi-step reasoning, tool use, and autonomous problem-solving. Resources Resources Resources Resource library Guides, papers, and tools for data-centric AI Events Upcoming talks, workshops, and conferences Reading Group AI discussions for researchers and practitioners Blog News, updates, and perspectives from our team featured BLOG Terminal-Bench 2.0 Developed by Stanford and Laude Institute with contributions from Snorkel AI, it’s a major leap forward in evaluating AI coding agents. Company Company Company About Our mission, story, and values Careers Open roles and life at our company Press Media resources and announcements Partners Organizations we work with Security How we keep data safe Contact us Get in touch with our team Join our expert community Get paid to shape safer, smarter AI Learn more Get started Get started See all articles Research How Tool Discipline Let a 4B Model Outsmart a 235B Giant on Financial Tasks Chris Glaze Published: February 18, 2026 The Snorkel research team collaborated with the rLLM team at UC Berkeley on the Agentica project, using their open-source rLLM framework to fine-tune Qwen3-4B-Instruct-2507, delivering a model that beats Qwen3-235B-A22B on Snorkel AI’s expert-curated financial benchmarks – at 1/60th the size. A full breakdown of the results are published in the rLLM blog here . The key insight? Just focus on tool use. Why tool discipline beats scale Large generalist models have excellent reasoning but poor tool discipline. They hallucinate column names, ignore constraints, and generate SQL that returns nonsensical results. The problem isn’t intelligence—it’s reliability. Rather than training on expensive multi-table examples, the team focused on teaching reliable tool use with simple, single-table queries – and those skills generalized. In internal ablations, single-table-only training achieved the best results (66.3% internal Pass@1), outperforming both single + multi-table (61.6%) and a single→multi curriculum (64.8%). The fundamentals generalize: explore tables before querying, validate data before proceeding, retry on failure rather than giving up. Trained on simple tasks, verified in complex environments The Snorkel team’s contributions were (1) the agentic environment for eval and RL, and (2) our Snorkel Finance benchmark and Finance Reasoning benchmark , containing expert-curated financial analysis tasks for evaluating the agent’s performance, so we could be confident that the lift we saw was relevant to realistic, complex tasks. The rLLM team developed single-table queries that focused on using the relevant tools correctly, then completed the RL fine-tuning of the model under test in the environment. Enterprise implications The economics shift substantially. For a firm processing 50,000 analyst queries monthly, this approach could reduce costs by 90% while improving accuracy and keeping data on-premises. A 4B model runs on a single GPU; its 235B counterpart requires a multi-node cluster. The methodology isn’t finance-specific either. Healthcare, legal, insurance – anywhere structured data and tool use intersect – the same pipeline applies: convert documents into queryable structures, teach tool-calling fundamentals on simple queries, verify aggressively, and fine-tune. Build your own domain specialists Qwen3-4B-Instruct-2507 was fine-tuned using the rLLM framework on a cluster of 8x H100 GPUs. By using small, specialized judges (GPT-5-nano) for simpler verifications and reserving larger models only for complex multi-table queries, the team kept the total training cost under $500 per run. This fundamentally changes the accessibility of domain adaptation. You do not need a massive pre-training cluster to build a state-of-the-art specialist; instead, you need the right domain expertise, a well-engineered RL environment, and a smart verification pipeline. The team is open-sourcing everything. Check out the rLLM blog here for links to their repository and the full details. Snorkel is thrilled to have collaborated with Berkeley’s Sky Computing Lab and the rLLM team on this research project. For more details, check out their homepage here. For more information on how Snorkel can help you with RL environments and expert-curated datasets, come talk to us! Share this article Chris Glaze Chris Glaze is Applied Research Scientist at Snorkel AI. He is an experienced PhD with a demonstrated history of developing novel machine learning tools and mathematical models in academia and industry. Accomplishments span data mining, experimental research, and application to digital technologies. Recommended articles See all articles Research Why coding agents need better data, evals, and environments Coding agents have moved from tab-complete to teammate. They autonomously inspect repositories, edit files, run commands, diagnose failures, and work through multi-step engineering tasks. That creates a harder reliability problem. A model that only suggests code is easy for a human to evaluate. A coding agent refactoring your repository and testing its own changes is much harder to supervise –… Justin Bauer May 11, 2026 Research Understanding Olmix: A Framework for Data Mixing Throughout Language Model Development At our latest Snorkel AI Reading Group, Mayee Chen (Stanford, Hazy Research) stopped by our San Francisco office to walk us through Olmix: A Framework for Data Mixing Throughout LM Development — work she contributed to during her internship at Ai2 on OLMo 3. Olmix tackles one of the messiest, least-documented levers in LLM pre-training: how to set the ratios… David Burch May 1, 2026 Research Benchmarks should shape the frontier, not just measure it Since launching the Open Benchmarks Grants, we’ve received more than 100 applications from academic groups and industry labs spanning a wide range of domains and capabilities. As the best benchmarks drive how the field allocates research effort, the bar for benchmarks has risen as well. Here, we share what’s now table stakes for useful benchmarks, and what separates the ones… Vincent Sunn Chen April 7, 2026 Join our newsletter for expert advice, the latest research, and exclusive events. By submitting this form, I acknowledge I will receive email updates from Snorkel AI, and I agree to the Terms of Use and acknowledge that my information will be used in accordance with the Privacy Policy . Get started How do you want to work with Snorkel? AI data development services Accelerate the development of frontier AI models with expert-curated, enterprise-grade data. Learn how Snorkel’s Data-as-a-Service helps teams label, refine, and evaluate high-quality, domain-specific datasets for your projects. Talk to a data researcher Build specialized agents Explore how Snorkel can collaborate with your product and development teams to build and deploy custom AI and agentic systems. Solutions undergo rigorous testing based on your business criteria to ensure positive ROI, faster. Talk to a strategist Become an expert contributor Join the Snorkel Expert Contributor community and help shape the future of AI with your expertise. Contribute to groundbre Snorkel AI Leaderboards | Snorkel AI Capabilities Capabilities How it works Research-led data and environment development for the frontier's hardest problems Learn more Data development Overview Expert-curated datasets for frontier AI Use cases See how our data improves frontier models Specialized agents Overview Custom AI systems built to unlock ROI fast Customer stories Real-world results from enterprise deployments Research Research Research Research hub Our latest papers and data-centric AI findings Leaderboards Compare model performance across benchmarks Open Benchmarks Grants Funding for open-source AI research Featured BENCHMARK Agentic Coding benchmark A benchmark for evaluating AI models on complex, real-world coding tasks that require multi-step reasoning, tool use, and autonomous problem-solving. Resources Resources Resources Resource library Guides, papers, and tools for data-centric AI Events Upcoming talks, workshops, and conferences Reading Group AI discussions for researchers and practitioners Blog News, updates, and perspectives from our team featured BLOG Terminal-Bench 2.0 Developed by Stanford and Laude Institute with contributions from Snorkel AI, it’s a major leap forward in evaluating AI coding agents. Company Company Company About Our mission, story, and values Careers Open roles and life at our company Press Media resources and announcements Partners Organizations we work with Security How we keep data safe Contact us Get in touch with our team Join our expert community Get paid to shape safer, smarter AI Learn more Get started Get started Back to Leaderboard Agentic Coding A benchmark for evaluating AI models on complex, real-world coding tasks that require multi-step reasoning, tool use, and autonomous problem-solving. Overview The Snorkel Agentic Coding benchmark comprises 100 multi-step coding tasks, evenly distributed across four difficulty tiers, designed to evaluate models across a diverse range of capabilities germane to real-world software engineering work. Taking insights from our contributions to the Terminal-Bench project , our Agentic Coding tasks evaluate agents in fully sandboxed execution environments. Each task is paired with a human-validated reference solution, comprehensive unit tests, and scoring rubrics that assess both final outputs and the agent’s trajectory. The current version of the benchmark spans a wide range of task categories, from typical software engineering related tasks, to advanced ML and data analytics, as well as build and dependency management tasks, and tests agents on long-horizon planning, tracking tasks, evaluating and executing their own solutions, and recovering from potential errors and incorrect previous steps. Our benchmark is built to challenge even the most advanced frontier models. Tasks are constructed with experts in the loop, confirming every challenge to be solvable in the environments in which they run, and verifying the reliability of all dependencies. We have calibrated the tasks so they deliver a range of difficulties, providing meaningful feedback for agents and models across the cost/performance spectrum -- from those pursuing Pareto-optimal results, to those that are delivering truly frontier-level capabilities. Data Sample Below is a sample of the kinds of coding challenges provided in the dataset. Each sample question like the one below is accompanied by all data and dependencies required for execution, a suite of tests, a rubric used by both human and LLM evaluators to score the result, and a golden solution. question Incident Commander: Payments Canary Rollback You are the on-call SRE responding to a spike in 5xx errors after a canary rollout of payments-api . All observability artifacts, runbooks, and config files are already packaged inside the container. No network access is allowed. Requirements Diagnose the incident using logs, metrics, and traces to identify the root cause and blast radius. Execute mitigation per the runbook ( /app/runbooks/payments-canary-rollback.md ): Update /app/config/service_state.json as specified in the runbook Only modify payments-api - do not change any other services Only change fields specified in the runbook - preserve all other fields exactly Create any required configuration files specified in the runbook Produce output files documenting your diagnosis, actions, and validation. Inputs /app/data/logs/app.log — application logs /app/data/metrics/before.json — pre-incident metrics /app/data/metrics/after.json — post-mitigation metrics /app/data/traces/trace_sample.json — distributed traces /app/data/deployments/ — deployment manifests /app/runbooks/payments-canary-rollback.md — mitigation runbook /app/config/service_state.json — service configuration Outputs All outputs must be created exactly as specified below. /app/outputs/incident_summary.md Markdown document with these sections: Overview, Root Cause, Blast Radius, Evidence, Actions Taken, Validation, Next Steps. Cite full artifact paths used in analysis Include pod, DB host, and failure mode in root cause Include the primary trace ID (the first payments-api canary error trace with DB connection issues, by timestamp), rollback_token, and correlation ID List affected transaction IDs and count Include the incident time window (start and end timestamps of payments-api canary DB errors) Confirm unaffected services explicitly - check all other services in service_state.json (excluding payments-api) and confirm they are unaffected Show before/after metrics with deltas Include an "SLO Budget" section containing the words "SLO" and "budget", plus the calculated remaining budget value (monthly_budget_pct minus consumed_pct, rounded to 2 decimal places) Use "disable" or "disabled" when describing the canary action /app/outputs/actions.json JSON array with exactly FIVE action objects: {"action": "disable_canary", "target": "payments-api", "status": "...", "details": "...", "evidence": "...", "rollback_token": "...", "correlation_id": "..."} evidence must reference app.log and include the primary trace ID correlation_id must be extracted from the relevant log entries {"action": "create_alert_suppression", "target": "payments-api", "status": "...", "details": "..."} {"action": "create_followup_ticket", "target": "payments-api", "status": "...", "details": "...", "priority": "...", "assigned_team": "..."} priority and assigned_team must be determined per the runbook {"action": "notify_stakeholders", "target": "payments-api", "status": "...", "details": "...", "channel": "...", "escalation_level": "..."} channel and escalation_level must be determined per the runbook {"action": "update_deployment_status", "target": "payments-api", "status": "...", "details": "...", "previous_status": "...", "new_status": "...", "canary_version": "..."} previous_status : the status from the deployment manifest before rollback new_status : "rolled_back" canary_version : the version from the deployment manifest /app/outputs/postcheck.json JSON object with fields: error_rate_pct , p99_latency_ms , validation_passed , notes , affected_txn_count , impact_pct , incident_window , slo_budget , mttr_seconds , error_velocity error_rate_pct and p99_latency_ms : values from after-metrics for payments-api validation_passed : boolean based on runbook thresholds notes : must contain "after.json", the literal metric names ("error_rate_pct", "p99_latency_ms"), and their threshold values ("5" and "400") affected_txn_count : count of unique transactions from payments-api canary DB error log entries impact_pct : (affected_txn_count / total_requests_from_before_metrics) * 100, rounded to 2 decimal places incident_window : object with start and end fields containing the first and last payments-api canary DB error timestamps from logs slo_budget : object with these exact fields: monthly_budget_pct : 0.1 (representing Chat With the Terminal-Bench Team | Snorkel AI Capabilities Capabilities How it works Research-led data and environment development for the frontier's hardest problems Learn more Data development Overview Expert-curated datasets for frontier AI Use cases See how our data improves frontier models Specialized agents Overview Custom AI systems built to unlock ROI fast Customer stories Real-world results from enterprise deployments Research Research Research Research hub Our latest papers and data-centric AI findings Leaderboards Compare model performance across benchmarks Open Benchmarks Grants Funding for open-source AI research Featured BENCHMARK Agentic Coding benchmark A benchmark for evaluating AI models on complex, real-world coding tasks that require multi-step reasoning, tool use, and autonomous problem-solving. Resources Resources Resources Resource library Guides, papers, and tools for data-centric AI Events Upcoming talks, workshops, and conferences Reading Group AI discussions for researchers and practitioners Blog News, updates, and perspectives from our team featured BLOG Terminal-Bench 2.0 Developed by Stanford and Laude Institute with contributions from Snorkel AI, it’s a major leap forward in evaluating AI coding agents. Company Company Company About Our mission, story, and values Careers Open roles and life at our company Press Media resources and announcements Partners Organizations we work with Security How we keep data safe Contact us Get in touch with our team Join our expert community Get paid to shape safer, smarter AI Learn more Get started Get started See all articles Research A chat with the Terminal-Bench team Kobie Crawford , Fred Sala Published: November 19, 2025 https://www.youtube.com/watch?v=Tk3-VKzPJOM Snorkel Chief Scientist Fred Sala and Kobie Crawford had a chance to catch up with Mike Merrill and Alex Shaw on November 5, just before they released Terminal-Bench 2.0 and the Harbor framework . They gave us a peek into the thought process and design decisions underpinning the benchmark, and shared insights about how Harbor supports a wide range of use cases beyond benchmarking. Jump to: — Kobie Crawford: I’m really glad to be here, really excited to have Alex and Mike here, from Laude Institute and Stanford. Guys, do you want to do just the quick intros, for yourselves, and let us know a little bit about where you come from for this call? Alex Shaw: I’ll introduce myself first. Yeah, my name’s Alex, I work at Laude Institute. I’ve been at Laude for about a year. I was at Google for a while before that. Working on, like, ad recommendations and, and conversion modeling stuff, and then, joined Laude Institute right around when it was being founded, and I’m happy to talk more about what exactly Laude Institute is, if you guys are interested later. But, yeah, we work on, a couple different directions, but we do kind of what we call impact research. Which is… which we define as research that doesn’t stop at the paper, so we love publishing papers, but we also love producing actual artifacts and products and projects that real users can use in order to adopt the inventions that made it into the research paper. So, yeah. Mike Merrill: And I’m Mike Merrill, I’m a postdoc at Stanford, excited about all things agents and evals and autonomy, and yeah, really excited to chat with you guys today. Kobie Crawford: Right on. Thanks so much. Fred, do you want to do a quick intro for yourself as well? Fred Sala: Awesome. Yeah, hey folks, I’m Fred Sala. I’m a professor at the University of Wisconsin-Madison, and I’m also the chief scientist at Snorkel AI. I’m also super interested in all data-centric and agent-focused things. Kobie Crawford: Super cool. Thank you, thank you. So, Kobie Crawford here, I’m Developer Advocate at Snorkel.  Rationale for creating Terminal-Bench Kobie Crawford: Let me throw this first question out: how did you come up with the initial idea for Terminal Bench? And, you know, what aspects of the existing benchmarks did you feel were missing? Mike Merrill: So, I think there was one insight that started Terminal Bench, which is that if you go back a year or so, the primary way that people were interested in getting language models to control computers was through operating computers’ GUIs. So, you know, you had these systems which took visual reasoning models, and they would move a cursor around, and it would click on drop-downs, and it would do things like configure EC2 instances, or go book you a flight, or something like this. And this made for very flashy demos, but if you ever actually tried using one of these systems, you found that they fell apart pretty quickly. Like, they weren’t able to maintain a long chain of thought, they weren’t very good at, like, precisely clicking on GUI elements, and you know, I think about a year ago, we were thinking, like, what’s a better way of doing this? And one of the insights that led us towards building Terminal Bench was that, like, even if reasoning visually is not so good, language models are really good at writing code, and they’ve been good at writing code for a number of years. And so, why don’t we write code to control a computer instead of relying on this dragging and dropping and clicking through GUIs that might not even be well-designed in the first place? And I think that’s where, like, the core heart of Terminal Bench came from, was the terminal is this interface that allows us to control computers through text, which is the modality that still works best for language models. And I think Ludwig and I have been talking about promoting this as a way of doing agentic use. A benchmark seems like a really good place to get started, and through some previous connections that Ludwig and I had, we were able to meet Andy, who is the… one of the founders of Laude Institute, who introduced us to Alex, who, like, really helped get the benchmark off the ground, and, has been a great collaborator ever since. I don’t know if you’d add anything to that.  “…the terminal is this interface that allows us to control computers through text, which is the modality that still works best for language models.” — Mike Merrill Alex Shaw: Yeah, I guess I would just add from Laude Institute’s perspective, or from our perspective, when I first joined Laude, we spent 3 months building this thing called the K Prize, I guess technically the Konwinski Prize, which is this $1 million prize around SWE-Bench, where we built, like, a continual… continually updating version of SWE-Bench that was contamination-free, that people could compete on for a $1 million prize. And SWE-Bench specifically is… you take GitHub issues that have been closed by some PR that added unit tests, and you give the language model a GitHub issue, and then you ask it to solve the problem, and then you run it against the unit tests that were contributed as part of the original pull request that solved it. And we just spent a lot of time looking at that benchmark and thinking about how they implemented it, and it became clear that there was maybe a broader abstraction that didn’t constrain tasks to be, GitHub-specific with open source Python repos and pull requests that had to close it, but what if you actually just had, like, a generic instruction, container, and some sort of test executable, like a test script? Then, in theory, you should be able to frame almost any task on a computer, which a container is essentially just, like, a simulated computer. You should be able to frame any task on a computer in this task format. And then that became the task format for Terminal Bench, and the terminal became the tool by which these agents solved these tasks. And, yeah, that was… that was right before the release of all these CLI agents, and then they started to come out. I think Snorkel AI Security Portal | Powered by SafeBase Security Portal Start your security review View & download sensitive information Ask for information Get access ​ ​ Overview This Trust Center provides resources demonstrating Snorkel AI's commitment to data security, privacy, and compliance. In our trust portal, you can access our comprehensive compliance documentation, find answers to frequently asked questions related to security and privacy, and explore our robust security practices. We believe in exceeding compliance and regulatory requirement. This portal is designed to provide the information and assurance you need to feel confident in our ability to protect your data. Compliance AWS Qualified Software HIPAA NIST CSF SOC 2 Documents All Public Private Get access Bulk download DOCUMENTS SOC2 Type II Letter of Understanding REPORTS Security Whitepaper REPORTS SOC 2 Report COMPLIANCE AWS Qualified Software APP SECURITY Vulnerability & Patch Management LEGAL Data Processing Agreement CORPORATE SECURITY Penetration Testing Product Security Audit Logging Data Security Integrations View more Reports HIPAA Report Pentest Report Security Whitepaper View more Data Security Data Backups Encryption-at-rest Encryption-in-transit View more App Security Code Analysis Credential Management Secure Development Training View more Legal Subprocessors Cyber Insurance Data Processing Agreement View more Data Privacy Data Privacy Officer Employee Privacy Training Access Control Data Access Logging Password Security Infrastructure Amazon Web Services BC/DR Infrastructure Security Endpoint Security Disk Encryption Endpoint Detection & Response Mobile Device Management View more Corporate Security Asset Management Practices Email Protection Employee Training View more If you need help using this Security Portal, please contact us. Contact support Built on Research | Snorkel AI Capabilities Capabilities How it works Research-led data and environment development for the frontier's hardest problems Learn more Data development Overview Expert-curated datasets for frontier AI Use cases See how our data improves frontier models Specialized agents Overview Custom AI systems built to unlock ROI fast Customer stories Real-world results from enterprise deployments Research Research Research Research hub Our latest papers and data-centric AI findings Leaderboards Compare model performance across benchmarks Open Benchmarks Grants Funding for open-source AI research Featured BENCHMARK Agentic Coding benchmark A benchmark for evaluating AI models on complex, real-world coding tasks that require multi-step reasoning, tool use, and autonomous problem-solving. Resources Resources Resources Resource library Guides, papers, and tools for data-centric AI Events Upcoming talks, workshops, and conferences Reading Group AI discussions for researchers and practitioners Blog News, updates, and perspectives from our team featured BLOG Terminal-Bench 2.0 Developed by Stanford and Laude Institute with contributions from Snorkel AI, it’s a major leap forward in evaluating AI coding agents. Company Company Company About Our mission, story, and values Careers Open roles and life at our company Press Media resources and announcements Partners Organizations we work with Security How we keep data safe Contact us Get in touch with our team Join our expert community Get paid to shape safer, smarter AI Learn more Get started Get started NEW Continual Learning Bench by Berkeley & Snorkel our research lab From cutting-edge research to enterprise and frontier impact Our research team advances the science of data-centric AI in partnership with leading enterprises and frontier labs. We translate these breakthroughs into production, powering the next generation of AI systems across industries, research, and government. Browse our research library Deep research roots Born out of the Stanford AI lab in 2019 and in collaboration with leading research institutions, Snorkel-affiliated researchers have published more than 170 peer-reviewed research papers on weak supervision, AI data development techniques, foundation models, and more—with special recognition at events such as NeurlPS, ICML, and ICLR. Our researchers are closely affiliated with academic institutions including Stanford University, University of Washington, Brown University, and the University of Wisconsin-Madison. Featured benchmarks Exclusive to Snorkel, these benchmarks are meticulously designed and validated by subject matter experts to probe frontier AI models on demanding, specialized tasks. These are just a few of our featured benchmarks—new ones are added regularly, so check back often to see the latest from our research team. Agentic Coding A benchmark for evaluating AI models on complex, real-world coding tasks that require multi-step reasoning, tool use, and autonomous problem-solving. View all results Finance Reasoning A benchmark co-created with Snorkel's financial expert network, to test agents on financial reasoning questions, through tool-calling and planning. View all results View all benchmarks Leaderboards Challenging benchmarks for models and agents Snorkel benchmarks are built with human expertise to test models on realistic tasks ranging from coding and financial analysis to healthcare and more. For example, our SnorkelUnderwrite benchmark includes multi-turn agentic tasks germane to the insurance industry. See the latest score on our Agentic coding leaderboard. Learn more Rubrics Aligning human expertise and automated evaluation We investigate how to scalably develop rubrics that are both comprehensive of the desired agentic capabilities and reliably assessed by both human experts and AI judges. Learn more about our findings. Learn more RL ENvironments Environments give agents a fully realized simulation As tool-calling and more open-ended application requirements break simple test frameworks, agent validation must be done with techniques that reproduce real-world variability. For example, our contributions to Terminal-Bench (tbench.ai) include containerized simulation environments. Read more in our blog post. Learn more Browse blog posts and 100+ peer reviewed academic papers All Types Blog Research Papers All Topics Benchmarking & Evaluation Benchmarks Computer Vision Data Labeling & Annotation Data-centric AI Foundation Models Generative AI LLM Specialization LLMs Natural Language Processing Open Source RAG Synthetic Data Sort by Newest Sort by Oldest Sort by Title (A-Z) Sort by Title (Z-A) Blog NEW Why coding agents need better data, evals, and environments Read more Learn More about Why coding agents need better data, evals, and environments Blog Understanding Olmix: A Framework for Data Mixing Throughout Language Model Development Read more Learn More about Understanding Olmix: A Framework for Data Mixing Throughout Language Model Development Research Paper Learning from Less: Measuring the Effectiveness of RLVR in Low Data and Compute Regimes Read more Learn More about Learning from Less: Measuring the Effectiveness of RLVR in Low Data and Compute Regimes Blog Benchmarks should shape the frontier, not just measure it Read more Learn More about Benchmarks should shape the frontier, not just measure it Research Paper RIFT: A Rubric Failure Mode Taxonomy and Automated Diagnostics Read more Learn More about RIFT: A Rubric Failure Mode Taxonomy and Automated Diagnostics Blog Benchtalks #1: Alex Shaw (Terminal-Bench, Harbor) – Building the Benchmark Factory Read more Learn More about Benchtalks #1: Alex Shaw (Terminal-Bench, Harbor) – Building the Benchmark Factory Blog Building FinQA: An Open RL Environment for Financial Reasoning Agents Read more Learn More about Building FinQA: An Open RL Environment for Financial Reasoning Agents Blog How Tool Discipline Let a 4B Model Outsmart a 235B Giant on Financial Tasks Read more Learn More about How Tool Discipline Let a 4B Model Outsmart a 235B Giant on Financial Tasks Blog Coding agents don’t need to be perfect, they need to recover Read more Learn More about Coding agents don’t need to be perfect, they need to recover Blog Closing the Evaluation Gap in Agentic AI Read more Learn More about Closing the Evaluation Gap in Agentic AI Research Paper Benchmarking Agents in Insurance Underwriting Environments Read more Learn More about Benchmarking Agents in Insurance Underwriting Environments Research Paper Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces Read more Learn More about Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces 1 2 3 ... 27 Backed by a $3M commitment, the Open Benchmarks Grants program — in partnership with Hugging Face, Prime Intellect, Together AI and Factory HQ and Harbor— funds open-source datasets, benchmarks, and evaluation artifacts that shape how frontier AI systems are built and evaluated. Applications are rolling — starting March 1st. Apply for a grant Learn more In collaboration with: Coming Fall 2026 Frontier Data Summit is on the horizon Get updates on grants, deadlines, and the Frontier Data Summit. Distinguished Affiliates Stephen Bach Eliot Horowitz Assistant Professor, Computer Science Department, Brown University Jason Fries Assistant Professor of Biomedical Data Science and of Medicine at Stanford University Changho Shin Postdoctoral Scholar at Princeton University Jared Dunnmon Co-Founder and Chief Scientist of a Maritime Logistics Startup View all researchers Join the Snorkel research team Join our team of leading researchers and help shape the future of AI. View all careers Role Department Location Research Engineer – Training Infra Research Redwood City, CA (Hybrid); San Francisco, CA (Hybrid); United States (Remote) Research Scientist Research Redwood City, CA (Hybrid); San Expert Data Development for Frontier AI | Snorkel AI Capabilities Capabilities How it works Research-led data and environment development for the frontier's hardest problems Learn more Data development Overview Expert-curated datasets for frontier AI Use cases See how our data improves frontier models Specialized agents Overview Custom AI systems built to unlock ROI fast Customer stories Real-world results from enterprise deployments Research Research Research Research hub Our latest papers and data-centric AI findings Leaderboards Compare model performance across benchmarks Open Benchmarks Grants Funding for open-source AI research Featured BENCHMARK Agentic Coding benchmark A benchmark for evaluating AI models on complex, real-world coding tasks that require multi-step reasoning, tool use, and autonomous problem-solving. Resources Resources Resources Resource library Guides, papers, and tools for data-centric AI Events Upcoming talks, workshops, and conferences Reading Group AI discussions for researchers and practitioners Blog News, updates, and perspectives from our team featured BLOG Terminal-Bench 2.0 Developed by Stanford and Laude Institute with contributions from Snorkel AI, it’s a major leap forward in evaluating AI coding agents. Company Company Company About Our mission, story, and values Careers Open roles and life at our company Press Media resources and announcements Partners Organizations we work with Security How we keep data safe Contact us Get in touch with our team Join our expert community Get paid to shape safer, smarter AI Learn more Get started Get started New Continual Learning Bench by Berkeley & Snorkel We build the data that pushes the frontier Snorkel helps frontier labs and AI teams develop specialized training data and environments that set their models and agents apart. Request dataset samples Explore research Proud to partner with top frontier AI and research teams Frontier models break at the edges. We build for that. Most data pipelines are built for volume, not difficulty. Frontier models fail on distributional gaps in specialized domains, benchmark blind spots, and tasks where correctness is hard to define. Snorkel is built specifically for these problems. Founded out of Stanford AI Lab, we've been shaping and benchmarking frontier AI for nearly a decade. See our research Open Benchmarks Grants What we've found New RL Research RLVR in Low Data & Compute Regimes Better data beats more compute — measured across low-resource settings. Evaluation Research RIFT: Rubric Failure Mode Taxonomy A diagnostic framework for when AI evaluation rubrics break down. Domain Agents Benchmarking Agents in Insurance Underwriting Environment-first benchmarking for agents in a genuinely high-stakes domain. Collaborations New Research Collaboration Continual Learning Bench Expert-validated tasks for agents that learn across task sequences, not isolated prompts. Code Quality SlopCodeBench Generic code evals miss sloppy code. This measures what they ignore. Agentic Coding Terminal-Bench 2.0 Real terminal tasks — exposing where today's coding agents fail. Legal AI Harvey BigLaw Bench Expert data for the hardest agentic legal research benchmark. Built with Harvey AI. The Frontier AI Data Lab Data development for the frontier Snorkel partners with frontier AI teams to build research-grade datasets, evaluation systems, and runnable environments where generic coverage runs out. Explore capabilities Snorkel Data Series Curriculum-structured datasets for the task areas frontier models are pushing hardest, with rubrics, reviewer guidance, difficulty tiers, and eval slices built in. Custom data development When off-the-shelf coverage runs out, we build bespoke datasets, evals, and benchmark expansions for the exact failure surface you need to close. Specialized agents Custom agents built on specialized data and evaluated in real workflows, with pass/fail criteria tied to the performance standards that moves ROI. Data Expert Demonstrations & Reasoning Human solution traces Reasoning traces SME Q&A rationales Workflow demos and decision workflows Tool-use demos Preference Labels & Rankings Patch/draft/report quality ranking Trajectory QA Risk/safety/style calibration Helpful/harmless ranking Grounding & style Rubrics & Verifiable Outcomes Unit tests / compile Deterministic graders Citation correctness Numerical consistency/scorable math/science Long-horizon tasks Environments Standard & Custom Environments Repo + CLI tools Browser/GUI harness Multi-step/stateful workflows Simulated environments Your tools, codebase, corpus, data & permissions DATA DEVELOPMENT Good data is a set of design choices Request dataset samples Most data quality problems are design problems. Ambiguous task definitions produce inconsistent labels. Uncalibrated reviewers introduce systematic bias. Missing provenance makes failure analysis guesswork. Snorkel's proprietary process is built around the decisions that determine whether training data actually drives model improvement: Well-specified expert-level tasks Tasks scoped to actual model failure modes, with target distributions, acceptance criteria, and verifier definitions. Each spec is a research artifact, written before data work begins. Calibrated expert review Calibration is a research workflow, not a staffing layer. Reviewers are trained against gold sets authored by Snorkel researchers, scored for agreement and bias, and re-calibrated per task. Rubrics and programmatic checks Fine-tuned evaluator models, not just human spot-checks. Snorkel researchers co-design rubrics with domain experts, then distill them into programmatic graders and specialized evaluator models. Adjudication and provenance Author, multi-reviewer, and final-adjudicator pipeline with full audit trails. Every label is traceable to who decided what, when, and on what evidence. Edge-case coverage Distributional precision, not volume. Expert-authored seeds expand into controlled coverage across difficulty bands, edge cases, and failure modes through templated generation. Benchmarks and evals Eval harnesses built alongside the data: task-specific rubrics, deterministic graders, and runnable environments that produce reproducible scores across model versions. Custom AGENTS Specialized agents grounded in expert data Explore agent solutions The same data development system we use to improve frontier models powers our specialized agents. That means agents evaluated against task-specific rubrics and programmatic checks – not generic benchmarks – and refined through the same adjudication and provenance practices used in production model development. Built for specialized workflows and high-consequence decisions, not generic copilots Evaluation on environment-grounded tasks with programmatic pass/fail criteria Same rigor used to train frontier-class models, applied to your enterprise deployment PUBLISHED RESEARCH Research that shapes the work Every dataset, benchmark, and environment we create is the output of active research co-developed and peer-reviewed with leading academic teams and frontier labs. Explore research Benchmark Agentic coding benchmark: Evaluating AI models on complex, real-world coding tasks Blog SlopCodeBench: Measuring Code Erosion as Agents Iterate Blog Terminal-Bench 2.0: Raising the bar for AI agent evaluation Research Paper Learning from Less: Measuring the Effectiveness of RLVR in Low Data and Compute Regimes Research Paper RIFT: A Rubric Failure Mode Taxonomy and Automated Diagnostics of For models that need to be right. Not just good enough. Request dataset samples Explore research Get started How do you want to work with Snorkel? AI data development services Accelerate the development of frontier AI models with expert-curated, enterprise-grade data. Learn how Snorkel’s Data-as-a-Service helps teams label, refine, and evaluate high-quality, domain-specific datasets for your projects. Talk to a da Partnerships | Snorkel AI Capabilities Capabilities How it works Research-led data and environment development for the frontier's hardest problems Learn more Data development Overview Expert-curated datasets for frontier AI Use cases See how our data improves frontier models Specialized agents Overview Custom AI systems built to unlock ROI fast Customer stories Real-world results from enterprise deployments Research Research Research Research hub Our latest papers and data-centric AI findings Leaderboards Compare model performance across benchmarks Open Benchmarks Grants Funding for open-source AI research Featured BENCHMARK Agentic Coding benchmark A benchmark for evaluating AI models on complex, real-world coding tasks that require multi-step reasoning, tool use, and autonomous problem-solving. Resources Resources Resources Resource library Guides, papers, and tools for data-centric AI Events Upcoming talks, workshops, and conferences Reading Group AI discussions for researchers and practitioners Blog News, updates, and perspectives from our team featured BLOG Terminal-Bench 2.0 Developed by Stanford and Laude Institute with contributions from Snorkel AI, it’s a major leap forward in evaluating AI coding agents. Company Company Company About Our mission, story, and values Careers Open roles and life at our company Press Media resources and announcements Partners Organizations we work with Security How we keep data safe Contact us Get in touch with our team Join our expert community Get paid to shape safer, smarter AI Learn more Get started Get started Partnerships Advancing data-centric AI with partners Together with a select group of partners, we empower our joint customers to transform data into new insights, innovations, and business value. We are building an ecosystem of cloud providers, technology innovators, and world-class consulting and service providers for data-centric AI. Our cloud and technology partners expand our platform’s interoperability within the modern ML stack, offering our Fortune 500 and government customers with seamless and security connections. Featured partnerships Build and deploy powerful, customized AI applications that solve your most critical problems with Snorkel AI and Google Cloud. Speed the development of intelligent, AI-driven applications customized with your own data with Snorkel AI and Microsoft. Build better AI with a data-centric approach. Efficiently transform unstructured data in your Databricks Data Intelligence Platform into custom ML and GenAI applications. Turn your data into your greatest asset for custom AI development with Snorkel Flow and Amazon Web Services. Ecosystem highlights Aimpoint Digital partners with Snorkel AI Read more Snorkel AI x Hugging Face: unlock foundation models for enterprises Read more Four Inc partners with Snorkel AI as federal aggregator Read more Cohere’s Alammar encourages effective strategy for Generative AI Read more MasterPeace hosts AI platform workshops in conjunction with Snorkel AI Read more NVIDIA’s product manager, Nyla Worker, on leveraging synthetic data to train perception models Read more Accelerating AI development with Snorkel Flow and AWS SageMaker Read more ML model benchmarking on GCP for Snorkel AI Read more Sambanova and Snorkel AI partner to advance foundation models Read more Seldon and Snorkel AI partner to advance data-centric AI Read more Snorkel AI partners with Snowflake to bring data-centric AI to the Snowflake Data Cloud Read more Snorkel AI and Together AI empower enterprises to build proprietary LLMs Read more Partner with our research lab to build and evaluate systems that perform where it matters most—in the real world. Contact us Get started How do you want to work with Snorkel? AI data development services Accelerate the development of frontier AI models with expert-curated, enterprise-grade data. Learn how Snorkel’s Data-as-a-Service helps teams label, refine, and evaluate high-quality, domain-specific datasets for your projects. Talk to a data researcher Build specialized agents Explore how Snorkel can collaborate with your product and development teams to build and deploy custom AI and agentic systems. Solutions undergo rigorous testing based on your business criteria to ensure positive ROI, faster. Talk to a strategist Become an expert contributor Join the Snorkel Expert Contributor community and help shape the future of AI with your expertise. Contribute to groundbreaking projects, share domain-specific insights, and get rewarded for your impact. Join now Capabilities How it works Data development Specialized agents Use cases Customer stories Research Research hub Leaderboards Open Benchmarks Grants Resources Resource library Events Reading Group Blog Company About Careers Press Partners Security Contact us Contact Get started Join expert network Compliance Copyright © 2026 Snorkel AI, Inc. All rights reserved. Terms of Use Privacy Cookie Policy Find Careers in AI | Snorkel AI Careers Capabilities Capabilities How it works Research-led data and environment development for the frontier's hardest problems Learn more Data development Overview Expert-curated datasets for frontier AI Use cases See how our data improves frontier models Specialized agents Overview Custom AI systems built to unlock ROI fast Customer stories Real-world results from enterprise deployments Research Research Research Research hub Our latest papers and data-centric AI findings Leaderboards Compare model performance across benchmarks Open Benchmarks Grants Funding for open-source AI research Featured BENCHMARK Agentic Coding benchmark A benchmark for evaluating AI models on complex, real-world coding tasks that require multi-step reasoning, tool use, and autonomous problem-solving. Resources Resources Resources Resource library Guides, papers, and tools for data-centric AI Events Upcoming talks, workshops, and conferences Reading Group AI discussions for researchers and practitioners Blog News, updates, and perspectives from our team featured BLOG Terminal-Bench 2.0 Developed by Stanford and Laude Institute with contributions from Snorkel AI, it’s a major leap forward in evaluating AI coding agents. Company Company Company About Our mission, story, and values Careers Open roles and life at our company Press Media resources and announcements Partners Organizations we work with Security How we keep data safe Contact us Get in touch with our team Join our expert community Get paid to shape safer, smarter AI Learn more Get started Get started Careers at Snorkel Build the future of AI with us. See open positions Why Snorkel Help us redefine how AI is built. Our mission: Safely define and advance the frontier of AI with expert data development. We work with some of the world’s largest organizations to empower data scientists and subject matter experts to build specialized AI fueled by their data and expertise, fast. Current job openings Reset filters Team Engineering Finance IT Marketing Operations - Data-as-a-Service People Product Professional Services Research Sales Location Mexico New York City Redwood City San Francisco United States Washington Engineering Back to top --> Infrastructure Security Engineer Redwood City, CA (Hybrid); San Francisco, CA (Hybrid) Senior Software Engineer - AI / ML Redwood City, CA (Hybrid); San Francisco, CA (Hybrid) Senior Software Engineer - Developer Experience Redwood City, CA (Hybrid); San Francisco, CA (Hybrid) Senior Software Engineer - Expert Contributor Lifecycle Redwood City, CA (Hybrid); San Francisco, CA (Hybrid) Senior Software Engineer - Expert Data Collection Platform Redwood City, CA (Hybrid); San Francisco, CA (Hybrid) Software Engineer - Core Services Redwood City, CA (Hybrid); San Francisco, CA (Hybrid) Software Engineer in Test - Infrastructure Redwood City, CA (Hybrid); San Francisco, CA (Hybrid) Software Engineer — Summer Intern Redwood City, CA (Hybrid) Staff Software Engineer - Core Services Redwood City, CA (Hybrid); San Francisco, CA (Hybrid) Staff Software Engineer - Expert Contributor Lifecycle Redwood City, CA (Hybrid); San Francisco, CA (Hybrid) Finance Back to top --> Accounts Receivable Specialist New York City, NY (Hybrid); Redwood City, CA (Hybrid); San Francisco, CA (Hybrid); United States (Remote) Deal Desk Manager Redwood City, CA (Hybrid); San Francisco, CA (Hybrid) Manager, Revenue Accounting New York City, NY (Hybrid); Redwood City, CA (Hybrid); San Francisco, CA (Hybrid); United States (Remote) Revenue Operations Lead Redwood City, CA (Hybrid); San Francisco, CA (Hybrid) Senior Business Operations Manager New York City, NY (Hybrid); Redwood City, CA (Hybrid); San Francisco, CA (Hybrid) Senior Business Systems Analyst Redwood City, CA (Hybrid); San Francisco, CA (Hybrid); United States (Remote) Senior GTM Engineer Redwood City, CA (Hybrid); San Francisco, CA (Hybrid); United States (Remote) IT Back to top --> IT Support Engineer New York City, NY (Hybrid); Redwood City, CA (Hybrid); San Francisco, CA (Hybrid) Marketing Back to top --> AI Advocate, Open-Source & Research Redwood City, CA (Hybrid); San Francisco, CA (Hybrid) Lead Field Marketing & Events Manager San Francisco, CA (Hybrid) Principal Growth Marketing Manager Redwood City, CA (Hybrid); San Francisco, CA (Hybrid) Operations - Data-as-a-Service Back to top --> Delivery Operations Manager Mexico (Remote); New York City, NY (Hybrid); Redwood City, CA (Hybrid); San Francisco, CA (Hybrid); United States (Remote) Engagement Manager - Data as a Service New York City, NY (Hybrid); Redwood City, CA (Hybrid); San Francisco, CA (Hybrid); United States (Remote) Forward Deployed Engineer - Data as a Service New York City, NY (Hybrid); Redwood City, CA (Hybrid); San Francisco, CA (Hybrid) Head of Engagement Management - Data as a Service New York City, NY (Hybrid); Redwood City, CA (Hybrid); San Francisco, CA (Hybrid); United States (Remote) Head of Forward Deployed Engineering New York City, NY (Hybrid); Redwood City, CA (Hybrid); San Francisco, CA (Hybrid) Manager, Forward Deployed Engineering New York City, NY (Hybrid); Redwood City, CA (Hybrid); San Francisco, CA (Hybrid) Manager, Technical Delivery New York City, NY (Hybrid); Redwood City, CA (Hybrid); San Francisco, CA (Hybrid); United States (Remote) Operations Manager - Enablement GTM New York City, NY (Hybrid); Redwood City, CA (Hybrid); San Francisco, CA (Hybrid); United States (Remote) Senior Analytics Manager, Delivery - Data as a Service New York City, NY (Hybrid); Redwood City, CA (Hybrid); San Francisco, CA (Hybrid) Senior Operations Manager - Growth United States (Remote) Senior Strategy & Operations Manager - Delivery New York City, NY (Hybrid); Redwood City, CA (Hybrid); San Francisco, CA (Hybrid) Strategic AI Lead New York City, NY (Hybrid); Redwood City, CA (Hybrid); San Francisco, CA (Hybrid); United States (Remote) Technical Delivery Manager New York City, NY (Hybrid); Redwood City, CA (Hybrid); San Francisco, CA (Hybrid); United States (Remote) People Back to top --> Recruiting Coordinator United States (Remote) Product Back to top --> Senior Product Manager - Data & Quality San Francisco, CA (Hybrid) Senior Product Manager - Platform Redwood City, CA (Hybrid); San Francisco, CA (Hybrid) Staff Product Designer Redwood City, CA (Hybrid); San Francisco, CA (Hybrid); United States (Remote) Staff Product Manager - Marketplace Redwood City, CA (Hybrid); San Francisco, CA (Hybrid) Professional Services Back to top --> Applied AI Engineer - AI Solutions New York City, NY (Hybrid); Redwood City, CA (Hybrid); San Francisco, CA (Hybrid) Applied AI Engineer - Federal (TS Required) United States (Remote); Washington, D.C. (Remote) Staff Applied AI Engineer - Pre-Sales New York City, NY (Hybrid); Redwood City, CA (Hybrid); San Francisco, CA (Hybrid); United States (Remote) Research Back to top --> Research Engineer – Training Infra Redwood City, CA (Hybrid); San Francisco, CA (Hybrid); United States (Remote) Research Scientist Redwood City, CA (Hybrid); San Francisco, CA (Hybrid); United States (Remote) Research Scientist - RL Training Redwood City, CA (Hybrid); San Francisco, CA (Hybrid); United States (Remote) Senior Manager - Research San Francisco, CA (Hybrid) Senior/Staff Research Scientist - Frontier Benchmarks Redwood City, CA (Hybrid); San Francisco, CA (Hybrid); United States (Remote) Staff/Principal Research Scientist Redwood City, CA (Hybrid); San Francisco, CA (Hybrid); United States (Remote) Sales Back to top --> Account Executive- Data as a Service New York City, NY (Hybrid); Redwood City, CA (Hybrid); San Francisco, CA (Hybrid) Director, Forward Deployed Researcher - Data as a Service New York City, NY (Hybrid); Redwood City, CA (Hybrid); San Francisco, CA (Hybrid) Enteprise Account Executive - Federal Washington, DC Forward Deployed Researcher - Data as a Service New York City, NY (Hybrid); Redwood City, CA (Hybrid); San Fran About us | Our mission, founders, and more! | Snorkel AI Capabilities Capabilities How it works Research-led data and environment development for the frontier's hardest problems Learn more Data development Overview Expert-curated datasets for frontier AI Use cases See how our data improves frontier models Specialized agents Overview Custom AI systems built to unlock ROI fast Customer stories Real-world results from enterprise deployments Research Research Research Research hub Our latest papers and data-centric AI findings Leaderboards Compare model performance across benchmarks Open Benchmarks Grants Funding for open-source AI research Featured BENCHMARK Agentic Coding benchmark A benchmark for evaluating AI models on complex, real-world coding tasks that require multi-step reasoning, tool use, and autonomous problem-solving. Resources Resources Resources Resource library Guides, papers, and tools for data-centric AI Events Upcoming talks, workshops, and conferences Reading Group AI discussions for researchers and practitioners Blog News, updates, and perspectives from our team featured BLOG Terminal-Bench 2.0 Developed by Stanford and Laude Institute with contributions from Snorkel AI, it’s a major leap forward in evaluating AI coding agents. Company Company Company About Our mission, story, and values Careers Open roles and life at our company Press Media resources and announcements Partners Organizations we work with Security How we keep data safe Contact us Get in touch with our team Join our expert community Get paid to shape safer, smarter AI Learn more Get started Get started About us In the world of enterprise AI, specialization is key. While big generalist models are gaining mass-market attention, we believe an enterprise's data is the key to AI's true potential. That’s why we’ve pioneered the shift from manual data labeling to programmatic data development for AI. This approach empowers enterprises to use their proprietary data to build production-quality, specialized AI models, faster and cheaper. Snorkel in the news Press Snorkel AI Raises $100 Million To Build Better Evaluators For AI Models May 29, 2025 Press The most innovative companies in artificial intelligence for 2025 Mar 18, 2025 Press For AI Giants, Smaller is Sometimes Better Jul 6, 2024 Proud to partner with top frontier AI and research teams World-leading enterprises work with Snorkel to remove AI roadblocks and unlock new possibilities and better performance. Systems built with Snorkel’s technology process billions (and trillions) of queries, records, images, and more at industry titans across the private and public sectors. Rooted in research Revolutionizing AI Snorkel AI’s founders spent half a decade researching programmatic labeling, weak supervision, and other techniques to break through one of the biggest bottlenecks in AI: the lack of labeled training data. Our research team has continued this work, resulting in 170+ peer-reviewed publications. Snorkel has developed and deployed its technology with Google, Apple, DARPA, Stanford Medicine, and more. Learn more about our founding story on Forbes Our Mission Make AI data development programmatic , like any other type of software development. Our investors Building a world-class team At Snorkel AI, we’re growing a unique team of passionate and creative individuals. We strive to foster a diverse community of colleagues that welcomes, represents, and empowers all. We're as dedicated to communication and mutual support as we are to crafting a world-class product. We cultivate autonomy across the entire team by being open about our goals, wins, and challenges. We get to answers fast, focusing on what works—not what’s fancy. Join us Get started How do you want to work with Snorkel? AI data development services Accelerate the development of frontier AI models with expert-curated, enterprise-grade data. Learn how Snorkel’s Data-as-a-Service helps teams label, refine, and evaluate high-quality, domain-specific datasets for your projects. Talk to a data researcher Build specialized agents Explore how Snorkel can collaborate with your product and development teams to build and deploy custom AI and agentic systems. Solutions undergo rigorous testing based on your business criteria to ensure positive ROI, faster. Talk to a strategist Become an expert contributor Join the Snorkel Expert Contributor community and help shape the future of AI with your expertise. Contribute to groundbreaking projects, share domain-specific insights, and get rewarded for your impact. Join now Capabilities How it works Data development Specialized agents Use cases Customer stories Research Research hub Leaderboards Open Benchmarks Grants Resources Resource library Events Reading Group Blog Company About Careers Press Partners Security Contact us Contact Get started Join expert network Compliance Copyright © 2026 Snorkel AI, Inc. All rights reserved. Terms of Use Privacy Cookie Policy Snorkel AI Reading Group | AI Research & Evaluation Community Capabilities Capabilities How it works Research-led data and environment development for the frontier's hardest problems Learn more Data development Overview Expert-curated datasets for frontier AI Use cases See how our data improves frontier models Specialized agents Overview Custom AI systems built to unlock ROI fast Customer stories Real-world results from enterprise deployments Research Research Research Research hub Our latest papers and data-centric AI findings Leaderboards Compare model performance across benchmarks Open Benchmarks Grants Funding for open-source AI research Featured BENCHMARK Agentic Coding benchmark A benchmark for evaluating AI models on complex, real-world coding tasks that require multi-step reasoning, tool use, and autonomous problem-solving. Resources Resources Resources Resource library Guides, papers, and tools for data-centric AI Events Upcoming talks, workshops, and conferences Reading Group AI discussions for researchers and practitioners Blog News, updates, and perspectives from our team featured BLOG Terminal-Bench 2.0 Developed by Stanford and Laude Institute with contributions from Snorkel AI, it’s a major leap forward in evaluating AI coding agents. Company Company Company About Our mission, story, and values Careers Open roles and life at our company Press Media resources and announcements Partners Organizations we work with Security How we keep data safe Contact us Get in touch with our team Join our expert community Get paid to shape safer, smarter AI Learn more Get started ↳ Get Started Reading group The Snorkel AI Reading Group (SARG) is a recurring forum for researchers and practitioners to explore the latest frontier developments in AI while building meaningful connections within the community. We’ll dive into the most talked-about research in benchmarking and evaluation, pressure-test the ideas, share our POV, and bring in the authors themselves for open discussion.  Get notified about upcoming paper readings // First name (Required) Last Name (Required) Email (Required) City (Required) Δ Paper reading // May 13, 2026, San Francisco, CA Improving LLM Agents with Code World Models & AutoHarness Carter Wendelken Google Deepmind ↳ Register now Past Olmix: A Framework for Data Mixing Throughout LM Development with Mayee Chen, PhD Candidate, Stanford AI Lab; Founding Scientist, Stealth Startup Get started How do you want to work with Snorkel? AI data development services Accelerate the development of frontier AI models with expert-curated, enterprise-grade data. Learn how Snorkel’s Data-as-a-Service helps teams label, refine, and evaluate high-quality, domain-specific datasets for your projects. Talk to a data researcher Build specialized agents Explore how Snorkel can collaborate with your product and development teams to build and deploy custom AI and agentic systems. Solutions undergo rigorous testing based on your business criteria to ensure positive ROI, faster. Talk to a strategist Become an expert contributor Join the Snorkel Expert Contributor community and help shape the future of AI with your expertise. Contribute to groundbreaking projects, share domain-specific insights, and get rewarded for your impact. Join now Capabilities How it works Data development Specialized agents Use cases Customer stories Research Research hub Leaderboards Open Benchmarks Grants Resources Resource library Events Reading Group Blog Company About Careers Press Partners Security Contact us Contact Get started Join expert network Compliance Copyright © 2026 Snorkel AI, Inc. All rights reserved. Terms of Use Privacy Cookie Policy Leaderboard | Snorkel AI Capabilities Capabilities How it works Research-led data and environment development for the frontier's hardest problems Learn more Data development Overview Expert-curated datasets for frontier AI Use cases See how our data improves frontier models Specialized agents Overview Custom AI systems built to unlock ROI fast Customer stories Real-world results from enterprise deployments Research Research Research Research hub Our latest papers and data-centric AI findings Leaderboards Compare model performance across benchmarks Open Benchmarks Grants Funding for open-source AI research Featured BENCHMARK Agentic Coding benchmark A benchmark for evaluating AI models on complex, real-world coding tasks that require multi-step reasoning, tool use, and autonomous problem-solving. Resources Resources Resources Resource library Guides, papers, and tools for data-centric AI Events Upcoming talks, workshops, and conferences Reading Group AI discussions for researchers and practitioners Blog News, updates, and perspectives from our team featured BLOG Terminal-Bench 2.0 Developed by Stanford and Laude Institute with contributions from Snorkel AI, it’s a major leap forward in evaluating AI coding agents. Company Company Company About Our mission, story, and values Careers Open roles and life at our company Press Media resources and announcements Partners Organizations we work with Security How we keep data safe Contact us Get in touch with our team Join our expert community Get paid to shape safer, smarter AI Learn more Get started Get started Snorkel expert data-as-a-service Featured leaderboards Exclusive to Snorkel, these benchmarks are meticulously designed and validated by subject matter experts to probe frontier AI models on demanding, specialized tasks. These are just a few of our featured benchmarks—new ones are added regularly, so check back often to see the latest from our research team. Agentic Coding A benchmark for evaluating AI models on complex, real-world coding tasks that require multi-step reasoning, tool use, and autonomous problem-solving. Claude Opus 4.6 65.2% Claude Opus 4.5 58% Claude Sonnet 4.5 57.6% Gemini 3 Pro Preview 51.6% gpt-5.2 49.4% gpt-5 45.2% Kimi-K2-Thinking 36.8% Devstral 2 33.2% Grok 4.1 Fast 25.2% Qwen 3 Coder 480B 18.8% Mistral Large 3 13.8% Show full rating SnorkelUnderwrite An expert-verified frontier benchmark with multi-turn conversations, focused on agentic reasoning and tool use in commercial underwriting settings. GPT-5.4 91% Claude Opus 4.1 86.3% gpt-5 83.33% Grok 4 83.3% Grok 4 Fast Reasoning 81.33% Grok 3 78% o4 mini 78% Claude Opus 4 77% o3 77% Claude Sonnet 3.7 74.6% Claude Sonnet 4 72.3% gpt-5-mini 71.67% Kimi-K2-Thinking 71.3% GPT-4.1 70.6% Gemini 2.5 Flash 61% Nova Premier 57% Gemini 2.5 Pro 56.3% Nova Pro 52.3% gpt-5-nano 47% Llama 3.3 70B 46.3% Llama 4 Maverick 46.3% Llama 4 Scout 44.3% o3-mini 44.3% Nova Lite 40% Mistral Large 38.3% Codestral 34% Nova Micro 31% gpt-oss-120b 30% Magistral Medium 29.3% Command R+ 25.7% Qwen 3 235B 21.3% Llama 3.1 405B 20% Command R 15.3% Show full rating Finance Reasoning A benchmark co-created with Snorkel's financial expert network, to test agents on financial reasoning questions, through tool-calling and planning. GPT-5.4 52% Grok 4 53.1% Claude Sonnet 3.7 51.89% gpt-5 51% Claude Sonnet 4 49.37% Claude Opus 4 48.1% Gemini 3 Pro 46.84% gpt-5-mini 46.8% o4 mini 45.57% Claude Opus 4.1 45.56% GPT-4.1 44.3% o3 43.04% Grok 3 41.8% Grok 4 Fast Reasoning 40.51% NVIDIA Nemotron Super 49B v1.5 35.443% Kimi-K2-Thinking 35% Gemini 2.5 Pro 34.6% Nova Premier 34.17% Gemini 2.5 Flash 32% gpt-oss-120b 31.6% o3-mini 30.37% gpt-5-nano 26.6% Qwen 3 235B 17.7% Magistral Medium 13.92% Nova Pro 12.65% Mistral Large 10.12% Show full rating SnorkelSequences A procedurally-generated and expert-verified benchmark for evaluating mathematical reasoning and compositional capabilities in LLMs. gpt-5 77.6% gpt-5-mini 77.6% gpt-5-nano 72% GPT-5.4 71.6% o3-mini 71.2% Gemini 2.5 Flash 70.8% Claude Sonnet 4 70.4% Grok 4 Fast Reasoning 70.2% o4 mini 68.8% NVIDIA Nemotron Super 49B v1.5 66.8% Gemini 2.5 Pro 66% Claude Opus 4 65.6% o3 65.2% Grok 4 63.2% Llama 4 Maverick 62% Nova Premier 51.8% Llama 4 Scout 48.4% Claude Sonnet 3.7 47.6% Magistral Medium 47.6% Nvidia nemotron super 49B 44.8% Nova Pro 41.2% Nova Lite 40% Grok 3 39.2% Llama 3.3 70B 38.8% Mistral Large 38.8% Codestral 38.4% GPT-4.1 36.8% Nvidia 70B Instruct 36.4% Kimi-K2-Thinking 36% Llama 3.1 405B 35.2% Nova Micro 33.6% Qwen 3 235B 28% Show full rating SnorkelSpatial A procedurally-generated benchmark for evaluating allocentric and egocentric spatial reasoning capabilities in LLMs. GPT-5.4 99% Grok 4 Fast Reasoning 84.85% o3 76.67% gpt-5 73.94% gpt-oss-120b 52.73% gpt-5-mini 45.45% Claude Opus 4.1 45.15% Magistral Medium 1.2 44.24% Claude Opus 4 40.3% o3-mini 37.88% Claude Sonnet 4 33.33% gpt-5-nano 26.67% Claude Sonnet 3.7 21.52% Gemini 2.5 Flash 18.79% Llama 4 Scout 15.45% Gemini 2.5 Pro 15.15% gpt-5-chat 14.85% Mistral Large 14.85% o4 mini 14.85% GPT-4.1 14.55% Llama 3.3 70B 14.55% Mistral Medium 3.1 14.55% Nova Micro 14.55% Command R+ 14.24% Nova Premier 14.24% Qwen 3 235B 13.94% Codestral 13.64% Nova Lite 13.33% Grok 3 12.73% Magistral Medium 12.42% Llama 4 Maverick 12.12% Nova Pro 12.12% Command R 11.82% Show full rating SnorkelWordle A benchmark designed to evaluate linguistic reasoning and instruction-following capabilities in language models through the iterative and constrained gameplay of Wordle. gpt-5 94% Grok 4 93% o3 92.9% o4 mini 91.9% Gemini 3 Pro 91% gpt-5-mini 91% o3-mini 90% Grok 4 Fast Reasoning 88% Claude Opus 4 85.6% Kimi-K2-Thinking 85% Claude Sonnet 4 83% gpt-oss-120b 81.6% gpt-5-nano 79% Gemini 2.5 Pro 74% Grok 3 71% Claude Sonnet 3.7 68% gpt-oss-20b 65.9% GPT-4.1 62% Gemini 2.5 Flash 61.9% Kimi-K2 54% Llama 3.3 70B 10.2% Show full rating SnorkelGraph A procedurally-generated and expert verified benchmark for evaluating mathematical and spatial reasoning capabilities of LLMs through graph reasoning problems. GPT-5.4 84.5% Grok 4 Fast Reasoning 75% o4 mini 75% gpt-5-mini 72.5% gpt-5 72% o3 71.5% o3-mini 71% Claude Opus 4 64.5% Grok 3 64% GPT-4.1 63% gpt-5-nano 62.5% Qwen 3 235B 61.5% Grok 4 61% Claude Sonnet 4 58% Gemini 2.5 Pro 58% Gemini 2.5 Flash 55% Magistral Medium 53.5% Claude Sonnet 3.7 50% Nova Premier 34.5% Llama 4 Maverick 34% Mistral Large 30% Nvidia nemotron super 49B 29% Nova Pro 28% Llama 4 Scout 26% Codestral 24.5% Llama 3.3 70B 23.5% Nvidia 70B Instruct 22.5% Llama 3.1 405B 20.5% Nova Lite 19% Nova Micro 17.5% Command R+ 15% Command-Light 10.5% Command 10% Show full rating SnorkelFinance A benchmark of expert-verified financial QA created from financial reports for evaluating AI agents on tool-calling and reasoning capabilities. gpt-5 81% o3 81% Gemini 3 Pro 80.34% Claude Opus 4.1 80.3% gpt-5-mini 79.3% Claude Opus 4 78.3% Claude Sonnet 3.7 77.9% Claude Sonnet 4 76.6% o4 mini 76.6% Grok 4 74.04% Grok 4 Fast Reasoning 73.45% Kimi-K2-Thinking 71.7% gpt-oss-120b 66.6% Grok 3 65.86% o3-mini 63.79% GPT-4.1 62.7% Nova Premier 62.06% Gemini 2.5 Pro 60.6% Gemini 2.5 Flash 53.1% Qwen 3 235B 51.37% gpt-5-nano 50% NVIDIA Nemotron Super 49B v1.5 44% Nova Pro 40.34% Codestral 27.6% Nova Lite 16.89% Magistral Medium 16.2% Nova Micro 14.48% Mistral Large 13.4% Show full rating Performance per Dollar Model Select a domain Cost Output Tokens Model Value Comparison Compare Select a domain Vs Output Tokens Snorkel Expert Data-as-a-Service Accelerate the evaluation and development of frontier AI models with a scalable, white-glove service that provides model development teams with high quality, expert data. Talk to Snorkel Get started How do you want to work with Snorkel? AI data development services Accelerate the development of frontier AI models with expert-curated, enterprise-grade data. Learn how Snorkel’s Data-as-a-Service helps teams label, refine, and evaluate high-quality, domain-specific datase Privacy Policy | Snorkel AI Capabilities Capabilities How it works Research-led data and environment development for the frontier's hardest problems Learn more Data development Overview Expert-curated datasets for frontier AI Use cases See how our data improves frontier models Specialized agents Overview Custom AI systems built to unlock ROI fast Customer stories Real-world results from enterprise deployments Research Research Research Research hub Our latest papers and data-centric AI findings Leaderboards Compare model performance across benchmarks Open Benchmarks Grants Funding for open-source AI research Featured BENCHMARK Agentic Coding benchmark A benchmark for evaluating AI models on complex, real-world coding tasks that require multi-step reasoning, tool use, and autonomous problem-solving. Resources Resources Resources Resource library Guides, papers, and tools for data-centric AI Events Upcoming talks, workshops, and conferences Reading Group AI discussions for researchers and practitioners Blog News, updates, and perspectives from our team featured BLOG Terminal-Bench 2.0 Developed by Stanford and Laude Institute with contributions from Snorkel AI, it’s a major leap forward in evaluating AI coding agents. Company Company Company About Our mission, story, and values Careers Open roles and life at our company Press Media resources and announcements Partners Organizations we work with Security How we keep data safe Contact us Get in touch with our team Join our expert community Get paid to shape safer, smarter AI Learn more Get started Get started Privacy Policy Last Modified: October 20, 2025 Index Scope Information We Collect About You Information We Collect From Expert Contributors How We Collect Information About You How We Use Your Information Disclosure of Your Information to Third Parties Choices About How We Use and Disclose Your Information California Privacy Rights Additional U.S. Privacy Rights Additional Rights for Individuals Located in the EEA, Switzerland, or the UK Exercising Your Data Rights Data Security Data Retention Children Under the Age of 18 Changes to Our Privacy Policy Contact Information Snorkel AI, Inc. (“Snorkel AI”, the “Company”, “we,” “us,” or “our”) respects and values your privacy. This Privacy Policy explains our online and offline information practices, the kinds of information we may collect, how we intend to use and share that information, and how you can opt-out of a use or correct or change such information. This Privacy Policy is incorporated by reference into our Website Terms of Service (“Terms of Service”). For privacy-related inquiries or to exercise your rights as detailed below, contact us at [email protected] . 1. Scope This Privacy Policy applies to information that we process from or regarding our current, prospective, and former customers, users, visitors, guests, business partners, and employees, in the course of our business, including on our website (our “Website”), through our forms, applications and platforms (including the platform accessed by Expert Contributors (the “Platform”)), by phone, texts, or in-person, and any other offerings that link to this Privacy Policy (collectively, the “Services”). This Privacy Policy does not apply to information collected by any third party, including through any application or content (including advertising), that may link to or be accessible from or through the Services. 2. Information We Collect About You We collect limited categories of personal information based on your relationship with us. The categories of information identified in this Privacy Policy will be referred to as “personal information.” Identifiers, e.g. real name, alias, account name Contact Information, e.g., physical and mailing address, email address, phone number. Account Information, e.g. usernames and account numbers. Internet or other similar network activity, e.g. internet or other electronic network activity information such as information regarding your interaction with the Services. Communications, e.g. the content of mail, email, or text messages you send to us or we send to you. Sensory data, e.g. audio, electronic, visual, or similar information, including audio recordings of customer service calls. Demographic Information, e.g. age, race, marital status, mental or physical disability, sex or gender, sexuality, employment status. Professional or employment-related information, e.g. current or past employment history, performance evaluations, and income and salary details. Education information, e.g. current or past education or credentialing information. Some or all of the types of personal information listed above may be considered “sensitive” personal information according to some state laws. While personal information is broadly defined, other categories are specifically excluded from this definition including: Publicly available information, meaning information that is available from federal, state, or local government records. Pseudonymized and de-identified information or aggregated and de-identified information that cannot be reasonably linked to an individual. 3. Information We Collect From Expert Contributors If you engage with us as an Expert Contributor, including but not limited to domain specialists, data scientists, annotators, researchers, engineers, or contractors providing subject matter expertise or dataset curation, we may collect additional categories of personal and professional information. This may include: Identifiers, e.g. real name, alias, account name Contact Information, e.g., physical and mailing address, email address, phone number. Employer or Affiliations, e.g., current employer, institutional affiliation, or contracting entity (if applicable). Account Information, e.g., usernames, passwords, account credentials, or account numbers. Professional or Employment-Related Information, e.g., job title, current or past employment history, professional certifications or licenses, performance feedback, and income or compensation details. Education Information, e.g., academic degrees, schools attended, technical certifications, or areas of subject-matter expertise. Expert Contributor Data, e.g., curriculum vitae (CV), professional bios, links to public profiles (e.g., LinkedIn, GitHub, ORCID), labeling functions, heuristics, or other programmatic contributions; annotation decisions; feedback on datasets; prompt or model evaluation input; and attribution information (e.g., name, title, image) where agreed. Communications, e.g., the content of emails, messages, or other interactions sent to or received from us, including editorial feedback or platform correspondence. Internet or Other Electronic Network Activity, e.g., information regarding your interactions with our services, such as IP address, browser type, access logs, usage behavior, and device identifiers. Geolocation Data, e.g., general location derived from IP address or device settings. Sensory Data, e.g., video recordings (e.g., content meetings) or visual data (e.g., profile photos submitted for publication). Demographic Information, e.g., age, gender, race or ethnicity, marital status, disability status, sexual orientation, or employment status, where voluntarily disclosed or collected for compliance, research, or diversity reporting. Legal, Financial, and Operational Information, e.g., contributor agreements, tax or payment documentation, intellectual property contributions, activity logs, revision history, audit trails, and attribution details for work products submitted or delivered under contract. Some or all of the types of personal information listed above may be considered “sensitive” personal information according to some state laws. While personal information is broadly defined, other categories are specifically excluded from this definition including: Publicly available information, meaning information that is available from federal, state, or local government records. Pse Talk to an expert - Expert Data-as-a-Service | Snorkel AI Capabilities Capabilities How it works Research-led data and environment development for the frontier's hardest problems Learn more Data development Overview Expert-curated datasets for frontier AI Use cases See how our data improves frontier models Specialized agents Overview Custom AI systems built to unlock ROI fast Customer stories Real-world results from enterprise deployments Research Research Research Research hub Our latest papers and data-centric AI findings Leaderboards Compare model performance across benchmarks Open Benchmarks Grants Funding for open-source AI research Featured BENCHMARK Agentic Coding benchmark A benchmark for evaluating AI models on complex, real-world coding tasks that require multi-step reasoning, tool use, and autonomous problem-solving. Resources Resources Resources Resource library Guides, papers, and tools for data-centric AI Events Upcoming talks, workshops, and conferences Reading Group AI discussions for researchers and practitioners Blog News, updates, and perspectives from our team featured BLOG Terminal-Bench 2.0 Developed by Stanford and Laude Institute with contributions from Snorkel AI, it’s a major leap forward in evaluating AI coding agents. Company Company Company About Our mission, story, and values Careers Open roles and life at our company Press Media resources and announcements Partners Organizations we work with Security How we keep data safe Contact us Get in touch with our team Join our expert community Get paid to shape safer, smarter AI Learn more Get started Get started Data Development Better data is developed, not collected. Let's build yours. Snorkel helps frontier AI teams build the data and environments needed for domain-specific, high-consequence problems where generic data isn't enough. Work with our researchers to develop specialized datasets, benchmarks, and evaluation environments built for how your models actually need to perform. Proud to partner with top frontier AI and research teams What we help with Snorkel Data Series access Custom agent development Domain-specific benchmarks Evaluation environments High-precision labeling & adjudication Edge case coverage Rubrics, verifiers & provenance Calibrated expert signal Scalable eval foundation Talk to a Researcher " * " indicates required fields First name * Last name * Work email * Company * Vertical * Select... Finance Media & Entertainment Government & Public Sector Coding / Software Development Medical / Healthcare Legal Education & Research Retail & E-commerce Technology & Cloud Services Other / Cross-industry What is your immediate use case? * Select... Text: Chat / multi-turn, information extraction / tagging, expert Q&A Agentic / tool-augmented interactions / RL environments Multimodal (text/images/audio/video), speech (TTS, ASR) Request dataset samples: Request dataset samples By submitting this form, I agree to the Terms of Use and acknowledge that my information will be used in accordance with the Privacy Policy . Δ Not more data. Better data. Talk to a researcher Explore research Get started How do you want to work with Snorkel? AI data development services Accelerate the development of frontier AI models with expert-curated, enterprise-grade data. Learn how Snorkel’s Data-as-a-Service helps teams label, refine, and evaluate high-quality, domain-specific datasets for your projects. Talk to a data researcher Build specialized agents Explore how Snorkel can collaborate with your product and development teams to build and deploy custom AI and agentic systems. Solutions undergo rigorous testing based on your business criteria to ensure positive ROI, faster. Talk to a strategist Become an expert contributor Join the Snorkel Expert Contributor community and help shape the future of AI with your expertise. Contribute to groundbreaking projects, share domain-specific insights, and get rewarded for your impact. Join now Capabilities How it works Data development Specialized agents Use cases Customer stories Research Research hub Leaderboards Open Benchmarks Grants Resources Resource library Events Reading Group Blog Company About Careers Press Partners Security Contact us Contact Get started Join expert network Compliance Copyright © 2026 Snorkel AI, Inc. All rights reserved. Terms of Use Privacy Cookie Policy Expert Data Development for Frontier AI | Snorkel AI Capabilities Capabilities How it works Research-led data and environment development for the frontier's hardest problems Learn more Data development Overview Expert-curated datasets for frontier AI Use cases See how our data improves frontier models Specialized agents Overview Custom AI systems built to unlock ROI fast Customer stories Real-world results from enterprise deployments Research Research Research Research hub Our latest papers and data-centric AI findings Leaderboards Compare model performance across benchmarks Open Benchmarks Grants Funding for open-source AI research Featured BENCHMARK Agentic Coding benchmark A benchmark for evaluating AI models on complex, real-world coding tasks that require multi-step reasoning, tool use, and autonomous problem-solving. Resources Resources Resources Resource library Guides, papers, and tools for data-centric AI Events Upcoming talks, workshops, and conferences Reading Group AI discussions for researchers and practitioners Blog News, updates, and perspectives from our team featured BLOG Terminal-Bench 2.0 Developed by Stanford and Laude Institute with contributions from Snorkel AI, it’s a major leap forward in evaluating AI coding agents. Company Company Company About Our mission, story, and values Careers Open roles and life at our company Press Media resources and announcements Partners Organizations we work with Security How we keep data safe Contact us Get in touch with our team Join our expert community Get paid to shape safer, smarter AI Learn more Get started Get started New Continual Learning Bench by Berkeley & Snorkel We build the data that pushes the frontier Snorkel helps frontier labs and AI teams develop specialized training data and environments that set their models and agents apart. Request dataset samples Explore research Proud to partner with top frontier AI and research teams Frontier models break at the edges. We build for that. Most data pipelines are built for volume, not difficulty. Frontier models fail on distributional gaps in specialized domains, benchmark blind spots, and tasks where correctness is hard to define. Snorkel is built specifically for these problems. Founded out of Stanford AI Lab, we've been shaping and benchmarking frontier AI for nearly a decade. See our research Open Benchmarks Grants What we've found New RL Research RLVR in Low Data & Compute Regimes Better data beats more compute — measured across low-resource settings. Evaluation Research RIFT: Rubric Failure Mode Taxonomy A diagnostic framework for when AI evaluation rubrics break down. Domain Agents Benchmarking Agents in Insurance Underwriting Environment-first benchmarking for agents in a genuinely high-stakes domain. Collaborations New Research Collaboration Continual Learning Bench Expert-validated tasks for agents that learn across task sequences, not isolated prompts. Code Quality SlopCodeBench Generic code evals miss sloppy code. This measures what they ignore. Agentic Coding Terminal-Bench 2.0 Real terminal tasks — exposing where today's coding agents fail. Legal AI Harvey BigLaw Bench Expert data for the hardest agentic legal research benchmark. Built with Harvey AI. The Frontier AI Data Lab Data development for the frontier Snorkel partners with frontier AI teams to build research-grade datasets, evaluation systems, and runnable environments where generic coverage runs out. Explore capabilities Snorkel Data Series Curriculum-structured datasets for the task areas frontier models are pushing hardest, with rubrics, reviewer guidance, difficulty tiers, and eval slices built in. Custom data development When off-the-shelf coverage runs out, we build bespoke datasets, evals, and benchmark expansions for the exact failure surface you need to close. Specialized agents Custom agents built on specialized data and evaluated in real workflows, with pass/fail criteria tied to the performance standards that moves ROI. Data Expert Demonstrations & Reasoning Human solution traces Reasoning traces SME Q&A rationales Workflow demos and decision workflows Tool-use demos Preference Labels & Rankings Patch/draft/report quality ranking Trajectory QA Risk/safety/style calibration Helpful/harmless ranking Grounding & style Rubrics & Verifiable Outcomes Unit tests / compile Deterministic graders Citation correctness Numerical consistency/scorable math/science Long-horizon tasks Environments Standard & Custom Environments Repo + CLI tools Browser/GUI harness Multi-step/stateful workflows Simulated environments Your tools, codebase, corpus, data & permissions DATA DEVELOPMENT Good data is a set of design choices Request dataset samples Most data quality problems are design problems. Ambiguous task definitions produce inconsistent labels. Uncalibrated reviewers introduce systematic bias. Missing provenance makes failure analysis guesswork. Snorkel's proprietary process is built around the decisions that determine whether training data actually drives model improvement: Well-specified expert-level tasks Tasks scoped to actual model failure modes, with target distributions, acceptance criteria, and verifier definitions. Each spec is a research artifact, written before data work begins. Calibrated expert review Calibration is a research workflow, not a staffing layer. Reviewers are trained against gold sets authored by Snorkel researchers, scored for agreement and bias, and re-calibrated per task. Rubrics and programmatic checks Fine-tuned evaluator models, not just human spot-checks. Snorkel researchers co-design rubrics with domain experts, then distill them into programmatic graders and specialized evaluator models. Adjudication and provenance Author, multi-reviewer, and final-adjudicator pipeline with full audit trails. Every label is traceable to who decided what, when, and on what evidence. Edge-case coverage Distributional precision, not volume. Expert-authored seeds expand into controlled coverage across difficulty bands, edge cases, and failure modes through templated generation. Benchmarks and evals Eval harnesses built alongside the data: task-specific rubrics, deterministic graders, and runnable environments that produce reproducible scores across model versions. Custom AGENTS Specialized agents grounded in expert data Explore agent solutions The same data development system we use to improve frontier models powers our specialized agents. That means agents evaluated against task-specific rubrics and programmatic checks – not generic benchmarks – and refined through the same adjudication and provenance practices used in production model development. Built for specialized workflows and high-consequence decisions, not generic copilots Evaluation on environment-grounded tasks with programmatic pass/fail criteria Same rigor used to train frontier-class models, applied to your enterprise deployment PUBLISHED RESEARCH Research that shapes the work Every dataset, benchmark, and environment we create is the output of active research co-developed and peer-reviewed with leading academic teams and frontier labs. Explore research Benchmark Agentic coding benchmark: Evaluating AI models on complex, real-world coding tasks Blog SlopCodeBench: Measuring Code Erosion as Agents Iterate Blog Terminal-Bench 2.0: Raising the bar for AI agent evaluation Research Paper Learning from Less: Measuring the Effectiveness of RLVR in Low Data and Compute Regimes Research Paper RIFT: A Rubric Failure Mode Taxonomy and Automated Diagnostics of For models that need to be right. Not just good enough. Request dataset samples Explore research Get started How do you want to work with Snorkel? AI data development services Accelerate the development of frontier AI models with expert-curated, enterprise-grade data. Learn how Snorkel’s Data-as-a-Service helps teams label, refine, and evaluate high-quality, domain-specific datasets for your projects. Talk to a da Cookie Notice | Snorkel AI Capabilities Capabilities How it works Research-led data and environment development for the frontier's hardest problems Learn more Data development Overview Expert-curated datasets for frontier AI Use cases See how our data improves frontier models Specialized agents Overview Custom AI systems built to unlock ROI fast Customer stories Real-world results from enterprise deployments Research Research Research Research hub Our latest papers and data-centric AI findings Leaderboards Compare model performance across benchmarks Open Benchmarks Grants Funding for open-source AI research Featured BENCHMARK Agentic Coding benchmark A benchmark for evaluating AI models on complex, real-world coding tasks that require multi-step reasoning, tool use, and autonomous problem-solving. Resources Resources Resources Resource library Guides, papers, and tools for data-centric AI Events Upcoming talks, workshops, and conferences Reading Group AI discussions for researchers and practitioners Blog News, updates, and perspectives from our team featured BLOG Terminal-Bench 2.0 Developed by Stanford and Laude Institute with contributions from Snorkel AI, it’s a major leap forward in evaluating AI coding agents. Company Company Company About Our mission, story, and values Careers Open roles and life at our company Press Media resources and announcements Partners Organizations we work with Security How we keep data safe Contact us Get in touch with our team Join our expert community Get paid to shape safer, smarter AI Learn more Get started Get started Cookies Policy Effective as of 8/9/2021 This Cookie Notice explains how Snorkel AI ( "Snorkel AI" , "we" , "us" or "our" ) uses cookies and similar technologies in connection with the www.snorkel.ai website and any other website that we own or control and which posts or links to this Cookie Notice (collectively, the "Sites" ). What are cookies? Cookies are small data files that are placed on your computer or mobile device when you visit a website. Cookies serve different purposes, like helping us understand how a site is being used, letting you navigate between pages efficiently, remembering your preferences, and generally improving your browsing experience. Our Sites may use both session cookies (which expire once you close your web browser) and persistent cookies (which stay on your computer or mobile device until you delete them). We use two broad categories of cookies: (1) first party cookies, served directly by us to your computer or mobile device, which we use to recognize your computer or mobile device when it revisits our Sites; and (2) third-party cookies, which are served by service providers or business partners on our Sites, and can be used by these parties to recognize your computer or mobile device when it visits other websites. Third-party cookies can be used for a variety of purposes, including site analytics, advertising, and social media features. What types of cookies and similar tracking technologies does Snorkel AI use on the Sites? On the Sites, we use cookies and other tracking technologies in the following categories described in the table below. Type Description Who serves the cookies (link to privacy policy/site) How to control them Analytics These cookies help us understand how our services is performing and being used.These cookies may work with web beacons included in emails we send to track which emails are opened and which links are clicked by recipients. Google Analytics You can find out more information about Google Analytics cookies here and about how Google protects your data here . You can prevent the use of Google Analytics relating to your use of our Sites by downloading and installing a browser plugin available here . See 'your choices' below. Essential These cookies are necessary to allow the technical operation of our services (e.g., they enable you to move around on a website and to use its features). JSDelivr See 'your choices' below. Functionality/ performance Enhance the performance and functionality of our services. Google Analytics Google Tag Manager Zoom Info Marketo You can find out more information about Google Analytics cookies here and about how Google protects your data here . You can prevent the use of Google Analytics relating to your use of our Sites by downloading and installing a browser plugin available here . See 'your choices' below. Type Analytics Description These cookies help us understand how our services is performing and being used.These cookies may work with web beacons included in emails we send to track which emails are opened and which links are clicked by recipients. Who serves the cookies (link to privacy policy/site) Google Analytics How to control them You can find out more information about Google Analytics cookies here and about how Google protects your data here . You can prevent the use of Google Analytics relating to your use of our Sites by downloading and installing a browser plugin available here . See 'your choices' below. Type Essential Description These cookies are necessary to allow the technical operation of our services (e.g., they enable you to move around on a website and to use its features). Who serves the cookies (link to privacy policy/site) JSDelivr How to control them See 'your choices' below. Type Functionality/ performance Description Enhance the performance and functionality of our services. Who serves the cookies (link to privacy policy/site) Google Analytics Google Tag Manager Zoom Info Marketo How to control them You can find out more information about Google Analytics cookies here and about how Google protects your data here . You can prevent the use of Google Analytics relating to your use of our Sites by downloading and installing a browser plugin available here . See 'your choices' below. Other technologies In addition to cookies, our Sites may use other technologies, such as Flash technology to pixel tags to collect information automatically. Browser Web Storage We may use browser web storage (including via HTML5), also known as locally stored objects ("LSOs"), for similar purposes as cookies. Browser web storage enables the storage of a larger amount of data than cookies. Your web browser may provide functionality to clear your browser web storage. Flash Technology We may use Flash cookies (which are also known as Flash Local Shared Object ("Flash LSOs")) on our Sites to collect and store information about your use of our Sites. Unlike other cookies, Flash cookies cannot be removed or rejected via your browser settings. If you do not want Flash LSOs stored on your computer or mobile device, you can adjust the settings of your Flash player to block Flash LSO storage using the tools contained in the Website Storage Settings Panel. You can also control Flash LSOs by going to the Global Storage Settings Panel and following the instructions. Please note that setting the Flash Player to restrict or limit acceptance of Flash LSOs may reduce or impede the functionality of some Flash applications, including, potentially, Flash applications used in connection with our Sites. Web Beacons We may also use web beacons (which are also known as pixel tags and clear GIFs) on our Sites and in our HTML formatted emails to track the actions of users on our Sites and interactions with our emails. Unlike cookies, which are stored on the hard drive of your computer or mobile device by a website, pixel tags are embedded invisibly on webpages or within HTML formatted emails. Pixel tags are used to demonstrate that a webpage was accessed or that certain content was viewed, typically to measure the success of our marketing campaigns or engagement with our emails and to compile statistics about usage of the Sites, so that we can manage our content more effectively. Mobile Application Software Development Kits (SDKs) We may use third-party software development kits ("SDKs") in our mobile applications. A SDK is third-par Press, News, & Awards | Snorkel AI Capabilities Capabilities How it works Research-led data and environment development for the frontier's hardest problems Learn more Data development Overview Expert-curated datasets for frontier AI Use cases See how our data improves frontier models Specialized agents Overview Custom AI systems built to unlock ROI fast Customer stories Real-world results from enterprise deployments Research Research Research Research hub Our latest papers and data-centric AI findings Leaderboards Compare model performance across benchmarks Open Benchmarks Grants Funding for open-source AI research Featured BENCHMARK Agentic Coding benchmark A benchmark for evaluating AI models on complex, real-world coding tasks that require multi-step reasoning, tool use, and autonomous problem-solving. Resources Resources Resources Resource library Guides, papers, and tools for data-centric AI Events Upcoming talks, workshops, and conferences Reading Group AI discussions for researchers and practitioners Blog News, updates, and perspectives from our team featured BLOG Terminal-Bench 2.0 Developed by Stanford and Laude Institute with contributions from Snorkel AI, it’s a major leap forward in evaluating AI coding agents. Company Company Company About Our mission, story, and values Careers Open roles and life at our company Press Media resources and announcements Partners Organizations we work with Security How we keep data safe Contact us Get in touch with our team Join our expert community Get paid to shape safer, smarter AI Learn more Get started Get started Press & awards Snorkel in the news Read our latest press releases and news coverage and browse our awards and recognition. All news articles All topics Awards In the news Press releases Sort: Newest Newest Oldest Awards Snorkel named among Fast Company’s Most Innovative AI Companies for 2026 March 24, 2026 Learn more about Snorkel named among Fast Company’s Most Innovative AI Companies for 2026 Awards Snorkel named to Forbes’ America’s Best Startup Employers 2026 list March 03, 2026 Learn more about Snorkel named to Forbes’ America’s Best Startup Employers 2026 list Awards Snorkel AI completes Defense Innovation Unit (DIU) Challenge December 10, 2025 Learn more about Snorkel AI completes Defense Innovation Unit (DIU) Challenge Awards Snorkel AI recognized on Deloitte’s 2025 Technology Fast 500™ November 19, 2025 Learn more about Snorkel AI recognized on Deloitte’s 2025 Technology Fast 500™ Awards Snorkel AI named one of six winners in Army xTech AI Grand Challenge August 18, 2025 Learn more about Snorkel AI named one of six winners in Army xTech AI Grand Challenge In the news Accenture invests in Snorkel AI to accelerate AI in financial services August 06, 2025 Learn more about Accenture invests in Snorkel AI to accelerate AI in financial services In the news The Fragmented Frontier: Why Rival AI Data Providers Are Poised to Thrive July 02, 2025 Learn more about The Fragmented Frontier: Why Rival AI Data Providers Are Poised to Thrive In the news OpenAI Takes a Page From Palantir, Doubles Down on Consulting Services June 30, 2025 Learn more about OpenAI Takes a Page From Palantir, Doubles Down on Consulting Services In the news The 11 Largest US Funding Rounds of May 2025 June 26, 2025 Learn more about The 11 Largest US Funding Rounds of May 2025 1 2 … 15 16 Join our newsletter For expert advice, the latest research, and exclusive events. By submitting this form, I acknowledge I will receive email updates from Snorkel AI, and I agree to the Terms of Use and acknowledge that my information will be used in accordance with the Privacy Policy . Email address (Required) Δ Get started How do you want to work with Snorkel? AI data development services Accelerate the development of frontier AI models with expert-curated, enterprise-grade data. Learn how Snorkel’s Data-as-a-Service helps teams label, refine, and evaluate high-quality, domain-specific datasets for your projects. Talk to a data researcher Build specialized agents Explore how Snorkel can collaborate with your product and development teams to build and deploy custom AI and agentic systems. Solutions undergo rigorous testing based on your business criteria to ensure positive ROI, faster. Talk to a strategist Become an expert contributor Join the Snorkel Expert Contributor community and help shape the future of AI with your expertise. Contribute to groundbreaking p