Harbor Agentic Coding Eval: LibYAML Resource Limits

中文 | English

What is Harbor?

Harbor Framework is a framework for evaluations, post-training, and prompt optimization using agentic environments. It prepares reproducible task environments, runs agents with terminal and code-editing tools, invokes verifiers against the final artifacts, and aggregates rewards, execution steps, and logs.

Harbor can compare end-to-end behavior across models, agents, prompts, and tool configurations.

How this project uses Harbor

This repository contains a local Agentic Coding evaluation that can be run directly with Harbor. The task archive packages the specification, a pinned LibYAML source environment, a reproducible reference solution, and automated black-box tests so that different models can attempt the same real-world engineering change against an identical codebase and verifier.

In this evaluation, a model works through an agent that uses terminal, editing, and testing tools to explore the repository, implement the feature, debug failures, and verify the result. Harbor orchestrates the task and environment, the agent drives the tools, the model supplies the reasoning, and the verifier evaluates the final implementation.

The recorded Qwen and Claude Opus runs both used the claude-code agent. The benchmark therefore compares end-to-end engineering performance under the same Harbor task, agent framework, and verification conditions; it is not a context-free ranking of model intelligence.

The concrete task is to add optional, per-parser YAML document complexity limits to LibYAML. Applications should be able to independently limit nesting depth, document node count, alias count, and decoded scalar bytes for each yaml_parser_t, while preserving existing behavior when the new API is not enabled.

Project overview

Field	Value
Language	C
Change type	Feature
Domain	Infrastructure / configuration security
Difficulty	L2
Estimated human completion time	4.5 hours
Upstream project	yaml/libyaml
Baseline commit	893682bb98d5ed663a3e314c46dceaf9b1c8802f
Task format	Harbor Agentic Coding Eval

Background

LibYAML provides a process-wide global nesting limit, but it cannot assign independent resource budgets to individual parsers. For long-running applications that process both trusted and untrusted YAML inputs, a global setting is too coarse and does not constrain the number of nodes, aliases, or total scalar data produced by a single document.

This task introduces per-parser limits and requires yaml_parser_parse(), yaml_parser_load(), and the scanner path to follow consistent document-boundary and counting semantics.

Target API

The task requires the following additions to LibYAML's public header:

typedef struct yaml_parser_limits_s {
    size_t max_nesting_depth;
    size_t max_document_nodes;
    size_t max_document_aliases;
    size_t max_document_scalar_bytes;
} yaml_parser_limits_t;

YAML_DECLARE(int)
yaml_parser_set_limits(yaml_parser_t *parser,
        const yaml_parser_limits_t *limits);

A value of 0 means unlimited for that dimension. The setter copies the configuration, so callers do not need to retain the input structure.

Required semantics

• Nesting depth: the number of simultaneously open block and flow collections, counted together.
• Document nodes: scalar, sequence, and mapping nodes; collections count when their start event is produced, while aliases are excluded.
• Document aliases: every alias event, including repeated references to the same anchor.
• Scalar bytes: the sum of decoded length values for all scalar events, including mapping keys and empty scalars.
• Exact boundaries: reaching a nonzero limit is valid; the next event beyond it fails.
• Per-document reset: node, alias, and scalar-byte counters reset at each document start.
• Parser isolation: each parser has independent limits and state.
• Backward compatibility: parsers that never use the new API retain existing behavior, including the legacy global nesting setting.

Limits may only be configured before parsing starts or at a complete document boundary. Updates during parsing, loading, or scanning—and updates after the parser enters an error state—must be rejected without overwriting the previous configuration.

Harbor task package

The complete task archive is available here:

• libyaml-parser-resource-limits_C_feature_基础设施_L2.zip

The extracted structure is:

libyaml-parser-resource-limits/
├── instruction.md
├── task.toml
├── environment/
│   └── Dockerfile
├── solution/
│   └── solve.sh
└── tests/
    ├── test.sh
    └── parser_limits_test.c

The components serve the following roles:

• instruction.md: task context, public API, behavioral boundaries, and compatibility requirements.
• task.toml: Harbor metadata, resources, and timeout configuration.
• environment/Dockerfile: reproducible C toolchain and the pinned LibYAML checkout.
• solution/solve.sh: reference implementation used by the Oracle agent.
• tests/test.sh: verifier entry point for upstream regressions, behavioral checks, and sanitizer runs.
• tests/parser_limits_test.c: C99 black-box tests of the public API and resource-limit semantics.

Quick start

Prerequisites

• Docker
• Harbor CLI
• Network access to GitHub and Debian package sources during image construction

Install the stable Harbor CLI as described in the official Getting Started guide:

uv tool install harbor
harbor --help

Extract the task

mkdir -p tasks
unzip libyaml-parser-resource-limits_C_feature_基础设施_L2.zip -d tasks

Run the reference solution

From the parent directory of tasks, run:

harbor run \
  -p ./tasks \
  -a oracle \
  -n 1

The verifier should report reward = 1. The untouched upstream baseline does not provide the new public API and should fail the task-specific tests.

Harbor CLI options may change between releases. If your installed version differs, refer to harbor run --help.

Test design

The verifier has three layers:

1. Upstream regression suite: build LibYAML with GCC and run its existing CTest suite.
2. Public API black-box tests: verify exact limits, document resets, parser isolation, valid and invalid reconfiguration points, error types and locations, and loader cleanup after failure.
3. Memory and undefined-behavior checks: repeat the behavioral tests under Clang ASan/UBSan.

Covered scenarios include:

• exact-boundary and overflow behavior for nodes, aliases, and scalar bytes;
• empty implicit scalars, mapping keys, and decoded UTF-8 escape byte counts;
• mixed block/flow nesting and explicit zero-as-unlimited behavior;
• parse, load, and scan consumption paths;
• counter reset and reconfiguration between documents in multi-document streams;
• rejected updates during active parsing or scanning without configuration loss;
• parser error states and empty loader-document cleanup;
• legacy global nesting behavior when the new API is not enabled.

See tests-analysis.md for the detailed test rationale and theoretical implementation path.

Benchmark results

Each model was evaluated in four independent trials using the same task checksum, claude-code agent, and verifier configuration.

Model	Trials	Exceptions	Rewards	Observed passes	Mean agent steps
Qwen-3.7-Max	4	0	0, 0, 0, 0	0/4	60.5
Claude Opus 4.6	4	0	1, 0, 1, 1	3/4	79.25

The combined mean across all eight runs was 69.875 agent steps. “Observed passes” is the number of trials whose reward equals 1; it is not the same concept as the statistical pass@k estimator.

Qwen-3.7-Max

Claude Opus 4.6

Reproducibility

• LibYAML is pinned to commit 893682bb98d5ed663a3e314c46dceaf9b1c8802f.
• The Docker base image is pinned by digest.
• Dependencies are installed during image construction; task execution does not rely on external services.
• Task checksum: b30492bb7ea76716ade20c6ea9a07cc852092d50fe40ae1ad7d21684196821c6
• Task archive SHA-256: 448c0fde86e941bdc6ff1002c338c8cb57ffd45d6130b84ccd4a30e59e615885

License and use

This repository currently provides no open-source license. Unless applicable law or hosting-platform terms state otherwise, the author retains all rights to the original material in this repository. Public visibility does not grant permission to copy, modify, distribute, or sublicense it.

LibYAML is distributed by its upstream authors under the license included in the upstream repository. That license applies only to the LibYAML material it covers and does not automatically license this repository's task specification, tests, reference solution, screenshots, or analysis.