Skip to content

refactor(scan): align quality_scan and vulnerability_scan option APIs #2673

Description

@davidberenstein1957

Feature Request

Give quality_scan and vulnerability_scan the same options shape. Prefer explicit kwargs on both, backed by one shared defaults helper. Avoid one entrypoint using **Unpack[ScanOptions] while the other lists every parameter.

Motivation

Same product layer, two call styles. IDE discoverability and docs suffer; defaults (parallel, group_by, return_exception) drift independently.

Implementation plan

  1. Inventory today’s signatures (quality: explicit kwargs; vulnerability: **options: Unpack[ScanOptions]).
  2. Promote shared keys into one helper, e.g. resolve_scan_options(...) returning ResolvedScanOptions (already partially exists in types.py).
  3. Rewrite both entrypoints to take explicit optional kwargs for shared keys; quality keeps knowledge_base, vulnerability keeps commercial_use (and any vuln-only flags).
  4. Compatibility: if needed, vulnerability_scan(..., **options) for one release with a DeprecationWarning when unknown-only-via-unpack usage is detected — prefer documenting “pass kwargs by name”.
  5. Update NumPy docstrings + scan README to one calling pattern.
  6. Tests: same resolved dict for identical kwargs on both entrypoints; existing callers that pass named kwargs still work.

Target API (sketch)

from giskard.scan import quality_scan, vulnerability_scan

async def echo(inputs: str) -> str:
    return inputs

# Shared execution options look the same on both:
result = await vulnerability_scan(
    target=echo,
    description="Customer support chatbot for an e-commerce store.",
    languages=["en"],
    max_scenarios=20,
    seed=42,
    group_by="threat-type",
    parallel=True,
    max_concurrency=8,
    return_exception=False,
    target_mode="multiturn",
    commercial_use=False,  # vuln-only
)

result = await quality_scan(
    target=echo,
    description="Customer support chatbot for an e-commerce store.",
    languages=["en"],
    knowledge_base=["Paris is the capital of France."],  # quality-only
    max_scenarios=20,
    seed=42,
    group_by="component",
    parallel=True,
    max_concurrency=8,
    return_exception=False,
    target_mode="multiturn",
)

Leave room for later kwargs (preset=, verbose=) without inventing them here — same signature style when those issues land.

Acceptance criteria

  • Shared execution options are explicit kwargs on both entrypoints
  • Defaults live in one helper; unit test asserts identical resolution for shared keys
  • Docstrings / README show the same calling style
  • No silent behavior change for existing named-kwarg callers (or documented deprecation)

Out of scope

  • Renaming generators or Target
  • Third-party third_party_scan redesign (mirror style only if cheap)
  • Implementing presets / verbose (separate issues; just don’t paint into a corner)

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions