Open
Conversation
… Starlark - Created `scriptable-pipelines.md` file to outline the concept of running pipelines written in Starlark instead of YAML. - Explained the need for a more expressive control flow beyond Tavern's current `finally` block and outlined examples of conditional logic. - Provided examples of using Starlark to define and execute pipelines, including `run_pipeline` function and integration with existing Tavern components such as stages and fixtures. - Elaborated on requirements for `run_stage` function, Starlark's `load` functionality, and experimental CLI flag `--experimental-starlark-pipeline`. - Included details on rendering Starlark pipelines to YAML for backward compatibility. - Discussed integration of Tavern fixtures, marking mechanisms, and direct definition of stages in Starlark files. - Linked to Starlark specification and outlined potential extensions for user-defined functions while noting their security concerns. This sets the foundation for enabling Starlark-based scriptable pipelines, improving flexibility and control in test workflows.
…tion notes - Added "Edge Cases and Implementation Details" section covering various aspects of scriptable pipelines using Starlark. - Documented `StageResponse` and `TestConfig` object structures with field-by-field explanations. - Included details for handling fixtures (`setup_fixtures`) and marks (`setup_marks`) in the pipeline execution flow. - Explained file loading mechanisms, including `include` for YAML files and `load` for `.star` files. - Outlined variable formatting, error handling using `fail()`, and stage definition format in YAML. - Listed limitations of the Starlark-based system, such as no exceptions, single-threaded execution, and JSON serialization constraints.
- Added `tavern/_core/starlark` module to handle Starlark-based pipeline execution. - Defined `StarlarkPipelineRunner` and `StageResponse` in `runner.py` for script execution and result encapsulation. - Implemented `setup_starlark_environment` to initialize Starlark environments and globals in `starlark_env.py`. - Integrated pytest plugin in `tavern/_core/starlark/pytest_plugin.py` for collecting `.tavern.star` files. - Added CLI flag `--experimental-starlark-pipeline` to enable the feature. - Added `StarlarkYamlFile` class to handle pipeline file collection and execution. - Updated `pytest_collect_file` in `tavern/_core/pytest/__init__.py` to support `.tavern.star` files. - Added the experimental flag to `parser.addoption` in `tavern/_core/pytest/hooks.py`. This commit establishes foundational support for Starlark-based scriptable pipelines, enabling more flexible and conditionally structured test workflows.
…tion - Removed `_add_starlark_parser_option` from `pytest_plugin.py` and relocated the Starlark pipeline CLI flag definition to `add_parser_options` in `util.py` for better organization. - Ensured no functional changes, preserving existing flag behavior and functionality.
… handling in `starlark_env.py` - Simplified global variable handling by replacing custom `self.globals` with `Globals.standard()` for better consistency with the Starlark module. - Introduced `get_sessions` in `StarlarkEnvironment` to retrieve session contexts from test specifications mirroring logic in `run.py`. - Updated `load_and_run` to utilize `ExitStack` for managing session contexts and improved error handling for Starlark script parsing and evaluation. - Refactored helper functions (`_setup_builtins`, `_setup_include`, `_setup_run_stage`, `_setup_fail`) to add methods directly to the `Module` instance instead of using a dictionary for globals. - Adjusted script parsing to use the file path (`test_path`) for error tracing during Starlark evaluations. refactor(starlark): Update `run_stage` in `runner.py` to support session handling - Added `sessions` argument to `run_stage` for injecting context-specific sessions during stage execution. - Made `sessions` an optional parameter, defaulting to an empty dictionary if not provided, ensuring backward compatibility.
… `run_stage` implementation - Added `README.md` in `tests/integration/starlark/` with instructions for setting up and running Starlark pipeline integration tests. - Created `stages.yaml` in `tests/integration/starlark/` to define sample stages for the pipeline tests. - Implemented `test_basic_pipeline.tavern.star` to test basic Starlark pipeline functionality using `include()` and `run_stage()` methods. - Updated `run_stage` function in `starlark_env.py` to return a tuple consisting of updated variables and `StageResponse`, enhancing usability and consistency with the pipeline execution flow.
…ution flow - Added `PipelineContext` dataclass in `starlark_env.py` to manage test configuration and session state across pipeline stages. - Updated `run_stage` definition to accept and return `PipelineContext` instead of raw dictionaries, ensuring improved state management and clarity. - Introduced `context` function in `starlark_env.py` to initialize the `PipelineContext`, now callable from Starlark scripts. - Modified `test_basic_pipeline.tavern.star` to utilize `PipelineContext`, reflecting the updated pipeline execution flow. - Enhanced `_setup_context` in `StarlarkPipelineRunner` to add the `context` function to the Starlark environment. - Updated imports in `__init__.py` to include `PipelineContext`.
…ndling in `runner.py` - Removed the unused `pytest_tavern_beta_before_every_request` hook from `pytest_plugin.py`, which was present for backward compatibility but is now redundant. - Updated exception handling in `runner.py` to only catch `TavernException`, ensuring non-Tavern exceptions are raised for better error clarity and debugging.
- In `tavern/_core/starlark/starlark_env.py`, the `ValueError` raised when parsing a Starlark script now uses a generic message `"Failed to parse starlark script"` instead of embedding the original exception text. - The `RuntimeError` raised when evaluating a Starlark script is similarly simplified to `"Failed to evaluate starlark script"` without the underlying exception details. - The `include` helper now raises a `ValueError` with a generic message `"Failed to include '<filename>'"` rather than appending the caught exception string. These changes rely on existing logging (which already records the detailed exception) to keep end‑user error messages concise while preserving full diagnostics in the logs.
- Upgraded `actionlint` to v1.7.11 in `.pre-commit-config.yaml`. - Upgraded `ruff-pre-commit` hooks (`ruff-format` and `ruff-check`) to v0.15.8. - Upgraded `mypy` to v1.19.1 with the `mirrors-mypy` hook. - Updated `uv-pre-commit` to v0.11.2. - Ensured dependencies are up-to-date for improved stability and features.
…ency - Updated `--experimental-starlark-pipeline` flag to `--tavern-experimental-starlark-pipeline` across the codebase for improved CLI consistency. - Modified relevant documentation files (`README.md`, `scriptable-pipelines.md`) to reflect the flag name change. - Adjusted flag handling in `tavern/_core/pytest/hooks.py` and `tavern/_core/pytest/util.py` to align with the new name. - Updated Starlark pipeline file collection logic in `tavern/_core/starlark/pytest_plugin.py` to use the renamed flag.
…lection in pytest plugin - Updated `tavern/_core/starlark/pytest_plugin.py` to use `Dialect.extended()` instead of `Dialect.standard()` for Starlark script parsing, enabling extended syntax support. - Implemented `_generate_items` for item collection in `pytest_plugin.py` and replaced the `NotImplementedError` in `collect` with a call to `_generate_items()`. - Made a similar change in `tavern/_core/starlark/starlark_env.py`, switching to `Dialect.extended()` for script parsing consistency.
…h PEP 518 standards - Renamed `[pytest]` section to `[tool.pytest]` in `pyproject.toml` for compliance with PEP 518.
…enablement check - Removed `logger.debug` call in `tavern/_core/starlark/pytest_plugin.py` that logged the `starlark_enabled` state, as it is unnecessary and provides no additional value to debug logs.
…handling - Refactored `YamlItem` and `StarlarkItem` to inherit from a new `BaseTavernItem` class in `tavern/_core/pytest/item.py`, centralizing shared functionality such as fixture handling, markers, and config loading. - Updated `BaseTavernItem` to include essential methods like `_load_fixture_values` and `initialise_fixture_attrs`. - Moved `_TavernFixtureRequest` class to ensure compatibility with both `YamlItem` and `StarlarkItem`. - Adjusted `StarlarkFile` in `tavern/_core/starlark/pytest_plugin.py` to generate `StarlarkItem` instead of `YamlItem` during test collection.
…unctions - Removed unused methods (`_setup_include`, `_setup_run_stage`, `_setup_fail`, `_setup_context`) in `tavern/_core/starlark/starlark_env.py` as their functionality was directly defined elsewhere in the code. - Simplified the `Module` initialization by consolidating relevant callable additions (`include`, `run_stage`) directly into the setup process. - Improves maintainability by eliminating unnecessary indirection and unused methods.
…oader modules - Replaced `_get_include_dirs` with a new `get_include_dirs` function in `tavern/_core/loader.py`, returning an `Iterable` instead of a `list` for improved flexibility and clarity. - Updated `find_include` in `tavern/_core/loader.py` to use the refactored `get_include_dirs` function. - Enhanced the docstring in `get_include_dirs` to provide detailed explanation of environment variable handling (`TAVERN_INCLUDE`) and loader path merging. - Modified `starlark_env.py` to integrate `get_include_dirs`, ensuring that include files are loaded correctly with environment variable paths. - Added error handling and enhanced logging when files cannot be included in `starlark_env.py`. - Improved code maintainability and readability by consolidating directory resolution logic.
…r includes - Updated `get_include_dirs` usage in `starlark_env.py` to correctly derive parent directories using `pathlib.Path(self.test_path).parent`. - Adjusted logging to clarify file inclusion attempts with absolute paths (`abs_filename`). - Fixed a bug in the file accessibility check, replacing `filename` with `abs_filename` for accurate results.
…runner usage - Removed the `setup_starlark_environment` function in `tavern/_core/starlark/starlark_env.py`, as it was redundant. - Updated `tavern/_core/starlark/pytest_plugin.py` to directly initialize `StarlarkPipelineRunner` instead of relying on `setup_starlark_environment`. - Adjusted imports in `tavern/_core/starlark/pytest_plugin.py` and `tavern/_core/starlark/__init__.py`, removing references to the now-deleted function. - Simplified code by eliminating unnecessary abstraction, improving readability and maintainability.
… unused `test_config`
- Updated `load_and_run` in `tavern/_core/starlark/pytest_plugin.py` to pass `global_cfg` to StarlarkPipelineRunner, ensuring global configuration is available during pipeline execution.
- Removed redundant test configuration creation in `pytest_plugin.py`.
- Modified constructor of `StarlarkPipelineRunner` in `tavern/_core/starlark/starlark_env.py` to no longer require `test_config`.
- Updated `load_and_run` and `get_sessions` methods in `starlark_env.py` to take `test_config` as an argument instead of being initialized in the runner.
- Added a call to `module.freeze().call("run_pipeline", test_config)` in `load_and_run` to execute the pipeline with the provided configuration.
- Improved maintainability by reducing reliance on class-level state and consolidating configuration handling.
… `TestConfig` and update pipeline context handling - Added `to_starlark` and `from_starlark` methods to `TestConfig` class in `tavern/_core/pytest/config.py` to support serialization and deserialization for Starlark. - Updated `PipelineContext` in `tavern/_core/starlark/starlark_env.py` to inherit from `TypedDict` instead of `dataclass` for stronger typing and compatibility with dictionary-style context management. - Modified `load_and_run` and pipeline execution logic in `tavern/_core/starlark/starlark_env.py` to utilize the new `TestConfig.to_starlark` and `TestConfig.from_starlark` methods for improved handling of `test_config` in Starlark pipelines. - Adjusted context creation to wrap `sessions` and `test_config` using `starlark.OpaquePythonObject` for consistent state management across stages. - Improved maintainability and readability by aligning serialization handling with Starlark requirements.
…tion - Changed return type of `to_starlark` in `tavern/_core/pytest/config.py` to `dict[str, Any]` for stronger typing. - Updated `test_config` and `sessions` attributes in `PipelineContext` within `tavern/_core/starlark/starlark_env.py` to use `dict[str, Any]` for consistent typing. - Simplified context dictionary construction during pipeline execution by removing redundant wrapping of `test_config` and `sessions`. - Ensured improved type clarity and consistency across the affected modules.
…ation - Updated `tavern/_core/starlark/starlark_env.py`: - Replaced `dict(stage)` with `copy.deepcopy(stage)` in `StarlarkEnv.run_stage`. - Ensures that nested dictionaries within `stage` are not unintentionally mutated, improving data isolation. - Added `copy` module import to support deep copy functionality.
- Updated `tavern/_core/starlark/starlark_env.py`: - Modified `re_sub` to directly return the result of `re.sub` on the decoded string when the input is of type `bytes`. - Prevents intermediate mutation of the `string` variable and ensures consistent behavior when handling both `str` and `bytes`.
…_response - Updated `tests/unit/starlark/test_starlark_env.py`: - Added `mock_response.json.assert_called_once()` in `test_create_response_struct_with_success`. - Ensures that the `json()` method is appropriately called on the mock response object during this unit test.
…truction - Updated `tavern/_core/starlark/starlark_env.py`: - Added error handling to safely parse `rest_response.json()` for JSON responses, with a fallback to `rest_response.content` in case of exceptions. - Ensures robust handling of response bodies by differentiating between JSON and raw content. - Prevents unhandled exceptions when JSON parsing fails, improving error resilience.
…tent stage IDs - Renamed `test_registry_raises_on_invalid_id` to `test_registry_returns_none_for_nonexistent_id` in `tests/unit/starlark/test_stage_registry.py`. - Updated the test to check for `None` instead of expecting an exception when querying a nonexistent stage ID. - Ensures that the test aligns with the intended behavior of `StageRegistry`.
- Updated `pyproject.toml` to include `.hypothesis` in the `tool.pytest.ini_options.exclude` list. - Prevents pytest from inadvertently detecting hypothesis-related files during test discovery.
…tests - Add `experimental_starlark_pipeline` field to `TestConfig` (default None) and populate it from the `--tavern-experimental-starlark-pipeline` pytest option (default False) - In `run_test`, raise `UnexpectedKeysError` when a test contains `control_flow` but the flag is not enabled; otherwise run the Starlark control flow handler - Exclude `starlark_control_flow` tests by default: update pytest `addopts` to also exclude this marker and add the marker definition - Mark the new inline control flow test with `is_defaults: True` and `starlark_control_flow` marker - Update unit test conftests to reflect the default flag value (`False`) and enable it in Starlark-specific unit tests - Modify `tox-integration.ini` to run `starlark_control_flow` tests separately with the flag and marker, removing the flag from the default test runs
Open
…ipts - Update pyproject.toml to use python 3.12 paths and target version - Update coverage script to use python 3.12 tox environments - Update starlark integration test documentation to reference python 3.12
- Move starlark helper functions from inline string to tavern_helpers.star file - Add importlib.resources import for loading starlark builtins - Refactor _get_starlark_builtins() to read from external file - Remove inline _STARLARK_BUILTINS string definition - Update load function to use _get_starlark_builtins() method This change improves maintainability by separating starlark builtins from python code and provides better organization for starlark helper functions. The tavern_helpers.star file now contains all built-in functions available to starlark scripts including run_stage, re, time, and log utilities.
- Create new test file test_control_flow_guard.py - Add test_control_flow_requires_experimental_flag() to verify BadSchemaError when control_flow is used without experimental flag - Add test_control_flow_works_with_experimental_flag() to ensure control_flow executes when experimental flag is enabled - Test both positive and negative cases for starlark pipeline experimental feature flag This ensures the control_flow feature is properly guarded behind the experimental flag and prevents accidental usage without explicit enablement.
- Added `scripting-api.md` to `docs/source/` to document utility modules in the Starlark pipeline feature. - Included details for `starlark.tavern_helpers`, detailing available functions (`run_stage`) and structs (`re`, `time`). - Provided usage examples for regex utilities, time controls, and executing test stages. chore(pre-commit): add Starlark docstring Markdown generation hook - Updated `.pre-commit-config.yaml` to include a local hook for generating Markdown from Starlark docstrings. - Hook uses `uv tool` and points to `tavern/tavern/_core/starlark/` and `docs/source/scripting-api.md`. - Configured hook to target files with `.star` extension. This improves the usability and maintainability of the Starlark scripting API and integrates automated documentation updates into the development workflow.
… header - Removed outdated list of available functions (`run_stage`, `re`, `time`, `log`) from the docstring in `tavern/_core/starlark/tavern_helpers.star`. - Simplifies the header and eliminates redundancy with inline `load` usage example.
…tory entries for hook configurations
- Updated the docstring in `tavern/_core/files.py` to provide a clearer and more detailed explanation of the return value format. - Structured the description to differentiate between tuple lengths (2, 3, or 4) and included notes on multipart upload group name handling. - Added explicit information about additional return values: form field name and absolute file path. - Improved clarity regarding the dictionary of headers passed to the `requests` 'files' argument.
…rity - Updated `FileSendSpec` docstring in `tavern/_core/files.py` to include detailed descriptions of attributes: `filename`, `file_obj`, `content_type`, and `content_encoding`. - Improved handling of `inferred_content_encoding` in `tavern/_plugins/rest/request.py`: - Added support for `dict` type `content_encoding` by updating `fspec["headers"]`. - Ensured non-dict `inferred_content_encoding` values are correctly assigned to `fspec["headers"]["content-encoding"]`. - Enhanced overall clarity and robustness of file upload functionality.
- Updated `tavern/_core/pytest/file.py` to include a conditional check ensuring that keyword arguments with `None` keys are excluded when creating the `kwargs` dictionary. - Adds robustness by avoiding potential runtime errors caused by `NoneType` keys during dictionary unpacking or usage.
- Updated pyproject.toml: changed ruff version constraint from `~=0.9.10` to `>=0.15.9` to allow newer versions - Updated uv.lock: updated to reflect the new dependency resolution for ruff
- Updated `pyproject.toml` to replace Python 3.11 with 3.12 and add support for Python 3.14 in `classifiers`. - Updated `requires-python` from `>=3.11` to `>=3.12`. - Changed MyPy's `python_version` setting to `3.12`. - Updated `tool.coverage.paths` to reflect Python 3.12 in `.tox` path definitions. - Updated `base_python` version in tox configuration to `3.12`. This ensures compatibility with newer Python versions and removes support for Python 3.11.
<description of changes In tavern/_core/dict_util.py, changed the return statement in _attempt_find_include to return the formatted string instead of None. In tavern/_core/pytest/util.py, added imports for logging, functools.lru_cache, pathlib.Path, and typing annotations.
- Added step in `.github/workflows/main.yml` to install the latest version of `uv` using `astral-sh/setup-uv@v7` in the `build` job. - Updated existing `uv` setup action from `v6` to `v7` in the `unit-tests` job for consistency.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is an attempt to fix https://github.com/taverntesting/tavern/milestone/3
See https://github.com/taverntesting/tavern/blob/feat/scriptable-pipeline/docs/source/scripting.md for details and problems
There is already have multiple keys to control execution:
tavern/tavern/_core/schema/tests.schema.yaml
Lines 43 to 58 in 4ef50bf
another similar story: https://cirrus-ci.org/guide/programming-tasks/
Summary by CodeRabbit
Release Notes
New Features
--tavern-experimental-starlark-pipelineflag).control_flowkeyword to define dynamic stage execution order with custom scripts.re.match,re.search,re.sub) for Starlark scripts.Documentation
Chores