Skip to content

Add initial BuildStream backend - #300

Open
juergbi wants to merge 9 commits into
ros-infrastructure:masterfrom
CodethinkLabs:bst
Open

Add initial BuildStream backend#300
juergbi wants to merge 9 commits into
ros-infrastructure:masterfrom
CodethinkLabs:bst

Conversation

@juergbi

@juergbi juergbi commented Mar 8, 2024

Copy link
Copy Markdown

This adds a generator for BuildStream elements. https://github.com/CodethinkLabs/ros2-bst is a working BuildStream project for a subset of ROS Iron where all elements for ROS packages have been generated by this branch of superflore. It also includes manually written elements for some external dependencies and depends on freedesktop-sdk for the base system and other external dependencies.

It currently cannot generate elements for all packages of a ROS distro as some packages require additional external dependencies. It's thus necessary to invoke superflore-gen-bst with the --only option to specify the target packages. Elements are also generated for internal dependencies of target packages.

The generated elements currently do not include test dependencies and test suites of packages are not run as part of the build process. This should be added as a future enhancement.

The generated elements do not yet include test dependencies and test
suites of packages are not run as part of the build process.
@robwoolley

Copy link
Copy Markdown
Contributor

Thanks for this contribution. Sorry for the delay. I am just ramping up as the new maintainer.

I tested your modifications with the following steps:

git clone https://github.com/CodethinkLabs/ros2-bst
git clone https://gitlab.com/freedesktop-sdk/freedesktop-sdk
superflore-gen-bst --ros-distro iron --output-repository-path ros2-bst --dry-run --only rclcpp

And got the error:

>>>> Regenerating package 'rti_connext_dds_cmake_module'...
Traceback (most recent call last):
  File "/home/rosuser/superflore/venv/bin/superflore-gen-bst", line 33, in <module>
    sys.exit(load_entry_point('superflore==0.3.3+14.ga343862', 'console_scripts', 'superflore-gen-bst')())
  File "/home/rosuser/superflore/venv/lib/python3.10/site-packages/superflore-0.3.3+14.ga343862-py3.10.egg/superflore/generators/buildstream/run.py", line 160, in main
  File "/home/rosuser/superflore/venv/lib/python3.10/site-packages/superflore-0.3.3+14.ga343862-py3.10.egg/superflore/generators/buildstream/gen_packages.py", line 109, in regenerate_pkg
  File "/home/rosuser/superflore/venv/lib/python3.10/site-packages/superflore-0.3.3+14.ga343862-py3.10.egg/superflore/generators/buildstream/gen_packages.py", line 215, in element_text
  File "/home/rosuser/superflore/venv/lib/python3.10/site-packages/superflore-0.3.3+14.ga343862-py3.10.egg/superflore/generators/buildstream/bst_element.py", line 261, in get_element_text
  File "/home/rosuser/superflore/venv/lib/python3.10/site-packages/superflore-0.3.3+14.ga343862-py3.10.egg/superflore/generators/buildstream/bst_element.py", line 222, in get_dependencies
superflore.exceptions.UnresolvedDependency: external dependency rti-connext-dds-6.0.1 missing

Can you provide any guidance on what is supported?

Cheers,
Rob

@juergbi

juergbi commented Nov 29, 2024

Copy link
Copy Markdown
Author

rti-connext-dds-6.0.1 is an external dependency that is not currently included in the ros2-bst repository. You can pass --skip-keys rmw_connextdds to disable rmw_connextdds and with that, avoid the missing external dependency. With this option, your command completes successfully here.

External dependencies can be added to ros2-bst as needed without requiring changes in superflore.

@robwoolley

robwoolley commented Nov 30, 2024

Copy link
Copy Markdown
Contributor

Thanks for the quick reply. Running the following completed successfully:

superflore-gen-bst --ros-distro iron --output-repository-path ros2-bst --dry-run --only rclcpp --skip-keys rmw_connextdds

which updated the following files:

ros2-bst/elements/generated/ament-cmake/ament-cmake-core.bst
ros2-bst/elements/generated/ament-cmake/ament-cmake-export-definitions.bst
ros2-bst/elements/generated/ament-cmake/ament-cmake-export-dependencies.bst
ros2-bst/elements/generated/ament-cmake/ament-cmake-export-include-directories.bst
ros2-bst/elements/generated/ament-cmake/ament-cmake-export-interfaces.bst
ros2-bst/elements/generated/ament-cmake/ament-cmake-export-libraries.bst
ros2-bst/elements/generated/ament-cmake/ament-cmake-export-link-flags.bst
ros2-bst/elements/generated/ament-cmake/ament-cmake-export-targets.bst
ros2-bst/elements/generated/ament-cmake/ament-cmake-gen-version-h.bst
ros2-bst/elements/generated/ament-cmake/ament-cmake-gmock.bst
ros2-bst/elements/generated/ament-cmake/ament-cmake-gtest.bst
ros2-bst/elements/generated/ament-cmake/ament-cmake-include-directories.bst
ros2-bst/elements/generated/ament-cmake/ament-cmake-libraries.bst
ros2-bst/elements/generated/ament-cmake/ament-cmake-pytest.bst
ros2-bst/elements/generated/ament-cmake/ament-cmake-python.bst
ros2-bst/elements/generated/ament-cmake/ament-cmake-target-dependencies.bst
ros2-bst/elements/generated/ament-cmake/ament-cmake-test.bst
ros2-bst/elements/generated/ament-cmake/ament-cmake-version.bst
ros2-bst/elements/generated/ament-cmake/ament-cmake.bst
ros2-bst/elements/generated/cyclonedds/cyclonedds.bst
ros2-bst/elements/generated/fastrtps/fastrtps.bst
ros2-bst/elements/generated/libstatistics-collector/libstatistics-collector.bst
ros2-bst/elements/generated/rcl/rcl-yaml-param-parser.bst
ros2-bst/elements/generated/rcl/rcl.bst
ros2-bst/elements/generated/rclcpp/rclcpp.bst
ros2-bst/elements/generated/rcpputils/rcpputils.bst
ros2-bst/elements/generated/rcutils/rcutils.bst
ros2-bst/elements/generated/rmw-fastrtps/rmw-fastrtps-cpp.bst
ros2-bst/elements/generated/rmw-fastrtps/rmw-fastrtps-dynamic-cpp.bst
ros2-bst/elements/generated/rmw-fastrtps/rmw-fastrtps-shared-cpp.bst
ros2-bst/elements/generated/rmw-implementation/rmw-implementation.bst
ros2-bst/elements/generated/ros2-tracing/tracetools.bst

@robwoolley

Copy link
Copy Markdown
Contributor

What are the future plans for the BuildStream backend?

Will you or CodeThink be maintaining it going forward?

Since ROS 2 Iron is now end-of-life, will you be adding support for Humble, Jazzy, or Rolling?

harrysarson and others added 2 commits January 8, 2025 08:59
For some buildstream elements generated by superflore we do not want to
use the normal source. The way that buildstream includes work means it
is not possible to overrite element sources.

This commit allows users of superflore-gen-bst to explicitly leave the
`sources` of an element blank so that the sources can be specified by an
include file.
Add new CLI flag to exclude sources from elements
@tom--pollard

Copy link
Copy Markdown

Hi @robwoolley sorry for the delayed reply! I work with @juergbi at Codethink and this is still work we'd like to press forward with, @harrysarson is also actively working on the superflore tooling & ros2-bst.

In-terms of the first two questions we're still doing some internal processing, for the last question we have moved the project forward to Jazzy but that's not been pushed publicly yet.

Will try to provide an update with specific answers to your questions when we can, and thanks for the interaction so far.

@robwoolley

Copy link
Copy Markdown
Contributor

@tom--pollard Thanks for the update. Glad to hear that you have moved the project forward to Jazzy. Whenever you are ready to submit the BuildStream backend, please let me know and I'll be happy to review it.

harrysarson and others added 6 commits September 3, 2025 10:32
This commits adds a new command line flag to force superflore (via the
rosdistro python package) to regenerate the distribution information.

This is useful when running superflore with $ROSDISTRO_INDEX_URL set to
get deterministic codegen. The rosdistro caches are updated to latest
and not versioned, see ros/rosdistro#47469.
Allow running superflore with uncached rosdistro
Superflore needs to handle conditional statements when walking
dependencies and when extracting package information for generating
buildstream elements. This commit achieves this by passing an
`evaluate_condition_context` when constructing `PackageMetadata` and
`DependencyWalker` objects.
Allow parsing of package.xml's with conditionals
If the .inc file defines the `cmake-extra` variable the element will
include the value of that variable in its `cmake-local definition`.
Allow tweaking the cmake flags in an .inc file
@robwoolley

Copy link
Copy Markdown
Contributor

@tom--pollard I see that you have been busy working on this. Please let me know when you are ready to submit the BuildStream backend for review and inclusion into Super Flore.

When you are ready, please help answer the following questions:

  • What are the future plans for the BuildStream backend?
  • Will you or CodeThink be maintaining it going forward?
  • Since ROS 2 Iron is now end-of-life, will you be adding support for Humble, Jazzy, or Rolling?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants