[GSoC] Automatic Extraction of OpenWrt Firmware Image Metadata

[nemesifier]

Official thread for this GSoC idea, ask questions and announce your interest here.

[nemesifier]

Nobody has expressed interest in this, but I believe it's a low hanging fruit. I spent a good amount of time researching this and both feasibility and specs look pretty good to me, should not be hard to achieve.

[asmodehn]

Also it will be a nice improvement to the firmware builds, as managing them will become easier to use correctly. since most things manual steps will be automated.

[atif09]

Hi im interested in this, and I was researching about dealing with non sysupgrade images, have questions regarding that too and ill get back soon

[Eeshu-Yadav]

@nemesifier
I Noticed FirmwareImageInline.has_change_permission Returns False Once Saved — What Was the Original Intent?

While reading admin.py, I saw that FirmwareImageInline.has_change_permission(self, request, obj) returns False when obj exists. This makes images fully immutable in the admin after creation — you can't edit any fields.
But the metadata extraction feature requires images in a "failed" state to have editable metadata fields so users can manually fill in board/target info. This seems to conflict with the current immutability design.
I'm curious about the original reasoning: was this to prevent accidental metadata changes that could break existing device pairings, or was it simply "images are binary blobs, nothing useful to edit"? Understanding this helps me figure out whether relaxing it conditionally (based on image status) is safe, or if there's a deeper invariant I should preserve by using a separate confirmation action/view instead of inline editing.

[nemesifier]

There's issues and pull requests dealing with these aspects in the fw-upgrader module.

[atif09]

since hardware.py is a static map of filename to board names, once fwtool extraction is done successfully, the long term plan is to keep it as fallback right?

[nemesifier]

I don't see this as viable, it would be cumbersome. The goal is to make it obsolete and remove it completely.

[atif09]

when fwtool trailer scan fails (no embedded metadata), the extractor falls back to locating and parsing the DTB inside the raw binary using magic byte scanning (0xd00dfeed), decompressing the kernel if needed (gzip → lzma → xz → bz2 → lz4), and parsing model and compatible via pyFDT. From what ive tested so far, fdt is also reliable and does the job.

and as the idea says, for images where both fwtool and DTB yield nothing (x86, armvirt), the image goes straight to failed requiring manual entry

can i get your opinion on this?

[asmodehn]

I m not sure I understand the actual question, so here is a somewhat generic comment, which hopefully answer your point:

In my opinion, one of the first thing to design, implement, test and get right, is the workflow itself from the user perspective... With possibilities to take over manually and fill in by hand at any step where automation could potentially fail.
The GSoC idea already does a pretty good job at describing it, including potential failure, retries, timeouts, etc.

Since some steps will happen in the background with celery (for performance and feasibility reasons), the workflow over various views might not always be simple.

Once that part is done, adding automation with tools will be just filling the gaps. These tools also might evolve and change in the future, but hopefully the workflow will not need to change much.

That being said, it is useful to know the current tools well at first, to be able to plan for all currently possible outcomes of the various steps of the workflow.
It also helps to plan for tools modularity. For instance, I think one celery task per tool run is a good starting point, we re relying on celery task modularity (one can call another when needed, each can be overridden separately, etc.).
The tricky part in my opinion, will be to tie up the frontend / api / db / background tasks without any loose ends...

[atif09]

Thanks, will keep this in mind and focus more on conveying how everything connects instead of more tool-specific info

[nemesifier]

@asmodehn's response is spot on 👍

[atif09]

please take a look at the prototype here

• Automatic metadata extraction on firmware image upload via a post-save Celery task
• Primary path: fwtool which extracts board, target, version, and compatible devices from standard OpenWrt sysupgrade images
• Fallback path: pure-Python DTB scan using fdt: scans for Device Tree Blobs to extract model and compatible strings for images - without embedded fwtool metadata
• Kernel decompression support (gzip, xz, lzma, bz2, lz4) before DTB scanning
• uImage header stripping
• FIT image support
• File validation: rejects invalid files (wrong magic bytes, rootfs-only images)
• extraction statuses: unconfirmed → in progress → success / failed / manual
• Manual override, admins can fill in metadata if automatic extraction fails
• Extraction log per image for debugging

ill make the changes asap if there is anything that needs to improve

[nemesifier]

Can't see much:

It's better if you provide a git repo and keep an updated video recording of the critical parts of your prototype in the README.
In the PDF of the application you can include some screenshots and a link to the prototype repository.

[atif09]

I've fixed the permission issues, the extraction functionality should work now, please have a look whenever possible.

I'll add a clear README with relevant gifs and include screenshots and a link to the repo in my application pdf.

[asmodehn]

Thanks for this.
I had a quick look and indeed I can see a list of various firmware image with their metadata extracted.
But only sysupgrade images ? We might want to come up with a list of images we want to work with for this, to make sure we cover everything that is needed...

Looking at the system info, I assume you made a clean django project for this prototype ?
I mean that is perfectly fine to test the various tools and celery, but I m interested in the reasons why you didn't start from the firmware-upgrader module directly ? Because if some things make it too complex to work with, it's something we probably will want to address...

You should be able run it locally with its test settings... otherwise let us know/open an issue.
By the way, if having to have the whole vpn setup is annoying, you can use:

OPENWISP_CONTROLLER_MANAGEMENT_IP_ONLY = False

This way you can simply experiment with image firmware upgrade on devices in your LAN and test your changes directly in your fork of the openwisp-firmware-upgrader app.

I think it would be worth doing in your case, especially to check how to modify the whole firmware image (add -> extract metadata -> check status -> device upgradable or not) workflow and check all possible points of failure...

And a last comment, as a developer, I find it more convenient to have a direct access at the code, and a obvious simple way to run it locally quickly to experiment, instead of a website frontend only.
Proper deployment is another challenge in itself, so lets not think about it for this work.

[atif09]

Thanks for this.
I had a quick look and indeed I can see a list of various firmware image with their metadata extracted.
But only sysupgrade images ? We might want to come up with a list of images we want to work with for this, to make sure we cover everything that is needed...

yes, Ive tested against other images and included the results in my proposal

Looking at the system info, I assume you made a clean django project for this prototype ?
I mean that is perfectly fine to test the various tools and celery, but I m interested in the reasons why you didn't start from the firmware-upgrader module directly ? Because if some things make it too complex to work with, it's something we probably will want to address...

right, initially i had made a separate django project to test the tools, but now ive shifted the logic to the fork

You should be able run it locally with its test settings... otherwise let us know/open an issue.
By the way, if having to have the whole vpn setup is annoying, you can use:

yes, thank you for this, it helped

And a last comment, as a developer, I find it more convenient to have a direct access at the code, and a obvious simple way to run it locally quickly to experiment, instead of a website frontend only.
Proper deployment is another challenge in itself, so lets not think about it for this work.

ive linked the fork with the readme above, please take a look whenever possible.

[atif09]

Hi @nemesifier @asmodehn, i can use downloads.openwrt.org directly for the prototype-level testing right? because currently downloads.openwisp.io only has ath79 in the firmware/ directory