Stream every module - #16
Merged
Merged
Conversation
Member
|
oh damn I've introduced a conflict :/ sorry about that |
lsd-cat
force-pushed
the
stream-every-module
branch
from
August 2, 2026 08:32
f9ee494 to
26e3e13
Compare
Only Files and Packages streamed their records; every other module built a results list during parse and walked it again in checkIndicators, so peak memory scaled with the artifact. Now every module emits each record and checks it on the spot, and Artifact.results is gone: a public list that is always empty invites a caller to read "no records" as "clean", so removing it turns that into a compile error instead. checkIndicators is no longer abstract. Only DumpsysAppops still overrides it, to order its risky-permission detections by timestamp once every package has been seen. DumpsysPackages needed real work rather than a mechanical swap: it buffered the whole Packages: section as one string before parsing. It now flushes a package when the next block starts, so only one block is ever held. A 24 MB package list needed more than 96 MB of heap before and now runs in 32 MB. A module that throws mid-parse keeps what it already found, alongside the ARTIFACT_PARSE_FAILED marker: records are checked as they stream, so a truncated artifact has already produced real detections, and discarding them made a partial read look clean. checkIndicators runs in a finally for the same reason, or a module holding detections back would lose them. recordCount replaces results as the way to tell "parsed nothing" from "parsed plenty and found nothing", which the runner tests relied on. Detection output is unchanged: the grouped JSON for the androidqf and bugreport fixtures is byte-identical, as is the raw ordered detection stream across every fixture, including the dumpsys_* ones no module path matches during a directory scan. Refs #15
parseXml handed the keystore= blob to a bare DocumentBuilderFactory, and that blob comes from the device. A DOCTYPE with an external SYSTEM identifier was resolved: pointing it at file:///... made libmvt open that path, and an http:// one turns into a request from the analyst's machine. So a compromised phone could read the analyst's disk, or exfiltrate it via an out-of-band DTD, just by shipping a crafted dumpsys.txt. Disallow DOCTYPE outright, as ManifestParser already does. A real keystore has none, so nothing legitimate is lost, and a hostile one now fails the module loudly and surfaces as ARTIFACT_PARSE_FAILED rather than silently reaching the filesystem.
lsd-cat
force-pushed
the
stream-every-module
branch
from
August 15, 2026 11:45
26e3e13 to
70f2759
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Address #15 and many other fixes