fix: real-time axis, non-UTF-8 file support, and OOM on large logs#12
Open
kaklik wants to merge 1 commit into
Open
fix: real-time axis, non-UTF-8 file support, and OOM on large logs#12kaklik wants to merge 1 commit into
kaklik wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
UnicodeDecodeError fix: log files are now opened with
encoding="ascii", errors="replace", so non-ASCII bytes in file headers no longer crash the parser.Real-time axis: the parser now reads the
$TIMErecord present in AIRDOS04C fw 2.x logs and extracts thertc_unix_offsetfield (= Unix timestamp − RTC counter). All timestamps intime_axisand telemetry are converted to Unix epoch.PlotCanvasdetects thehas_real_timemetadata flag and switches topg.DateAxisItemso graphs show actual UTC date/time instead of relative minutes. ASmartDateAxisItemsubclass adds the calendar date to the first tick of each new day, so the date is always visible at midnight transitions (or at the first visible tick after opening a file). Logs without a$TIMErecord (old format) are unaffected.OOM fix for large files:
AirdosV2LogParsernow runs a fast pre-scan pass over the file to find the highest channel index used in$Eand$STOPrecords, then allocateshistand everycurrent_histcopy only to that size withdtype=int32. For a file with max channel 7 647 and 11 001 records, thespectral_matrixshrinks from ~5.8 GB (65 536 ch × int64) to ~320 MB (7 648 ch × int32), making files that were previously killed by the OOM killer parseable in under a second.