feat(structured): add structured traceroute + comprehensive IP enrichment and UI Updates - #343
Closed
WilhelmZA wants to merge 107 commits into
Closed
feat(structured): add structured traceroute + comprehensive IP enrichment and UI Updates#343WilhelmZA wants to merge 107 commits into
WilhelmZA wants to merge 107 commits into
Conversation
- Split type checking into two separate conditions for clarity - Add early return when output contains non-string objects - Ensure structured output like MikrotikBGPRouteTable passes through unchanged - Prevent AttributeError on .strip() for tuple containing structured data Resolves remaining crashes when processing already-structured BGP route data.
- Add IPv6 address pattern matching alongside IPv4 - Extract SENT count from traceroute lines to prioritize complete data - Keep lines with highest SENT count (final results) instead of first occurrence - Maintain original hop order while using best data for each IP - Prefer non-timeout results when SENT counts are equal Fixes issues where IPv4 showed early results (SENT=1) instead of final (SENT=3) and IPv6 showed all intermediate steps instead of final consolidated results.
- Fix MED field showing N/A when value is 0 (now correctly shows 0) - Hide Originator field when empty instead of showing N/A - Hide Age field when not available instead of showing N/A - Add HideableField component for fields that should be hidden when empty - Update string output formatting to skip hidden fields in text export Resolves display issues where valid zero values and unavailable fields were showing as N/A instead of proper formatting or being hidden.
- Remove custom __init__ from HuaweiBGPRouteTable class - Use standard BGPRouteTable inheritance like MikroTik does - Add BGPRoute import and instantiate BGPRoute objects in bgp_table() - Enable proper Pydantic validation for both table and routes - Ensure consistent behavior across all BGP parsers - Update .gitignore to exclude dev-build.sh and dev-docker/
…riable naming - Add BGPRoute import to Juniper, Arista, and FRR parsers - Change all parsers to instantiate BGPRoute objects instead of raw dictionaries - Harmonize variable naming across all BGP parsers: - Use 'route_data' for dictionary variable - Use 'routes' for output list - Use 'route' for loop variable - Ensure consistent validation behavior across MikroTik, Huawei, Juniper, Arista, and FRR - All parsers now properly invoke BGP table class validation
- Add BGPRoute import and export in hyperglass.models.data.__init__.py - This allows BGP parsers to properly import BGPRoute for validation - Resolves ImportError when starting hyperglass application
MAJOR NEW ARCHITECTURE - STRUCTURED TRACEROUTE: - Complete rewrite of traceroute data processing with structured output - Dedicated TracerouteResult and TracerouteHop data models - Platform-specific parsers with unified output format - Rich metadata including ASN, organization, country, and prefix information - AS path visualization with organization names in React Flow charts SUPPORTED PLATFORMS: - TraceroutePluginMikrotik: Handles MikroTik's complex multi-table format * Progressive statistics parsing with deduplication * Timeout hop handling and continuation line processing * Loss percentage and RTT statistics extraction - TraceroutePluginHuawei: Unix-style traceroute format parser * Standard hop_number ip_address rtt format support * Timeout hop detection with * notation * Automatic cleanup of excessive trailing timeouts COMPREHENSIVE IP ENRICHMENT SYSTEM: - Offline enrichment using BGP.tools bulk data (1.3M+ CIDR entries) - PeeringDB integration for IXP detection and ASN organization data - Ultra-fast pickle cache system with combined data files - Integer-based bitwise IP matching for maximum performance - Bulk ASN organization lookup capabilities - Private/reserved IP handling with AS0 fallbacks - Country code mapping from ASN database - Graceful fallbacks for missing enrichment data FRONTEND ENHANCEMENTS: - New traceroute table components with consistent formatting - Enhanced AS path visualization with organization names - Improved copy-to-clipboard functionality with structured data - Unified table styling across BGP and traceroute results - Better error handling and loading states CONCURRENT PROCESSING INFRASTRUCTURE: - Thread executor implementation for blocking I/O operations - Query deduplication system to prevent resource conflicts - Non-blocking Redis cache operations using asyncio executors - Event coordination for waiting requests - Background cleanup for completed operations - Prevents website hangs during long-running queries PLUGIN ARCHITECTURE IMPROVEMENTS: - Platform-aware plugin system with proper execution restrictions - Enhanced MikroTik garbage output cleaning - IP enrichment plugins for both BGP routes and traceroute - Conditional plugin execution based on platform detection - Proper async/sync plugin method handling CRITICAL BUG FIXES: - Fixed double AS prefix bug (ASAS123456 → AS123456) - Resolved TracerouteHop avg_rtt field/property conflicts - Corrected Huawei traceroute source field validation - Fixed plugin platform restriction enforcement - Eliminated blocking I/O causing UI freezes - Proper timeout and empty response caching prevention - Enhanced private IP range detection and handling PERFORMANCE OPTIMIZATIONS: - Pickle cache system reduces startup time from seconds to milliseconds - Bulk processing for ASN organization lookups - Simplified IXP detection using single PeeringDB API call - Efficient CIDR network sorting and integer-based lookups - Reduced external API calls by 90%+ - Optimized memory usage for large datasets API & ROUTING ENHANCEMENTS: - Enhanced API routes with proper error handling - Improved middleware for concurrent request processing - Better state management and event handling - Enhanced task processing with thread pool execution This represents a complete transformation of hyperglass traceroute capabilities, moving from basic text output to rich, structured data with comprehensive network intelligence and concurrent processing support.
WilhelmZA
force-pushed
the
structured
branch
from
September 30, 2025 14:43
a259abf to
0398966
Compare
Summary: - Add structured traceroute support with comprehensive IP enrichment (ASN/org/RDNS). - Improve MikroTik traceroute cleaning and aggregation; collapse repeated tables into a single representative table. - Enhance traceroute logging for visibility and add traceroute-specific cleaning helpers. - Add/adjust IP enrichment plugins and BGP/traceroute enrichment integrations. - UI updates for traceroute output and path visualization; update docs and configuration for structured output. This commit squashes changes from 'structured-dev' into a single release commit.
Author
|
@jdhall75 I have finished my commits for the moment and ready for review. This commit covers all the aspects of the pull request opened by Carlos: #340 We should consider doing this as one pull to main @CarlosSuporteISP ? |
WilhelmZA
changed the title
Collaborator
|
So.. for future reference. PRs should be small and focused on a particular function of the application. It's considered the #1 rule for PRs. I know with AI it's easy to get carried away and make broad sweeping changes, but a human has to review it. I appreciate the all the contributions and I am sure many others will too. This is just something to keep in mind for the next one. As soon as we get some feedback from @CarlosSuporteISP I'll start the review and close out one of the PRs. |
jdhall75
self-requested a review
October 1, 2025 00:48
hcaldicott
reviewed
Oct 4, 2025
There was a problem hiding this comment.
Just a heads up that this PR contains modifications to the compose.yaml file which I don't feel should be included. I have only had a cursory glance at this so far in my own testing but a much more thorough review is needed with this in mind.
I don't have much feedback in the way of the PR as a whole, as I'm completely new to Hyperglass and I am using this PR in order to stand up the project for evaluation. Apologies for chiming in our of no-where with this comment otherwise :)
hcaldicott
reviewed
Oct 4, 2025
MAJOR ENHANCEMENTS: IP Enrichment Service (hyperglass/external/ip_enrichment.py): - Increase IXP data cache duration from 24 hours to 7 days (604800s) for better performance - Fix critical cache refresh logic: ensure_data_loaded() now properly checks expiry before using existing pickle files - Remove 'force' refresh parameters from public APIs and admin endpoints to prevent potential abuse/DDOS - Implement automatic refresh based on file timestamps and cache duration - Add comprehensive debug logging gated by Settings.debug throughout the module - Clean up verbose comments and improve code readability - Update configuration model to enforce 7-day minimum cache timeout MikroTik Traceroute Processing: - Refactor trace_route_mikrotik plugin to use garbage cleaner before structured parsing - Only log raw router output when Settings.debug is enabled to reduce log verbosity - Simplify MikrotikTracerouteTable parser to expect pre-cleaned input from garbage cleaner - Remove complex multi-table detection, format detection, and deduplication logic (handled by cleaner) - Add concise debug messages for processing decisions and configuration states Traceroute IP Enrichment (traceroute_ip_enrichment.py): - Implement concurrent reverse DNS lookups using asyncio.to_thread and asyncio.gather - Add async wrapper for reverse DNS with proper error handling and fallbacks - Significant performance improvement for multi-hop traceroutes (parallel vs sequential DNS) - Proper debug logging gates: only detailed logs when Settings.debug=True - Upgrade operational messages to log.info level (start/completion status) - Maintain compatibility with different event loop contexts and runtime environments Configuration Updates: - Update structured.ip_enrichment.cache_timeout default to 604800 seconds - Update documentation to reflect new cache defaults and behavior - Remove force refresh options from admin API endpoints MIGRATION NOTES: - Operators should ensure /etc/hyperglass/ip_enrichment directory is writable - Any code relying on force refresh parameters must be updated - Monitor logs for automatic refresh behavior and performance improvements - The 7-day cache significantly reduces PeeringDB API load PERFORMANCE BENEFITS: - Faster traceroute enrichment due to concurrent DNS lookups - Reduced external API calls with longer IXP cache duration - More reliable refresh logic prevents stale cache usage - Cleaner, more focused debug output when debug mode is disabled TECHNICAL DETAILS: - Uses asyncio.to_thread for non-blocking DNS operations - Implements process-wide file locking for safe concurrent cache updates - Robust fallbacks for various asyncio execution contexts - Maintains backward compatibility while improving performance FILES MODIFIED: - hyperglass/external/ip_enrichment.py - hyperglass/models/config/structured.py - hyperglass/api/routes.py - hyperglass/plugins/_builtin/trace_route_mikrotik.py - hyperglass/models/parsing/mikrotik.py - hyperglass/plugins/_builtin/traceroute_ip_enrichment.py - docs/pages/configuration/config/structured-output.mdx
Author
|
Manually cleaned up and reviewed the following files, fixing some bugs and issues there was with edge cases and found by users:
Details: 4a10576 |
… streamline configurations
- Remove ip_enrichment_status and ip_enrichment_refresh endpoints from API routes - Remove unused imports and handlers from API initialization - Background refresh functionality in query endpoint remains intact - Cleanup dead code that had no UI or client integration
IP Enrichment Changes: - Remove PeeringDB dependency and file caching infrastructure - Implement real-time BGP.TOOLS WHOIS API integration (port 43) - Simplify IPEnrichmentService with bulk query support - Add comprehensive error handling and logging - Reduce module from 1,545 to 641 lines (58% reduction) - Update documentation to reflect BGP.TOOLS-only approach Validation System Fixes: - Fix Pydantic v2 model validator signatures across multiple classes - Use Field(default_factory=...) pattern to prevent premature instantiation - Resolve KeyError: None in ThemeColors.validate_colors validator - Fix Credential.validate_credential method signature for 'after' mode - Ensure proper validation context for all model validators MikroTik Improvements: - Add intelligent retry logic for empty BGP route responses - Retry up to 2 times when execution time < 5 seconds - Include 2-second delay between retries for device stabilization Security Enhancements: - Sanitize connection error messages to remove sensitive network information - Remove 'Device settings: platform IP:port' from user-facing errors - Apply sanitization to DeviceTimeout, ScrapeError, AuthError, RestError - Provide generic fallback messages for empty errors UI Improvements: - Hide 'None' group filter when no directive groups exist - Filter out directives with null/empty names from query type dropdown - Improve UI cleanliness and user experience Configuration Updates: - Clean up structured output configuration comments - Remove deprecated cache_timeout and refresh settings - Update model field definitions for consistency
- Updated FRR traceroute parser to handle new simplified output format - Added comprehensive regex patterns for various traceroute line formats - Improved hop parsing with support for timeouts, hostnames, and multiple RTTs - Enhanced IP address validation and hostname handling - Added detailed debug logging for troubleshooting - Cleaned up duplicate hop handling and sorting - Maintained backward compatibility with existing traceroute output - Removed experimental MTR functionality that had timeout issues - Restored all BGP directives (BGP, BGPRoute, BGPCommunity, BGPASPath) - Streamlined plugin to focus on reliable traditional traceroute parsing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This pull request implements a complete structured traceroute subsystem and offline-first IP enrichment pipeline for Hyperglass. It transitions traceroute output from raw text dumps into typed, enriched data objects and adds end-to-end UI support to render and interact with structured traceroute results.
The change set includes parser implementations for multiple vendor formats, a fast offline enrichment backend (ASN/org/RDNS/RPKI), UI components to display enriched traceroute data, concurrency and caching improvements, and documentation/sample configuration to operate the feature.
This pull request includes the amazing work of https://github.com/CarlosSuporteISP who introduced structured bgp route for a number of platforms, this expands his existing PR: #340 with enhancements to structured traceroute and ip_enrichment.
Major components
Structured traceroute architecture
TracerouteResultandTracerouteHopwith rich per-hop metadata (ip, asn, org, rdns, prefix, rpkistatus, country, loss, sent, last, avg, best, worst, stddev, status).IP enrichment system
Supported platforms (parsers)
Frontend enhancements
traceroute-table,traceroute-fields,traceroute-cell.Infrastructure improvements
Files changed (summary)
Rough totals: ~94 files changed; 13,602 insertions and 1,912 deletions.
Notable additions:
hyperglass/models/data/traceroute.py(TracerouteResult, TracerouteHop models)hyperglass/models/parsing/mikrotik.py,huawei.py,traceroute.pyhyperglass/plugins/_builtin/trace_route_*.py(vendor traceroute plugins)hyperglass/plugins/_builtin/traceroute_ip_enrichment.pyhyperglass/external/ip_enrichment.py(enrichment backend, cache ingestion)hyperglass/ui/components/output/traceroute-*.tsxdocs/pages/...and.samples/Notable modifications:
hyperglass/plugins/_builtin/mikrotik_garbage_output.py— extensive cleaner and aggregation improvementshyperglass/main.py— plugin registration and startup wiringhyperglass/models/config/params.py,structured.py— new configuration optionsSee the branch diff for the full file list and exact changes.
Motivation and context
The prior traceroute implementation returned only raw text. That limited:
This feature introduces typed structured traceroute outputs combined with an offline-first enrichment strategy to provide accurate, fast, and stable metadata while significantly reducing network traffic to external services.
It transforms Hyperglass from a plain looking-glass utility to a structured, analysable traceroute platform with improved diagnostics and visualization.
Detailed technical description
Traceroute data model
The
TracerouteResultcontains metadata about the probe and an ordered list ofTracerouteHopobjects. Each hop includes network-level fields and enrichment fields. The model is serializable to JSON and validated with the project’s existing validation layer.Fields include (non-exhaustive):
The model intentionally keeps hop order fixed (hop position is significant). Parsers must not deduplicate by IP address alone since the same IP can appear at different hops.
Vendor parsers & normalization
Each vendor parser parses raw text and returns an ordered list of hop rows with the above fields. The normalization layer coerces vendor-specific differences (different column order, missing fields) into the unified hop model.
MikroTik specifics:
SENTcount (prefers non-timeout rows on ties, and later tables on equal SENT). Excessive trailing timeouts are collapsed into a single aggregation line like “... (N more timeout hops)”.Huawei specifics:
IP enrichment pipeline
The enrichment component is implemented as an offline-first service that:
The enrichment plugin batches lookups and uses a thread pool so the request pipeline remains responsive. Query deduplication ensures multiple hops with the same IP don’t trigger duplicate lookups concurrently.
Plugin & request lifecycle changes
structuredtop-level configuration acts as the authoritative switch; per-feature tri-state flags allow opt-outs (for enrichment subfeatures like RPKI or RDNS).Frontend & UX changes
Testing performed
Environment: Linux (Ubuntu/Debian), Python 3.11+, browsers: Chrome/Firefox/Safari.
Parser tests:
Enrichment tests:
Manual validation:
Load/Concurrency:
Automated:
Behavioral & operational impact
Configuration & deployment notes
hyperglass/models/config/params.pyandhyperglass/models/config/structured.pyas needed to enable or change enrichment behavior.Commit / PR context
This branch represents a single large feature release and can be squashed into one comprehensive commit for the upstream PR. The PR message included with the branch accurately reflects motivation, testing, and scope; this document provides a detailed technical explanation and operational notes.