Skip to content

perf: pre-compile APM tag regex filters at startup #1138

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Dogbu-cyber wants to merge 5 commits into
base: main
Choose a base branch
from
Open

perf: pre-compile APM tag regex filters at startup #1138

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
d0743b0
perf: pre-compile APM tag regex filters at startup
Dogbu-cyber Mar 28, 2026
2016ffe
fix: surface invalid regex filter patterns as warnings at startup
Dogbu-cyber Mar 29, 2026
a020ab9
Revert "fix: surface invalid regex filter patterns as warnings at sta…
Dogbu-cyber Mar 30, 2026
89807e9
style: format trace_processor after revert
Dogbu-cyber Mar 30, 2026
b638a2b
fix: address PR feedback on regex filter trace processor
Dogbu-cyber Apr 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions bottlecap/src/bin/bottlecap/main.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1134,9 +1134,10 @@ fn start_trace_agent(
obfuscation_redis_remove_all_args: false,
};

let trace_processor = Arc::new(trace_processor::ServerlessTraceProcessor {
obfuscation_config: Arc::new(obfuscation_config),
});
let trace_processor = Arc::new(trace_processor::ServerlessTraceProcessor::new(
config.as_ref(),
obfuscation_config,
));
Comment on lines +1137 to +1140
Copy link
Copy Markdown
Contributor

@lym953 lym953 Apr 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we test how long this takes on a Lambda function, before vs after? Just to make sure it doesn't increase cold start a lot.

Copy link
Copy Markdown
Contributor Author

@Dogbu-cyber Dogbu-cyber Apr 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ran bench.sh (20 forced cold starts) on a Lambda with 27 configured regex filters, both main and this branch built locally with identical flags (ARCHITECTURE=amd64 FIPS=false ALPINE=0 DEBUG=false). I didn't see any meaningful difference,

Init avg Init min Init max Runtime avg Runtime min Runtime max
main 375 ms 272 ms 416 ms 1603 ms 1496 ms 1707 ms
this branch 374 ms 345 ms 392 ms 1623 ms 1501 ms 1740 ms

Filters configured (DD_APM_FILTER_TAGS_REGEX_REQUIRE — 12 patterns):

env:^(prod|production)$
service:^(api|web|worker|processor|handler|consumer|producer|scheduler|gateway)-.+
http.url:^https?://.*\.(datadog|datadoghq)\.com(/.*)?$
version:^v[0-9]+\.[0-9]+\.[0-9]+(-[a-zA-Z0-9.]+)?$
http.method:^(GET|POST|PUT|DELETE|PATCH|HEAD|OPTIONS)$
http.status_code:^[2345][0-9]{2}$
span.type:^(web|db|cache|rpc|queue|grpc|graphql|serverless)$
resource:^[a-zA-Z][a-zA-Z0-9._/:-]{2,127}$
aws.region:^(us|eu|ap|sa|ca|me|af)-(east|west|north|south|central|northeast|southeast)-[1-9]$
peer.hostname:^[a-zA-Z0-9]([a-zA-Z0-9\-]{0,61}[a-zA-Z0-9])?(\.[a-zA-Z0-9]([a-zA-Z0-9\-]{0,61}[a-zA-Z0-9])?)*$
operation.name:^(aws\.lambda\.(invoke|coldstart)|http\.(request|response)|db\.(query|execute|transaction)|cache\.(get|set|delete))$
runtime:^(nodejs[0-9]+\.x|python[0-9]+\.[0-9]+|java[0-9]+|provided\.al2(023)?)$

Filters configured (DD_APM_FILTER_TAGS_REGEX_REJECT — 15 patterns):

http.url:.*(localhost|127\.0\.0\.1|0\.0\.0\.0|\[::1\]).*
http.url:.*\.(internal|local|corp|test|dev\.internal)\..*
http.url:.*/health(check|z)?(/.*)?$
http.url:.*/ready(ness)?(/.*)?$
http.url:.*/live(ness)?(/.*)?$
http.url:.*/metrics(/.*)?$
http.url:.*/ping(/.*)?$
http.url:.*[?&](debug|trace|test|mock|dryrun)=[^&]*.*
error.message:.*(timeout|connection refused|ECONNREFUSED|ETIMEDOUT|ENOTFOUND|EHOSTUNREACH|ECONNRESET).*
error.type:^(network\..+|dns\..+|ssl\..+|tls\..+|socket\..+)$
span.name:^(health_check|ping|liveness|readiness|metrics_scrape|telemetry_flush)$
peer.service:^(dd-agent|datadog-agent|healthcheck-.+|internal-.+|sidecar-.+)$
resource:^(GET|POST|PUT|DELETE) /(internal|private|admin|_health|_ready)(/.*)?$
aws.operation:^(HeadObject|HeadBucket|ListObjectsV2|GetBucketLocation|DescribeInstances|GetCallerIdentity)$
http.url:.*\.(png|jpg|jpeg|gif|ico|svg|woff|woff2|ttf|eot|css|js\.map)([?#].*)?$

Copy link
Copy Markdown
Contributor

@lym953 lym953 Apr 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add temporary logging to test this specific part? Total duration includes some randomness, and it's hard to see how long this part takes.
Like this:

datadog-lambda-extension/bottlecap/src/bin/bottlecap/main.rs

Lines 412 to 415 in d668142

debug!(
"Datadog Next-Gen Extension ready in {:}ms",
start_time.elapsed().as_millis().to_string()
);

Copy link
Copy Markdown
Contributor Author

@Dogbu-cyber Dogbu-cyber Apr 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Measured with same 27 regex filters

let trace_processor_start = Instant::now();
let trace_processor = Arc::new(trace_processor::ServerlessTraceProcessor::new(
    config.as_ref(),
    obfuscation_config,
));
debug!(
    "TRACE_PROCESSOR | Initialized in {:}ms",
    trace_processor_start.elapsed().as_millis().to_string()
);

Results across 10 cold starts:

Invocation main this branch
1 0ms 11ms
2 0ms 13ms
3 0ms 7ms
4 0ms 16ms
5 0ms 18ms
6 0ms 19ms
7 0ms 7ms
8 0ms 11ms
9 0ms 7ms
10 0ms 9ms
avg 0ms 12ms
range 0ms 7–19ms


let (span_dedup_service, span_dedup_handle) = span_dedup_service::DedupService::new();
tokio::spawn(span_dedup_service.run());
Expand Down
4 changes: 2 additions & 2 deletions bottlecap/src/config/env.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -226,7 +226,7 @@ pub struct EnvConfig {
/// @env `DD_APM_FILTER_TAGS_REQUIRE`
///
/// Space-separated list of key:value tag pairs that spans must match to be kept.
/// Only spans matching at least one of these tags will be sent to Datadog.
/// Only spans matching all of these tags will be sent to Datadog.
/// Example: "env:production service:api-gateway"
#[serde(deserialize_with = "deserialize_apm_filter_tags")]
pub apm_filter_tags_require: Option<Vec<String>>,
Expand All @@ -240,7 +240,7 @@ pub struct EnvConfig {
/// @env `DD_APM_FILTER_TAGS_REGEX_REQUIRE`
///
/// Space-separated list of key:value tag pairs with regex values that spans must match to be kept.
/// Only spans matching at least one of these regex patterns will be sent to Datadog.
/// Only spans matching all of these regex patterns will be sent to Datadog.
/// Example: "env:^prod.*$ service:^api-.*$"
#[serde(deserialize_with = "deserialize_apm_filter_tags")]
pub apm_filter_tags_regex_require: Option<Vec<String>>,
Expand Down
26 changes: 13 additions & 13 deletions bottlecap/src/lifecycle/invocation/processor.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1671,9 +1671,11 @@ mod tests {
// Create trace sender
let trace_sender = Arc::new(SendingTraceProcessor {
appsec: None,
processor: Arc::new(trace_processor::ServerlessTraceProcessor {
obfuscation_config: Arc::new(ObfuscationConfig::new().expect("Failed to create ObfuscationConfig")),
}),
processor: Arc::new(trace_processor::ServerlessTraceProcessor::new(
config.as_ref(),
ObfuscationConfig::new()
.expect("Failed to create ObfuscationConfig"),
)),
trace_tx: tokio::sync::mpsc::channel(1).0,
stats_generator: Arc::new(StatsGenerator::new(stats_concentrator_handle)),
});
Expand Down Expand Up @@ -1779,11 +1781,10 @@ mod tests {
tokio::spawn(stats_concentrator_service.run());
let trace_sender = Arc::new(SendingTraceProcessor {
appsec: None,
processor: Arc::new(trace_processor::ServerlessTraceProcessor {
obfuscation_config: Arc::new(
ObfuscationConfig::new().expect("Failed to create ObfuscationConfig"),
),
}),
processor: Arc::new(trace_processor::ServerlessTraceProcessor::new(
config.as_ref(),
ObfuscationConfig::new().expect("Failed to create ObfuscationConfig"),
)),
trace_tx: tokio::sync::mpsc::channel(1).0,
stats_generator: Arc::new(StatsGenerator::new(stats_concentrator_handle)),
});
Expand Down Expand Up @@ -2174,11 +2175,10 @@ mod tests {
tokio::spawn(stats_concentrator_service.run());
let trace_sender = Arc::new(SendingTraceProcessor {
appsec: None,
processor: Arc::new(trace_processor::ServerlessTraceProcessor {
obfuscation_config: Arc::new(
ObfuscationConfig::new().expect("Failed to create ObfuscationConfig"),
),
}),
processor: Arc::new(trace_processor::ServerlessTraceProcessor::new(
config.as_ref(),
ObfuscationConfig::new().expect("Failed to create ObfuscationConfig"),
)),
trace_tx,
stats_generator: Arc::new(StatsGenerator::new(stats_concentrator_handle)),
});
Expand Down
Loading
Loading