Update Spock Tests to run with `STRICT_TEST` mode by default by mhlidd · Pull Request #11032 · DataDog/dd-trace-java

mhlidd · 2026-04-01T20:02:33Z

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Format the title according to the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any other useful labels
Avoid using close, fix, or any linking keywords when referencing an issue
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, migration, or deletion
Update public documentation with any new configuration flags or behaviors

Jira ticket: [PROJ-IDENT]

Note: Once your PR is ready to merge, add it to the merge queue by commenting /merge. /merge -c cancels the queue request. /merge -f --reason "reason" skips all merge queue checks; please use this judiciously, as some checks do not run at the PR-level. For more information, see this doc.

pr-commenter · 2026-04-05T14:26:44Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	mhlidd/run_spock_with_strict
git_commit_date	1775155088	1775399846
git_commit_sha	`ca975c2`	`5ba6f69`
release_version	1.61.0-SNAPSHOT~ca975c2722	1.61.0-SNAPSHOT~5ba6f6958e

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1775401611	1775401611
ci_job_id	1568814453	1568814453
ci_pipeline_id	106117985	106117985
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-chux78se 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-chux78se 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 57 metrics, 14 unstable metrics.

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.61.0-SNAPSHOT~5ba6f6958e, baseline=1.61.0-SNAPSHOT~ca975c2722

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.061 s) : 0, 1061126
Total [baseline] (8.829 s) : 0, 8829256
Agent [candidate] (1.056 s) : 0, 1055562
Total [candidate] (8.864 s) : 0, 8863979
section iast
Agent [baseline] (1.223 s) : 0, 1223260
Total [baseline] (9.549 s) : 0, 9548726
Agent [candidate] (1.219 s) : 0, 1219290
Total [candidate] (9.533 s) : 0, 9532794

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.061 s	-
Agent	iast	1.223 s	162.134 ms (15.3%)
Total	tracing	8.829 s	-
Total	iast	9.549 s	719.47 ms (8.1%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.056 s	-
Agent	iast	1.219 s	163.728 ms (15.5%)
Total	tracing	8.864 s	-
Total	iast	9.533 s	668.815 ms (7.5%)

gantt
    title insecure-bank - break down per module: candidate=1.61.0-SNAPSHOT~5ba6f6958e, baseline=1.61.0-SNAPSHOT~ca975c2722

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.207 ms) : 0, 1207
crashtracking [candidate] (1.233 ms) : 0, 1233
BytebuddyAgent [baseline] (635.327 ms) : 0, 635327
BytebuddyAgent [candidate] (631.458 ms) : 0, 631458
AgentMeter [baseline] (29.716 ms) : 0, 29716
AgentMeter [candidate] (29.455 ms) : 0, 29455
GlobalTracer [baseline] (250.156 ms) : 0, 250156
GlobalTracer [candidate] (248.792 ms) : 0, 248792
AppSec [baseline] (32.295 ms) : 0, 32295
AppSec [candidate] (31.951 ms) : 0, 31951
Debugger [baseline] (59.657 ms) : 0, 59657
Debugger [candidate] (59.06 ms) : 0, 59060
Remote Config [baseline] (602.696 µs) : 0, 603
Remote Config [candidate] (598.105 µs) : 0, 598
Telemetry [baseline] (8.169 ms) : 0, 8169
Telemetry [candidate] (8.076 ms) : 0, 8076
Flare Poller [baseline] (7.526 ms) : 0, 7526
Flare Poller [candidate] (8.835 ms) : 0, 8835
section iast
crashtracking [baseline] (1.212 ms) : 0, 1212
crashtracking [candidate] (1.238 ms) : 0, 1238
BytebuddyAgent [baseline] (800.897 ms) : 0, 800897
BytebuddyAgent [candidate] (798.831 ms) : 0, 798831
AgentMeter [baseline] (11.395 ms) : 0, 11395
AgentMeter [candidate] (11.356 ms) : 0, 11356
GlobalTracer [baseline] (239.005 ms) : 0, 239005
GlobalTracer [candidate] (238.094 ms) : 0, 238094
IAST [baseline] (26.642 ms) : 0, 26642
IAST [candidate] (25.744 ms) : 0, 25744
AppSec [baseline] (30.784 ms) : 0, 30784
AppSec [candidate] (30.802 ms) : 0, 30802
Debugger [baseline] (57.704 ms) : 0, 57704
Debugger [candidate] (57.895 ms) : 0, 57895
Remote Config [baseline] (540.389 µs) : 0, 540
Remote Config [candidate] (522.723 µs) : 0, 523
Telemetry [baseline] (14.572 ms) : 0, 14572
Telemetry [candidate] (14.536 ms) : 0, 14536
Flare Poller [baseline] (4.032 ms) : 0, 4032
Flare Poller [candidate] (3.942 ms) : 0, 3942

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.61.0-SNAPSHOT~5ba6f6958e, baseline=1.61.0-SNAPSHOT~ca975c2722

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.054 s) : 0, 1053676
Total [baseline] (11.02 s) : 0, 11019634
Agent [candidate] (1.055 s) : 0, 1054624
Total [candidate] (10.955 s) : 0, 10955227
section appsec
Agent [baseline] (1.246 s) : 0, 1246490
Total [baseline] (11.152 s) : 0, 11151809
Agent [candidate] (1.25 s) : 0, 1250050
Total [candidate] (11.005 s) : 0, 11005152
section iast
Agent [baseline] (1.226 s) : 0, 1225656
Total [baseline] (11.315 s) : 0, 11315416
Agent [candidate] (1.228 s) : 0, 1228500
Total [candidate] (11.246 s) : 0, 11246172
section profiling
Agent [baseline] (1.182 s) : 0, 1182101
Total [baseline] (10.983 s) : 0, 10983120
Agent [candidate] (1.179 s) : 0, 1178905
Total [candidate] (11.128 s) : 0, 11128432

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.054 s	-
Agent	appsec	1.246 s	192.814 ms (18.3%)
Agent	iast	1.226 s	171.98 ms (16.3%)
Agent	profiling	1.182 s	128.425 ms (12.2%)
Total	tracing	11.02 s	-
Total	appsec	11.152 s	132.175 ms (1.2%)
Total	iast	11.315 s	295.782 ms (2.7%)
Total	profiling	10.983 s	-36.514 ms (-0.3%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.055 s	-
Agent	appsec	1.25 s	195.426 ms (18.5%)
Agent	iast	1.228 s	173.876 ms (16.5%)
Agent	profiling	1.179 s	124.281 ms (11.8%)
Total	tracing	10.955 s	-
Total	appsec	11.005 s	49.925 ms (0.5%)
Total	iast	11.246 s	290.945 ms (2.7%)
Total	profiling	11.128 s	173.205 ms (1.6%)

gantt
    title petclinic - break down per module: candidate=1.61.0-SNAPSHOT~5ba6f6958e, baseline=1.61.0-SNAPSHOT~ca975c2722

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.206 ms) : 0, 1206
crashtracking [candidate] (1.256 ms) : 0, 1256
BytebuddyAgent [baseline] (631.34 ms) : 0, 631340
BytebuddyAgent [candidate] (630.93 ms) : 0, 630930
AgentMeter [baseline] (29.446 ms) : 0, 29446
AgentMeter [candidate] (29.49 ms) : 0, 29490
GlobalTracer [baseline] (248.494 ms) : 0, 248494
GlobalTracer [candidate] (248.262 ms) : 0, 248262
AppSec [baseline] (31.922 ms) : 0, 31922
AppSec [candidate] (32.002 ms) : 0, 32002
Debugger [baseline] (59.858 ms) : 0, 59858
Debugger [candidate] (59.889 ms) : 0, 59889
Remote Config [baseline] (604.917 µs) : 0, 605
Remote Config [candidate] (605.503 µs) : 0, 606
Telemetry [baseline] (8.086 ms) : 0, 8086
Telemetry [candidate] (7.96 ms) : 0, 7960
Flare Poller [baseline] (6.666 ms) : 0, 6666
Flare Poller [candidate] (8.031 ms) : 0, 8031
section appsec
crashtracking [baseline] (1.2 ms) : 0, 1200
crashtracking [candidate] (1.222 ms) : 0, 1222
BytebuddyAgent [baseline] (659.819 ms) : 0, 659819
BytebuddyAgent [candidate] (660.796 ms) : 0, 660796
AgentMeter [baseline] (12.092 ms) : 0, 12092
AgentMeter [candidate] (12.079 ms) : 0, 12079
GlobalTracer [baseline] (248.487 ms) : 0, 248487
GlobalTracer [candidate] (250.231 ms) : 0, 250231
IAST [baseline] (24.502 ms) : 0, 24502
IAST [candidate] (24.783 ms) : 0, 24783
AppSec [baseline] (184.509 ms) : 0, 184509
AppSec [candidate] (185.175 ms) : 0, 185175
Debugger [baseline] (66.538 ms) : 0, 66538
Debugger [candidate] (66.814 ms) : 0, 66814
Remote Config [baseline] (628.167 µs) : 0, 628
Remote Config [candidate] (599.941 µs) : 0, 600
Telemetry [baseline] (8.728 ms) : 0, 8728
Telemetry [candidate] (8.486 ms) : 0, 8486
Flare Poller [baseline] (3.621 ms) : 0, 3621
Flare Poller [candidate] (3.553 ms) : 0, 3553
section iast
crashtracking [baseline] (1.186 ms) : 0, 1186
crashtracking [candidate] (1.233 ms) : 0, 1233
BytebuddyAgent [baseline] (802.197 ms) : 0, 802197
BytebuddyAgent [candidate] (803.304 ms) : 0, 803304
AgentMeter [baseline] (11.396 ms) : 0, 11396
AgentMeter [candidate] (11.503 ms) : 0, 11503
GlobalTracer [baseline] (238.587 ms) : 0, 238587
GlobalTracer [candidate] (240.144 ms) : 0, 240144
IAST [baseline] (25.931 ms) : 0, 25931
IAST [candidate] (26.171 ms) : 0, 26171
AppSec [baseline] (31.005 ms) : 0, 31005
AppSec [candidate] (32.745 ms) : 0, 32745
Debugger [baseline] (59.652 ms) : 0, 59652
Debugger [candidate] (57.447 ms) : 0, 57447
Remote Config [baseline] (531.483 µs) : 0, 531
Remote Config [candidate] (528.852 µs) : 0, 529
Telemetry [baseline] (14.833 ms) : 0, 14833
Telemetry [candidate] (14.763 ms) : 0, 14763
Flare Poller [baseline] (3.872 ms) : 0, 3872
Flare Poller [candidate] (3.792 ms) : 0, 3792
section profiling
crashtracking [baseline] (1.178 ms) : 0, 1178
crashtracking [candidate] (1.186 ms) : 0, 1186
BytebuddyAgent [baseline] (689.867 ms) : 0, 689867
BytebuddyAgent [candidate] (688.458 ms) : 0, 688458
AgentMeter [baseline] (9.108 ms) : 0, 9108
AgentMeter [candidate] (9.078 ms) : 0, 9078
GlobalTracer [baseline] (206.698 ms) : 0, 206698
GlobalTracer [candidate] (206.226 ms) : 0, 206226
AppSec [baseline] (32.434 ms) : 0, 32434
AppSec [candidate] (32.381 ms) : 0, 32381
Debugger [baseline] (65.038 ms) : 0, 65038
Debugger [candidate] (65.205 ms) : 0, 65205
Remote Config [baseline] (579.281 µs) : 0, 579
Remote Config [candidate] (566.28 µs) : 0, 566
Telemetry [baseline] (7.825 ms) : 0, 7825
Telemetry [candidate] (7.774 ms) : 0, 7774
Flare Poller [baseline] (4.417 ms) : 0, 4417
Flare Poller [candidate] (3.506 ms) : 0, 3506
ProfilingAgent [baseline] (93.724 ms) : 0, 93724
ProfilingAgent [candidate] (93.452 ms) : 0, 93452
Profiling [baseline] (94.295 ms) : 0, 94295
Profiling [candidate] (94.013 ms) : 0, 94013

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	mhlidd/run_spock_with_strict
git_commit_date	1775155088	1775399846
git_commit_sha	`ca975c2`	`5ba6f69`
release_version	1.61.0-SNAPSHOT~ca975c2722	1.61.0-SNAPSHOT~5ba6f6958e

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1775402212	1775402212
ci_job_id	1568814454	1568814454
ci_pipeline_id	106117985	106117985
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-u6nhgjfi 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-u6nhgjfi 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 19 metrics, 17 unstable metrics.

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.61.0-SNAPSHOT~5ba6f6958e, baseline=1.61.0-SNAPSHOT~ca975c2722
    dateFormat X
    axisFormat %s
section baseline
no_agent (18.273 ms) : 18086, 18460
.   : milestone, 18273,
appsec (18.742 ms) : 18553, 18931
.   : milestone, 18742,
code_origins (17.935 ms) : 17760, 18109
.   : milestone, 17935,
iast (17.915 ms) : 17738, 18091
.   : milestone, 17915,
profiling (19.305 ms) : 19109, 19500
.   : milestone, 19305,
tracing (17.81 ms) : 17632, 17988
.   : milestone, 17810,
section candidate
no_agent (19.019 ms) : 18819, 19218
.   : milestone, 19019,
appsec (18.654 ms) : 18465, 18843
.   : milestone, 18654,
code_origins (17.695 ms) : 17522, 17868
.   : milestone, 17695,
iast (17.886 ms) : 17710, 18063
.   : milestone, 17886,
profiling (18.935 ms) : 18750, 19121
.   : milestone, 18935,
tracing (17.701 ms) : 17529, 17873
.   : milestone, 17701,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	18.273 ms [18.086 ms, 18.46 ms]	-
appsec	18.742 ms [18.553 ms, 18.931 ms]	469.331 µs (2.6%)
code_origins	17.935 ms [17.76 ms, 18.109 ms]	-338.093 µs (-1.9%)
iast	17.915 ms [17.738 ms, 18.091 ms]	-358.309 µs (-2.0%)
profiling	19.305 ms [19.109 ms, 19.5 ms]	1.032 ms (5.6%)
tracing	17.81 ms [17.632 ms, 17.988 ms]	-463.049 µs (-2.5%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	19.019 ms [18.819 ms, 19.218 ms]	-
appsec	18.654 ms [18.465 ms, 18.843 ms]	-364.728 µs (-1.9%)
code_origins	17.695 ms [17.522 ms, 17.868 ms]	-1.324 ms (-7.0%)
iast	17.886 ms [17.71 ms, 18.063 ms]	-1.132 ms (-6.0%)
profiling	18.935 ms [18.75 ms, 19.121 ms]	-83.287 µs (-0.4%)
tracing	17.701 ms [17.529 ms, 17.873 ms]	-1.318 ms (-6.9%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.61.0-SNAPSHOT~5ba6f6958e, baseline=1.61.0-SNAPSHOT~ca975c2722
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.247 ms) : 1234, 1259
.   : milestone, 1247,
iast (3.279 ms) : 3239, 3319
.   : milestone, 3279,
iast_FULL (5.973 ms) : 5912, 6034
.   : milestone, 5973,
iast_GLOBAL (3.731 ms) : 3676, 3785
.   : milestone, 3731,
profiling (2.182 ms) : 2162, 2201
.   : milestone, 2182,
tracing (1.926 ms) : 1910, 1943
.   : milestone, 1926,
section candidate
no_agent (1.215 ms) : 1204, 1226
.   : milestone, 1215,
iast (3.38 ms) : 3332, 3428
.   : milestone, 3380,
iast_FULL (6.089 ms) : 6028, 6150
.   : milestone, 6089,
iast_GLOBAL (3.601 ms) : 3541, 3660
.   : milestone, 3601,
profiling (2.425 ms) : 2399, 2451
.   : milestone, 2425,
tracing (1.885 ms) : 1870, 1900
.   : milestone, 1885,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.247 ms [1.234 ms, 1.259 ms]	-
iast	3.279 ms [3.239 ms, 3.319 ms]	2.032 ms (163.0%)
iast_FULL	5.973 ms [5.912 ms, 6.034 ms]	4.726 ms (379.1%)
iast_GLOBAL	3.731 ms [3.676 ms, 3.785 ms]	2.484 ms (199.2%)
profiling	2.182 ms [2.162 ms, 2.201 ms]	935.011 µs (75.0%)
tracing	1.926 ms [1.91 ms, 1.943 ms]	679.342 µs (54.5%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.215 ms [1.204 ms, 1.226 ms]	-
iast	3.38 ms [3.332 ms, 3.428 ms]	2.165 ms (178.3%)
iast_FULL	6.089 ms [6.028 ms, 6.15 ms]	4.874 ms (401.3%)
iast_GLOBAL	3.601 ms [3.541 ms, 3.66 ms]	2.386 ms (196.4%)
profiling	2.425 ms [2.399 ms, 2.451 ms]	1.21 ms (99.7%)
tracing	1.885 ms [1.87 ms, 1.9 ms]	670.175 µs (55.2%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	mhlidd/run_spock_with_strict
git_commit_date	1775155088	1775399846
git_commit_sha	`ca975c2`	`5ba6f69`
release_version	1.61.0-SNAPSHOT~ca975c2722	1.61.0-SNAPSHOT~5ba6f6958e

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1775401858	1775401858
ci_job_id	1568814455	1568814455
ci_pipeline_id	106117985	106117985
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-1-dpqavi88 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-1-dpqavi88 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.61.0-SNAPSHOT~5ba6f6958e, baseline=1.61.0-SNAPSHOT~ca975c2722
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.48 s) : 15480000, 15480000
.   : milestone, 15480000,
appsec (14.691 s) : 14691000, 14691000
.   : milestone, 14691000,
iast (18.375 s) : 18375000, 18375000
.   : milestone, 18375000,
iast_GLOBAL (17.579 s) : 17579000, 17579000
.   : milestone, 17579000,
profiling (15.114 s) : 15114000, 15114000
.   : milestone, 15114000,
tracing (14.955 s) : 14955000, 14955000
.   : milestone, 14955000,
section candidate
no_agent (15.563 s) : 15563000, 15563000
.   : milestone, 15563000,
appsec (14.76 s) : 14760000, 14760000
.   : milestone, 14760000,
iast (18.691 s) : 18691000, 18691000
.   : milestone, 18691000,
iast_GLOBAL (18.669 s) : 18669000, 18669000
.   : milestone, 18669000,
profiling (15.127 s) : 15127000, 15127000
.   : milestone, 15127000,
tracing (15.101 s) : 15101000, 15101000
.   : milestone, 15101000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.48 s [15.48 s, 15.48 s]	-
appsec	14.691 s [14.691 s, 14.691 s]	-789.0 ms (-5.1%)
iast	18.375 s [18.375 s, 18.375 s]	2.895 s (18.7%)
iast_GLOBAL	17.579 s [17.579 s, 17.579 s]	2.099 s (13.6%)
profiling	15.114 s [15.114 s, 15.114 s]	-366.0 ms (-2.4%)
tracing	14.955 s [14.955 s, 14.955 s]	-525.0 ms (-3.4%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.563 s [15.563 s, 15.563 s]	-
appsec	14.76 s [14.76 s, 14.76 s]	-803.0 ms (-5.2%)
iast	18.691 s [18.691 s, 18.691 s]	3.128 s (20.1%)
iast_GLOBAL	18.669 s [18.669 s, 18.669 s]	3.106 s (20.0%)
profiling	15.127 s [15.127 s, 15.127 s]	-436.0 ms (-2.8%)
tracing	15.101 s [15.101 s, 15.101 s]	-462.0 ms (-3.0%)

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.61.0-SNAPSHOT~5ba6f6958e, baseline=1.61.0-SNAPSHOT~ca975c2722
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.5 ms) : 1488, 1511
.   : milestone, 1500,
appsec (3.848 ms) : 3625, 4071
.   : milestone, 3848,
iast (2.278 ms) : 2209, 2346
.   : milestone, 2278,
iast_GLOBAL (2.324 ms) : 2255, 2393
.   : milestone, 2324,
profiling (2.104 ms) : 2050, 2159
.   : milestone, 2104,
tracing (2.084 ms) : 2031, 2137
.   : milestone, 2084,
section candidate
no_agent (1.496 ms) : 1485, 1508
.   : milestone, 1496,
appsec (3.825 ms) : 3603, 4046
.   : milestone, 3825,
iast (2.289 ms) : 2220, 2358
.   : milestone, 2289,
iast_GLOBAL (2.32 ms) : 2251, 2389
.   : milestone, 2320,
profiling (2.133 ms) : 2076, 2189
.   : milestone, 2133,
tracing (2.11 ms) : 2056, 2164
.   : milestone, 2110,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.5 ms [1.488 ms, 1.511 ms]	-
appsec	3.848 ms [3.625 ms, 4.071 ms]	2.349 ms (156.6%)
iast	2.278 ms [2.209 ms, 2.346 ms]	777.932 µs (51.9%)
iast_GLOBAL	2.324 ms [2.255 ms, 2.393 ms]	824.559 µs (55.0%)
profiling	2.104 ms [2.05 ms, 2.159 ms]	604.55 µs (40.3%)
tracing	2.084 ms [2.031 ms, 2.137 ms]	584.522 µs (39.0%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.496 ms [1.485 ms, 1.508 ms]	-
appsec	3.825 ms [3.603 ms, 4.046 ms]	2.328 ms (155.6%)
iast	2.289 ms [2.22 ms, 2.358 ms]	792.753 µs (53.0%)
iast_GLOBAL	2.32 ms [2.251 ms, 2.389 ms]	823.967 µs (55.1%)
profiling	2.133 ms [2.076 ms, 2.189 ms]	636.363 µs (42.5%)
tracing	2.11 ms [2.056 ms, 2.164 ms]	613.612 µs (41.0%)

mhlidd added 14 commits April 1, 2026 15:38

updating config helper to have descriptive exception in strict mode

4c8153c

enabling ConfigHelper strict mode by default for spock tests

f29cdf5

updating test-utils to include config-utils dependency at runtime

1a30830

overriding strictness for tests using fake env vars

da699c2

test adding real env vars

650e7d5

updating tests using fake env vars and adding more env vars

14e0405

overriding test mode for elasticsearch and opensearch

b998bec

adding final integration configs and overriding fake config tests

3e2ccfe

updating strictness in configurePreAgent

92e6d26

update config helper to guarantee throw an error that won't be caught

3582e00

configuring more fake config tests

e7b23e9

adding final configurations

b618154

Merge branch 'master' into mhlidd/run_spock_with_strict

84dc914

revert supported-configurations

9d52790

mhlidd changed the base branch from master to mhlidd/update_strictness_policy April 2, 2026 21:09

mhlidd changed the base branch from mhlidd/update_strictness_policy to master April 2, 2026 21:10

mhlidd mentioned this pull request Apr 5, 2026

Update supported-configurations.json w/ Missing Configs #11046

Open

testing

8b5647e

Update supported-configurations.json

5ba6f69

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update Spock Tests to run with `STRICT_TEST` mode by default#11032

Update Spock Tests to run with `STRICT_TEST` mode by default#11032
mhlidd wants to merge 16 commits intomasterfrom
mhlidd/run_spock_with_strict

mhlidd commented Apr 1, 2026

Uh oh!

pr-commenter bot commented Apr 5, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

mhlidd commented Apr 1, 2026

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

pr-commenter bot commented Apr 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

pr-commenter bot commented Apr 5, 2026 •

edited

Loading