Skip to content

Commit d0872b8

Browse files
nkeplingnkepling
authored
Merge pull request #37 from JayLi1209/feature/tutorial
Leaderboard + tutorial/Evaluation finished
2 parents 3e593a0 + 52c37dd commit d0872b8

8 files changed

Lines changed: 383 additions & 9 deletions

File tree

‎docs/source/_static/css/custom.css‎

Lines changed: 151 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,4 +12,154 @@
1212
/* Vertically center the content in our middle column */
1313
.table-vcenter {
1414
vertical-align: middle;
15-
}
15+
}
16+
17+
/* Leaderboard layout */
18+
.leaderboard-overview {
19+
display: grid;
20+
gap: 1rem;
21+
grid-template-columns: repeat(auto-fit, minmax(220px, 1fr));
22+
margin: 1.5rem 0 2rem;
23+
}
24+
25+
.leaderboard-summary-card,
26+
.leaderboard-card {
27+
border: 1px solid #d7e3da;
28+
border-radius: 16px;
29+
background: linear-gradient(180deg, #fcfdfc 0%, #f5f9f6 100%);
30+
box-shadow: 0 10px 24px rgba(15, 23, 42, 0.04);
31+
}
32+
33+
.leaderboard-summary-card {
34+
padding: 1.1rem 1.2rem;
35+
}
36+
37+
.leaderboard-summary-card h2 {
38+
margin: 0 0 0.5rem;
39+
font-size: 1.1rem;
40+
}
41+
42+
.leaderboard-summary-card p {
43+
margin: 0;
44+
}
45+
46+
.leaderboard-summary-meta {
47+
margin-top: 0.75rem !important;
48+
color: #446251;
49+
font-size: 0.95rem;
50+
font-weight: 600;
51+
}
52+
53+
.leaderboard-card {
54+
margin: 1rem 0 2rem;
55+
padding: 1.1rem 1.2rem;
56+
}
57+
58+
.leaderboard-badges {
59+
display: flex;
60+
flex-wrap: wrap;
61+
gap: 0.5rem;
62+
margin-bottom: 0.45rem;
63+
}
64+
65+
.leaderboard-badge,
66+
.leaderboard-tabs > label {
67+
display: inline-flex;
68+
align-items: center;
69+
padding: 0.45rem 0.9rem;
70+
border: 1px solid #c8d4cc;
71+
border-radius: 999px;
72+
background: #ffffff;
73+
color: #395143;
74+
font-size: 0.95rem;
75+
font-weight: 600;
76+
line-height: 1.2;
77+
}
78+
79+
.leaderboard-tabs {
80+
display: flex;
81+
flex-wrap: wrap;
82+
gap: 0.55rem;
83+
align-items: center;
84+
}
85+
86+
.leaderboard-tabs > input[type="radio"] {
87+
position: absolute;
88+
opacity: 0;
89+
pointer-events: none;
90+
}
91+
92+
.leaderboard-tabs > label {
93+
cursor: pointer;
94+
transition: background-color 0.2s ease, border-color 0.2s ease, color 0.2s ease;
95+
}
96+
97+
.leaderboard-tabs > label:hover {
98+
border-color: #8ca996;
99+
color: #234d33;
100+
}
101+
102+
.leaderboard-tabs > input:nth-of-type(1):checked + label,
103+
.leaderboard-tabs > input:nth-of-type(2):checked + label {
104+
border-color: #2f6e4d;
105+
background: #2f6e4d;
106+
color: #ffffff;
107+
}
108+
109+
.leaderboard-tab-panels {
110+
width: 100%;
111+
margin-top: 1rem;
112+
}
113+
114+
.leaderboard-panel {
115+
display: none;
116+
}
117+
118+
.leaderboard-tabs > input:nth-of-type(1):checked ~ .leaderboard-tab-panels > .leaderboard-panel:nth-of-type(1),
119+
.leaderboard-tabs > input:nth-of-type(2):checked ~ .leaderboard-tab-panels > .leaderboard-panel:nth-of-type(2) {
120+
display: block;
121+
}
122+
123+
.leaderboard-table-wrap {
124+
overflow-x: auto;
125+
}
126+
127+
.leaderboard-table {
128+
width: 100%;
129+
min-width: 680px;
130+
border-collapse: collapse;
131+
}
132+
133+
.leaderboard-table caption {
134+
margin-bottom: 0.75rem;
135+
color: #1f2937;
136+
text-align: left;
137+
font-size: 1rem;
138+
font-weight: 700;
139+
}
140+
141+
.leaderboard-table th,
142+
.leaderboard-table td {
143+
padding: 0.8rem 0.75rem;
144+
border-bottom: 1px solid #dfe9e1;
145+
text-align: left;
146+
vertical-align: top;
147+
}
148+
149+
.leaderboard-table thead th {
150+
background: rgba(234, 244, 237, 0.9);
151+
color: #264534;
152+
font-size: 0.95rem;
153+
font-weight: 700;
154+
}
155+
156+
.leaderboard-table tbody tr:last-child td {
157+
border-bottom: none;
158+
}
159+
160+
.leaderboard-empty-row td {
161+
color: #58695f;
162+
text-align: center;
163+
font-style: italic;
164+
padding: 1.15rem 0.75rem;
165+
}

‎docs/source/aamas2026_competition.md‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,8 @@ Use these guides in order:
4343
1. [Setup](aamas2026_competition/setup.md)
4444
2. [Build Your Agent](aamas2026_competition/build_agents.md)
4545
3. [Create a Custom Environment](aamas2026_competition/make_env.md)
46-
4. [Submit Your Agent](aamas2026_competition/submission.md)
46+
4. [Evaluation Criteria](aamas2026_competition/evaluation.md)
47+
5. [Submit Your Agent](aamas2026_competition/submission.md)
4748

4849
```{toctree}
4950
:maxdepth: 1
@@ -52,6 +53,7 @@ Use these guides in order:
5253
aamas2026_competition/setup
5354
aamas2026_competition/build_agents
5455
aamas2026_competition/make_env
56+
aamas2026_competition/evaluation
5557
aamas2026_competition/submission
5658
```
5759

@@ -112,6 +114,8 @@ We evaluate agents based on how well they handle the transition from a stable en
112114
3. **Resilience**: We assess how well the agent recovers from unexpected shifts in dynamics. We specifically look at the magnitude of the performance drop between the stable pre-shift return and the post-shift episode return.
113115
4. **Efficiency**: We track the computational cost of the adaptation mechanism. This includes CPU/GPU utilization per decision step and the total memory utilization across the evaluation episode.
114116

117+
For detailed information, see [Evaluation Criteria](aamas2026_competition/evaluation.md).
118+
115119
## Example Environments
116120

117121
The competition will focus on environments that have parametric changes. The parametric changes involve gradual or abrupt variations in continuous parameters that affect the environment's dynamics. Examples include changes in friction coefficients, mass distributions, or gravity. The magnitude and frequency of these changes will vary across different environments to test the robustness of the agents.

‎docs/source/aamas2026_competition/build_agents.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,8 @@ Welcome! This is the second tutorial in a series for the full submission workflo
66
1. [Environment Setup](setup.md): create your repository, configure Python, and build Docker images.
77
2. This tutorial: implement a model-based or model-free agent and register it.
88
3. [Create a Custom Environment](make_env.md): define non-stationarity with schedulers and update functions.
9-
4. [Submit Your Agent](submission.md): run final checks and send your repository for evaluation.
9+
4. [Evaluation Criteria](evaluation.md): understand how submissions are scored and ranked.
10+
5. [Submit Your Agent](submission.md): run final checks and send your repository for evaluation.
1011

1112
This is the part you will spend most time on. Let's get started!
1213

‎docs/source/aamas2026_competition/evaluation.md‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
# Evaluating Your Agent
2+
3+
Welcome! This is the fourth tutorial in a series for the full submission workflow:
4+
5+
1. [Environment Setup](setup.md): create your repository, configure Python, and build Docker images.
6+
2. [Build Your Agent](build_agents.md): implement a model-based or model-free agent and register it.
7+
3. [Create a Custom Environment](make_env.md): define non-stationarity with schedulers and update functions.
8+
4. This tutorial: understand how submissions are scored and ranked.
9+
5. [Submit Your Agent](submission.md): run final checks and send your repository for evaluation.
10+
11+
12+
> “Jade must be polished to become a gem.” - Three Character Classic (San Zi Jing)
13+
14+
15+
For each submission, we will evaluate and rank according to four criteria. When you run the evaluator through Docker, you should see your scoring metrics on your machine. We will officiate your result after submission.
16+
17+
We mainly evaluate based on:
18+
19+
1. **Adaptability**: a measure of how fast an agent could adapt to change. At an unknown timestep, we will make a change to the non-stationary environment parameter. Your algorithm needs to learn to recover from initial failures, and adapt to find a good solution.
20+
21+
To evaluate, we consider the following:
22+
- Regret. The difference between our oracle solution and your submission.
23+
- Recovery Time. The time your algorithm takes to adapt to the change.
24+
25+
*Unnotify category only.*
26+
27+
2. **Performance**: this is the average undiscounted episodic reward achieved under non-stationary conditions. The environment we use to evaluate this will be more non-stationary.
28+
29+
*Unnotify and partial-notify categories.*
30+
31+
3. **Resilience**: a good algorithm needs a robust policy that still thrives with slight perturbations. We want to measure the agent's performance immediately after the change, before the agent has time to adapt. To measure this, we froze your policy (or your current estimation of the MDP, or both) and perturbs the non-stationarity settings. We consider both the retention ratio and its relative performance with respect to our oracle solution. To get ranked on this leaderboard, agents have to pass a specific performance threshold.
32+
33+
*Fully-notify only.*
34+
35+
4. **Efficiency**: this is measured in two aspects: fewer timesteps and less wall-clock (real-life) time the agent consumes in finding a solution. We rank submissions based on the ratio between the two. To get ranked on this leaderboard, agents have to pass a specific performance threshold.
36+
37+
*Unnotify and partial-notify categories.*
38+
39+
## Next step
40+
41+
Think you can beat one or more categories? Go to [Submit Your Agent](submission.md) for the final checklist and submission steps!

‎docs/source/aamas2026_competition/make_env.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,8 @@ Welcome! This is the third tutorial in a series for the full submission workflow
55
1. [Environment Setup](setup.md): create your repository, configure Python, and build Docker images.
66
2. [Build Your Agent](build_agents.md): implement a model-based or model-free agent and register it.
77
3. This tutorial: define non-stationarity with schedulers and update functions.
8-
4. [Submit Your Agent](submission.md): run final checks and send your repository for evaluation.
8+
4. [Evaluation Criteria](evaluation.md): understand how submissions are scored and ranked.
9+
5. [Submit Your Agent](submission.md): run final checks and send your repository for evaluation.
910

1011
This is an in-depth tutorial on the settings of NS-Gym. Let's get started!
1112

@@ -137,4 +138,4 @@ Need help? Join in our [Office Hour sessions](../aamas2026_competition.md#office
137138

138139
## Next step
139140

140-
Confident about your algorithm? Go to [Submit Your Agent](submission.md) and show us what you got!
141+
Confident about your environment and algorithm? Go to [Evaluation Criteria](evaluation.md) to see how submissions are scored.

‎docs/source/aamas2026_competition/setup.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,8 @@ This tutorial series will walk you through the full submission workflow:
77
1. This tutorial: create your repository, configure Python, and build Docker images.
88
2. [Build Your Agent](build_agents.md): implement a model-based or model-free agent and register it.
99
3. [Create a Custom Environment](make_env.md): define non-stationarity with schedulers and update functions.
10-
4. [Submit Your Agent](submission.md): run final checks and send your repository for evaluation.
10+
4. [Evaluation Criteria](evaluation.md): understand how submissions are scored and ranked.
11+
5. [Submit Your Agent](submission.md): run final checks and send your repository for evaluation.
1112

1213
Let's get started!
1314

‎docs/source/aamas2026_competition/submission.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ This tutorial series will walk you through the full submission workflow:
66
- [Environment Setup](setup.md): create your repository, configure Python, and build Docker images.
77
- [Build Your Agent](build_agents.md): implement a model-based or model-free agent and register it.
88
- [Create a Custom Environment](make_env.md): define non-stationarity with schedulers and update functions.
9+
- [Evaluation Criteria](evaluation.md): understand how submissions are scored and ranked.
910
- This tutorial: run final checks and send your repository for evaluation.
1011

1112
Let's get started!

0 commit comments

Comments
 (0)