Skip to content

Fail loudly in mape and define smape's 0/0 term - #1050

Open
arpitjain099 wants to merge 2 commits into
uber:masterfrom
arpitjain099:fix/mape-smape-degenerate-targets
Open

arpitjain099 wants to merge 2 commits into
uber:masterfrom
arpitjain099:fix/mape-smape-degenerate-targets

Conversation

@arpitjain099

Copy link
Copy Markdown
Contributor

Fixes #1047.

mape filtered out near-zero targets and then averaged what was left. When every target is at or below EPS the filter selects nothing, and np.mean of an empty slice returns NaN with only a RuntimeWarning, which is easy to lose in training logs. The scalar sibling ape already asserts against this, so mape now raises ValueError.

smape has the related problem one element along, and it is the more likely one to bite: where target and prediction are both zero the term is 0/0, so a single such row turns the whole metric into NaN even when every other row is fine. That row is an exact prediction rather than an undefined one, so the term is defined as 0.

The issue was filed from static analysis, so I ran it first. Before:

mape(zeros, zeros)   -> nan   ['Mean of empty slice.', 'invalid value encountered in scalar divide']
smape(zeros, zeros)  -> nan
smape([0, 2, 0, 8], [0, 1.8, 0.5, 7])  -> nan

After: mape raises, smape(zeros, zeros) is 0.0, and the mixed case is 0.5596. Both metrics are unchanged on non-degenerate input (mape 0.10625 and smape 0.10727 before and after), and mape still ignores zero targets when others are present.

Five tests added to tests/test_metrics.py. Three fail on master, the two that pin unchanged behaviour pass either way.

mape filtered out near-zero targets and then took the mean of what was left.
With every target at or below EPS the filter selects nothing and np.mean of an
empty slice returns NaN with only a RuntimeWarning, while the scalar sibling
ape asserts against exactly this case. Raise instead.

smape had the related problem one step along: an element where target and
prediction are both zero gives 0/0, so a single such row turns the whole
metric into NaN even when every other row is fine. That case is an exact
prediction rather than an undefined one, so the term is defined as 0.

Signed-off-by: Arpit Jain <arpitjain099@gmail.com>
Signed-off-by: Arpit Jain <arpitjain099@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

mape silently returns NaN when all targets are near-zero (empty filtered slice)

1 participant