Fail loudly in mape and define smape's 0/0 term - #1050
Open
arpitjain099 wants to merge 2 commits into
Open
arpitjain099 wants to merge 2 commits into
arpitjain099 wants to merge 2 commits into
Conversation
mape filtered out near-zero targets and then took the mean of what was left. With every target at or below EPS the filter selects nothing and np.mean of an empty slice returns NaN with only a RuntimeWarning, while the scalar sibling ape asserts against exactly this case. Raise instead. smape had the related problem one step along: an element where target and prediction are both zero gives 0/0, so a single such row turns the whole metric into NaN even when every other row is fine. That case is an exact prediction rather than an undefined one, so the term is defined as 0. Signed-off-by: Arpit Jain <arpitjain099@gmail.com>
Signed-off-by: Arpit Jain <arpitjain099@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1047.
mapefiltered out near-zero targets and then averaged what was left. When every target is at or belowEPSthe filter selects nothing, andnp.meanof an empty slice returns NaN with only a RuntimeWarning, which is easy to lose in training logs. The scalar siblingapealready asserts against this, somapenow raisesValueError.smapehas the related problem one element along, and it is the more likely one to bite: where target and prediction are both zero the term is0/0, so a single such row turns the whole metric into NaN even when every other row is fine. That row is an exact prediction rather than an undefined one, so the term is defined as 0.The issue was filed from static analysis, so I ran it first. Before:
After:
maperaises,smape(zeros, zeros)is0.0, and the mixed case is0.5596. Both metrics are unchanged on non-degenerate input (mape0.10625 andsmape0.10727 before and after), andmapestill ignores zero targets when others are present.Five tests added to
tests/test_metrics.py. Three fail on master, the two that pin unchanged behaviour pass either way.