Supporting disabling tests conditionally at compile time

[arlyon]

Hi!

Avid user of test_case so its fun to see some competition. One feature I have always wanted from that lib is the ability to mark tests as ignored or inconclusive per test based on some flag (such as cfg!(target=windows)). Since you have a relatively simple syntax perhaps there is room to figure out nice ergonomics here

[foresterre]

Thanks for the suggestion.

Considering just ignoring an individual test case in Yare, I've been trying out some syntactic ideas:

Ignore with ignore(...)

#[parameterized(
  ignore(apple = { Fruit::Apple, "apple" }),
  pear = { Fruit::Pear, "pear" },
  blackberry = { Fruit::Bramble(BrambleFruit::Blackberry), "blackberry" },
)]
fn a_fruity_test(fruit: Fruit, name: &str) {
    assert_eq!(fruit.name_of(), name)
}

Advantages:

• Clearly scoped to a test case
• Can add more options to ignores parameters

Disadvantages:

• Dense, making it harder to read
• Placing the closing parenthesis can be frustrating since it needs to be placed before the comma, and requires two edits

Ignore with #

#[parameterized(
  # apple = { Fruit::Apple, "apple" }),
  # pear = { 
    Fruit::Pear, "pear"
  },
  blackberry = { Fruit::Bramble(BrambleFruit::Blackberry), "blackberry" },
)]
fn a_fruity_test(fruit: Fruit, name: &str) {
    assert_eq!(fruit.name_of(), name)
}

Advantages:

• Easy on the eyes (to me)

Disadvantages:

• Using a token over a name requires learning the meaning of the token, while a keyword is more self-explanatory
• Multiline test cases may be less clear cut: do you require a # on every test case line or for example require the first line and be optional on all others`:
• Could it communicate to a user: commented out instead of #[ignore]'ed?

#[parameterized(
  # pear = { 
  #  Fruit::Pear, "pear"
  # },
  blackberry = { Fruit::Bramble(BrambleFruit::Blackberry), "blackberry" },
)]
fn a_fruity_test(fruit: Fruit, name: &str) {
    assert_eq!(fruit.name_of(), name)
}

Variation: ignore with #ignore

#[parameterized(
  #ignore apple = { Fruit::Apple, "apple" }),
  #ignore pear = { 
    Fruit::Pear, "pear"
  },
  blackberry = { Fruit::Bramble(BrambleFruit::Blackberry), "blackberry" },
)]
fn a_fruity_test(fruit: Fruit, name: &str) {
    assert_eq!(fruit.name_of(), name)
}

Let the # act like a tag.

Extra advantage:

• Could add different functionality with a different tag name

With cfg: ignore with #ignore(cfg(target=windows)) (or #ignore(target=windows))

#[parameterized(
  #ignore(target=windows) apple = { Fruit::Apple, "apple" }),
  #ignore(target=windows) pear = { 
    Fruit::Pear, "pear"
  },
  blackberry = { Fruit::Bramble(BrambleFruit::Blackberry), "blackberry" },
)]
fn a_fruity_test(fruit: Fruit, name: &str) {
    assert_eq!(fruit.name_of(), name)
}

Variation on with cfg: postfix 1

#[parameterized(
  apple = { Fruit::Apple, "apple" }) #ignore(target=windows),
  pear = { 
    Fruit::Pear, "pear"
  } #ignore(target=windows),
  blackberry = { Fruit::Bramble(BrambleFruit::Blackberry), "blackberry" },
)]
fn a_fruity_test(fruit: Fruit, name: &str) {
    assert_eq!(fruit.name_of(), name)
}

Variation on with cfg: postfix 2

#[parameterized(
  apple = { Fruit::Apple, "apple" }).ignore(target=windows),
  pear = { 
    Fruit::Pear, "pear"
  }.ignore(target=windows),
  blackberry = { Fruit::Bramble(BrambleFruit::Blackberry), "blackberry" },
)]
fn a_fruity_test(fruit: Fruit, name: &str) {
    assert_eq!(fruit.name_of(), name)
}

Inspired by .await

Scope the placement in an "_ignore" directive

#[parameterized(
  blackberry = { Fruit::Bramble(BrambleFruit::Blackberry), "blackberry" },
  _ignore = {
      apple = { Fruit::Apple, "apple" }),
	  pear = { 
	    Fruit::Pear, "pear"
	  },
  }
)]
fn a_fruity_test(fruit: Fruit, name: &str) {
    assert_eq!(fruit.name_of(), name)
}

A disadvantage:

• test case with name _ignore either disallowed or requiring some extra syntax like \_ignore

Variation: Scope the placement in an "_ignore" directive with cfg

#[parameterized(
   blackberry = { Fruit::Bramble(BrambleFruit::Blackberry), "blackberry" },
  _ignore(cfg(target=windows)) = {
      apple = { Fruit::Apple, "apple" }),
	  pear = { 
	    Fruit::Pear, "pear"
	  },
  },
  _ignore(cfg(target=unix)) = {
  	banana = { Fruit::Banana, "banana" }),
  },
)]
fn a_fruity_test(fruit: Fruit, name: &str) {
    assert_eq!(fruit.name_of(), name)
}