Merge pull request #3 from robert-kisteleki/feature/measurement-scheduling

Feature/measurement scheduling

Author: robert-kisteleki

CHANGES.md

@@ -2,7 +2,11 @@
 
 ## next
 
-* NEW: add support for measurement status checks (status subcommand)
+* NEW: add support for measurement status checks (`status` subcommand)
+* NEW: add support for measurement scheduling (`measure` subcommand)
+  * support all measurement types with lots of options and sane defaults
+  * support all probe selection criteria
+  * support default probe selection specification in config file
 
 ## v0.3.0
 

README.md

@@ -6,9 +6,10 @@ It supports:
 * finding probes
 * finding anchors
 * finding measurements
-* downloading and displaying results of measurements and turning them into Go objects
-* tuning in to result streaming and turning them into Go objects
-* loading a local file containing measurement results and turning them into Go objects
+* scheduling new measurements
+* downloading and displaying results of measurements
+* tuning in to result streaming
+* loading a local file containing measurement results
 * various kinds of output formatters for displaying and aggregating measurement results
 * (more features to come)
 
@@ -146,9 +147,96 @@ The output formatters are extensible, feel free to write your own -- and contrib
 
 New output processors need to be registered in `goatcli.go`. See `some.go` or `native.go` for examples.
 
+## Schedule a New Measurement
+
+Use the `measurement` subcommand. Quick examples:
+
+```sh
+# create a one-off with default set of probes
+$ ./goatcli measure --ping --target ping.ripe.net
+
+# create an ongoing with default set of probes and predefined start and stop times
+$ ./goatcli measure --ping --target ping.ripe.net --ongoing --start 2023-10-23T19 --stop tomorrow
+
+# create a one-off with 15 probes from AS33260
+$ ./goatcli measure --ping --target ping.ripe.net --probeasn 15@3320
+
+# create a one-off with 15 probes from AS33260 and 20 probes from NL
+$ ./goatcli measure --trace --target ping.ripe.net --probeasn 15@3320 --probecc 20@nl
+```
+
+### Authentication
+
+A valid API key for creating measurements is needed. It should be defined in the configuration file (`/.config/goat.ini`):
+
+```
+[apikeys]
+# Create new measurement(s)
+create_measurements = "12345678-xxxx-xxxx-xxxx-something"
+```
+
+It is also possible to supply a key on the command line (`--key KEY`).
+
+### Probe Selection
+
+The following probe selection options are available:
+* `--probearea` select from areas. A comma separated list of `amount@area` where `area` can be `ww`, `west`, `nc`, `sc`, `ne`, `se`
+* `--probeasn` select from ASNs. A comma separated list of `amount@ASN`
+* `--probecc` select from countries. A comma separated list of `amount@CC` where CC is a valid country code
+* `--probelist` provides an explicit list of probe IDs to include as a comma separated list.
+* `--probeprefix` select from prefixes (IPv4 or IPv6). A comma separated list of `amount@prefix`
+* `--probereuse` reuse probes from a precvious measurement. A comma separated list of `amount@msmID`
+
+Multiple probe selection criteria can be sepcified; each of them add more probes to the selection.
+
+`--probetaginc` and `--probetagexc` can be used to filter for probes that have been tagged (or not tagged) with those tags. Both are comma separated lists.
+
+A default probe selection can be expressed in the configuration file (`/.config/goat.ini`) using entries with the above names in the `[probespec]` section, e.g.:
+
+```
+# default probe specifications for new measurements
+[probespec]
+probecc = ""
+probearea = "9@ww"
+probeasn = ""
+probeprefix = ""
+probelist = ""
+probereuse = ""
+```
+
+### Timing
+
+By default one-offs are scheduled, starting as soon as possible. You can specify a start time (`start`) in the future and for ongoings perhaps even a stop time (`stop`). Times can be specifies as:
+* UNIX timestamps
+* ISO8601 variants:
+  * `YYYY-mm-ddTHH:MM:SS` - leaving time details from the right makes them default to 0 (e.g. `2023-10-24` is valid and becomes `2023-10-24T00:00:00`)
+
+### Measurement Types and Options
+
+You can define one measurement per invocation. This can be one of: `ping`, `trace`, `dns`, `tls`, `ntp` or `http` (the last one with restrictions by the system). Each measurement needs a `target`, except DNS that needs a `name` to look up and uses `target` as the server/resolver to use if it's specified, otherwise uses the local resolver.
+
+Common options for measurements include `interval`, `spread`, `tags` and some more.
+
+Each mesurement type accepts a number of options, such as `abuf`, `qbuf`, `nsid`, `rd` for DNS, `minhop` and `maxhop` for trace, etc. Check the help page for the complete list of these.
+
+### Output
+
+For now the measurement scheduling request returns either the ID of the new measurement or an error if something went wrong.
+
+
+## Status Checks
+
+Provide a short summary of the results for a [measurement status check](https://atlas.ripe.net/docs/apis/rest-api-manual/measurements/status-checks.html). It needs a measurement ID, and by default it summarises the results (is there an alert, how many probes are in that state out of how many total, and with the `most` output formatter the list of alerting probes).
+
+```sh
+$ ./goatcli status -id 61953517 -output most
+true	1	9	[1005382]
+```
+
+
 # Future Additions / TODO
 
-* schedule a new measurement, stop existing measurements
+* stop existing measurements
 * modify participants of an existing measurement (add/remove probes)
 * check credit balance, transfer credits, ...
 

command.go

@@ -36,6 +36,8 @@ func commandSelector() {
 		commandResult(args[1:])
 	case args[0] == "status":
 		commandStatusCheck(args[1:])
+	case args[0] == "measure":
+		commandMeasure(args[1:])
 	default:
 		commandHelp()
 	}
@@ -61,6 +63,7 @@ func printUsage() {
 	fmt.Println("	fm|findmsm       search for measurements")
 	fmt.Println("	result           download results")
 	fmt.Println("	status           measurement status check")
+	fmt.Println("	measure          start new measurement(s)")
 	fmt.Println("")
 	fmt.Println("Options:")
 	flag.PrintDefaults()

config.go

@@ -32,9 +32,17 @@ var (
 	flagsFindMsm     *flag.FlagSet
 	flagsGetResult   *flag.FlagSet
 	flagsStatusCheck *flag.FlagSet
+	flagsMeasure     *flag.FlagSet
 
 	apiKey  *uuid.UUID           // specified on the command line explicitly or via env
 	apiKeys map[string]uuid.UUID // collected from config file
+
+	probeSpecCc     string
+	probeSpecArea   string
+	probeSpecAsn    string
+	probeSpecPrefix string
+	probeSpecList   string
+	probeSpecReuse  string
 )
 
 const version = "v0.3.0+"
@@ -89,6 +97,7 @@ func configure() {
 	flagsFindMsm = flag.NewFlagSet("measurement", flag.ExitOnError)
 	flagsGetResult = flag.NewFlagSet("result", flag.ExitOnError)
 	flagsStatusCheck = flag.NewFlagSet("status", flag.ExitOnError)
+	flagsMeasure = flag.NewFlagSet("measure", flag.ExitOnError)
 
 	Subcommands = map[string]*flag.FlagSet{
 		flagsVersion.Name():     flagsVersion,
@@ -97,6 +106,7 @@ func configure() {
 		flagsFindMsm.Name():     flagsFindMsm,
 		flagsGetResult.Name():   flagsGetResult,
 		flagsStatusCheck.Name(): flagsStatusCheck,
+		flagsMeasure.Name():     flagsStatusCheck,
 	}
 	setupFlags()
 
@@ -134,6 +144,7 @@ func readConfig(confFile string) bool {
 
 	// record stuff that was in the config file
 	loadApiKey(cfg, "list_measurements")
+	loadApiKey(cfg, "create_measurements")
 	// TODO: add more API key variations here
 
 	// allow config to override where the API is
@@ -156,6 +167,14 @@ func readConfig(confFile string) bool {
 	// we deliberately ignore errors on creating this dir as it may exist
 	_ = os.MkdirAll(CacheDir, os.FileMode(0755))
 
+	// load probe specification defaults
+	probeSpecCc = cfg.Section("probespec").Key("probecc").MustString("")
+	probeSpecArea = cfg.Section("probespec").Key("probearea").MustString("")
+	probeSpecAsn = cfg.Section("probespec").Key("probeasn").MustString("")
+	probeSpecPrefix = cfg.Section("probespec").Key("probeprefix").MustString("")
+	probeSpecList = cfg.Section("probespec").Key("probelist").MustString("")
+	probeSpecReuse = cfg.Section("probespec").Key("probereuse").MustString("")
+
 	return true
 }
 
@@ -192,6 +211,18 @@ cachedir = ""
 
 # List your measurements
 list_measurements = ""
+
+# Create new measurement(s)
+create_measurements = ""
+
+# default probe specifications for new measurements
+[probespec]
+probecc = ""
+probearea = ""
+probeasn = ""
+probeprefix = ""
+probelist = ""
+probereuse = ""
 `)
 
 	if flagVerbose {
@@ -232,3 +263,22 @@ func getApiKey(function string) *uuid.UUID {
 
 	return nil
 }
+
+func getProbeSpecDefault(spec string) string {
+	switch spec {
+	case "cc":
+		return probeSpecCc
+	case "area":
+		return probeSpecArea
+	case "asn":
+		return probeSpecAsn
+	case "prefix":
+		return probeSpecPrefix
+	case "list":
+		return probeSpecList
+	case "reuse":
+		return probeSpecReuse
+	default:
+		return ""
+	}
+}

findanchor.go

@@ -43,7 +43,7 @@ func commandFindAnchor(args []string) {
 
 	// counting only
 	if _, ok := options["count"]; ok {
-		count, err := filter.GetAnchorCount(flagVerbose)
+		count, err := filter.GetAnchorCount()
 		if err != nil {
 			fmt.Fprintf(os.Stderr, "ERROR: %v\n", err)
 			os.Exit(1)
@@ -54,7 +54,7 @@ func commandFindAnchor(args []string) {
 
 	// most of the work is done by goatAPI
 	anchors := make(chan goatapi.AsyncAnchorResult)
-	go filter.GetAnchors(flagVerbose, anchors)
+	go filter.GetAnchors(anchors)
 
 	// produce output; exact format depends on the "format" option
 	output.Setup(formatter, flagVerbose, flags.outopts)
@@ -78,6 +78,7 @@ func processFindAnchorFlags(flags *findAnchorFlags) (
 ) {
 	options = make(map[string]any)
 	filter = goatapi.NewAnchorFilter()
+	filter.Verbose(flagVerbose)
 
 	// options
 

findmsm.go

@@ -79,7 +79,7 @@ func commandFindMsm(args []string) {
 
 	// counting only
 	if _, ok := options["count"]; ok {
-		count, err := filter.GetMeasurementCount(flagVerbose)
+		count, err := filter.GetMeasurementCount()
 		if err != nil {
 			fmt.Fprintf(os.Stderr, "ERROR: %v\n", err)
 			os.Exit(1)
@@ -90,7 +90,7 @@ func commandFindMsm(args []string) {
 
 	// most of the work is done by goatAPI
 	measurements := make(chan goatapi.AsyncMeasurementResult)
-	go filter.GetMeasurements(flagVerbose, measurements)
+	go filter.GetMeasurements(measurements)
 
 	// produce output; exact format depends on the "format" option
 	output.Setup(formatter, flagVerbose, []string(flags.outopts))
@@ -114,6 +114,7 @@ func parseFindMsmFlags(flags *findMsmFlags) (
 ) {
 	options = make(map[string]any)
 	filter = goatapi.NewMeasurementFilter()
+	filter.Verbose(flagVerbose)
 
 	// options
 

findprobe.go

@@ -73,7 +73,7 @@ func commandFindProbe(args []string) {
 
 	// counting only
 	if _, ok := options["count"]; ok {
-		count, err := filter.GetProbeCount(flagVerbose)
+		count, err := filter.GetProbeCount()
 		if err != nil {
 			fmt.Fprintf(os.Stderr, "ERROR: %v\n", err)
 			os.Exit(1)
@@ -84,7 +84,7 @@ func commandFindProbe(args []string) {
 
 	// most of the work is done by goatAPI
 	probes := make(chan goatapi.AsyncProbeResult)
-	go filter.GetProbes(flagVerbose, probes)
+	go filter.GetProbes(probes)
 
 	// produce output; exact format depends on the "format" option
 	output.Setup(formatter, flagVerbose, flags.outopts)
@@ -108,6 +108,7 @@ func parseFindProbeFlags(flags *findProbeFlags) (
 ) {
 	options = make(map[string]any)
 	filter = goatapi.NewProbeFilter()
+	filter.Verbose(flagVerbose)
 
 	// options