How to create a search space dictionary where one key-value depends on another key-value?

[wjlgatech]

I want to create a search space for a neural network with n layers (randomly chosen from [2,3,4,5]) and each layer has certain units (randomly chosen from [100, 200, 300])

For example:

if the network has 2 layers, then the units could be [100, 300], [300, 200], [200, 200] , etc.

if the network has 3 layers, then the units could be [100, 200, 100], [300, 100, 200], [200,300,300], etc.

The code could look like

def train_nn(config):
      ...

# define search space dictionary
space = {"n_layers": tune.choice([2,3,4,5])}

# add an additional item to the search space dictionary
space["n_units"= [tune.choice([100, 200,300]) for i in range(space['n_layers'])]

analysis = tune.run(
    train_nn,
    config = space,
)

Here I try to create space['n_units'] by the randomly selected value of space['n_layers']. But there is syntax error:

TypeError                                 Traceback (most recent call last)
<ipython-input-13-e7959554a989> in <module>()
      1 # search space
      2 space = {"n_layers": tune.choice([2,3,4,5]), 
----> 3          "n_units": [tune.choice([100, 200,300]) for i in range(space['n_layers'])]
TypeError: 'Categorical' object cannot be interpreted as an integer

It's because tune.choice([2,3,4,5]) is of type ray.tune.sample.Categorical which I do not know how to turn it into int.
Your advise is much appreciated!

The implement in optuna is as following; what is the corresponding implementation in FLAML/ Ray?

def objective(trial: optuna.Trial):
    num_layers = trial.suggest_int('n_layers', 2, 5)  # `num_layers` is  2, 3, 4, or 5.
    layers= []  # define the number of unit of each layer / the ratio of dropout of each layer
    for i in range(n_layers - 1):  # `TabularModel` automatically adds the last layer.
        num_units = trial.suggest_categorical(f'num_units_layer_{i}', [100, 200, 300])
        layers.append(num_units)
        
    emb_drop = trial.suggest_discrete_uniform('emb_drop', 0, 1, 0.05)
    learn = tabular_learner(data, layers=layers, emb_drop=emb_drop, y_range=y_range, metrics=exp_rmspe)
    
    learn.fit_one_cycle(5, 1e-3, wd=0.2)
    return learn.validate()[-1].item()  # Of course you can use the last record of `learn.recorder`.
    
study = optuna.create_study()
study.optimize(objective)
best_trial = study.best_trial

[sonichi]

I'd suggest this:

space = {"n_layers": tune.randint(2, 6)}
for i in range(4):
    space[f"num_units_layer_{i}"] = tune.qrandint(100, 300, q=100)

and in the trainable function, simply ignore the unused hps.

[wjlgatech]

@sonichi Smart solution, thanks!

I found another solution using Ray's custom/conditional search space here:

import ray
from ray import tune

def train_clf(config):
     ...


analysis = tune.run(
    train_clf,
    config = {"n_layers": tune.sample_from(lambda _: np.random.choice([2,3,4,5])), 
              "layers": tune.sample_from(lambda spec: [np.random.choice([100, 200, 300, 400, 500]) for i in range(spec.config.n_layers-1)]),
              "n_epochs": tune.lograndint(lower=1, upper=100),
              "emb_drop": tune.loguniform(lower=1e-4, upper=8e-1)
              },
    num_samples=100,
    time_budget_s=600 # time budget in seconds
)

Here config['layers'] is conditional to config['n_layers'] by using the tune.sample_from(lambda spec:...) method here: https://docs.ray.io/en/latest/tune/api_docs/search_space.html#tune-custom-search

[wjlgatech]

However, the conditional search space syntax is not working under flaml.tune.run(). In contract, it works under tune.run() (see code above)

Do you have advice on how to fix it?

I also keep the experiment on the previous colab notebook.

import time
import flaml
import ray
from ray import tune
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from fastai.tabular import * 

# load a sample dataset
path = untar_data(URLs.ADULT_SAMPLE)
df = pd.read_csv(path/'adult.csv')
df.head()

# simple split data into train & valid
valid_idx = range(len(df)-2000, len(df))

# define local variables
dep_var = 'salary'
cat_names = ['workclass', 'education', 'marital-status', 'occupation', 'relationship', 'race', 'sex', 'native-country']

def train_clf_config2(config, path, df, dep_var, valid_idx, cat_names):
    """this function is inherited from train_clf_config() with 'config' added """
    # data preprocessing
    procs = [FillMissing, Categorify, Normalize]

    # prep data for tabular_learner()
    data = TabularDataBunch.from_df(path, df, dep_var=dep_var, valid_idx=valid_idx, procs=procs, cat_names=cat_names)

    #metrics
    f1=FBeta()
    precision = Precision()
    recall = Recall()

    print('================ training classifier================')

    # choose hyperparameters: layers, ps, emb_drop
    layers=config['layers'], 
    ps = config['ps']
    emb_drop = config['emb_drop']

    # define a tabular classifier
    learn = tabular_learner(data, layers=layers, ps=ps, emb_drop=emb_drop, emb_szs={'native-country': 10}, metrics=[accuracy, precision, recall, f1])
    
    # auto find learning rate
    try:
        lr = find_appropriate_lr(model=learn, plot=True)
        print(f'clf uses estimated lr={lr}')
    except:
        lr = 1e-2
        print(f'clf uses pre-defined lr={lr}')
    
    # train n_epoch
    n_epochs = config['n_epochs']
    learn.fit_one_cycle(n_epochs, moms=(lr*0.01,lr))
    
    # build validation performance metrics
    valid_metrics = dict(zip(['accuracy',	'precision',	'recall',	'f1'], [x.item() for x in learn.recorder.metrics[0]]))
    
    # send metrics to tune
    tune.report(**valid_metrics)

######################## tuning#########################
analysis = flaml.tune.run(
    tune.with_parameters(train_clf_config2, path=path, df=df, dep_var=dep_var, valid_idx=valid_idx, cat_names=cat_names, return_learner=False),
    config = {"n_layers": tune.sample_from(lambda _: np.random.choice([2,3,4,5,6])), 
              "layers": tune.sample_from(lambda spec: [np.random.choice([100, 200, 300, 400, 500, 600, 700, 800, 900, 1000, 1100, 1200]) for i in range(spec.config.n_layers-1)]),
              "ps": tune.sample_from(lambda spec: [float(loguniform.rvs(1e-4, 1e0, size=1)) for i in range(spec.config.n_layers-1)]),
              "n_epochs": tune.lograndint(lower=1, upper=100),
              "emb_drop": tune.loguniform(lower=1e-4, upper=8e-1)
              },
    metric="f1", # select "f1" as metric for optimization
    mode="max", # we want to maximize metric selected e.g. 'f1'
    low_cost_partial_config={"n_epochs": 1},
    resources_per_trial={"cpu": 1, "gpu": 0.5},# number of gpus for each trial; 0.5 means two training jobs can share one gpu
    local_dir='logs/',
    num_samples=600, # maximal number of trials
    time_budget_s=600, # time budget in seconds
    use_ray=True)

ray.shutdown()

Here is error:

You passed a `space` parameter to OptunaSearch that contained unresolved search space definitions. OptunaSearch should however be instantiated with fully configured search spaces only. To use Ray Tune's automatic search space conversion, pass the space definition as part of the `config` argument to `tune.run()` instead.
You passed a `space` parameter to OptunaSearch that contained unresolved search space definitions. OptunaSearch should however be instantiated with fully configured search spaces only. To use Ray Tune's automatic search space conversion, pass the space definition as part of the `config` argument to `tune.run()` instead.
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)

...
ValueError: Optuna search does not support parameters of type `Function` with samplers of type `NoneType`