Inconsistent number of samples with customized learner

[hanhanwu]

Hello FLAML team

I tried to create a customized learner like this:

But got error saying inconsistent number of samples:

Any idea on how to fix this problem?

Another question is, what's the relationship between the estimator's objective function, the metrics in settings and the model's validation loss? With customized learner, should I use "my_balanced_accuracy" in the metrics of settings too?

Thank you!

[sonichi]

For objective of LGBMClassifier, according to https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.LGBMClassifier.html?highlight=custom%20objective,

A custom objective function can be provided for the objective parameter. In this case, it should have the signature objective(y_true, y_pred) -> grad, hess or objective(y_true, y_pred, group) -> grad, hess:

For metric of flaml.AutoML, according to https://microsoft.github.io/FLAML/#flaml.AutoML.fit,

if passing a customized metric function, the function needs to
have the follwing signature:

                    def custom_metric(
                        X_test, y_test, estimator, labels,
                        X_train, y_train, weight_test=None, weight_train=None
                    ):
                        return metric_to_minimize, metrics_to_log

            which returns a float number as the minimization objective,
            and a tuple of floats or a dictionary as the metrics to log

BTW, you can remove the min_child_samples from the search space if you do not want to search for it.