WIP, ENH: Add support for gamma scaling and iterative solvers. by nray · Pull Request #70 · lanl/GFDL

nray · 2026-03-19T15:28:47Z

This draft MR contains the changes needed for gamma scaling experiments.

Addresses issue: #71

tylerjereddy · 2026-03-19T16:25:08Z

src/gfdl/model.py

        return out


+class GFDLRegressor(RegressorMixin, MultiOutputMixin, GFDL):


Has something changed about GFDLRegressor? The diff seems quite messy here, making it hard to judge what has actually changed and what hasn't. This class was deleted below and then pasted here?

Yes, the class was at the end of the file, but I moved it after GFDLClassifier to be have similar estimators one after the other. The only other change is the extra argument "gamma" to the constructor and "solver" to fit method.

It would be best not to reorganize the classes for now I think, so we can focus on what actually changed in the diff.

Reverted to the old organization.

tylerjereddy · 2026-03-19T16:54:56Z

src/gfdl/model.py

        return np.random.default_rng(seed)

+
+    def stochastic_gradient_descent(self, X, y):    


What's the motivation for rolling our own SGD instead of using upstream SGDRegressor from sklearn, which has learning_rate, eta0, etc.? Does its verbose argument not provide enough output?

You mention it briefly in the matching issue, but my intuition would always be to try the upstream/well-tested solution first before investing time on a hand-rolled version.

Created a sub issue.

tylerjereddy · 2026-03-19T17:56:47Z

src/gfdl/model.py

+
+        return self
+
+    def gradient_descent(self, X, y):


Would also be good to clarify why we can't use i.e., scipy.optimize.minimize (which has some output options and custom callback support), and if we ever need mini-batch/partial_fit() support for these experiments or not.

tylerjereddy · 2026-03-29T20:31:55Z

src/gfdl/model.py

        seed: int = None,
        reg_alpha: float = None,
        rtol: float | None = None,
+        gamma: float = None, 


Kostas' email on March 29/2026 suggests that there may be interest in the ability to provide different scaling values for different layers and/or to only apply scaling to a subset of layers, so the API design may need careful consideration.

The api can now pass different gamma for each layer.

tylerjereddy · 2026-03-31T21:57:18Z

Beyond the various cleanups and clarifications noted above, this will also need regression tests before it is seriously ready for review.

Given the amount of confusion I've seen around gamma scaling, the documentation should also explain how it works with crystal clarity.

nray · 2026-04-08T02:23:33Z

src/gfdl/model.py

            Z = H_prev @ w.T + b  # (n_samples, n_hidden)
            H_prev = self._activation_fn(Z)
+            if self.gamma is not None:
+                H_prev *= 1.0 / (H_prev.shape[1] ** self.gamma)


@shahyadk-bu, @eiviani-lanl : I would appreciate some feedback about the gamma scaling api implemented here. To my understanding, this is equivalent to Shahyad's implementation here, but still it would nice to get a confirmation from both of you.

shahyadk-bu · 2026-04-08T13:37:55Z

Hi Ray, This seems correct. I also have begun using the non-training method of solving the RVFL directly and have that code in my Git as well if you wanted to look at that reference as well. It is the folder called "RVFL_Solved" on my Git and there is a python script called "RVFL_model.py" which holds my RVFL class and the scaling is implemented there as well. Sincerely, Shahyad

…

On Wed, Apr 8, 2026 at 4:23 AM Navamita Ray ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In src/gfdl/model.py <#70 (comment)>: > Z = H_prev @ w.T + b # (n_samples, n_hidden) H_prev = self._activation_fn(Z) + if self.gamma is not None: + H_prev *= 1.0 / (H_prev.shape[1] ** self.gamma) @shahyadk-bu <https://github.com/shahyadk-bu>, @eiviani-lanl <https://github.com/eiviani-lanl> : I would appreciate some feedback about the gamma scaling api implemented here. To my understanding, this is equivalent to Shahyad's implementation here <https://github.com/shahyadk-bu/RVFL_Research/blob/main/RVFL_Trained/RVFL_Model.py#L45>, but still it would nice to get a confirmation from both of you. — Reply to this email directly, view it on GitHub <#70 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BZ2CZWVMIEDQOE5CSISBSJD4UWZ3ZAVCNFSM6AAAAACWYC7PESVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHM2DANZSGQZDENRXGY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

nray · 2026-04-09T02:44:33Z

Hi Ray, This seems correct. I also have begun using the non-training method of solving the RVFL directly and have that code in my Git as well if you wanted to look at that reference as well. It is the folder called "RVFL_Solved" on my Git and there is a python script called "RVFL_model.py" which holds my RVFL class and the scaling is implemented there as well. Sincerely, Shahyad
…
On Wed, Apr 8, 2026 at 4:23 AM Navamita Ray @.> wrote: @.* commented on this pull request. ------------------------------ In src/gfdl/model.py <#70 (comment)>: > Z = H_prev @ w.T + b # (n_samples, n_hidden) H_prev = self._activation_fn(Z) + if self.gamma is not None: + H_prev = 1.0 / (H_prev.shape[1] ** self.gamma) @shahyadk-bu https://github.com/shahyadk-bu, @eiviani-lanl https://github.com/eiviani-lanl : I would appreciate some feedback about the gamma scaling api implemented here. To my understanding, this is equivalent to Shahyad's implementation here https://github.com/shahyadk-bu/RVFL_Research/blob/main/RVFL_Trained/RVFL_Model.py#L45, but still it would nice to get a confirmation from both of you. — Reply to this email directly, view it on GitHub <#70 (review)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/BZ2CZWVMIEDQOE5CSISBSJD4UWZ3ZAVCNFSM6AAAAACWYC7PESVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHM2DANZSGQZDENRXGY . You are receiving this because you were mentioned.Message ID: @.**>

Hi Shahyad,
Thanks for the confirmation, I will also take a look at your RVFL implementation that you pointed to.

…t gamma per layer.

tylerjereddy · 2026-04-14T22:59:03Z

src/gfdl/model.py

        seed: int = None,
        reg_alpha: float = None,
        rtol: float | None = None,
+        gamma: float | np.typing.ArrayLike | None = None,


here and elsewhere, note that ArrayLike already includes floats and other scalars: https://numpy.org/devdocs/reference/typing.html#numpy.typing.ArrayLike

tylerjereddy · 2026-04-14T23:00:48Z

src/gfdl/model.py

+        before inputting to the next layer. None implies no
+        scaling is applied. A single value implies applying
+        the scaling each layer with the same value. For different
+        gammas per layer, pass an array like type.


a float is also an array-like in NumPy speak, so we should probably reword a bit

WIP, ENH: Add support for gamma scaling and iterative solvers.

041c542

nray self-assigned this Mar 19, 2026

tylerjereddy reviewed Mar 19, 2026

View reviewed changes

tylerjereddy added the enhancement New feature or request label Mar 27, 2026

tylerjereddy reviewed Mar 29, 2026

View reviewed changes

tylerjereddy mentioned this pull request Mar 29, 2026

ENH: Gamma Scaling Experiments #71

Open

nray commented Apr 8, 2026

View reviewed changes

nray added 3 commits April 8, 2026 22:20

MAINT: Remove regressor to end as in current model.

e18a7b9

ENH, MNT: Remove solvers from this MR. Extend api to support differen…

9373d96

…t gamma per layer.

Merge branch 'main' into nray/gamma_scaling

4009ff1

nray mentioned this pull request Apr 14, 2026

ENH: Gamma scaling iterative solvers #84

Open

MNT, TST: Add tests for gamma scaling.

ea76c88

tylerjereddy reviewed Apr 14, 2026

View reviewed changes

		return out


		class GFDLRegressor(RegressorMixin, MultiOutputMixin, GFDL):

		return np.random.default_rng(seed)


		def stochastic_gradient_descent(self, X, y):

Conversation

nray commented Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tylerjereddy commented Mar 31, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shahyadk-bu commented Apr 8, 2026 via email

Uh oh!

nray commented Apr 9, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nray commented Mar 19, 2026 •

edited

Loading