Carbon Benchmarking the GAW

[ArneTR]

This issue describes the idea of benchmarking the energy cost and carbon emissions of a reduced version of the GAW worker.

Currently the technical setup is to have transformation done in a cloudflare worker and then sending a reduced HTML/JS/CSS setup to the client.

Since it is very hard to estimate how much the energy and carbon cost in a real cloudflare worker will be this issue recommends a simpler setup that will measure the minimum cost of making the page transformation.
The theory is that if that minimum cost is already higher than the gains we can see in the browser for rendering, than the cost of brining this to a cloudflare worker setup with all the network and separate system overhead will be even worse.

Measurement setup description

The measurement shall be done with the Green Metrics Tool

All measurements will be done on the same hardware to make it comparable.

What will be compared in the end is:

• Cost of rendering the full fledged frontend without any carbon aware optimizations
  vs.
• Cost of rendering the reduced carbon aware frontend + the cost of the transformation (nodeJS code)

Current version

In order to benchmark the current version the live website can be used.
GMT can spin up a firefox browser and measure all frontend costs (page rendering, AJAX requests etc.) and give out an energy and carbon value

What is needed here is a working link to the live website which is guaranteed to have the "most expensive" version of the website (aka the one with carbon aware features off). If that is not configurable the test can also be timed to run at a specific point in time to get this version of the website.

Carbon aware version

Here either a live endpoint where the reduced version can be seen is needed, or, if simpler, a full HTML dump of the reduced site.
This assumes that reduced site has no AJAX requests in the page anymore and all content is either static HTML/CSS or javascript which does not make any external requests anymore.

Furthermore the nodeJS code is needed that does the actual transformation along with the expected input.
This measurement is done separately and the energy/carbon values are thn later added to the rendering costs to get a comparable number to compare with the current version.

Alternate approach

If live endpoints cannot be provided the GMT can also orchestrate a full webserver setup. Here all server files and configuration need to be supplied.

@mrchrisadams @fershad

[fershad]

Hi @ArneTR thanks for this.

One question, can GMT be configured to simulate user action on a website (e.g. clicking a toggle, selecting an option)?

[ArneTR]

GMT is just a benchmarking tool. It needs a driver to do that. Sorry for not mentioning that.

The driver I plan to use is a Firefox browser automated with Playwright, which for instance can do all the interactions you mentioned.

If you have any other automation tool already configured it can also be used (Selenium, Puppeteer, curl etc.)

[mrchrisadams]

@ArneTR do you folks have a reference playright image that you use for GMT test runs?

I see the GCB Playright container in use in this example here:

https://github.com/green-coding-solutions/example-applications/tree/main/wordpress-official-starter

Is that the preferred one to use if we were to create a usage scenario to record any user journeys?

What I'm basing this on:

the usage scenario file has this content here:

services:
  gcb-playwright:
    image: greencoding/gcb_playwright:v16
    networks:
      - gcb-mariadb-wordpress-network
    depends_on:
      gcb-wordpress:
        condition: service_healthy

flow:
  - name: Check Website
    container: gcb-playwright
    commands:
      - type: console
        command: python3 /tmp/repo/wordpress-official-starter/playwright-flow.py
        note: Starting User Journey
        read-notes-stdout: true
        log-stdout: true
        log-stderr: true

Which I read as telling GMT to drive playright inside the gcb-playright container, using the steps in playwright-flow.py to access the wordpress site being tested.

I get that this is checking the whole stack, not just the client, so if there is a simplified example for just checking resource usage on the client, I'd welcome a pointer to it, as I don't know how we do simpler lookups with GMT at present.

[ArneTR]

Yes, you can use this container (https://hub.docker.com/r/greencoding/gcb_playwright)
We have created it as it is for Python.

If you use JS to interface with the library you can also use the microsoft native one (e.g. mcr.microsoft.com/playwright:v1.54.0-noble)

2. In the example that you extracted only one service is started: gcb-playwright .
   So no other services will be created on the measurement machine. This will in turn only do client energy and carbon accounting. (You need to change what the playwright-flow.py contains of course, as this is expecting a running local wordpress installation)

3. My offer still stands: I am happy to set this all up in GMT. I just need the files as described in the issue from your website.

[mrchrisadams]

Thanks @ArneTR - we've just launched Branch magazine in its shiny new GAW incarnation and that's been taking up quite a lot of our attention.

Thank you again for the offer btw - I'll catch up with @fershad this week to discuss user journeys and getting the necessary files to you 👍

[fershad]

Hi @ArneTR just a note that we've not forgotten about this. I've got some code working and have deployed it to our staging site so that we can use staging to test a "client-side" implementation versus production to test "server-side".

We've got a blocker at the moment that we need to update the CSP for the staging website to allow it to get the energy data it needs. Once that's done I'll update this issue.

[fershad]

Hi @ArneTR okay we've got two working links setup.

Client-side grid-aware code: https://branch-staging.climateaction.tech/
Edge-side grid-aware code: https://branch.climateaction.tech/

Both versions have grid-aware turned off by default for first time visitors.

Our hypothesis:
Running grid-aware logic and HTML changes on the Edge should result in lower energy consumption when compared to running the logic and HTML changes in the client.

Code:

The client side code can be found here for reference:

• Logic: https://github.com/climateaction-tech/branch-theme/blob/staging/assets/js/gaw.js
• DOM changes: https://github.com/climateaction-tech/branch-theme/blob/staging/assets/js/gaw-views.js

The edge implementation can be found here:

• https://github.com/thegreenwebfoundation/branch-gaw-worker/blob/main/src/index.js

The edge implementation runs through a Cloudflare Worker which performs both the logic checks & HTML changes before returning a response to the browser.

We can test the energy consumption on the client first, and in the meantime explore how a Workers like environment could be created for testing the energy used for those processes.

Does this feel like a reasonable approach? I can write up some playwright tests (in JavaScript) after lunch.

[ArneTR]

Uh, nice to see this moving forward :)

My remarks / questions:

• Yes, a playwright file would be nice. You can factor in what you are imagining for the browser to do on the page if it shall be any more than going to one page. I would recommend one super simple setup where you really visit the page and do nothing and one more complex if you like with scrolling down, clicking etc.
• What constitutues a "first time" visitor and what not? I think for the test it would be best to have two deterministic and easy instrumentable versions. Aka it would be nice to have a url param ala ?gaw=off that never executes the grid aware functionality and one ?gaw=on that forces it so that I can trigger a fixed behaviour. Is that possible? Or maybe achievable through a different way?
• Can you make me a CLI example for the nodeJS code that I can energy-benchmark directly? Or does the code rely on custom cloudflare libraries that are not available locally in nodeJS? Can it be run then in Cloudflares local test tool Miniflare?

[fershad]

Thanks Arne.

1. I'll create a few playwright tests so that we can test different functionality.
2. The on/off functionality is controlled via cookies which should possible to add to the browser context when setting up the tests. I'll include those in the test files.
3. I think that should be possible, but might not be something I can get around to this week.

[fershad]

@ArneTR let me know if you need these test provided in any specific way, or if you can work with the blog below.

I've written 3 tests, each to be run on the Branch Production (server-side GAW) and Branch Staging (client-side GAW) websites.

Test 1: Launch the site with no cookies (default view - no GAW changes applied)
Test 2: Launch the site with GAW enabled (set via cookies)
Test 3: Launch the site with GAW enabled and navigate to the Issues page.

I think this should be enough to test the hypothesis above. I'll work on something to replicate a Cloudflare Workers runtime over the next few days.

const { firefox } = require("playwright");

// Test 1 - BRANCH: Launch Branch website with no cookies
(async () => {
  const browser = await firefox.launch({ headless: false }); // Or 'firefox' or 'webkit'.
  const context = await browser.newContext();
  await context.clearCookies();
  const page = await context.newPage();
  await page.goto("https://branch.climateaction.tech");
  await page.waitForTimeout(5000);
  await context.close();
  await browser.close();
})();

// Test 1 - BRANCH STAGING: Launch Branch website with no cookies
(async () => {
  const browser = await firefox.launch({ headless: false }); // Or 'firefox' or 'webkit'.
  const context = await browser.newContext();
  await context.clearCookies();
  const page = await context.newPage();
  await page.goto("https://branch-staging.climateaction.tech");
  await page.waitForTimeout(5000);
  await context.close();
  await browser.close();
})();

// Test 2 - BRANCH: Launch Branch website with grid-aware cookies set
(async () => {
  const browser = await firefox.launch({ headless: false }); // Or 'firefox' or 'webkit'.
  const context = await browser.newContext();
  await context.clearCookies();
  // Set the cookie so that grid-awareness is triggered
  await context.addCookies([
    {
      name: "gaw-user-opt-in",
      value: "true",
      domain: "branch.climateaction.tech",
      path: "/",
    },
  ]);
  const page = await context.newPage();
  await page.goto("https://branch.climateaction.tech");
  await page.waitForTimeout(5000);
  await context.close();
  await browser.close();
})();

// Test 2 - BRANCH STAGING: Launch Branch website with grid-aware cookies set
(async () => {
  const browser = await firefox.launch({ headless: false }); // Or 'firefox' or 'webkit'.
  const context = await browser.newContext();
  await context.clearCookies();
  // Set the cookie so that grid-awareness is triggered
  await context.addCookies([
    {
      name: "gaw-user-opt-in",
      value: "true",
      domain: "branch-staging.climateaction.tech",
      path: "/",
    },
  ]);
  const page = await context.newPage();
  await page.goto("https://branch-staging.climateaction.tech");
  await page.waitForTimeout(5000);
  await context.close();
  await browser.close();
})();

// Test 3 - BRANCH: Launch Branch website with grid-aware cookies set and visit Issues page
(async () => {
  const browser = await firefox.launch({ headless: false }); // Or 'firefox' or 'webkit'.
  const context = await browser.newContext();
  await context.clearCookies();
  // Set the cookie so that grid-awareness is triggered
  await context.addCookies([
    {
      name: "gaw-user-opt-in",
      value: "true",
      domain: "branch.climateaction.tech",
      path: "/",
    },
  ]);
  const page = await context.newPage();
  await page.goto("https://branch.climateaction.tech");
  await page.getByRole("link", { name: "Issues" }).click();
  await page.waitForNavigation("https://branch.climateaction.tech/issues");
  await page.waitForTimeout(5000);
  await context.close();
  await browser.close();
})();

// Test 3 - BRANCH STAGING: Launch Branch website with grid-aware cookies set and visit Issues page
(async () => {
  const browser = await firefox.launch({ headless: false }); // Or 'firefox' or 'webkit'.
  const context = await browser.newContext();
  await context.clearCookies();
  // Set the cookie so that grid-awareness is triggered
  await context.addCookies([
    {
      name: "gaw-user-opt-in",
      value: "true",
      domain: "branch-staging.climateaction.tech",
      path: "/",
    },
  ]);
  const page = await context.newPage();
  await page.goto("https://branch-staging.climateaction.tech");
  await page.getByRole("link", { name: "Issues" }).click();
  await page.waitForNavigation(
    "https://branch-staging.climateaction.tech/issues",
  );
  await page.waitForTimeout(5000);
  await context.close();
  await browser.close();
})();

[ArneTR]

This looks good! Given my current planning I can set the tests up by the end of the week / saturday.

2. Let me know once you have the nodeJS CLI / miniflare runnable script for testing the cloudflare worker

[fershad]

Thanks @ArneTR ~~

I'm working on the Miniflare thing now, might have something over the weekend for you.
Just had a thought though, would you be able to test using the wrangler dev (or npx wrangler dev) command? That runs the worker locally anyway at localhost:8787. Is that something you can work with?

[ArneTR]

An intermediate update on this.

• I created a repository that works with GMT at: https://github.com/green-coding-solutions/branch-magazine-energy-tests
• First dummy measurements are done at https://metrics.green-coding.io/runs.html?uri=branch&show=repositories
• I also set up a local wrangler test at https://github.com/green-coding-solutions/branch-magazine-energy-tests/tree/main/cloudflare-worker

This is just a work in progress so far. The measurements are just to verify that it works in general, but they need validation and tuning.

At the moment some stuff is still not working. Here my questions:

• I split up the tests for staging and "normal" into two files. I suppose that is OK?

  • Could you explain again what the difference between staging and normal is and if they are independent, or need to be touched sequentially?

• I commented out the third tests. Going to the Issues produced a timeout for me with playwright. Is this a local issue for me? Or was there a typo or something in the test script?

• I created a simple test application to test the worker with a rick-roll youtube. I docuemented in the Readme how to run it. My expectation would be to see a Youtube-Lite container instead of a Youtube one. But sadly that is not the case. Am I instrumenting your Cloudflare Worker plugin wrongly?

Screenshot of my browser debug:

Sorry for the many questions. But am sadly not to used to use cloudflare workers ... :/

[fershad]

Thanks for this @ArneTR! No worries about the questions. One at a time:

I split up the tests for staging and "normal" into two files. I suppose that is OK?

This is perfect. The two sites are independent, and have different functionality which is what we are testing for.

Could you explain again what the difference between staging and normal is and if they are independent, or need to be touched sequentially?

That staging site is running all the grid-aware code & performing DOM manipulation in the client when the page loads. The normal site is running all the grid-aware code & manipulating the DOM inside the Cloudflare Worker. Part of the purpose of these tests is to see how much (if any) energy is saved on the client by moving all that compute to the edge.

I commented out the third tests. Going to the Issues produced a timeout for me with playwright. Is this a local issue for me? Or was there a typo or something in the test script?

I'll have a look at that test & if there's a fix will submit a PR to your repo.

I created a simple test application to test the worker with a rick-roll youtube. I docuemented in the Readme how to run it. My expectation would be to see a Youtube-Lite container instead of a Youtube one. But sadly that is not the case. Am I instrumenting your Cloudflare Worker plugin wrongly?

I'll submit a PR to update the config.