Last week my coworker Andrew Halberstadt talked me through the process of configuring Firefox CI to run a given test suite with Fission enabled on additional platforms.

I am working on a patch to do this for our telemetry integration tests which are set up with mozharness and use treeherder symbol tt(c). Since the process should be close to identical for similar test suites, I decided to summarize what I’ve learned in this post, so next time someone on my team wants to do this, we don’t need to bug Andrew again. 😅

The overall taskgraph documentation can be found in the Firefox Source Docs. 📝

Configure a test variant

The first step is to configure a test variant for Fission for our test suite. When a test suite configures test variants, Firefox CI will create a clone for all of the tasks of this test suite. The default configuration will always run and does not need to be listed as a variant.

The available variants are defined in taskcluster/taskgraph/transforms/tests.py. We can see in this file that the name of the variant we want to enable is "fission".

The telemetry-tests-client suite is defined in taskcluster/ci/test/misc.yml and doesn’t configure any variants yet:

telemetry-tests-client:
    description: "Telemetry tests client run"
    suite: telemetry-tests-client
    treeherder-symbol: tt(c)
    max-run-time: 1200
    tier: default
    python-3: true
    mozharness:
        script: telemetry/telemetry_client.py
        config:
            by-test-platform:
                linux.*:
                    - remove_executables.py
                windows.*: []
                macosx.*: []

We add a new key named variants to the telemetry-tests-client object and assign a list with a single item:

variants: [fission]

Specify projects by variant and test platform

The platforms and projects that we want the Fission enabled tests to run on are listed in bug 1696040. Let’s create a table to make it easier to verify our patch later on. I use 🤖 here to indicate that task sets need to be generated for a project and 🚫 to indicate that task sets should not be generated for a project:

The next step is to update the test suite configuration to specify the projects we want the tests to run on by variant and test platform. We can do this by combining by-variant with by-test-platform. The contained keys are regular expressions, which means we can specify projects for multiple platforms in a single line:

run-on-projects:
    by-variant:
        fission:
            by-test-platform:
                linux.*-64/debug: [trunk]
                linux.*-64(?:-shippable|-asan)?/opt: [trunk]
                windows.*-64/debug: [trunk]
                windows.*-64(?:-shippable)?/opt: [trunk]
                default: []
        default: built-projects

Note that the value [trunk] is a shorthand for [autoland, mozilla-central].

The taskgraph generation procedure includes a step in which a filter removes shippable from autoland and non-shippable from mozilla-central , so it’s OK to specify both projects here and leave it to the filter to clean up the configuration for us, see taskcluster/taskgraph/target_tasks.py.

We need to make sure we don’t run the fission variant for the default platform by assigning an empty list []. Likewise, we need to assign built-projects to the variant default to restore the default configuration for non-fission builds. This special case uses the parent build task’s run-on-projects, meaning that tests run only on platforms that are built.

Set name for Treeherder group

Before we can generate a Taskcluster taskgraph for our updated test suite configuration, we need to add a name for the new Treeherder group in taskcluster/ci/config.yml.

We can copy the entry for the existing telemetry tests group and add the -fis suffix for Fission variants:

'tt-fis': 'Telemetry tests with fission enabled'

Generate taskgraphs

Next we generate a taskcluster taskgraph locally to see which task sets are generated for the updated test suite configuration. We commit the above changes and run the following command. It will generate and diff the current taskgraph against the previous revision for the autoland project:

./mach taskgraph target --target-kind test --fast --diff .~1 -p project=autoland

The following new task sets were generated for autoland:

+test-linux1804-64-asan/opt-telemetry-tests-client-fis-e10s
+test-linux1804-64/debug-telemetry-tests-client-fis-e10s
+test-linux1804-64/opt-telemetry-tests-client-fis-e10s
+test-windows10-64/debug-telemetry-tests-client-fis-e10s
+test-windows10-64/opt-telemetry-tests-client-fis-e10s

We repeat this step for the mozilla-central project:

./mach taskgraph target --target-kind test --fast --diff .~1 -p project=mozilla-central

The following new task sets were generated for mozilla-central:

+test-linux1804-64-asan/opt-telemetry-tests-client-fis-e10s
+test-linux1804-64-shippable/opt-telemetry-tests-client-fis-e10s
+test-linux1804-64/debug-telemetry-tests-client-fis-e10s
+test-windows10-64-shippable/opt-telemetry-tests-client-fis-e10s
+test-windows10-64/debug-telemetry-tests-client-fis-e10s

Verify generated task sets

Once we have verified that task sets for the specified platforms are generated for the requested project (🤖) and not generated for the other one (🚫), we cab submit our changes to Phabricator for review:

For the sake of completeness, here’s the updated test suite configuration:

telemetry-tests-client:
    description: "Telemetry tests client run"
    suite: telemetry-tests-client
    treeherder-symbol: tt(c)
    variants: [fission]
    run-on-projects:
        by-variant:
            fission:
                by-test-platform:
                    linux.*-64/debug: [trunk]
                    linux.*-64(?:-shippable|-asan)?/opt: [trunk]
                    windows.*-64/debug: [trunk]
                    windows.*-64(?:-shippable)?/opt: [trunk]
                    default: []
            default: built-projects
    max-run-time: 1200
    tier: default
    python-3: true
    mozharness:
        script: telemetry/telemetry_client.py
        config:
            by-test-platform:
                linux.*:
                    - remove_executables.py
                windows.*: []
                macosx.*: []

Back to posts