[FEATURE] Improve and optimize NuttX CI #20483

Activity

1. acassis

   added

   Type: EnhancementNew feature or requestNew feature or request

   on Oct 7, 2026
2. acassis

   changed the title [-][FEATURE] Improve and optimize our CI[/-] [+][FEATURE] Improve and optimize NuttX CI[/+] on Oct 7, 2026
3. Swatantra-66 commented on Oct 10, 2026

   

   Swatantra-66

   on Oct 10, 2026

   Contributor

   More actions

   Hi @acassis,

   I investigated the current CI pipeline (.github/workflows/arch.yml, build.yml, labeler.yml, and tools/ci/testlist/) to see why chip-isolated PRs like #20477 end up triggering all 14 arm-xx jobs.

   Current Pipeline Bottleneck

   1. Broad Labeling: .github/labeler.yml labels any change in arch/arm/** as Arch: arm.
   2. Coarse-Grained Filter: In .github/workflows/arch.yml, when Arch: arm is present, it skips non-ARM architectures (RISC-V, Xtensa, Simulator, etc.), but it passes all 14 arm-01 through arm-14 jobs to the matrix.
   3. Alphanumeric Partitioning: tools/ci/testlist/arm-*.dat divides chips into arbitrary alphanumeric buckets ([a]*, [b-h]*, etc.) rather than chip families, so a one-line change in imxrt triggers all 14 container jobs building hundreds of unrelated STM32, NRF52, LPC, and SAMA5 defconfigs.

   ---

   Proposed Improvement Approaches

   Option A: Family-Based Testlists + Sub-Labels (Direct Implementation of Your Suggestion)

   1. Refactor tools/ci/testlist/:
      Group ARM and RISC-V testlists by major vendor/chip families rather than arbitrary numbers:
      • arm-stm32f.dat, arm-stm32h.dat, arm-stm32l_u_g.dat
      • arm-imxrt.dat
      • arm-nrf.dat (nrf52, nrf53, nrf91)
      • arm-nxp.dat (lpc*, kinetis, s32k)
      • arm-sam.dat (sam34, sama5, samd, samv7)
      • arm-rp2040.dat (rp2040, rp23xx)
      • arm-other.dat
   2. Update labeler.yml:
      Apply chip family labels (e.g. Chip: imxrt, Chip: stm32) when changes are isolated to arch/<arch>/src/<chip>/** and boards/<arch>/<chip>/**.
   3. Update arch.yml:
      When a Chip: <family> label is detected without core OS changes, filter the matrix to only run arm-<family>.dat.

   Option B: Dynamic Chip Filter in testbuild.sh (Zero Matrix Bloat)

   Instead of creating dozens of separate GitHub Actions matrix jobs (which could hit GitHub Actions concurrency/queue limits during complex PRs):

   1. Detect modified chip directories from the PR changeset (e.g., imxrt).
   2. If changes are strictly confined to arch/<arch>/src/<chip>/** or boards/<arch>/<chip>/** (and no common files like sched/, fs/, include/nuttx/, or arch/*/src/common were touched):
      Pass a target filter flag to testbuild.sh (e.g., -F imxrt) so it skips board configs outside that chip family.
   3. For PR arch/arm/imxrt: Fix nxstyle issues on imxrt_usdhc.c #20477, this would compile only imxrt defconfigs in 2–3 minutes while keeping the existing matrix structure intact.

   ---

   I would be happy to work on this and submit a PR. Which approach aligns best with your vision for the CI?
4. acassis commented on Oct 10, 2026

   

   acassis

   on Oct 10, 2026

   ContributorAuthor

   More actions

   Hi @Swatantra-66 I think all improvements to CI will be beneficial. Also the idea from @casaroli to use ccache between CI runs is a good idea.

   👍React with 👍1Reacted by Swatantra Yadav
5. Swatantra-66 commented on Oct 10, 2026

   

   Swatantra-66

   on Oct 10, 2026

   Contributor

   More actions

   Thank you @acassis!

   I investigated the current setup and noticed that ccache and compiler wrappers are already configured in tools/ci/platforms/ubuntu.sh inside the Docker image, but GitHub Actions currently discards the cache on container teardown.

   I will structure this into two focused phases:

   1. Enable ccache persistence: Point CCACHE_DIR to the workspace and add actions/cache in .github/workflows/build.yml so matrix jobs preserve and restore cached objects between runs.
   2. Chip/Family build filtering: Optimize testlists and runner filtering so PRs modifying isolated chip peripherals skip unrelated targets.

   I will start prototyping the ccache cache integration and follow up with a PR.
6. Swatantra-66

   added a commit that references this issue on Oct 11, 2026

   ci: enable ccache persistence across CI workflow runs

   ...

   2ae45d1
7. xiaoxiang781216

   added a commit that references this issue on Oct 12, 2026

   ci: enable ccache persistence across CI workflow runs

   ...

   1a2d22e
8. Swatantra-66 commented on Oct 12, 2026

   

   Swatantra-66

   on Oct 12, 2026

   Last edited by Swatantra-66

   Contributor

   More actions

   Hi @acassis, @xiaoxiang781216,

   Update: Phase 1 has been completed and merged into master via PR #20580 (persistent ccache caching across GitHub Actions workflow runs).

   I am ready to proceed with Phase 2: Chip / Architecture Build Filtering so that chip-isolated PRs (like #20477) don't trigger builds for all unrelated chip defconfigs.

   To ensure this aligns with your vision for the CI infrastructure, I'd like your thoughts on the implementation approach:

   • Approach 1 (Granular Testlists & Matrix Routing):
     Reorganize tools/ci/testlist/arm-*.dat into vendor/family groups (e.g., arm-stm32, arm-nxp-imx, arm-sam, arm-nrf, arm-rp2040, arm-other) and update .github/workflows/arch.yml to only launch the matching matrix job for isolated chip changes.

   • Approach 2 (Dynamic Filter in testbuild.sh / cibuild.sh):
     Keep the existing matrix structure, but detect if the changeset is strictly confined to arch/<arch>/src/<chip>/** and boards/<arch>/<chip>/**. If so, pass a filter to skip non-matching board defconfigs during the run.

   Would you prefer Approach 1 (family-based testlists) or Approach 2 (runner target filtering)? Once aligned, I will prepare the implementation.
9. simbit18 commented on Oct 12, 2026

   

   simbit18

   on Oct 12, 2026

   Last edited by simbit18

   Contributor

   More actions

   @Swatantra-66
   What tests have been run on GitHub?

   Please see “Showing caches from all workflows”
   Every PR that is merged adds a cache file, and that doesn't help

   

   Approaching total cache storage limit (10.02 GB of 10 GB used)
   Least recently used caches will be automatically evicted to limit the total cache storage to 10 GB.
   https://github.com/apache/nuttx/actions/caches

   @acassis @xiaoxiang781216

   Cache usage in Nuttx was removed after this
   #14376

   👍React with 👍1Reacted by Swatantra Yadav
10. Swatantra-66 commented on Oct 12, 2026

    

    Swatantra-66

    on Oct 12, 2026

    Contributor

    More actions

    Hi @simbit18, thanks for checking the cache storage metrics and raising this!

    Regarding the tests run: PR #20580 was tested on forks to verify that ccache saved and restored cleanly without runner permission errors, and bounded each job with CCACHE_MAXSIZE=1G to prevent individual jobs from ballooning.

    Regarding the 10 GB limit: GitHub Actions enforces a 10 GB cache storage limit per repository and automatically evicts Least Recently Used (LRU) caches when this threshold is reached, without failing workflow runs.

    However, to prevent high churn from PR runs and keep total cache storage well under the 10 GB quota, here are two immediate improvements we can apply:

    1. Only save cache on master branch (Read-only for PRs):
       We can configure actions/cache so that PR runs only restore the cache from master and do not save new cache entries. Only merges to master will update the cache. This guarantees that total cache entries are strictly bounded to the matrix size (e.g. ~10–14 entries total, keeping usage well under 5–6 GB).
    2. Selective Caching:
       Only enable caching for the heaviest/slowest matrix jobs (e.g. arm-01 to arm-04) where compilation time is largest.
    3. Revert if preferred:
       If the maintainers prefer not to use GitHub Actions caching at all as per past [URGENT] Reducing our usage of GitHub Runners #14376 discussions, we can also cleanly revert ci: enable ccache persistence across CI workflow runs #20580.

    @acassis @xiaoxiang781216 Would you prefer updating the workflow so that caching only saves on master (read-only for PRs), or would you prefer reverting PR #20580?
11. acassis commented on Oct 12, 2026

    

    acassis

    on Oct 12, 2026

    ContributorAuthor

    More actions

    @simbit18 @Swatantra-66 what is the drawback of keeping the ccache as in #20580 ?

    Maybe we should keep this way for more a few days to see how this goes, what do you think?

    👍React with 👍1Reacted by Swatantra Yadav
12. Swatantra-66 commented on Oct 13, 2026

    

    Swatantra-66

    on Oct 13, 2026

    Contributor

    More actions

    Hi @acassis,

    To answer your question directly:

    1. No build failures or billing penalties: There is no operational failure or extra cost. When cache storage reaches the 10 GB quota, GitHub Actions automatically applies a Least Recently Used (LRU) eviction policy, silently dropping the oldest unused cache entries to make space for new ones.
    2. The only minor drawback is cache churn: High PR frequency means newer cache files can evict older caches from less frequently run matrix jobs, resulting in occasional cold builds for those specific targets. However, active targets will consistently hit warm caches and reduce compilation time.

    I completely agree with your suggestion to keep it running as-is for a few days to observe real-world CI compilation times and telemetry. If we notice too much churn later, we can always refine it so only master merges save new caches.

    In the meantime, we can monitor the runner metrics while moving forward with Phase 2!