Potential For Automation During Release Process

[krzywon]

A number of things could be automated during the release process using the on: tags: workflow.

• Upload wheels to pypi
  • PyPi allows owners to link Github workflows that can push directly to a repo. This is already set up for sasview, sasmodels, and sasdata.
  • Need to add the appropriate action step to our workflows.
  • This process bypasses all github secrets in favor or known, trusted, workflows linked to pypi.
• Generate zenodo(?)
  • Currently one account manages the zenodo and everyone has the password. Need to distribute ownership/management to other accounts.
  • Add a step to the CI that creates the record, and updates the DOI in the repo. This record generation would have to be before sasview is built, and then finalized at the end of the process once the binaries exist.
  • Authorship list - order matters. Who should be first, last, and in between?
• Link binaries to a releases
  • Our nightly build already attaches binaries to a release. We should be able to do the same for full releases.
• Open a PR to merge the release branch into main/master/etc.

[krzywon]

Zenodo authorship order suggestion:

• First author should be the release manager
• 2nd through nth are the n-1 people who directly contributed to the current release, in alphabetical order
  • 'Direct' means people who had a commit in any of sasview, sasdata, sasmodels, or the tutorials repos, regardless of the type of commit made, including code changes, documentation, addition of example data, etc.
  • Other potential direct contributors include people who had a significant impact on the release (led meetings, reviewed PRs, extensive testing, etc)
• n + 1 though the end of authors should be all other authors, in alphabetical order

[butlerpd]

After thinking about this a bit now I would argue that we should NOT try splitting into two lists. From what I understand about the purpose, doing this will not actually help and will add a not insignificant burden to every release. However, putting the name of the release manager first does make sense as it is trivial to do, requiring no discussion, and will actually do what is intended: recognizing that person.

Details:

• Citations will never list all the names but instead cite first name et al. Thus splitting the list is not generally visible and will not really serve the purpose of recognizing the active vs inactive creators. But the first author as release manager will be seen and thus achieve its purpose?
• For those who pull the full citation list with the number of names currently, it is not clear that anybody will notice the subtlety of the alphabet starting over again. If anybody does it is not clear that they will understand why? If this is established as our new routine, every future release will require that much extra work and discussion to sort names into three categories not just two for no clear gain.
• Finally, there is the question of what we mean by "directly contributed to the current release?" I assume what is meant is those who added new features or fixed bugs for things that are new in this release vs previous releases. That however seems problematic to me since many features in the current version will not have had any new input. If I'm using the latest release, as recommended, but utilizing a feature has not been altered since the previous version, why would I be focusing on referencing a bunch of people that contributed nothing to what I'm using? I.e. why would I be making the distinction?

[krzywon]

Opening PR might require using a bot account - @krzywon to work with @ajj about changing the @sasview-bot ownership

[jamescrake-merani]

The other day, I came across a project called Gitleaks. In short, it checks, and prevents secrets from being committed accidentally to the repository. While the risk of this happening on the SasView repository is very small, it also isn't zero. We may be handling Zenodo secrets, for example, and we really don't want to leak these.

The problem with leaking secrets on GitHub is that because we're an open source project, once we do it, its very difficult to clawback, and we'd likely have to change our secrets altogether. Even worse, if they are committed accidentally but no one notices, a malicious actor may find them, and could try to use them.

While Gitleaks would require a little bit of setup, it gives us peace of mind later that there is a protection against secrets leaking. It may be useful further down the line if we need to implement API access for other sites, and need to handle secrets again.

There are a few different ways we could configure Gitleaks. The two main options seem to be: adding it to the CI, or as a pre-commit. We can probably discuss exactly the approach we'd prefer to take, but I would lean towards setting it up as a pre-commit as we're already configuring Ruff as a pre-commit.

[jamescrake-merani]

I was discussing this on the cross government slack this morning, and some interesting points came up:

• The official Gitleaks Github action is actually licenced under a proprietary 'source available' licence. This is not ideal in my view. Though if we did want to go down this road, someone from NHS England showed me a workaround that wraps the open source Gitleaks executable in a custom Github action.
• Github does have some builtin functionality for detecting secrets in a repository. I believe it creates an 'alert' if it finds something it thinks is a secret which has to be remedied. Though something like a pre-commit may be better in making sure the secret never reaches the repository in the first place.

[krzywon]

Automating Release Processes

Preamble

To decrease the number of person-hours required for each release, many processes can and should be automated.

Proposal

All processes that can be automated should be automated within our CI.

Processes that can be automated

• Generating the DOI
  • A separate workflow should be triggered by the release manager prior to generating the first release.
  • This process will need commit access to the current branch to add the new DOI to the code base.
• Generating wheels and tar.gz files and uploading them to PyPi
  • Wheels should be automatically generated during the build process.
  • When a new tag is created, the wheels from the tag action should be uploaded to PyPi.
  • To minimize the number of secrets kept on GitHub, PyPi repo maintainers must allow direct push access from the uploading workflows.
• Generating conda packages and uploading them to conda-forge
  • Conda packages should be automatically built during the CI process.
  • When a new tag is created, the conda package should be automatically uploaded to conda-forge.
  • Conda-forge maintainers must allow
• Attaching binaries to releases
  • All release files should be added to the release page when a new release is created.
• Open a pull request to merge the release branch into the default branch for the repo
  • A new branch merging the default branch in to the release branch should be created
  • The new branch should be used to open a pull request to ensure the default branch is up-to-date with the release branch.
• In order to enable all of these steps, all repositories should allow GitHub action runners to create branches, commit changes, and open pull requests.

[krzywon]

Discussion from todays technical meeting:

• Does this need to be an ADR or just an on-going discussion on what can and should be automated? Is there a better place for this?
• Alternately, ADRs are designed to be architecture structures and how we came to these decisions. They can be changed, but they help developers to understand what needs to be done. 'Single place of truths.'
• Should this be a list of issues instead? A series of feature requests in each repo would also capture everything set in here, without limiting the set of tasks to a long-term discussion that is required for the ADRs.
• Shorten the discussion to 'we agree to automate as much as possible' (in a few more words)
• How do we manage issues and ADRs to keep them both visible, clear, yet still distinct from one another.

[krzywon]

After today's discussion, the decision is this will not be an ADR. I will add issues to each repository related to each step for automating many of our CI processes that aren't already part of the build process.