Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -327,7 +327,7 @@ local-stovepipe-stop: ## Stop the Stovepipe service

mocks: ## Generate mock files using mockgen
@echo "Generating mocks..."
@$(BAZEL) run @rules_go//go -- generate ./submitqueue/extension/storage/... ./submitqueue/extension/buildrunner/... ./submitqueue/extension/changeprovider/... ./platform/extension/counter/... ./platform/extension/messagequeue/... ./submitqueue/extension/queueconfig/... ./submitqueue/extension/mergechecker/... ./submitqueue/extension/pusher/... ./submitqueue/extension/scorer/... ./submitqueue/extension/conflict/... ./platform/consumer/... ./stovepipe/extension/storage/...
@$(BAZEL) run @rules_go//go -- generate ./submitqueue/extension/storage/... ./submitqueue/extension/buildrunner/... ./submitqueue/extension/changeprovider/... ./platform/extension/counter/... ./platform/extension/messagequeue/... ./submitqueue/extension/queueconfig/... ./submitqueue/extension/mergechecker/... ./submitqueue/extension/pusher/... ./submitqueue/extension/scorer/... ./submitqueue/extension/conflict/... ./platform/consumer/... ./stovepipe/extension/storage/... ./stovepipe/extension/sourcecontrol/...
@echo "Mocks generated successfully!"

proto: ## Generate protobuf files from .proto definitions
Expand Down
8 changes: 8 additions & 0 deletions platform/base/page/BUILD.bazel
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
load("@rules_go//go:def.bzl", "go_library")

go_library(
name = "page",
srcs = ["page.go"],
importpath = "github.com/uber/submitqueue/platform/base/page",
visibility = ["//visibility:public"],
)
30 changes: 30 additions & 0 deletions platform/base/page/page.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
// Copyright (c) 2025 Uber Technologies, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

// Package page defines a generic, cursor-paginated result envelope shared across
// domains. Producers return one bounded Page at a time; callers walk further by
// passing the page's NextCursor back to the producing call until it is empty.
package page

// Page is one bounded slice of a larger sequence, plus an opaque cursor for
// fetching the next page. The element type T is the domain value being paged
// (e.g. a commit URI string, or an entity).
type Page[T any] struct {
// Items are the elements in this page, in the producer's defined order.
Items []T
// NextCursor is an opaque token for fetching the next page, passed back to
// the producing call. It is empty when this page is the last one. Its
// encoding is defined and interpreted solely by the producer.
NextCursor string
}
9 changes: 9 additions & 0 deletions stovepipe/extension/sourcecontrol/BUILD.bazel
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
load("@rules_go//go:def.bzl", "go_library")

go_library(
name = "sourcecontrol",
srcs = ["sourcecontrol.go"],
importpath = "github.com/uber/submitqueue/stovepipe/extension/sourcecontrol",
visibility = ["//visibility:public"],
deps = ["//platform/base/page"],
)
21 changes: 21 additions & 0 deletions stovepipe/extension/sourcecontrol/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# SourceControl

Vendor-agnostic interface through which Stovepipe talks to a version control system. It is the **sole owner of URI semantics**: a URI is an opaque, VCS-agnostic locator of a commit. The `git://` scheme used by the reference backend is just one encoding — a Mercurial or Perforce backend mints its own behind the same contract. Nothing outside an implementation parses a URI; it is a token you hand back to ask questions about a ref.

A `SourceControl` is **bound to a single queue** (a repo+ref) when its `Factory` constructs it from a `Config`, so the behavioral methods take no queue argument. Per the repository's extension rules, this package holds the `SourceControl` interface, its `Config`, and the `Factory` *interface* only — concrete `Factory` implementations and the per-queue routing that picks a backend for a `Config.QueueName` live in the wiring layer.

## Behavior

- **Latest** resolves the queue's ref to the URI of its latest commit — the commit a new validation `Request` is minted against during `ingest`.
- **IsAncestor** answers whether one URI is an ancestor of another. The `process` stage uses it to choose a build strategy: if the queue's last-green URI is no longer an ancestor of the latest commit, history was rewritten and a full build is required rather than an incremental one.
- **History** returns a bounded, newest-first page of commit URIs on the ref, using the shared generic `page.Page[string]` (`platform/base/page`). It is paginated with an opaque cursor: callers pass an empty cursor for the newest page and the page's `NextCursor` to walk further back, stopping when it is empty. Pagination keeps a remote backend cheap; callers join the URIs against the request store to render the greenness of each commit.

## Errors

Implementations return plain errors and use the package sentinel `ErrNotFound` (with the `IsNotFound` / `WrapNotFound` helpers) when a queue, ref, or URI cannot be resolved. They do not classify errors as user- or infra-caused — the calling controller does that.

## Implementations

- **fake** — an in-memory backend seeded with a queue's ref history (newest first), for examples and tests.

To add a backend, create `sourcecontrol/{backend}/`, implement the `SourceControl` interface, and return it from a `New(...)` constructor.
23 changes: 23 additions & 0 deletions stovepipe/extension/sourcecontrol/fake/BUILD.bazel
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
load("@rules_go//go:def.bzl", "go_library", "go_test")

go_library(
name = "fake",
srcs = ["fake.go"],
importpath = "github.com/uber/submitqueue/stovepipe/extension/sourcecontrol/fake",
visibility = ["//visibility:public"],
deps = [
"//platform/base/page",
"//stovepipe/extension/sourcecontrol",
],
)

go_test(
name = "fake_test",
srcs = ["fake_test.go"],
embed = [":fake"],
deps = [
"//stovepipe/extension/sourcecontrol",
"@com_github_stretchr_testify//assert",
"@com_github_stretchr_testify//require",
],
)
102 changes: 102 additions & 0 deletions stovepipe/extension/sourcecontrol/fake/fake.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
// Copyright (c) 2025 Uber Technologies, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

// Package fake provides an in-memory sourcecontrol.SourceControl seeded with a
// single queue's linear ref history, ordered newest-first. It is intended for
// examples and tests only, never production. Ancestry is decided by position in
// the seeded slice: an earlier commit (larger index) is an ancestor of a later
// one (smaller index).
package fake

import (
"context"

"github.com/uber/submitqueue/platform/base/page"
"github.com/uber/submitqueue/stovepipe/extension/sourcecontrol"
)

// sourceControlFake serves a single queue's linear history. history[0] is the
// latest commit; higher indices are progressively older ancestors.
type sourceControlFake struct {
history []string
}

// New returns a sourcecontrol.SourceControl backed by the given ref history,
// ordered newest-first (history[0] is the latest commit). The slice is copied so
// later mutation by the caller does not affect the fake.
func New(history []string) sourcecontrol.SourceControl {
cp := make([]string, len(history))
copy(cp, history)
return sourceControlFake{history: cp}
}

// Latest returns the newest commit URI, or ErrNotFound when the history is empty.
func (s sourceControlFake) Latest(_ context.Context) (string, error) {
if len(s.history) == 0 {
return "", sourcecontrol.ErrNotFound
}
return s.history[0], nil
}

// IsAncestor reports whether ancestor is an ancestor of descendant. Both URIs
// must be on the ref; an unknown URI yields ErrNotFound. Since the history is
// newest-first, ancestor is an ancestor of descendant when its index is greater
// than or equal to descendant's (older-or-equal commit).
func (s sourceControlFake) IsAncestor(_ context.Context, ancestor, descendant string) (bool, error) {
ai := s.indexOf(ancestor)
di := s.indexOf(descendant)
if ai < 0 || di < 0 {
return false, sourcecontrol.ErrNotFound
}
return ai >= di, nil
}

// History returns one page of commit URIs, newest first. The cursor is the URI
// of the first commit of the page to return; an empty cursor starts at the latest
// commit. A limit of zero or less returns the rest of the history from the cursor
// in a single page. The returned NextCursor is the URI of the next, older commit,
// or empty when the page reaches the end of the history.
func (s sourceControlFake) History(_ context.Context, cursor string, limit int) (page.Page[string], error) {
start := 0
if cursor != "" {
start = s.indexOf(cursor)
if start < 0 {
return page.Page[string]{}, sourcecontrol.ErrNotFound
}
}

end := len(s.history)
if limit > 0 && start+limit < end {
end = start + limit
}

uris := make([]string, end-start)
copy(uris, s.history[start:end])

next := ""
if end < len(s.history) {
next = s.history[end]
}
return page.Page[string]{Items: uris, NextCursor: next}, nil
}

// indexOf returns the index of uri in the history, or -1 if absent.
func (s sourceControlFake) indexOf(uri string) int {
for i, u := range s.history {
if u == uri {
return i
}
}
return -1
}
139 changes: 139 additions & 0 deletions stovepipe/extension/sourcecontrol/fake/fake_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,139 @@
// Copyright (c) 2025 Uber Technologies, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package fake

import (
"context"
"testing"

"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/uber/submitqueue/stovepipe/extension/sourcecontrol"
)

func TestNew_ImplementsInterface(t *testing.T) {
var _ sourcecontrol.SourceControl = New(nil)
}

// history is ordered newest-first: c is the latest, a is the oldest ancestor.
var history = []string{"git://repo/ref/c", "git://repo/ref/b", "git://repo/ref/a"}

func TestLatest(t *testing.T) {
tests := []struct {
name string
history []string
want string
wantErr bool
}{
{name: "newest first", history: history, want: "git://repo/ref/c"},
{name: "empty history", history: nil, wantErr: true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, err := New(tt.history).Latest(context.Background())
if tt.wantErr {
require.ErrorIs(t, err, sourcecontrol.ErrNotFound)
return
}
require.NoError(t, err)
assert.Equal(t, tt.want, got)
})
}
}

func TestIsAncestor(t *testing.T) {
tests := []struct {
name string
ancestor string
descendant string
want bool
wantErr bool
}{
{name: "older is ancestor of newer", ancestor: "git://repo/ref/a", descendant: "git://repo/ref/c", want: true},
{name: "newer is not ancestor of older", ancestor: "git://repo/ref/c", descendant: "git://repo/ref/a", want: false},
{name: "equal is ancestor of itself", ancestor: "git://repo/ref/b", descendant: "git://repo/ref/b", want: true},
{name: "unknown ancestor", ancestor: "git://repo/ref/x", descendant: "git://repo/ref/a", wantErr: true},
{name: "unknown descendant", ancestor: "git://repo/ref/a", descendant: "git://repo/ref/x", wantErr: true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, err := New(history).IsAncestor(context.Background(), tt.ancestor, tt.descendant)
if tt.wantErr {
require.ErrorIs(t, err, sourcecontrol.ErrNotFound)
return
}
require.NoError(t, err)
assert.Equal(t, tt.want, got)
})
}
}

func TestHistory(t *testing.T) {
tests := []struct {
name string
cursor string
limit int
wantItems []string
wantCursor string
wantErr bool
}{
{
name: "first page with next cursor",
cursor: "",
limit: 2,
wantItems: []string{"git://repo/ref/c", "git://repo/ref/b"},
wantCursor: "git://repo/ref/a",
},
{
name: "second page reaches end",
cursor: "git://repo/ref/a",
limit: 2,
wantItems: []string{"git://repo/ref/a"},
wantCursor: "",
},
{
name: "limit larger than remaining returns rest",
cursor: "",
limit: 10,
wantItems: history,
wantCursor: "",
},
{
name: "zero limit returns rest in one page",
cursor: "",
limit: 0,
wantItems: history,
wantCursor: "",
},
{
name: "unknown cursor",
cursor: "git://repo/ref/x",
limit: 2,
wantErr: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, err := New(history).History(context.Background(), tt.cursor, tt.limit)
if tt.wantErr {
require.ErrorIs(t, err, sourcecontrol.ErrNotFound)
return
}
require.NoError(t, err)
assert.Equal(t, tt.wantItems, got.Items)
assert.Equal(t, tt.wantCursor, got.NextCursor)
})
}
}
13 changes: 13 additions & 0 deletions stovepipe/extension/sourcecontrol/mock/BUILD.bazel
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
load("@rules_go//go:def.bzl", "go_library")

go_library(
name = "mock",
srcs = ["sourcecontrol_mock.go"],
importpath = "github.com/uber/submitqueue/stovepipe/extension/sourcecontrol/mock",
visibility = ["//visibility:public"],
deps = [
"//platform/base/page",
"//stovepipe/extension/sourcecontrol",
"@org_uber_go_mock//gomock",
],
)
5 changes: 5 additions & 0 deletions stovepipe/extension/sourcecontrol/mock/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# sourcecontrol mocks

Generated gomock mock for the `sourcecontrol.SourceControl` interface, used by controller and pipeline tests.

Mocks are **checked in** and produced by [mockgen](https://github.com/uber-go/mock) from the `//go:generate` directive on `sourcecontrol.go`. After changing the interface, run `make mocks` to regenerate, then `make gazelle` to update `BUILD.bazel`, and commit the result.
Loading
Loading