ykcheungab
c37efd7a44
The pre-push gate linted only the dirty tree, so the commits about to be pushed went unchecked; and every failed overlay projection was sticky, which cost a finger result its overlay over one bad frame while box results were retried nowhere. gate.ps1 now takes -LintBase (default origin/main, exit 2 on an unknown ref), the projector declares whether its own failures are permanent, and serve_stored_view decides before it sets the view. Verified with gate.ps1 (PASS: 569 studio + 485 shared tests, 632 ui_check checks) and the two round-2 red samples kept in scratch.
382 lines
16 KiB
PowerShell
382 lines
16 KiB
PowerShell
<#
|
|
.SYNOPSIS
|
|
Merge gate for orisys-app: lint the change, run both test suites, run the UI check.
|
|
|
|
.DESCRIPTION
|
|
One command that a change cannot skip. It runs, in order:
|
|
|
|
1. tools/lint_gate.py the narrow lint rule set over the changed files
|
|
2. pytest apps/studio/tests full, with the ``timing`` tests deselected
|
|
3. pytest shared/tests full, with the ``timing`` tests deselected
|
|
4. apps/studio/tools/ui_check.py the layout / theme / record-channel checks
|
|
|
|
and then holds the run to three contracts:
|
|
|
|
* tools/known_failures.txt must be empty. Any content fails the gate: a known
|
|
failure list is the thing this gate exists to make unnecessary.
|
|
* the skip count per suite must stay under tools/skip_budget.txt. A skip that
|
|
grows past the budget is a suite that stopped testing what it claims.
|
|
* the collected test count per suite must stay above the floor in the same file.
|
|
A module that becomes uncollectable silently shrinks the suite; the floor is
|
|
how that shows up as a failure instead of a faster green run.
|
|
|
|
Every command's output goes to <repo>/.scratch/<Task>/gate-<run-id>/, and the
|
|
one-page summary is both printed and written there as summary.md. The scratch
|
|
marker <repo>/.scratch/<Task>/.active is removed in a finally block, so an
|
|
interrupted or failed run does not leave the task looking busy.
|
|
|
|
Platform: **Windows**. The suite renders real Qt widgets offscreen and two tests
|
|
are platform-sensitive (see docs/test-repair-plan.zh.md §3 #11 and #13); Linux
|
|
numbers are not equivalent.
|
|
|
|
.PARAMETER Task
|
|
Scratch task id, i.e. the <task-id> in .scratch/<task-id>/. Defaults to "gate".
|
|
|
|
.PARAMETER Python
|
|
The interpreter to run the suites with. Defaults to .venv-build/Scripts/python.exe.
|
|
|
|
.PARAMETER LintBase
|
|
The commit to lint against, passed to tools/lint_gate.py as ``--base``. Defaults to
|
|
``origin/main``, so the pre-push hook checks every commit about to be pushed, not
|
|
only the dirty working tree. An invalid ref is a configuration error (exit 2); it
|
|
never falls back silently to "changed files only".
|
|
|
|
.PARAMETER SkipLint
|
|
Run only the suites and the UI check (used when diagnosing a red gate).
|
|
|
|
.EXAMPLE
|
|
powershell -NoProfile -ExecutionPolicy Bypass -File tools/gate.ps1
|
|
|
|
.EXAMPLE
|
|
powershell -NoProfile -ExecutionPolicy Bypass -File tools/gate.ps1 -LintBase HEAD~1
|
|
|
|
.NOTES
|
|
Exit codes: 0 gate passed, 1 gate failed, 2 environment or configuration problem.
|
|
#>
|
|
[CmdletBinding()]
|
|
param(
|
|
[string]$Task = 'gate',
|
|
[string]$Python = '',
|
|
[string]$LintBase = 'origin/main',
|
|
[switch]$SkipLint
|
|
)
|
|
|
|
$ErrorActionPreference = 'Stop'
|
|
$Repo = Split-Path -Parent $PSScriptRoot
|
|
Set-Location $Repo
|
|
|
|
function Write-Step([string]$Text) {
|
|
Write-Host ''
|
|
Write-Host "== $Text" -ForegroundColor Cyan
|
|
}
|
|
|
|
function Invoke-Native([string]$Exe, [string[]]$Arguments, [string]$LogPath, [switch]$Quiet) {
|
|
# Every tool here writes progress and Qt diagnostics to stderr, and Windows
|
|
# PowerShell turns a native command's stderr into error records: with
|
|
# ``$ErrorActionPreference = 'Stop'`` a Qt warning ("This plugin does not support
|
|
# raise()") aborted the whole gate. The exit code is the signal; stderr is data.
|
|
$previous = $ErrorActionPreference
|
|
$ErrorActionPreference = 'Continue'
|
|
try {
|
|
if ($Quiet) {
|
|
& $Exe @Arguments 2>&1 | Tee-Object -FilePath $LogPath | Out-Null
|
|
} else {
|
|
& $Exe @Arguments 2>&1 | Tee-Object -FilePath $LogPath | Out-Host
|
|
}
|
|
return $LASTEXITCODE
|
|
} finally {
|
|
$ErrorActionPreference = $previous
|
|
}
|
|
}
|
|
|
|
# ------------------------------------------------------------------ environment
|
|
if ([string]::IsNullOrWhiteSpace($Python)) {
|
|
$Python = Join-Path $Repo '.venv-build\Scripts\python.exe'
|
|
}
|
|
if (-not (Test-Path -LiteralPath $Python)) {
|
|
Write-Host "gate: no interpreter at $Python (set -Python or build .venv-build)" -ForegroundColor Red
|
|
exit 2
|
|
}
|
|
|
|
# R1: the lint scope is a commit range, not "whatever happens to be dirty". A pre-push
|
|
# gate exists to check the commits being pushed, and a base that does not resolve is a
|
|
# configuration problem — silently linting only the working tree is how the change that
|
|
# reaches main goes unchecked.
|
|
$previous = $ErrorActionPreference
|
|
$ErrorActionPreference = 'Continue'
|
|
try {
|
|
git rev-parse --verify --quiet $LintBase *> $null
|
|
$baseExit = $LASTEXITCODE
|
|
} finally {
|
|
$ErrorActionPreference = $previous
|
|
}
|
|
if ($baseExit -ne 0) {
|
|
Write-Host "gate: -LintBase '$LintBase' is not a valid git ref; refusing to lint only the working tree" -ForegroundColor Red
|
|
exit 2
|
|
}
|
|
|
|
# UTC: the run id is read beside logs written by other machines and by CI, so it must not
|
|
# depend on the local time zone.
|
|
$RunId = (Get-Date).ToUniversalTime().ToString('yyyyMMddTHHmmssZ')
|
|
$TaskDir = Join-Path $Repo (Join-Path '.scratch' $Task)
|
|
$Scratch = Join-Path $TaskDir "gate-$RunId"
|
|
New-Item -ItemType Directory -Force -Path $Scratch | Out-Null
|
|
# AGENTS.md rule 1: the scratch directory is marked while a task is using it. Removed in
|
|
# the finally block below, so a failed or interrupted gate leaves no stale marker.
|
|
$ActiveMarker = Join-Path $TaskDir '.active'
|
|
Set-Content -LiteralPath $ActiveMarker -Value "gate $RunId"
|
|
|
|
try {
|
|
$KnownFailures = Join-Path $Repo 'tools\known_failures.txt'
|
|
$BudgetFile = Join-Path $Repo 'tools\skip_budget.txt'
|
|
|
|
$Results = New-Object System.Collections.ArrayList
|
|
function Add-Result([string]$Name, [string]$Status, [string]$Detail) {
|
|
[void]$Results.Add([pscustomobject]@{ Name = $Name; Status = $Status; Detail = $Detail })
|
|
}
|
|
|
|
# -------------------------------------------------------------- budgets
|
|
function Read-Budget([string]$Path) {
|
|
$values = @{}
|
|
if (-not (Test-Path -LiteralPath $Path)) {
|
|
return $values
|
|
}
|
|
foreach ($line in Get-Content -LiteralPath $Path) {
|
|
$text = $line.Trim()
|
|
if ($text -eq '' -or $text.StartsWith('#')) { continue }
|
|
if ($text -notmatch '^([A-Za-z_]+)\s*=\s*(\d+)$') {
|
|
Write-Host "gate: unreadable line in ${Path}: $text" -ForegroundColor Red
|
|
throw "bad budget line: $text"
|
|
}
|
|
$values[$Matches[1]] = [int]$Matches[2]
|
|
}
|
|
return $values
|
|
}
|
|
|
|
# -------------------------------------------------------------- known failures
|
|
Write-Step 'known failures'
|
|
$KnownStatus = 'passed'
|
|
$KnownDetail = 'tools/known_failures.txt is empty'
|
|
if (-not (Test-Path -LiteralPath $KnownFailures)) {
|
|
$KnownStatus = 'failed'
|
|
$KnownDetail = 'tools/known_failures.txt is missing'
|
|
} else {
|
|
$known = (Get-Content -LiteralPath $KnownFailures -Raw)
|
|
if ($null -eq $known) { $known = '' }
|
|
if ($known.Trim() -ne '') {
|
|
$KnownStatus = 'failed'
|
|
$KnownDetail = 'tools/known_failures.txt is not empty: ' + ($known.Trim() -replace '\r?\n', ' | ')
|
|
}
|
|
}
|
|
Add-Result 'known failures' $KnownStatus $KnownDetail
|
|
if ($KnownStatus -ne 'passed') {
|
|
Write-Host $KnownDetail -ForegroundColor Red
|
|
} else {
|
|
Write-Host $KnownDetail
|
|
}
|
|
|
|
# -------------------------------------------------------------- lint
|
|
if (-not $SkipLint) {
|
|
Write-Step "lint (changed files since $LintBase)"
|
|
$lintLog = Join-Path $Scratch 'lint.log'
|
|
$lintExit = Invoke-Native $Python @(
|
|
(Join-Path $Repo 'tools\lint_gate.py'), '--base', $LintBase
|
|
) $lintLog
|
|
# Name the files the range covered, in the same log a reviewer reads. The list is
|
|
# not re-derived here — the gate must not keep a second copy of lint_gate's
|
|
# selection rule — it is read from lint_gate's own --json answer for the same
|
|
# range, in a quiet second pass (ruff over a handful of files).
|
|
$lintJson = Join-Path $Scratch 'lint.json'
|
|
$null = Invoke-Native $Python @(
|
|
(Join-Path $Repo 'tools\lint_gate.py'), '--base', $LintBase, '--json'
|
|
) $lintJson -Quiet
|
|
$checkedPaths = @()
|
|
try {
|
|
$payload = (Get-Content -LiteralPath $lintJson -Raw) | ConvertFrom-Json
|
|
if ($payload -and $payload.checked) { $checkedPaths = @($payload.checked) }
|
|
} catch {
|
|
$checkedPaths = @()
|
|
}
|
|
if ($checkedPaths.Count -gt 0) {
|
|
Add-Content -LiteralPath $lintLog -Value ''
|
|
Add-Content -LiteralPath $lintLog -Value " files checked against $LintBase ($($checkedPaths.Count)):"
|
|
foreach ($lintedPath in $checkedPaths) {
|
|
Add-Content -LiteralPath $lintLog -Value " $lintedPath"
|
|
}
|
|
}
|
|
# lint_gate prints "checked N target(s)"; the summary repeats N so the range a
|
|
# green lint actually covered is on the one page an operator reads.
|
|
$checked = $checkedPaths.Count
|
|
$scope = "base $LintBase, $checked file(s) checked"
|
|
if ($lintExit -eq 0) {
|
|
Add-Result 'lint' 'passed' "no findings ($scope)"
|
|
} elseif ($lintExit -eq 1) {
|
|
Add-Result 'lint' 'failed' "findings in the changed files ($scope; see lint.log)"
|
|
} else {
|
|
Add-Result 'lint' 'failed' "lint_gate.py exited $lintExit (environment problem, e.g. ruff missing; $scope)"
|
|
}
|
|
}
|
|
|
|
# -------------------------------------------------------------- test suites
|
|
function Invoke-Suite([string]$Name, [string]$Path) {
|
|
$safe = $Name.Replace(' ', '-')
|
|
$log = Join-Path $Scratch "$safe.log"
|
|
$xml = Join-Path $Scratch "$safe.xml"
|
|
$base = Join-Path $Scratch "pytest-$safe"
|
|
Write-Step "$Name"
|
|
$exit = Invoke-Native $Python @(
|
|
'-m', 'pytest', $Path,
|
|
'-m', 'not timing',
|
|
'-p', 'no:cacheprovider',
|
|
"--basetemp=$base",
|
|
"--junitxml=$xml"
|
|
) $log
|
|
|
|
$counts = [pscustomobject]@{
|
|
Tests = 0; Failures = 0; Errors = 0; Skipped = 0; Exit = $exit; Xml = (Test-Path -LiteralPath $xml)
|
|
}
|
|
if ($counts.Xml) {
|
|
try {
|
|
[xml]$doc = Get-Content -LiteralPath $xml
|
|
$suites = @($doc.testsuites.testsuite)
|
|
foreach ($suite in $suites) {
|
|
$counts.Tests += [int]$suite.tests
|
|
$counts.Failures += [int]$suite.failures
|
|
$counts.Errors += [int]$suite.errors
|
|
$counts.Skipped += [int]$suite.skipped
|
|
}
|
|
} catch {
|
|
$counts.Xml = $false
|
|
}
|
|
}
|
|
return $counts
|
|
}
|
|
|
|
#: An access violation while Qt tears the window down after the last assertion has
|
|
#: been seen (exit 0xC0000005 as a signed int). It is allowed and reported; every
|
|
#: other non-zero exit from a suite with a clean JUnit report is a gate failure.
|
|
$TeardownCrash = -1073741819
|
|
|
|
function Test-Suite([string]$Name, $Counts, $Budget, [string]$MinKey, [string]$MaxKey) {
|
|
# JUnit is authoritative for which assertions failed: the interpreter has been
|
|
# observed to crash in Qt teardown *after* reporting every test. The exit code is
|
|
# still checked, because a suite that died for another reason has not passed.
|
|
if (-not $Counts.Xml) {
|
|
Add-Result $Name 'failed' "no JUnit report (pytest exit $($Counts.Exit))"
|
|
return
|
|
}
|
|
$detail = "$($Counts.Tests) tests, $($Counts.Failures) failed, $($Counts.Errors) errors, $($Counts.Skipped) skipped"
|
|
if ($Counts.Failures -ne 0 -or $Counts.Errors -ne 0) {
|
|
Add-Result $Name 'failed' $detail
|
|
return
|
|
}
|
|
$note = $detail
|
|
if ($Counts.Exit -ne 0) {
|
|
if ($Counts.Exit -ne $TeardownCrash) {
|
|
Add-Result $Name 'failed' "$detail; process exit $($Counts.Exit) is not the allowed teardown crash"
|
|
return
|
|
}
|
|
$note = "$detail; process exit 0xC0000005 after the summary (Qt teardown, allowed)"
|
|
}
|
|
$ok = $true
|
|
if ($Budget.ContainsKey($MinKey)) {
|
|
$floor = [int]$Budget[$MinKey]
|
|
if ($Counts.Tests -lt $floor) {
|
|
$ok = $false
|
|
$note = "$detail; only $($Counts.Tests) tests collected, floor is $floor"
|
|
}
|
|
}
|
|
if ($Budget.ContainsKey($MaxKey)) {
|
|
$ceiling = [int]$Budget[$MaxKey]
|
|
if ($Counts.Skipped -gt $ceiling) {
|
|
$ok = $false
|
|
$note = "$detail; more than the $ceiling skips allowed"
|
|
}
|
|
}
|
|
if ($ok) {
|
|
Add-Result $Name 'passed' $note
|
|
} else {
|
|
Add-Result $Name 'failed' $note
|
|
}
|
|
}
|
|
|
|
try {
|
|
$budget = Read-Budget $BudgetFile
|
|
} catch {
|
|
Write-Host "gate: $($_.Exception.Message)" -ForegroundColor Red
|
|
exit 2
|
|
}
|
|
if ($budget.Count -eq 0) {
|
|
Write-Host "gate: no budgets read from $BudgetFile (floors and skip ceilings are part of the gate)" -ForegroundColor Red
|
|
exit 2
|
|
}
|
|
|
|
Write-Step 'studio suite'
|
|
$studio = Invoke-Suite 'studio suite' 'apps/studio/tests'
|
|
Test-Suite 'studio suite' $studio $budget 'studio_min_tests' 'studio_max_skips'
|
|
|
|
Write-Step 'shared suite'
|
|
$shared = Invoke-Suite 'shared suite' 'shared/tests'
|
|
Test-Suite 'shared suite' $shared $budget 'shared_min_tests' 'shared_max_skips'
|
|
|
|
# -------------------------------------------------------------- ui_check
|
|
Write-Step 'ui_check'
|
|
$uiLog = Join-Path $Scratch 'ui_check.log'
|
|
$uiExit = Invoke-Native $Python @((Join-Path $Repo 'apps\studio\tools\ui_check.py')) $uiLog
|
|
# ui_check prints "scenes: 4 checks: 631 failures: 2"; the last number is the
|
|
# verdict. Its exit code is also the verdict: a script that dies partway through
|
|
# (missing worker signature, an import error) must not pass on a stale summary line.
|
|
$uiSummary = Select-String -LiteralPath $uiLog -Pattern 'checks:\s*\d+.*failures:\s*(\d+)'
|
|
if ($uiExit -ne 0) {
|
|
Add-Result 'ui_check' 'failed' "ui_check.py exited $uiExit"
|
|
} elseif ($uiSummary) {
|
|
$match = $uiSummary[-1].Matches[0]
|
|
$failures = [int]$match.Groups[1].Value
|
|
$line = $uiSummary[-1].Line.Trim()
|
|
if ($failures -eq 0) {
|
|
Add-Result 'ui_check' 'passed' $line
|
|
} else {
|
|
Add-Result 'ui_check' 'failed' $line
|
|
}
|
|
} else {
|
|
Add-Result 'ui_check' 'failed' "no 'checks: N ... failures: M' summary (exit $uiExit)"
|
|
}
|
|
|
|
# -------------------------------------------------------------- summary
|
|
$failed = @($Results | Where-Object { $_.Status -ne 'passed' })
|
|
$verdict = if ($failed.Count -eq 0) { 'PASS' } else { 'FAIL' }
|
|
|
|
Write-Step "summary: $verdict"
|
|
foreach ($result in $Results) {
|
|
$colour = if ($result.Status -eq 'passed') { 'Green' } else { 'Red' }
|
|
Write-Host (" {0,-16} {1,-7} {2}" -f $result.Name, $result.Status, $result.Detail) -ForegroundColor $colour
|
|
}
|
|
|
|
$lines = New-Object System.Collections.ArrayList
|
|
[void]$lines.Add("# Gate $RunId : $verdict")
|
|
[void]$lines.Add('')
|
|
[void]$lines.Add("Platform : Windows")
|
|
[void]$lines.Add("Python : $Python")
|
|
[void]$lines.Add("LintBase : $LintBase")
|
|
[void]$lines.Add("Scratch : $Scratch")
|
|
[void]$lines.Add('')
|
|
[void]$lines.Add('| Check | Result | Detail |')
|
|
[void]$lines.Add('|---|---|---|')
|
|
foreach ($result in $Results) {
|
|
[void]$lines.Add("| $($result.Name) | $($result.Status) | $($result.Detail) |")
|
|
}
|
|
[void]$lines.Add('')
|
|
[void]$lines.Add('Logs: lint.log, studio-suite.log, shared-suite.log, ui_check.log, and the JUnit XMLs, in this directory.')
|
|
$summaryPath = Join-Path $Scratch 'summary.md'
|
|
Set-Content -LiteralPath $summaryPath -Value $lines -Encoding UTF8
|
|
Write-Host ''
|
|
Write-Host "gate: $verdict (summary: $summaryPath)"
|
|
|
|
if ($failed.Count -eq 0) { exit 0 }
|
|
exit 1
|
|
} finally {
|
|
if (Test-Path -LiteralPath $ActiveMarker) {
|
|
Remove-Item -LiteralPath $ActiveMarker -Force -ErrorAction SilentlyContinue
|
|
}
|
|
}
|