Testing¶

The whole suite runs on Linux, against models of macOS and of servers that encode every behaviour measured so far. The Mac VM only measures: it seeds the models, and a green Linux suite is what proves a change.

The principle¶

In the owner's words:

"I would like as much of the testing as possible to be done in a way that encodes mocks measured/known macOS behaviour and then exercises the codebase against all known issues identified in the past. If future macOS releases cause breakage, we can then add their behaviour into the encoded mocks and thereby unit test across all supported macOS versions (and each one's quirks) without needing to spin up VMs and such. The tests should be able to run 100% isolated on a Linux box. We should only use the VM to seed the macOS mocks into the tests."

Running the suite¶

On Linux, where the suite is the gate:

. ~/.local/share/swiftly/env.sh
cd Packages/SSHDriveCore
swift test
cd ../../helper && cargo test && cargo clippy -- -D warnings

No Mac, no VM, no network, no testbed, no docker compose. The run is 819 XCTest tests with 41 skipped, 65 swift-testing tests in twelve suites, and 54 crate tests for the helper. The 41 skips are the testbed-backed tests in SFTPTests and SSHProcessTests, gated on SSHDRIVE_TESTBED=1. The testbed answers the build VM and nothing else, so they are inert here.

On the build VM¶

The same swift test must be green on the VM too. It is the same scenarios against a different box, not a second suite:

• Extra tests: the mirrored-constant assertions, which need Apple's frameworks (see the extension's seams).
• Extra skips: rows whose premise a Mac cannot hold, each skipping by name (see the box is a seam too).
• macOS has no timeout and none of the shells some rows want, so start a long run detached on the VM and poll it.

scripts/mac-build.sh test     # sync, xcodegen, swift build + swift test on the VM
scripts/mac-build.sh app      # sync, xcodegen, xcodebuild, ad-hoc sign

Apps/ needs a Mac only to compile and sign.

The testbed and CI¶

The testbed is twelve real SSH servers in Docker Compose, run on the Mac that hosts the build VM, never on the Linux box. It exists to seed SQ rows and to answer questions the model cannot; no ordinary run needs it. testbed/README.md has the account table, the ~/.ssh/config stanzas and the per-service smoke tests.

.github/workflows/helper.yml builds and tests the Rust crate and cross-compiles its targets.

No decision under Apps/¶

Apps/ is adapters: Apple types in, package types out, package types in, Apple calls out. A reviewer should be able to read any file there in one sitting and find no branch worth testing.

A new rule the extension follows belongs in ProviderCore with a scenario. A new rule the agent follows belongs in AgentRuntime with a scenario. Nowhere else.

Packages/SSHDriveCore holds Logging, XPCProtocols, XPCInterfaces, Config, Index, SFTP, Secrets, SSHProcess, AgentCore, ProviderCore, AgentRuntime, AgentRuntimeTestSupport, SystemModel and ServerModel. All of it builds and tests on Linux. XPCInterfaces (the @objc NSXPC protocols and the configured NSXPCInterface whitelists, linked only by the four app targets) has #if canImport(Darwin) file bodies and compiles to nothing off Darwin.

The seams¶

The extension's (ProviderCore)¶

Everything Apps/FileProvider would otherwise decide is in ProviderCore: the two enumerators, the working-set change path and its fallback, the reader store and its readiness rule, item construction, the trash contract, error selection and the two Finder actions. It sits behind platform-free protocols:

Apple's identifiers, capability bits, fileSystemFlags bits, changedFields bits, content policy and error codes are mirrored as ProviderItemIdentifier, ProviderSyncAnchor, ProviderCapabilities, ProviderFileSystemFlags, ProviderItemFields and ProviderContentPolicy, so AgentCore and ProviderCore need no import FileProvider.

A mirror that drifted would be silent: the adapter keeps compiling and the Linux suite keeps passing. Tests/ProviderCoreTests/MirroredProviderConstantsTests.swift and Tests/AgentRuntimeTests/MirroredAgentConstantsTests.swift are one assert per constant against Apple's own, compiled only where Apple's frameworks exist. They are the reason the suite also runs on the Mac.

The agent's (AgentRuntime)¶

LocationRuntime and its extensions, DomainManager, ChangeDetector, CacheEvictor, IndexReconcile, ReconnectingTransport and its gate, SSHBackedTransport, the channel budget, the helper's deployment and stream, the collect connection and both command handlers live in AgentRuntime behind fifteen protocols:

Two protocols sit beside the fifteen: TerminalRelaying (the CLI as the collect connection sees it, one note and one prompt; ScriptedTerminal off Darwin) and LiveConnection (an SFTPTransport the gate hands out or does not).

Details that matter when writing a scenario:

• AgentClock answers now() (wall clock, what the schedules are in), uptime() (monotonic, what the breaker's backoff is in) and sleep(seconds:), so a scenario drives both readings together.
• AgentEndpoint.terminate(status:) does not return Never: the real one calls exit, the harness records the status, and P4 asserts that SIGTERM exits 0.
• The seams are one value, AgentEnvironment, and DomainManager is constructed with it rather than reading singletons. That is what lets one process run two agents.
• DomainManager.shared still exists for the XPC command layer, which is reached from an object made per connection and has nothing else to hold. A scenario binds its own through AgentCommandContext, a task local, for the duration of one call.

AgentRuntimeTestSupport is the in-memory half: the fakes above plus AgentHarness, which gives a scenario an AgentEnvironment it can drive, a DomainManager on it, a real SQLite index and a temporary directory standing in for the group container. Config.GroupContainer is process-wide, so the harness installs its locator on set-up and puts it back on tear-down, and the suites that use it are .serialized.

SystemModel¶

A simulated fileproviderd, Finder and launchd. It drives ProviderCore through the protocols above exactly as the real one drives Apps/FileProvider, and behaves the way macOS was measured to behave, with the version's quirk table deciding every difference.

The agent on the other side is ModelAgent: a scripted AgentChannel whose answers come from a real IndexWriter on a real SQLite file. "An index with rows the replica lacks" means real rows, and the extension's reader reads the same file. Only two things are scripted: whether the agent is reachable, and what it answers to indexReady.

let harness = try ScenarioHarness(macOS: .v26_4)
try harness.serverCreates("note.txt")
let domain = try harness.addDomain()   // fileproviderd creates it and launches a provider
domain.openFolder()                    // one enumerateItems, ever
harness.clock.advance(60)              // the driven clock; nothing sleeps
XCTAssertEqual(harness.finderListing(), ["note.txt"])

VirtualClock is the only clock in the model. advance moves time, releases whatever the simulated schedulers owe, and returns when everything queued has run. There is no real sleeping in the suite.

Quirks and versions¶

Every behaviour the model implements is keyed to an MQ-### id from the macOS quirk catalogue; a version whose table says otherwise gets the other behaviour.

public struct Quirk {
    public let id: QuirkID                   // "MQ-005"
    public let statement: String
    public let measurements: [QuirkMeasurement]   // versions, value, source
}

QuirkTable(for:) resolves each id to the value measured nearest that version and refuses to resolve an id with no measurement covering it. A scenario that depends on an unmeasured quirk on a supported version fails loudly with the id rather than quietly taking someone's guess.

MacOSVersion has two columns, 26.4 and 26.6, identical unless a quirk says otherwise. macOS 14 and 15 are the supported minimum and nothing has been measured on either. Every empty cell in the catalogue is that work item, and a scenario reaching for one of those columns fails rather than guesses.

Where a value was taken once, or bracketed rather than measured, the rule that reads it and the catalogue row both carry a confidence: note. Five do:

• the window in which an eviction straight after a modifyItem reply is refused (MQ-017)
• the single deferred atime advance (MQ-022)
• the reading that the dividing line in MQ-029 is a new container rather than a new item
• the unpin settle window (MQ-034)
• the length of the window unregister() leaves open (MQ-063)

ServerModel¶

The remote half: a whole SSH/SFTP server universe in process, plus the parts that are honest to run for real on Linux.

ServerProfile¶

One value describing a server: sftp implementation, the advertised extension list in order, findFlavour, loginShell, forceCommand, maxSessions, sessionGrouping, clientAliveInterval, reapsOrphans, clockOffset, umask, caseInsensitive, renameOverwrites, hasSHA256Sum, hasMkfifo, unameSM, identificationString, auth, and the quirks list naming the SQ-### rows it carries.

There are twenty-one constants:

• the twelve testbed services
• five account variants of them that behave differently: debianPassword, debianKeyOrPassword, debianBackgroundHolder, debianForceCommand, debianExternalSFTPQuiet
• the owner's own two servers, ownerDebian and ownerTailscale
• two we have no access to and model anyway, freeBSD and synologyDSM

The extension sets are exact, because sshdrive status reads them. Go pkg/sftp advertises exactly hardlink@openssh.com, posix-rename@openssh.com and statvfs@openssh.com; OpenSSH's own server adds fsync, lsetstat, limits, expand-path, copy-data, home-directory and users-groups-by-id.

The three stand-ins¶

FakeSFTPServer and FakeExecChannel speak the same SSHProcess.ByteStream a mux client's stdio does, so SFTPClient, RemoteScript, the sweep script, the sentinel, the heartbeat wrapper and the helper's NDJSON reader all run unmodified.

FakeSFTPServer is a real SFTP v3 wire server over a ByteStream pair. It speaks the protocol, not a mock of it:

• SSH_FXP_VERSION with the profile's extension list
• status classes and no errno, so ENOSPC/EEXIST/ENOTEMPTY/EXDEV all arrive as a bare FAILURE
• OpenSSH's reversed SSH2_FXP_SYMLINK argument order
• an opendir that follows a symlink, and a readdir carrying attributes but no link target
• limits@openssh.com values, posix-rename present or absent, and a plain rename that does or does not overwrite
• ETXTBSY on a write over a running executable, mkdir attributes filtered by the umask, and case folding

SFTP.FakeTransport remains the fast in-memory double for tests that do not care about the wire.

FakeExecChannel is an exec channel whose remote end is a real POSIX shell on the box running the tests. It picks the profile's shell, prepends the profile's rc-file noise, applies the ForceCommand refusal sentence where the profile has one, enforces MaxSessions, and models the session's process-group policy with a real setsid or a shared pgid, so kill -TERM 0 really does or does not reach a bystander. A scenario needing a shell the box does not have (fish, tcsh, zsh) skips with a named reason.

The shell scripts are tested against real shells because three of the worst bugs this project has had lived there: the ;; dash rejects, the { … } group the heartbeat reader would otherwise eat off its own stdin, and the printf "\0<sentinel>" that ate its own sentinel.

FakeSSH is a small executable the package builds, installed at SSHProcess.sshBinaryPath as the TransportLauncher's "ssh", so SSHInvocation's argv assembly, the spawn, ExitClassification and ControlSocket run for real. It:

• binds a real ControlPath socket and answers -O check and -O exit
• honours or ignores ControlPersist
• prints the measured stderr sentences with CRLF line endings, and remote software version … only at DEBUG1 and above
• invokes SSH_ASKPASS with the measured prompt strings, trailing spaces included
• applies OpenSSH's readconf first-setting-wins rule, and percent-expands a ProxyCommand before handing it to /bin/sh
• exits 255 with nothing on stderr when its remote command was killed by a signal

The box is a seam too¶

ServerModel.HostTools measures once per process what the machine running the suite can do. A scenario that meets one of those bounds skips with the sentence it returns: a skip naming the platform fact, never a silent #if. Three bounds apply:

FakeSSH also tells ControlSocket what the kernel will call a running stub. The stub is a shell script named ssh; Linux takes a process's short name from the script and XNU from the interpreter binary, and macOS launch constraints SIGKILL a copy of any system shell, so there is no binary to point a shebang at (SQ-082). ControlSocket.masterProcessName is the seam, and install() sets it to a name measured from a probe script that prints before it sleeps.

The harness suppresses one artifact rather than asserting around it. A shell prints Killed/Terminated on its own stderr when a foreground child dies by a signal, and dash writes it to the command's redirected stderr (SQ-084), while SQ-011 says a signal-killed remote command prints nothing at all. FakeSSH hands the session its real stderr on fd 4 and points its own at /dev/null.

Process-wide state in a concurrent suite¶

swift test runs the swift-testing suites concurrently with XCTest in one process, so anything installed process-wide is shared. Two rules follow:

• A scenario that sweeps or kills by prefix passes the directory it acts on. ControlSocket.sweepOrphans(in:), liveMasterPIDs(in:) and killStrayMasters(in:) default to the real $TMPDIR, which is the blast radius agent stop needs; a test passes one of its own.
• A suite that installs SSHProcess.sshBinaryPath (which decides which stub every ssh in the process is) is .serialized, and its bed stops its agent before putting the path back. Otherwise a DomainManager handed to a detached task outlives its bed, goes on reconnecting on the breaker's schedule and dials the next scenario's stub.

The quirk catalogue¶

docs/quirks/ is the inventory of measured behaviour: macos.md (80 rows) and servers.md (82). The catalog's README has the row format and the steps that turn a measurement into a model rule and a scenario.

MQ rows are keyed to a macOS version. SQ rows are keyed to a testbed service or a real server, not to an OS version, so a new testbed image is the same kind of work as a new macOS and wants the same walk.

The models name the ids they read: SystemModel's Quirks.swift and QuirkCatalogue.swift, ServerModel's ServerQuirks.swift. Two tests keep the halves from drifting:

• QuirkCatalogueTests asserts that every id the macOS model resolves is a row of macos.md and that every id resolves on both columns.
• ServerProfileScenarios asserts that every id ServerModel implements is a row of servers.md and that every profile cites only implemented ids.

The VM measures; it never proves. A VM session that ends without a catalogue row, a model rule and a green Linux suite has not finished.

Adding a macOS version¶

1. Add the version to MacOSVersion.
2. Measure on a VM of that version against the testbed.
3. Walk every row of macos.md and record a value: confirmed, changed, or not measured. Not-measured is a legitimate answer and shows as an empty cell.
4. Walk every row of servers.md too where the session touched a server.
5. For each changed value, add a measurement and let the model branch on it.
6. Mark any value taken once, or bracketed, with a confidence: note in both the row and the rule that reads it.
7. swift test --filter QuirkCatalogueTests checks that no id is in the model and missing from the catalogue.
8. swift test on Linux, green.
9. Update the catalog README's supported-versions table, and the platform page if the minimum moved.

Scenarios¶

A scenario is a numbered regression: a setup, an action and an assertion, named after the failure it defends against. Ids are stable for ever - a scenario is never renumbered, only retired with a reason - and the quirk rows cite them.

Scenarios live beside the module they exercise:

There are 143 ids in fifteen families, and 126 of them are named in Tests/. The seventeen that are not are the outstanding work: C3-C6, D7, D8, E1, E4-E6, F3, G7, K8, L2, M1, P6, P7. Several of the behaviours behind them are covered by a unit test that does not name the id.

A - the working set, anchors and enumeration¶

B - trash¶

C - extension lifecycle and XPC¶

D - writes, conflicts, atomicity¶

E - the index¶

F - offline, the breaker, reconnection¶

G - eviction and pinning¶

H - change detection¶

J - remote execution and the helper¶

K - transport and ssh¶

L - paths, names and attributes¶

M - symlinks¶

N - the capability report and probes¶

P - packaging and lifecycle¶

The family whose ground truth is a Mac. Each asserts the state machine on Linux; the fact behind it is a runbook item.

Q - add, askpass and the collect connection¶

Each drives the shipping add end to end, and nothing between the CLI's arguments and the server is a stub of ours:

• FakeSSH at SSHProcess.sshBinaryPath answers ssh -G, raises the real prompt strings and binds a real control socket.
• An askpass-shaped program, with a file mailbox in place of the XPC connection, hands each prompt to the real broker.
• The mount that follows runs the real SFTP transport over FakeSFTPServer on a real SFTP v3 wire.

What stays a manual check¶

These stay VM runbook items. They are listed so that nobody mistakes a green Linux suite for proof of them.

A real BSD find is not on this list: macOS's /usr/bin/find is one, and H4 and H7 exercise the .bsd flavour against it when the suite runs on the Mac.