| 1 | |
|---|
| 2 | # Continuous Integration for Swift |
|---|
| 3 | |
|---|
| 4 | **Table of Contents** |
|---|
| 5 | |
|---|
| 6 | - [Introduction](#introduction) |
|---|
| 7 | - [Pull Request Testing](#pull-request-testing) |
|---|
| 8 | - [@swift-ci](#swift-ci) |
|---|
| 9 | - [Smoke Testing](#smoke-testing) |
|---|
| 10 | - [Validation Testing](#validation-testing) |
|---|
| 11 | - [Linting](#linting) |
|---|
| 12 | - [Source Compatibility Testing](#source-compatibility-testing) |
|---|
| 13 | - [Sourcekit Stress Testing](#sourcekit-stress-testing) |
|---|
| 14 | - [Build Swift Toolchain](#build-swift-toolchain) |
|---|
| 15 | - [Build and Test Stdlib against Snapshot Toolchain](#build-and-test-stdlib-against-snapshot-toolchain) |
|---|
| 16 | - [Specific Preset Testing](#specific-preset-testing) |
|---|
| 17 | - [Specific Preset Testing against a Snapshot Toolchain](#specific-preset-testing-against-a-snapshot-toolchain) |
|---|
| 18 | - [Running Non-Executable Device Tests using Specific Preset Testing](#running-non-executable-device-tests-using-specific-preset-testing) |
|---|
| 19 | - [Build and Test the Minimal Freestanding Stdlib using Toolchain Specific Preset Testing](#build-and-test-the-minimal-freestanding-stdlib-using-toolchain-specific-preset-testing) |
|---|
| 20 | - [Testing Compiler Performance](#testing-compiler-performance) |
|---|
| 21 | - [Swift Community Hosted CI Pull Request Testing](#swift-community-hosted-ci-pull-request-testing) |
|---|
| 22 | - [Cross Repository Testing](#cross-repository-testing) |
|---|
| 23 | - [ci.swift.org bots](#ciswiftorg-bots) |
|---|
| 24 | |
|---|
| 25 | |
|---|
| 26 | ## Introduction |
|---|
| 27 | |
|---|
| 28 | This page is designed to assist in the understanding of proper practices for testing for the Swift project. |
|---|
| 29 | |
|---|
| 30 | ## Pull Request Testing |
|---|
| 31 | |
|---|
| 32 | In order for the Swift project to be able to advance quickly, it is important that we maintain a green build [[1]](#footnote-1). In order to help maintain this green build, the Swift project heavily uses pull request (PR) testing. Specifically, an important general rule is that **all** non-trivial checkins to any Swift Project repository should at least perform a [smoke test](#smoke-testing) if simulators will not be impacted *or* a full [validation test](#validation-testing) if simulators may be impacted. If in addition one is attempting to make a source breaking change across multiple repositories, one should follow the cross repo source breaking changes workflow. We now continue by describing the Swift system for Pull Request testing, @swift-ci: |
|---|
| 33 | |
|---|
| 34 | ### @swift-ci |
|---|
| 35 | |
|---|
| 36 | Users with [commit access](https://swift.org/contributing/#commit-access) can trigger pull request testing by writing a comment on a PR addressed to the GitHub user @swift-ci. Different tests will run depending on the specific comment used. The current test types are: |
|---|
| 37 | |
|---|
| 38 | 1. Smoke Testing |
|---|
| 39 | 2. Validation Testing |
|---|
| 40 | 3. Benchmarking. |
|---|
| 41 | 4. Linting |
|---|
| 42 | 5. Source Compatibility Testing |
|---|
| 43 | 6. Specific Preset Testing |
|---|
| 44 | 7. Testing Compiler Performance |
|---|
| 45 | |
|---|
| 46 | We describe each in detail below: |
|---|
| 47 | |
|---|
| 48 | ### Smoke Testing |
|---|
| 49 | |
|---|
| 50 | Platform | Comment | Check Status |
|---|
| 51 | ------------ | ------- | ------------ |
|---|
| 52 | All supported platforms | @swift-ci Please smoke test | Swift Test Linux Platform (smoke test)<br>Swift Test macOS Platform (smoke test) |
|---|
| 53 | All supported platforms | @swift-ci Please clean smoke test | Swift Test Linux Platform (smoke test)<br>Swift Test macOS Platform (smoke test) |
|---|
| 54 | macOS platform | @swift-ci Please smoke test macOS platform | Swift Test macOS Platform (smoke test) |
|---|
| 55 | macOS platform | @swift-ci Please clean smoke test macOS platform | Swift Test macOS Platform (smoke test) |
|---|
| 56 | Linux platform | @swift-ci Please smoke test Linux platform | Swift Test Linux Platform (smoke test) |
|---|
| 57 | Linux platform | @swift-ci Please clean smoke test Linux platform | Swift Test Linux Platform (smoke test) |
|---|
| 58 | |
|---|
| 59 | A smoke test on macOS does the following: |
|---|
| 60 | |
|---|
| 61 | 1. Builds LLVM/Clang incrementally. |
|---|
| 62 | 2. Builds Swift clean. |
|---|
| 63 | 3. Builds the standard library clean only for macOS. Simulator standard libraries and |
|---|
| 64 | device standard libraries are not built. |
|---|
| 65 | 4. lldb is not built. |
|---|
| 66 | 5. The test and validation-test targets are run only for macOS. The optimized |
|---|
| 67 | version of these tests are not run. |
|---|
| 68 | |
|---|
| 69 | A smoke test on Linux does the following: |
|---|
| 70 | |
|---|
| 71 | 1. Builds LLVM/Clang incrementally. |
|---|
| 72 | 2. Builds Swift clean. |
|---|
| 73 | 3. Builds the standard library clean. |
|---|
| 74 | 4. lldb is built incrementally. |
|---|
| 75 | 5. Foundation, SwiftPM, LLBuild, XCTest are built. |
|---|
| 76 | 6. The swift test and validation-test targets are run. The optimized version of these |
|---|
| 77 | tests are not run. |
|---|
| 78 | 7. lldb is tested. |
|---|
| 79 | 8. Foundation, SwiftPM, LLBuild, XCTest are tested. |
|---|
| 80 | |
|---|
| 81 | ### Validation Testing |
|---|
| 82 | |
|---|
| 83 | Platform | Comment | Check Status |
|---|
| 84 | ------------ | ------- | ------------ |
|---|
| 85 | All supported platforms | @swift-ci Please test | Swift Test Linux Platform (smoke test)<br>Swift Test macOS Platform (smoke test)<br>Swift Test Linux Platform<br>Swift Test macOS Platform<br> |
|---|
| 86 | All supported platforms | @swift-ci Please clean test | Swift Test Linux Platform (smoke test)<br>Swift Test macOS Platform (smoke test)<br>Swift Test Linux Platform<br>Swift Test macOS Platform<br> |
|---|
| 87 | macOS platform | @swift-ci Please test macOS platform | Swift Test macOS Platform (smoke test)<br>Swift Test macOS Platform |
|---|
| 88 | macOS platform | @swift-ci Please clean test macOS platform | Swift Test macOS Platform (smoke test)<br>Swift Test macOS Platform |
|---|
| 89 | macOS platform | @swift-ci Please benchmark | Swift Benchmark on macOS Platform (many runs - rigorous) |
|---|
| 90 | macOS platform | @swift-ci Please smoke benchmark | Swift Benchmark macOS Platform (few runs - sanity) |
|---|
| 91 | Linux platform | @swift-ci Please test Linux platform | Swift Test Linux Platform (smoke test)<br>Swift Test Linux Platform |
|---|
| 92 | Linux platform | @swift-ci Please clean test Linux platform | Swift Test Linux Platform (smoke test)<br>Swift Test Linux Platform |
|---|
| 93 | macOS platform | @swift-ci Please ASAN test | Swift ASAN Test macOS Platform |
|---|
| 94 | |
|---|
| 95 | The core principles of validation testing is that: |
|---|
| 96 | |
|---|
| 97 | 1. A validation test should build and run tests for /all/ platforms and all |
|---|
| 98 | architectures supported by the CI. |
|---|
| 99 | 2. A validation test should not be incremental. We want there to be a |
|---|
| 100 | definitiveness to a validation test. If one uses a validation test, one |
|---|
| 101 | should be sure that there is no nook or cranny in the code base that has not |
|---|
| 102 | been tested. |
|---|
| 103 | |
|---|
| 104 | With that being said, a validation test on macOS does the following: |
|---|
| 105 | |
|---|
| 106 | 1. Removes the workspace. |
|---|
| 107 | 2. Builds the compiler. |
|---|
| 108 | 3. Builds the standard library for macOS and the simulators for all platforms. |
|---|
| 109 | 4. lldb is /not/ built/tested [[2]](#footnote-2) |
|---|
| 110 | 5. The tests, validation-tests are run for iOS simulator, watchOS simulator and macOS both with |
|---|
| 111 | and without optimizations enabled. |
|---|
| 112 | |
|---|
| 113 | A validation test on Linux does the following: |
|---|
| 114 | |
|---|
| 115 | 1. Removes the workspace. |
|---|
| 116 | 2. Builds the compiler. |
|---|
| 117 | 3. Builds the standard library. |
|---|
| 118 | 4. lldb is built. |
|---|
| 119 | 5. Builds Foundation, SwiftPM, LLBuild, XCTest |
|---|
| 120 | 6. Run the swift test and validation-test targets with and without optimization. |
|---|
| 121 | 7. lldb is tested. |
|---|
| 122 | 8. Foundation, SwiftPM, LLBuild, XCTest are tested. |
|---|
| 123 | |
|---|
| 124 | ### Benchmarking |
|---|
| 125 | |
|---|
| 126 | Platform | Comment | Check Status |
|---|
| 127 | ------------ | ------- | ------------ |
|---|
| 128 | macOS platform | @swift-ci Please benchmark | Swift Benchmark on macOS Platform (many runs - rigorous) |
|---|
| 129 | macOS platform | @swift-ci Please smoke benchmark | Swift Benchmark on macOS Platform (few runs - sanity) |
|---|
| 130 | |
|---|
| 131 | ### Linting |
|---|
| 132 | |
|---|
| 133 | Language | Comment | What it Does | Corresponding Local Command |
|---|
| 134 | ------------ | ------- | ------------ | ------------- |
|---|
| 135 | Python | @swift-ci Please Python lint | Lints Python sources | `./utils/python_lint.py` |
|---|
| 136 | |
|---|
| 137 | ### Source Compatibility Testing |
|---|
| 138 | |
|---|
| 139 | Platform | Comment | Check Status |
|---|
| 140 | ------------ | ------- | ------------ |
|---|
| 141 | macOS platform | @swift-ci Please Test Source Compatibility | Swift Source Compatibility Suite on macOS Platform (Release and Debug) |
|---|
| 142 | macOS platform | @swift-ci Please Test Source Compatibility Release | Swift Source Compatibility Suite on macOS Platform (Release) |
|---|
| 143 | macOS platform | @swift-ci Please Test Source Compatibility Debug | Swift Source Compatibility Suite on macOS Platform (Debug) |
|---|
| 144 | |
|---|
| 145 | ### Sourcekit Stress Testing |
|---|
| 146 | |
|---|
| 147 | Platform | Comment | Check Status |
|---|
| 148 | ------------ | ------- | ------------ |
|---|
| 149 | macOS platform | @swift-ci Please Sourcekit Stress test | Swift Sourcekit Stress Tester on macOS Platform |
|---|
| 150 | |
|---|
| 151 | ### Build Swift Toolchain |
|---|
| 152 | |
|---|
| 153 | Platform | Comment | Check Status |
|---|
| 154 | ------------ | ------- | ------------ |
|---|
| 155 | macOS platform | @swift-ci Please Build Toolchain macOS Platform| Swift Build Toolchain macOS Platform |
|---|
| 156 | Linux platform | @swift-ci Please Build Toolchain Linux Platform| Swift Build Toolchain Ubuntu 22.04 (x86_64) |
|---|
| 157 | |
|---|
| 158 | You can also build a toolchain for a specific Linux distribution |
|---|
| 159 | |
|---|
| 160 | Distro | Comment | Check Status |
|---|
| 161 | -------------- | ------------------------------------------------ | ---------------------------------------------- |
|---|
| 162 | UBI9 | @swift-ci Please Build Toolchain UBI9 | Swift Build Toolchain UBI9 (x86_64) |
|---|
| 163 | CentOS 7 | @swift-ci Please Build Toolchain CentOS 7 | Swift Build Toolchain CentOS 7 (x86_64) |
|---|
| 164 | Ubuntu 18.04 | @swift-ci Please Build Toolchain Ubuntu 18.04 | Swift Build Toolchain Ubuntu 18.04 (x86_64) |
|---|
| 165 | Ubuntu 20.04 | @swift-ci Please Build Toolchain Ubuntu 20.04 | Swift Build Toolchain Ubuntu 20.04 (x86_64) |
|---|
| 166 | Ubuntu 22.04 | @swift-ci Please Build Toolchain Ubuntu 22.04 | Swift Build Toolchain Ubuntu 22.04 (x86_64) |
|---|
| 167 | Amazon Linux 2 | @swift-ci Please Build Toolchain Amazon Linux 2 | Swift Build Toolchain Amazon Linux 2 (x86_64) |
|---|
| 168 | |
|---|
| 169 | ### Build and Test Stdlib against Snapshot Toolchain |
|---|
| 170 | |
|---|
| 171 | To test/build the stdlib for a branch that changes only the stdlib using a last known good snapshot toolchain: |
|---|
| 172 | |
|---|
| 173 | Platform | Comment | Check Status |
|---|
| 174 | ------------ | ------- | ------------ |
|---|
| 175 | macOS platform | @swift-ci Please test stdlib with toolchain| Swift Test stdlib with toolchain macOS Platform |
|---|
| 176 | |
|---|
| 177 | ### Specific Preset Testing |
|---|
| 178 | |
|---|
| 179 | Platform | Comment | Check Status |
|---|
| 180 | ------------ | ------- | ------------ |
|---|
| 181 | macOS platform | preset=<preset> <br> @swift-ci Please test with preset macOS Platform | Swift Test macOS Platform with preset |
|---|
| 182 | Linux platform | preset=<preset> <br> @swift-ci Please test with preset Linux Platform | Swift Test Linux Platform with preset |
|---|
| 183 | |
|---|
| 184 | |
|---|
| 185 | For example: |
|---|
| 186 | |
|---|
| 187 | ``` |
|---|
| 188 | preset=buildbot_incremental,tools=RA,stdlib=RD,smoketest=macosx,single-thread |
|---|
| 189 | @swift-ci Please test with preset macOS |
|---|
| 190 | ``` |
|---|
| 191 | |
|---|
| 192 | |
|---|
| 193 | ### Specific Preset Testing against a Snapshot Toolchain |
|---|
| 194 | |
|---|
| 195 | One can also run an arbitrary preset against a snapshot toolchain |
|---|
| 196 | |
|---|
| 197 | Platform | Comment | Check Status |
|---|
| 198 | ------------ | ------- | ------------ |
|---|
| 199 | macOS platform | preset=<preset> <br> @swift-ci Please test with toolchain and preset| Swift Test stdlib with toolchain macOS Platform (Preset) |
|---|
| 200 | |
|---|
| 201 | For example: |
|---|
| 202 | |
|---|
| 203 | ``` |
|---|
| 204 | preset=$PRESET_NAME |
|---|
| 205 | @swift-ci Please test with toolchain and preset |
|---|
| 206 | ``` |
|---|
| 207 | |
|---|
| 208 | ### Running Non-Executable Device Tests using Specific Preset Testing |
|---|
| 209 | |
|---|
| 210 | Using the specific preset testing, one can run non-executable device tests by |
|---|
| 211 | telling swift-ci: |
|---|
| 212 | |
|---|
| 213 | ``` |
|---|
| 214 | preset=buildbot,tools=RA,stdlib=RD,test=non_executable |
|---|
| 215 | @swift-ci Please test with preset macOS |
|---|
| 216 | ``` |
|---|
| 217 | |
|---|
| 218 | ### Build and Test the Minimal Freestanding Stdlib using Toolchain Specific Preset Testing |
|---|
| 219 | |
|---|
| 220 | To test the minimal freestanding stdlib on macho, you can use the support for running a miscellaneous preset against a snapshot toolchain. |
|---|
| 221 | |
|---|
| 222 | ``` |
|---|
| 223 | preset=stdlib_S_standalone_minimal_macho_x86_64,build,test |
|---|
| 224 | @swift-ci please test with toolchain and preset |
|---|
| 225 | ``` |
|---|
| 226 | |
|---|
| 227 | ### Testing Compiler Performance |
|---|
| 228 | |
|---|
| 229 | Platform | Comment | Check Status |
|---|
| 230 | ------------ | ------- | ------------ |
|---|
| 231 | macOS platform | @swift-ci Please test compiler performance | Compiles full source compatibility test suite and measures compiler performance |
|---|
| 232 | macOS platform | @swift-ci Please smoke test compiler performance | Compiles a subset of source compatibility test suite and measures compiler performance |
|---|
| 233 | |
|---|
| 234 | These commands will: |
|---|
| 235 | |
|---|
| 236 | 1. Build a set of projects from the compatibility test suite |
|---|
| 237 | 2. Collect counters and timers reported by the compiler |
|---|
| 238 | 3. Compare the obtained data to the baseline (stored in git) and HEAD (version of a compiler built without the PR changes) |
|---|
| 239 | 4. Report the results in a pull request comment |
|---|
| 240 | |
|---|
| 241 | For the detailed explanation of how compiler performance is measured, please refer to [this document](https://github.com/apple/swift/blob/main/docs/CompilerPerformance.md). |
|---|
| 242 | |
|---|
| 243 | ## Cross Repository Testing |
|---|
| 244 | |
|---|
| 245 | Simply provide the URL from corresponding pull requests in the same comment as "@swift-ci Please test" phrase. List all of the pull requests and then provide the specific test phrase you would like to trigger. Currently, it will only merge the main pull request you requested testing from as opposed to all of the PR's. |
|---|
| 246 | |
|---|
| 247 | For example: |
|---|
| 248 | |
|---|
| 249 | ``` |
|---|
| 250 | Please test with following pull request: |
|---|
| 251 | https://github.com/apple/swift/pull/4574 |
|---|
| 252 | |
|---|
| 253 | @swift-ci Please test Linux platform |
|---|
| 254 | ``` |
|---|
| 255 | |
|---|
| 256 | ``` |
|---|
| 257 | Please test with following PR: |
|---|
| 258 | https://github.com/apple/swift-lldb/pull/48 |
|---|
| 259 | https://github.com/apple/swift-package-manager/pull/632 |
|---|
| 260 | |
|---|
| 261 | @swift-ci Please test macOS platform |
|---|
| 262 | ``` |
|---|
| 263 | |
|---|
| 264 | ``` |
|---|
| 265 | apple/swift-lldb#48 |
|---|
| 266 | |
|---|
| 267 | @swift-ci Please test Linux platform |
|---|
| 268 | ``` |
|---|
| 269 | |
|---|
| 270 | 1. Create a separate PR for each repository that needs to be changed. Each should reference the main Swift PR and create a reference to all of the others from the main PR. |
|---|
| 271 | |
|---|
| 272 | 2. Gate all commits on @swift-ci smoke test. As stated above, it is important that *all* checkins perform PR testing since if breakage enters the tree PR testing becomes less effective. If you have done local testing (using build-toolchain) and have made appropriate changes to the other repositories then perform a smoke test should be sufficient for correctness. This is not meant to check for correctness in your commits, but rather to be sure that no one landed changes in other repositories or in swift that cause your PR to no longer be correct. If you were unable to make workarounds to the other repositories, this smoke test will break *after* Swift has built. Check the log to make sure that it is the expected failure for that platform/repository that coincides with the failure your PR is supposed to fix. |
|---|
| 273 | |
|---|
| 274 | 3. Merge all of the pull requests simultaneously. |
|---|
| 275 | |
|---|
| 276 | 4. Watch the public incremental build on [ci.swift.org](https://ci.swift.org/) to make sure that you did not make any mistakes. It should complete within 30-40 minutes depending on what else was being committed in the mean time. |
|---|
| 277 | |
|---|
| 278 | ### Swift Community Hosted CI Pull Request Testing |
|---|
| 279 | |
|---|
| 280 | Currently, supported pull request testing triggers: |
|---|
| 281 | |
|---|
| 282 | Platform | Comment | Check Status |
|---|
| 283 | ------------ | ------- | ------------ |
|---|
| 284 | Windows | @swift-ci Please test Windows platform | Swift Test Windows Platform |
|---|
| 285 | |
|---|
| 286 | ## ci.swift.org bots |
|---|
| 287 | |
|---|
| 288 | FIXME: FILL ME IN! |
|---|
| 289 | |
|---|
| 290 | <a name="footnote-1">[1]</a> Even though it should be without saying, the reason why having a green build is important is that: |
|---|
| 291 | |
|---|
| 292 | 1. A full build break can prevent other developers from testing their work. |
|---|
| 293 | 2. A test break can make it difficult for developers to know whether or not their specific commit has broken a test, requiring them to perform an initial clean build, wasting time. |
|---|
| 294 | 3. @swift-ci pull request testing becomes less effective since one can not perform a test and one must reason about the source of a given failure. |
|---|
| 295 | |
|---|
| 296 | <a name="footnote-2">[2]</a> This is due to unrelated issues relating to running lldb tests on macOS. |
|---|
| 297 | |
|---|
| 298 | |
|---|
| 299 | |
|---|
