UI Test Feasibility Assessment — MixBoard iOS

Date: 2026-03-16
Tester model: Claude Opus 4.6 (Tester mode)
Scope: Full app UI test coverage assessment

1. Current UI Test Coverage

Existing tests: `UITests/MixBoardUITests.swift` — 20 test methods

Area	Tests	What's Covered
App Launch	2	Navigation title, toolbar buttons
Empty State	1	No-playlists message
Playlist CRUD	4	Create, cancel, multiple creates, delete via swipe
Playlist Navigation	2	Navigate to detail, header/track count
Library	2	Open sheet, browse mode buttons
Settings	4	Open sheet, skin section, mix targets, skin switch, library stats
Mini Player	1	Not visible without track
Now Playing	1	Not shown initially
Orientation	1	Landscape rotation no crash
Performance	1	Launch metric

What's NOT covered (gaps)

Area	Gap	Risk
Cloud Browser	Zero tests	HIGH — the Years decoding bug lived here undetected
Cloud → Category drill-down	Zero tests	HIGH — all API-dependent navigation untested
Cloud → Album → Tracks	Zero tests	HIGH
Playback flow	Zero tests	MEDIUM — play a track, verify mini player appears
Queue management	Zero tests	MEDIUM — add to queue, reorder, remove
Now Playing transport	Zero tests	LOW — play/pause/skip buttons exist but untested
Lyrics panel	Zero tests	LOW
Settings: Chad Music config	Zero tests	MEDIUM — server URL, API key, test connection
Add to Playlist sheet	Zero tests	MEDIUM
Search (in lists)	Zero tests	LOW

Existing accessibility identifiers (good foundation)

The app already has accessibilityIdentifier on key elements:

PlaylistListView: libraryButton, cloudBrowserButton, settingsButton, newPlaylistButton, playlistList, emptyState, playlistRow_{name}
NowPlayingView: nowPlayingDismiss, queueButton, lyricsButton, nowPlayingTitle, nowPlayingArtist, shuffleButton, previousButton, playPauseButton, nextButton, lyricsPanel
MiniPlayerView: miniPlayer, miniPlayerPlayPause, miniPlayerNext, miniPlayerQueue
ContentView: ContentView

Missing identifiers: CloudBrowserView has no accessibility identifiers at all — they'd need to be added before UI tests can target cloud elements.

Infrastructure observation

-UITesting launch argument is passed in setUp() but never checked in app code — there's no mock data path or conditional behavior for UI testing
ChadMusicAPIClient is a concrete singleton (ChadMusicAPIClient.shared) with no protocol abstraction — cannot be mocked without changes
No mock server, stub data, or URLProtocol interception exists

2. Technical Approach: Handling the API Dependency

Three viable options, ranked by effort and value:

Option A: URLProtocol Stub (Recommended)

How: Register a custom URLProtocol subclass that intercepts outgoing requests and returns canned JSON. Activated via the -UITesting launch argument.

Pros: No external dependencies, runs offline, deterministic, fast
Cons: Requires adding ~100 lines of stub infrastructure to the app target + JSON fixture files
Catches the Years bug?: YES — if fixtures match real API shape, decoding failures surface immediately. If fixtures are fabricated, they might mask the problem. Best practice: capture real API responses as fixtures.

Option B: Test Against Real Server

How: Point UI tests at the actual Chad Music server (pre-configured URL + API key in test scheme environment variables).

Pros: Zero mock infrastructure, tests real behavior end-to-end
Cons: Tests are flaky (server down → tests fail), slow (network latency), non-deterministic (library content changes), can't run in CI without server access
Catches the Years bug?: YES — directly, because it hits the real API

Option C: Local Mock Server (e.g., Vapor/Hummingbird test server)

How: Spin up a lightweight Swift HTTP server in the test setUp that serves canned responses.

Pros: Full control, realistic HTTP layer
Cons: High complexity, process management in XCUITest is cumbersome
Catches the Years bug?: Only if mock data matches real API shapes

Recommendation

Start with Option B (real server) for immediate smoke tests that catch bugs like the Years issue. Migrate to Option A (URLProtocol stubs) once you want CI reliability and offline testing. Option A requires the app to check for -UITesting and inject a stub URLSession into the API client.

3. Recommended UI Test Scenarios (Prioritized)

P1 — Must Have (catches real bugs, high value)

T1: Cloud Browser — Navigate to Each Category

Type: UI / Integration
Criterion: All cloud browse categories (Albums, Artists, Genres, Years, etc.) should load without errors
Preconditions: App configured with valid Chad Music server URL + API key (or stubs)
Steps:
1. Launch app
2. Tap cloud browser button
3. For each category in [Artists, Genres, Years]: tap category, wait for list to load, verify no error message appears, go back
Expected Result: Each category shows a populated list (or empty list), never an error message
Automatable: Yes
Priority: P1
Note: This SINGLE test would have caught the Years decoding bug

T2: Cloud Browser — Album Drill-Down to Tracks

Type: UI / Integration
Criterion: User can browse albums and see track listing
Steps:
1. Open cloud browser
2. Tap "Albums" (or navigate via Artists → first artist → first album)
3. Verify album detail loads with track rows
Expected Result: Track list appears with track titles
Automatable: Yes
Priority: P1

T3: Cloud Browser — Play a Cloud Track

Type: UI / Integration
Criterion: Tapping a cloud track starts playback and shows mini player
Steps:
1. Navigate to an album in cloud browser
2. Tap a track
3. Verify mini player appears with track title
Expected Result: Mini player shows, track title matches
Automatable: Yes (verify UI state, not audio output)
Priority: P1

T4: Settings — Configure Chad Music Server

Type: UI
Criterion: User can enter server URL and API key in Settings
Steps:
1. Open Settings
2. Find Chad Music section
3. Enter server URL and API key
4. Tap "Test Connection"
5. Verify success/failure indicator
Expected Result: Connection test shows result
Automatable: Yes
Priority: P1

P2 — Should Have (core app flows)

T5: Playlist — Add Cloud Track to Playlist

Type: UI / Integration
Steps:
1. Create a playlist
2. Open cloud browser, navigate to tracks
3. Long-press or context-menu a track → "Add to Playlist"
4. Select the created playlist
5. Navigate to playlist, verify track appears
Expected Result: Track is in playlist with correct title
Automatable: Yes
Priority: P2

T6: Now Playing — Transport Controls

Type: UI
Steps:
1. Start playing a track (local or cloud)
2. Tap mini player to open Now Playing
3. Verify play/pause button, next, previous, shuffle, repeat are tappable
4. Tap play/pause — verify icon toggles
Expected Result: Controls respond, icon state changes
Automatable: Yes
Priority: P2

T7: Queue — Add and View Queue

Type: UI
Steps:
1. Start playing a track
2. Add another track to queue
3. Open queue view
4. Verify both tracks appear in correct sections
Expected Result: Now Playing + Up Next sections populated
Automatable: Yes
Priority: P2

T8: Library — Import Local Files

Type: UI
Preconditions: Test audio file accessible in Files app
Steps:
1. Open Library
2. Navigate to folder browser
3. Select a folder
4. Verify tracks appear
Expected Result: Track listing from local files
Automatable: Partial (file access in simulator is limited)
Priority: P2

P3 — Nice to Have

T9: Now Playing — Lyrics Toggle

Type: UI
Steps: Open Now Playing → tap lyrics button → verify lyrics panel appears
Automatable: Yes (with a track that has lyrics or stubs)
Priority: P3

T10: Settings — Skin Switching (already partially covered)

Type: UI
Already exists: testSwitchSkin() partially covers this
Priority: P3

T11: Cloud Browser — Search Filtering

Type: UI
Steps: Open an album list → type in search → verify list filters
Automatable: Yes
Priority: P3

4. The Years Bug — Specific Analysis

Could a UI test have caught it? YES, trivially.

The bug is in CategoryDetailView.load() which calls ChadMusicAPIClient.shared.fetchCategory(.year). The API returns year values in a format that ChadCategory.item: String can't decode. The view catches the error and displays errorMessage — so a UI test that navigates to Years and asserts "no error text is visible" would catch it immediately.

What that test looks like:

func testCloudBrowserYearsCategoryLoads() {
    // Precondition: app is configured with a valid Chad Music server
    let cloudButton = app.buttons["cloudBrowserButton"]
    XCTAssertTrue(cloudButton.waitForExistence(timeout: 5))
    cloudButton.tap()
    
    // Navigate to the category list
    let yearsCell = app.staticTexts["Years"]
    XCTAssertTrue(yearsCell.waitForExistence(timeout: 5), 
        "Years category should appear in cloud browser")
    yearsCell.tap()
    
    // Wait for the category to load — should show items, not an error
    // The view shows a red error text on decode failure
    let loadingIndicator = app.activityIndicators.firstMatch
    // Wait for loading to finish
    let loaded = NSPredicate(format: "exists == false")
    expectation(for: loaded, evaluatedWith: loadingIndicator, handler: nil)
    waitForExpectations(timeout: 10)
    
    // Verify: no error message visible
    let errorTexts = app.staticTexts.matching(
        NSPredicate(format: "label CONTAINS 'Failed to decode'")
    )
    XCTAssertEqual(errorTexts.count, 0, 
        "Years category should load without decoding errors")
    
    // Verify: at least one year item exists
    let listCells = app.cells
    XCTAssertTrue(listCells.count > 0, 
        "Years list should have at least one item")
}

Example: All-categories smoke test

func testCloudBrowserAllCategoriesLoad() {
    let cloudButton = app.buttons["cloudBrowserButton"]
    XCTAssertTrue(cloudButton.waitForExistence(timeout: 5))
    cloudButton.tap()
    
    // Verify the browse section exists
    let browseSection = app.staticTexts["Browse"]
    XCTAssertTrue(browseSection.waitForExistence(timeout: 5))
    
    // Test each non-album category (album has its own view)
    let categories = ["Artists", "Genres", "Years"]
    for categoryName in categories {
        let cell = app.staticTexts[categoryName]
        XCTAssertTrue(cell.waitForExistence(timeout: 3),
            "\(categoryName) should be visible")
        cell.tap()
        
        // Wait for load, check no error
        sleep(3) // Allow network request to complete
        
        let errorTexts = app.staticTexts.matching(
            NSPredicate(format: "label CONTAINS 'Failed' OR label CONTAINS 'Error'")
        )
        XCTAssertEqual(errorTexts.count, 0,
            "\(categoryName) should load without errors")
        
        // Navigate back
        app.navigationBars.buttons.element(boundBy: 0).tap()
    }
}

Example: Cloud playback smoke test

func testCloudTrackPlaybackShowsMiniPlayer() {
    let cloudButton = app.buttons["cloudBrowserButton"]
    XCTAssertTrue(cloudButton.waitForExistence(timeout: 5))
    cloudButton.tap()
    
    // Navigate: Albums → first album → first track
    let albumsCell = app.staticTexts["Albums"]
    XCTAssertTrue(albumsCell.waitForExistence(timeout: 5))
    albumsCell.tap()
    
    // Wait for albums to load, tap first one
    let firstAlbum = app.cells.firstMatch
    XCTAssertTrue(firstAlbum.waitForExistence(timeout: 10))
    firstAlbum.tap()
    
    // Wait for tracks to load, tap first one
    let firstTrack = app.cells.firstMatch
    XCTAssertTrue(firstTrack.waitForExistence(timeout: 10))
    firstTrack.tap()
    
    // Dismiss cloud browser
    let doneButton = app.buttons["Done"]
    if doneButton.waitForExistence(timeout: 2) {
        doneButton.tap()
    }
    
    // Verify mini player appears
    let miniPlayer = app.otherElements["miniPlayer"]
    XCTAssertTrue(miniPlayer.waitForExistence(timeout: 10),
        "Mini player should appear after tapping a cloud track")
}

5. Effort Estimate

Phase	Tests	Complexity	Notes
Phase 1: Cloud smoke tests (P1)	4 tests	Low	Requires: add accessibilityIdentifiers to CloudBrowserView, have a configured server
Phase 2: Playback + queue (P2)	4 tests	Medium	Need a playable track (cloud or local), state verification
Phase 3: URLProtocol stubs	Infrastructure	Medium	~100 lines stub code + JSON fixtures, app-side `-UITesting` check
Phase 4: Nice-to-have (P3)	3 tests	Low	Lyrics, search, additional settings

Prerequisites before writing any new UI tests:

Add accessibilityIdentifiers to CloudBrowserView — category cells, album rows, track rows, error labels, stats badges. Without these, XCUITest can't reliably target elements. (~15 identifiers needed)
Decide on API strategy — real server (fast to start) vs stubs (reliable for CI). Can start with real server and add stubs later.
Pre-configure the simulator — the test scheme needs a Chad Music server URL + API key already set, OR the tests must enter them in Settings first.

Total new test methods: ~11 (across P1–P3)

Infrastructure work: AccessibilityIdentifiers (~30 min), URLProtocol stubs if desired (~2 hours)

6. Findings

Finding F1: CloudBrowserView has zero accessibilityIdentifiers

Severity: Major (blocks UI test authoring)
Description: None of the interactive elements in CloudBrowserView, CategoryDetailView, AlbumListView, AlbumDetailView, or FilteredAlbumListView have accessibilityIdentifiers
Expected: Key elements (category rows, album rows, track rows, error labels) should have stable identifiers
Actual: XCUITest must rely on fragile text matching to find elements
Affects criterion: All cloud browser UI test scenarios

Finding F2: No `-UITesting` launch argument handling in app code

Severity: Minor (only matters if you want mock data / stubs)
Description: The UI test setUp passes -UITesting as a launch argument, but the app never checks ProcessInfo.processInfo.arguments for it
Expected: App should detect UI testing mode to enable stubs, seed data, or disable animations
Actual: Launch argument is ignored

Finding F3: ChadMusicAPIClient is a concrete singleton — not mockable

Severity: Minor (architectural concern for testability)
Description: ChadMusicAPIClient.shared is used directly in views. No protocol, no dependency injection. To use URLProtocol stubs, you'd need either: (a) make the client accept a custom URLSession, or (b) register a global URLProtocol
Affects: Option A (URLProtocol stubs) approach

Finding F4: Unit tests already cover ChadCategory decoding

Severity: Informational (positive finding)
Description: CloudStreamingTests.swift has thorough unit tests for ChadCategory, ChadAlbum, ChadTrack decoding. However, these test with fabricated JSON — if the real API returns a different shape (e.g., year as Int not String), unit tests wouldn't catch it. UI tests against the real API WOULD catch it.

7. Summary & Recommendation

Feasibility: HIGH. The app already has good accessibility identifiers on most views, an existing UI test infrastructure that works, and a test target properly configured in project.yml. The main gap is cloud browser coverage and API dependency handling.

Highest-impact, lowest-effort win: Add 4 smoke tests (T1–T4) that navigate the cloud browser categories and verify no decode errors. These run against the real server, require only adding accessibilityIdentifiers to CloudBrowserView, and would have caught the Years bug before a human ever saw it.

Longer-term: Implement URLProtocol stubs so cloud browser tests run reliably in CI without a live server.

test-assessment.md 16 KB Байнгын холболт Түүх Анхны өгөгдөл