Automatically Turn UI Tests Into Screen Recordings

In a previous post, I talked about making your status bar nice when automating your App Store screenshots. These days, images alone are not enough to tell your app’s story. The App Store supports preview videos, and if you’re trying to create one of those for every combination of screen size and language that your app supports, it can be a chore!

Video can also be incredibly effective at presenting your work to others. At Detroit Labs, we pride ourselves in polished demos of our work to our clients. One of the best ways to do that is to embed videos of the app in a slide deck and go over it during the presentation. This avoids awkwardness in live demos and allows the video to loop as the client asks questions (you should also have the code ready to run in case the client asks about something that isn’t in the video).

One downside of these videos is the amount of time it takes to prepare them. If you’re diligent, you can record these videos for each individual pull request, then at the end of a sprint, collect the videos and prepare the presentation from them. Even if that happens, you can still fall out of sync. If pull request A shows some new functionality but is merged before pull request B, which updates the client’s logo to their new branding, their old branding in the video for pull request A will confuse them and distract from the actual feature during the demo. It’s best to compile all of the videos from a single, stable codebase, and deliver that exact code to the client along with the demo so there are no surprises.

Manually running all of the code and capturing video is a chore, especially for large, productive teams. I wanted to figure out a way to automate these video captures, and last week while in my hotel room at CodeMash, I figured out how to do it. Xcode ships with the simctl utility, and you can use it to record video from the command line:

xcrun simctl io <device> recordVideo <options> <path>

Once this starts running, it’ll record the given device’s screen to the given file. All we need to do is trigger this command. Unfortunately, the UI tests that Xcode runs run as iOS applications, so we don’t have access to the host Mac to run simctl. That’s where SBTUITestTunnelHost comes in! Using this framework and the associated Mac app, we can pass messages locally from our UI test suite to our Mac. One of the things it can do is execute command-line utilities. This is perfect for our needs… almost.

Because simctl will keep recording video until you hit ^C in Terminal, we need to leave it open. SBTUITestTunelHost, by default, executes a command, waits for it to finish, and then returns. To get around this, I modified the server (and sent a pull request) to enable starting asynchronous commands, then later terminating them. With this in place and running, recording an individual unit test is straightforward! Here’s an example test suite:

import SBTUITestTunnelHost
import XCTest

class InstantReplay_iOSUITests: XCTestCase {
    
    var commandID: UUID!
    
    override func setUp() {
        // Put setup code here. This method is called before the invocation of each test method in the class.

        // In UI tests it is usually best to stop immediately when a failure occurs.
        continueAfterFailure = false
        
        // UI tests must launch the application that they test.
        let app = XCUIApplication()
        app.launch()

        commandID = host.launchCommand(
            "xcrun simctl io booted recordVideo --codec=h264 --mask=black --force /Users/Jeff/Desktop/test.mov"
        )
        
        _ = app.wait(for: .runningForeground, timeout: 5)
        
        _ = app.textFields.firstMatch.waitForExistence(timeout: 0.5)

        if let commandID = commandID {
            addTeardownBlock { [unowned self] in
                self.host.interruptCommand(with: commandID)
            }
        }
        else {
            print("No status!")
        }
    }

    func testExample() {
        let app = XCUIApplication()
            
        app.textFields.firstMatch.tap()
        
        let keyDelay = { return TimeInterval.random(in: 0...0.15) }

        "UITestExample@example.com".map(String.init).forEach {
            if ["@", "."].contains($0) {
                app.keys["more"].tap()
                XCTAssertTrue(app.keys[$0].waitForExistence(timeout: keyDelay()))
                app.keys[$0].tap()
                app.keys["more"].tap()
            }
            else if $0 == $0.uppercased(), !app.keys[$0].exists {
                app.buttons["shift"].tap()
                XCTAssertTrue(app.keys[$0].waitForExistence(timeout: keyDelay()))
                app.keys[$0].tap()
            }
            else {
                XCTAssertTrue(app.keys[$0].waitForExistence(timeout: keyDelay()))
                app.keys[$0].tap()
            }
        }
        
        app.buttons["Return"].tap()
        
        XCTAssertTrue(app.textFields.firstMatch.waitForExistence(timeout: 0.5))
    }
    
}

This suite will automatically run, perform its tests, and record the output to a video on my Desktop. Sure enough, once I’d given the host app sufficient permissions in macOS Catalina, it worked, and while I was drinking my coffee after pressing ⌘U in Xcode, this video appeared on my desktop:

This is useful for pull requests, demos, embedding videos into Deckset presentations, you name it! Because it’s generating new videos every time, you never have to worry about your demo being out of date, or spending all of your time creating videos for every language you support. Just set it and forget it!

UIAppearance and the Case of the Missing Method

Apple’s UIAppearance API is a great way to customize the look and feel of your iOS app. In one fell swoop, you can achieve common goals like styling the navigation bar across your entire app, setting custom fonts, and the like. You can even use it to style your own custom classes! However, I recently ran into a common issue with the protocol that left me wanting more.

Take the above example of styling a navigation bar. Here’s a code sample that produces a green navigation bar with a custom font:

typealias LaunchOptions = [UIApplication.LaunchOptionsKey: Any]

func application(
    _ application: UIApplication,
    didFinishLaunchingWithOptions launchOptions: LaunchOptions?
) -> Bool {
    UINavigationBar.appearance().barTintColor = .systemGreen
    UINavigationBar.appearance().barStyle = .black
    UINavigationBar.appearance().tintColor = .systemGreen
    UINavigationBar.appearance().titleTextAttributes = [
        NSAttributedString.Key.font: UIFont(name: "Marker Felt", size: 24)!
    ]
    
    return true
}

This produces this beautiful UI:

Our custom-styled navigation bar has a green background and a custom font.
Our custom-styled navigation bar has a green background and a custom font.

Looks perfect! This can cause problems, however, when we go to use system controls that also use UINavigationBar. Let’s add a button that brings up some system UI:

@IBAction func userTappedShowContact(_ sender: UIButton) {
    let contact = CNMutableContact()
    contact.givenName = "John"
    contact.familyName = "Appleseed"
    contact.emailAddresses = [
        CNLabeledValue(label: "work", value: "John-Appleseed@mac.com")
    ]
    
    let viewController = CNContactViewController(forUnknownContact: contact)
    
    viewController.allowsEditing = false
    viewController.isEditing = false
    
    let navigationController = UINavigationController(
        rootViewController: viewController
    )
    
    present(navigationController, animated: true, completion: nil)
}

Because we’ve already customized UINavigationBar, this displays with the same green background:

The system UI now also has that same custom style.
The system UI now also has that same custom style.

What’s worse is when you tap on “Create New Contact,” which brings up its own modal navigation controller. The tint color and background color are too close, making the bar button items nearly unreadable:

On the “New Contact” screen, the background color and tint color combine to make ugly green-on-green text that’s hard to read.
On the “New Contact” screen, the background color and tint color combine to make ugly green-on-green text that’s hard to read.

Trying to find a way around this, I tried a few different options, but nothing really worked the way I wanted it to. You can use the method appearance(whenContainedInInstancesOf:) to get an appearance proxy that only applies for certain containers, but as the “New Contact” screen is created by the system, it’ll always use a standard UINavigationController. What I really wanted was a way to reset the state of the appearance proxy for this class, display the system UI, and then restore my customizations when this was done. Unfortunately the appearance proxies, since they appear as the same class as the view you’re customizing, don’t have a reset() method. So, I built that.

We’re only using the “legacy customizations” for UINavigationBar, as our app supports iOS 12 and therefore can’t use the new options, so I wrote a method to restore each of these values to their defaults. When you’re done, simply call the returned closure and your custom settings will be restored:

extension UIAppearance {
    typealias DefaultAppearanceResetHandler = () -> Void
}

extension UINavigationBar {
    
    /// Resets the navigation bar’s appearance proxy to the default settings.
    /// Returns a closure to execute to restore the current settings. 
    static func resetToDefaultAppearance() -> UIAppearance.DefaultAppearanceResetHandler {
        let appearanceProxy = appearance()
        
        // The following settings are all part of the “legacy customizations”
        // for UINavigationBar’s appearance:
        // https://developer.apple.com/documentation/uikit/uinavigationbar/legacy_customizations
        //
        // First, we store the current values, then we reset them to their
        // defaults.
        
        // Setting the Bar’s Style
        let barStyle = appearanceProxy.barStyle
        
        // Configuring the Title
        let titleTextAttributes = appearanceProxy.titleTextAttributes
        let largeTitleTextAttributes = appearanceProxy.largeTitleTextAttributes
        
        let titleVerticalPositionAdjustments: [UIBarMetrics: CGFloat] = UIBarMetrics.allCases.reduce(into: [:]) { (result, metrics) in
            result[metrics] = appearanceProxy.titleVerticalPositionAdjustment(for: metrics)
        }
        
        // Configuring Bar Button items
        let tintColor = appearanceProxy.tintColor
        
        // Configuring the Back Button
        let backIndicatorImage = appearanceProxy.backIndicatorImage
        let backIndicatorTransitionMaskImage = appearanceProxy.backIndicatorTransitionMaskImage
        
        // Changing the Background
        let barTintColor = appearanceProxy.barTintColor
        
        let backgroundImages: [UIBarMetrics: UIImage] = UIBarMetrics.allCases.reduce(into: [:]) { (result, metrics) in
            result[metrics] = appearanceProxy.backgroundImage(for: metrics)
        }
        
        let backgroundImagesForBarPosition: [UIBarPosition: [UIBarMetrics: UIImage]] =
            UIBarPosition.casesForNavigationBarBackgroundImage.reduce(into: [:]) { (result, position) in
                result[position] = UIBarMetrics.allCases.reduce(into: [:]) { (result, metrics) in
                    result[metrics] = appearanceProxy.backgroundImage(
                        for: position,
                        barMetrics: metrics
                    )
                }
        }
        
        // Adding a Shadow
        let shadowImage = appearanceProxy.shadowImage
        
        // Now we reset the values to their defaults.
        appearanceProxy.barStyle = .default
        appearanceProxy.titleTextAttributes = nil
        appearanceProxy.largeTitleTextAttributes = nil
        
        UIBarMetrics.allCases.forEach { metrics in
            appearanceProxy.setTitleVerticalPositionAdjustment(0, for: metrics)
        }
        
        appearanceProxy.tintColor = nil
        appearanceProxy.backIndicatorImage = nil
        appearanceProxy.backIndicatorTransitionMaskImage = nil
        appearanceProxy.barTintColor = nil
        
        UIBarMetrics.allCases.forEach { metrics in
            appearanceProxy.setBackgroundImage(nil, for: metrics)
        }
        
        UIBarPosition.casesForNavigationBarBackgroundImage.forEach { position in
            UIBarMetrics.allCases.forEach { metrics in
                appearanceProxy.setBackgroundImage(nil,
                                                   for: position,
                                                   barMetrics: metrics)
            }
        }
        
        appearanceProxy.shadowImage = nil
        
        // The block that we return will restore the customizations that were
        // on the appearance proxy when this method first started.
        return {
            appearanceProxy.barStyle = barStyle
            appearanceProxy.titleTextAttributes = titleTextAttributes
            appearanceProxy.largeTitleTextAttributes = largeTitleTextAttributes
            
            titleVerticalPositionAdjustments.forEach { (metrics, value) in
                if value != 0 {
                    appearanceProxy.setTitleVerticalPositionAdjustment(value,
                                                                       for: metrics)
                }
            }
            
            appearanceProxy.tintColor = tintColor
            appearanceProxy.backIndicatorImage = backIndicatorImage
            appearanceProxy.backIndicatorTransitionMaskImage = backIndicatorTransitionMaskImage
            appearanceProxy.barTintColor = barTintColor
            
            backgroundImages.forEach { (metrics, image) in
                appearanceProxy.setBackgroundImage(image, for: metrics)
            }
            
            backgroundImagesForBarPosition.forEach { (position, images) in
                images.forEach { (metrics, image) in
                    appearanceProxy.setBackgroundImage(image,
                                                       for: position,
                                                       barMetrics: metrics)
                }
            }
            
            appearanceProxy.shadowImage = shadowImage
        }
    }
    
}

Now, simply call this method, present your system UI, and then restore your original style when it’s done—and your system UI will look as it’s supposed to:

The system UI now appears as designed when displayed over our custom UI.
The system UI now appears as designed when displayed over our custom UI.

Obviously, this is a lot of code, but as there’s no built-in way to reset to a standard UIAppearance, this is the best I could do for now. You’d have to write one of these for every UIKit class that’s giving you problems in system UI, but this approach lets you have your UI customization cake and eat it too.

Fixing Simulator Status Bars for App Store Screenshots With Xcode 11 and iOS 13

Today Apple released the Xcode 11 GM release, meaning that it’s now time to create our builds using the iOS 13 SDK and submit to the App Store! If you’re like me and you use Fastlane’s Snapshot tool to automate the creation of your App Store screenshots using Xcode’s UI testing infrastructure, then you may have noticed something that broke with this new SDK: using SimulatorStatusMagic to clean up the simulator’s status bar. I use it on Landmarked, a side project I have at Detroit Labs, to make the App Store screenshots just a little nicer:

A screenshot of Landmarked

As you can tell, the status bar doesn’t say “Carrier,” the battery is full (important if you’re developing on a laptop), and the time has been set. In this case, since Landmarked is about Detroit, I set the time to 3:13 PM, but you may want to set it to 9:41 AM as per tradition. Doing this with SimulatorStatusMagic is easy:

#if targetEnvironment(simulator)
SDStatusBarManager.sharedInstance().bluetoothState = .hidden
SDStatusBarManager.sharedInstance().batteryDetailEnabled = false
SDStatusBarManager.sharedInstance().timeString = "3:13 PM"
SDStatusBarManager.sharedInstance().enableOverrides()
#endif

I use the #if targetEnvironment(simulator) bit to ensure that this doesn’t run if you’re running the UI tests on a real device, as it isn’t supported. Up until now, this has worked great. Of course, you should clean this up when you’re done—otherwise, these overrides persist across app launches on the running simulator. The above code runs in the test case’s setUp() method, and in the tearDown() method, you can clean up your work:

override func tearDown() {
    super.tearDown()
    #if targetEnvironment(simulator)
    SDStatusBarManager.sharedInstance().disableOverrides()
    #endif
}

So how do we do this on Xcode 11? With SimulatorStatusMagic not working, the first step is to avoid running it on iOS 13:

override func setUp() {
    #if targetEnvironment(simulator)
    if #available(iOS 13.0, *) {
        
    }
    else {
        SDStatusBarManager.sharedInstance().bluetoothState = .hidden
        SDStatusBarManager.sharedInstance().batteryDetailEnabled = false
        SDStatusBarManager.sharedInstance().timeString = "3:13 PM"
        SDStatusBarManager.sharedInstance().enableOverrides()
    }
    #endif
}
override func tearDown() {
    super.tearDown()

    #if targetEnvironment(simulator)
    if #available(iOS 13.0, *) {
        
    }
    else {
        SDStatusBarManager.sharedInstance().disableOverrides()
    }
    #endif
}

Looking at the release notes for Xcode 11, there’s a new simctl command to control the simulator status bar, aptly named status_bar. At the command line, to replicate the above behavior, we’d call it like this:

xcrun simctl status_bar booted override \
    --time "3:13 PM" \
    --dataNetwork wifi \
    --wifiMode active \
    --wifiBars 3 \
    --cellularMode notSupported \
    --batteryState discharging \
    --batteryLevel 100

But if you want to run this automatically, where to put it? You can’t call out to Process on iOS, but you need to call something on your Mac to run this command. Fortunately, Xcode has a way to do this by adding a pre-action to your test scheme:

A screenshot of Xcode editing a scheme’s pre-run actions

And of course, just as before, you can use a post-run action with the clear command to reset everything:

A screenshot of Xcode editing a scheme’s post-run actions

It works! Huzzah! Of course, there are still some things we can do better. If you run these UI tests on a device, and you have a simulator open, then that simulator will get these overrides while it’s running thanks to the booted parameter passed to simctl. Also, this approach doesn’t work when automating with Fastlane. To avoid that, we can use the TARGET_DEVICE_IDENTIFIER parameter to specifically boot and configure the device that the tests will run on (if the device is not booted, then simctl will fail to set the overrides). In your pre-action, make sure you’re inheriting build settings from the UI testing target, then add this snippet:

xcrun simctl boot "${TARGET_DEVICE_IDENTIFIER}"

xcrun simctl status_bar "${TARGET_DEVICE_IDENTIFIER}" override \
    --time "3:13 PM" \
    --dataNetwork wifi \
    --wifiMode active \
    --wifiBars 3 \
    --cellularMode notSupported \
    --batteryState discharging \
    --batteryLevel 100

With that, we’re successfully configuring our status bar on the simulator when it runs. Of course, the value of TARGET_DEVICE_IDENTIFIER won’t make sense to simctl if the destination is a device. Let’s add a quick check that we’re running on a simulator, and if not, exit early:

if [[ "${SDKROOT}" != *"simulator"* ]]; then
    exit 0
fi

Perfect! The value of SDKROOT is something like iphonesimulator13.0 when running on a simulator and iphoneos13 when running on a device, so this if statement will call exit when the SDK isn’t a simulator. Next up, we want to avoid running this code when the destination isn’t running at least iOS 13, so to do that, we’ll need to check the target device’s iOS version. Luckily for me, that code was readily available on StackExchange. The final version of our script looks like this:

function version {
    echo "$@" | awk -F. '{ printf("%d%03d%03d%03d\n", $1,$2,$3,$4); }';
}

# Don’t run on iOS devices.
if [[ "${SDKROOT}" != *"simulator"* ]]; then
    exit 0
fi

# Don’t run on iOS versions before 13.
if [ $(version "${TARGET_DEVICE_OS_VERSION}") -ge $(version "13") ]; then
    xcrun simctl boot "${TARGET_DEVICE_IDENTIFIER}"

    xcrun simctl status_bar "${TARGET_DEVICE_IDENTIFIER}" override \
        --time "3:13 PM" \
        --dataNetwork wifi \
        --wifiMode active \
        --wifiBars 3 \
        --cellularMode notSupported \
        --batteryState discharging \
        --batteryLevel 100
fi

Now, if you run your UI tests on an iOS simulator target running anything earlier than iOS 13, the pre-run action will exit early, and SimulatorStatusMagic will take care of it. This is important for our purposes as Landmarked still supports iOS 9 and the iPhone 4S’s 3.5″ screen, so we need to generate a screenshot for that size to be complete.

With this, our pre-run script is complete! The post-run is much simpler:

function version {
    echo "$@" | awk -F. '{ printf("%d%03d%03d%03d\n", $1,$2,$3,$4); }';
}

# Don’t run on iOS devices.
if [[ "${SDKROOT}" != *"simulator"* ]]; then
    exit 0
fi

# Don’t run on iOS versions before 13.
if [ $(version "${TARGET_DEVICE_OS_VERSION}") -ge $(version "13") ]; then
    xcrun simctl boot "${TARGET_DEVICE_IDENTIFIER}"
    xcrun simctl status_bar "${TARGET_DEVICE_IDENTIFIER}" clear
fi

I was able to use the above code to generate all of the screenshots for the iOS 13 version of Landmarked. Keep an eye out for version 1.1.3 as soon as iOS 13 is released! Here’s what it looks like on the new iPhone 11 Pro Max:

A screenshot of Landmarked on an iPhone 11 Pro Max.