NSUserActivity with SwiftUI

When I was researching NSUserActivity in order to expand the content of the Companion for SwiftUI app, I found that there is a lot of outdated information and confusing examples out there. For instance, most articles about Handoff are from the time the feature was added. But when that happened, scenes did not exist yet. Everything was handled by the application delegate. When scenes appeared later, a lot of the code moved to the scene delegate and all those handoff examples just stopped working properly. When you are new to NSUserActivity, that is very confusing. And now that SwiftUI also supports user activities and the scene delegates are gone, there’s even more change. A good time then, for a new article.

Another aspect of NSUserActivity that makes it a little confusing, is the fact that it is an object that can be used for multiple but unrelated functions. Many of its properties are only relevant in some cases, but not in others.

A summary of some of the things that NSUserActivity is involved with:

• Universal Links: Universal links are URLs that open in the associated app (if installed), or in Safari otherwise.
• SiriKit: Siri can launch your app and tell it what it wants to do.
• Spotlight: Define actions that your app can do, so they are included in Spotlight search results.
• Handoff: An application can continue the work of another application (or the same application from a different device).

This article presents a series of examples, that will progressively introduce the methods SwiftUI offers to deal with NSUserActivity. There is one example for each of the cases described above.

Important Notice

The SwiftUI methods involved with NSUserActivity are onOpenURL(), userActivity(), onContinueUserActivity() and handlesExternalEvents(). Note that these methods can only be used successfully if your app was created with the SwiftUI Lifecycle. If your project still uses a scene delegate, the inclusion of these methods will produce the following console message:

Cannot use Scene methods for URL, NSUserActivity, and other External Events without using SwiftUI Lifecycle. Without SwiftUI Lifecycle, advertising and handling External Events wastes resources, and will have unpredictable results.

In my experience, the aforementioned unpredictable results, are very predictable: all methods are ignored.

The Two Faces of a User Activity

According to Apple’s documentation, a user activity object represents the state of an app at a moment in time:

An NSUserActivity object provides a lightweight way to capture the state of your app and put it to use later. You create user activity objects and use them to capture information about what the user was doing, such as viewing app content, editing a document, viewing a web page, or watching a video. When the system launches your app and an activity object is available, your app can use the information in that object to restore itself to an appropriate state.

With this in mind, we can distinguish two key moments in a user activity: At some point, user activities are created (we’ll see when and how later). At some other time, the system may decide to launch (or resume) an app, providing it with an NSUserActivity, so the application can show the relevant UI. We’ll also learn how to react to these user activities in the app.

Note that when an application can have multiple scenes, only one will get the user activity. We will see how that is determined… although, spoiler alert, at the time of this writing (Xcode 12, beta 6), that part of SwiftUI seems broken.

---

Universal Links

Introducing onOpenURL()

Universal links are useful to integrate your app with your website. Setting up universal links requires several steps that are very well documented by Apple here: Universal Links.

Of all the NSUserActivity uses, this is the easiest to implement with SwiftUI. Although universal links use an NSUserActivity to launch or resume your app, you will never get to see it if you are using the SwiftUI LifeCycle!

With UIKit, you would normally use the scene delegate:

func scene(_ scene: UIScene, continue userActivity: NSUserActivity) {
    if userActivity.activityType == NSUserActivityTypeBrowsingWeb  {
        doSomethingWith(url: userActivity.webpageURL)
    }
}

However, because now we no longer have a scene delegate, we can simply use the onOpenURL method. Now, instead of getting an NSUserActivity object, we will get the URL:

struct ContentView: View {
    var body: some View {
        SomeView()
            .onOpenURL { url in
                doSomethingWith(url: url)
            }
    }
    
    func doSomethingWith(url: URL?) {
        ...
    }
}

---

SiriKit

Introducing onContinueUserActivity()

Our application can define shortcuts to specific parts of our app. In iOS, these may be created with the Shortcuts app, but we can also create them from inside the application. UIKit has some UI elements to do so, but since these are not directly available in SwiftUI, the example in this section includes a UIViewControllerRepresentable. Its purpose is to place a button that opens the system modal that lets the user create (or edit) the shortcut.

Once the shortcut is created, when Siri is instructed to execute it, it will launch (or resume) our app and will provide an NSUserActivity with the details of the shortcut it wants us to execute. The SwiftUI method we use to provide a callback for that event is onContinueUserActivity()

In the example below, say “Hey Siri, show random animal” (or any other configured phrase). The system will launch our app and will navigate into a random animal view.

import SwiftUI
import Intents

// Remember to add this to the NSUserActivityTypes array in the Info.plist file
let aType = "com.example.show-animal"

struct Animal: Identifiable {
    let id: Int
    let name: String
    let image: String
}

let animals = [Animal(id: 1, name: "Lion", image: "lion"),
               Animal(id: 2, name: "Fox", image: "fox"),
               Animal(id: 3, name: "Panda", image: "panda-bear"),
               Animal(id: 4, name: "Elephant", image: "elephant")]

struct ContentView: View {
    @State private var selection: Int? = nil
    
    var body: some View {
        NavigationView {
            List(animals) { animal in
                NavigationLink(
                    destination: AnimalDetail(animal: animal),
                    tag: animal.id,
                    selection: $selection,
                    label: { AnimalRow(animal: animal) })
            }
            .navigationTitle("Animal Gallery")
            .onContinueUserActivity(aType, perform: { userActivity in
                self.selection = Int.random(in: 0...(animals.count - 1))
            })
            
        }.navigationViewStyle(StackNavigationViewStyle())
    }
}

struct AnimalRow: View {
    let animal: Animal
    
    var body: some View {
        HStack {
            Image(animal.image)
                .resizable()
                .frame(width: 60, height: 60)

            Text(animal.name)
        }
    }
}

struct AnimalDetail: View {
    @State private var showAddToSiri: Bool = false
    let animal: Animal
    
    let shortcut: INShortcut = {
        let activity = NSUserActivity(activityType: aType)
        activity.title = "Display a random animal"
        activity.suggestedInvocationPhrase = "Show Random Animal"

        return INShortcut(userActivity: activity)
    }()
    
    var body: some View {
        VStack(spacing: 20) {
            Text(animal.name)
                .font(.title)

            Image(animal.image)
                .resizable()
                .scaledToFit()
            
            SiriButton(shortcut: shortcut).frame(height: 34)

            Spacer()
        }
    }
}

The UIViewControllerRepresentable for the shortcut creation and edit modal:

import SwiftUI
import IntentsUI

struct SiriButton: UIViewControllerRepresentable {
    public let shortcut: INShortcut
    
    func makeUIViewController(context: Context) -> SiriUIViewController {
        return SiriUIViewController(shortcut: shortcut)
    }
    
    func updateUIViewController(_ uiViewController: SiriUIViewController, context: Context) {
    }
}

class SiriUIViewController: UIViewController {
    let shortcut: INShortcut
    
    init(shortcut: INShortcut) {
        self.shortcut = shortcut
        super.init(nibName: nil, bundle: nil)
    }
    
    required init?(coder: NSCoder) {
        fatalError("init(coder:) has not been implemented")
    }
    
    override func viewDidLoad() {
        super.viewDidLoad()
        
        let button = INUIAddVoiceShortcutButton(style: .blackOutline)
        button.shortcut = shortcut
        
        self.view.addSubview(button)
        view.centerXAnchor.constraint(equalTo: button.centerXAnchor).isActive = true
        view.centerYAnchor.constraint(equalTo: button.centerYAnchor).isActive = true
        button.translatesAutoresizingMaskIntoConstraints = false

        button.delegate = self
    }
}

extension SiriUIViewController: INUIAddVoiceShortcutButtonDelegate {
    func present(_ addVoiceShortcutViewController: INUIAddVoiceShortcutViewController, for addVoiceShortcutButton: INUIAddVoiceShortcutButton) {
        addVoiceShortcutViewController.delegate = self
        addVoiceShortcutViewController.modalPresentationStyle = .formSheet
        present(addVoiceShortcutViewController, animated: true)
    }
    
    func present(_ editVoiceShortcutViewController: INUIEditVoiceShortcutViewController, for addVoiceShortcutButton: INUIAddVoiceShortcutButton) {
        editVoiceShortcutViewController.delegate = self
        editVoiceShortcutViewController.modalPresentationStyle = .formSheet
        present(editVoiceShortcutViewController, animated: true)
    }
}

extension SiriUIViewController: INUIAddVoiceShortcutViewControllerDelegate {
    func addVoiceShortcutViewController(_ controller: INUIAddVoiceShortcutViewController, didFinishWith voiceShortcut: INVoiceShortcut?, error: Error?) {
        controller.dismiss(animated: true)
    }

    func addVoiceShortcutViewControllerDidCancel(_ controller: INUIAddVoiceShortcutViewController) {
        controller.dismiss(animated: true)
    }
}

extension SiriUIViewController: INUIEditVoiceShortcutViewControllerDelegate {
    func editVoiceShortcutViewController(_ controller: INUIEditVoiceShortcutViewController, didUpdate voiceShortcut: INVoiceShortcut?, error: Error?) {
        controller.dismiss(animated: true)
    }

    func editVoiceShortcutViewController(_ controller: INUIEditVoiceShortcutViewController, didDeleteVoiceShortcutWithIdentifier deletedVoiceShortcutIdentifier: UUID) {
        controller.dismiss(animated: true)
    }

    func editVoiceShortcutViewControllerDidCancel(_ controller: INUIEditVoiceShortcutViewController) {
        controller.dismiss(animated: true)
    }
}

---

Spotlight

Introducing userActivity()

Spotlight search results can include common activities from your app. To make Spotlight learn about them, you need to advertise these activities when they occur. This will make Spotlight aware of them. To advertise NSUserActivities with SwiftUI, we use the userActivity() modifier.

In the example below, we have an app that sells ice creams. Whenever we select one of the ice cream sizes, the app updates the advertised size. Whenever the user searches for ice creams, a result for our app will be shown. If the user selects it, the app will be launched and take the user to the last ice cream size advertised.

Note that the system optimizes the times at which it calls the closure from userActivity(). Unfortunately, that is not documented. The system tries to be clever enough to keep the information current, without doing constant updates. When debugging, it is a good idea to use a print statement in the userActivity closure.

The example also includes a “Forget” button, useful for debugging. It clears the advertised user activity to remove the app from search results. Note that NSUserActivity has an optional property expirationDate. If left as nil, the activity does not expire.

import SwiftUI
import Intents
import CoreSpotlight
import CoreServices

// Remember to add this to the NSUserActivityTypes array in the Info.plist file
let aType = "com.example.icecream-selection"

struct IceCreamSize: Identifiable {
    let id: Int
    let name: String
    let price: Float
    let image: String
}

let sizes = [
    IceCreamSize(id: 1, name: "Small", price: 1.0, image: "small"),
    IceCreamSize(id: 2, name: "Medium", price: 1.45, image: "medium"),
    IceCreamSize(id: 3, name: "Large", price: 1.9, image: "large")
]

struct ContentView: View {
    @State private var selection: Int? = nil
    
    var body: some View {
        NavigationView {
            List(sizes) { size in
                NavigationLink(destination: IceCreamDetail(icecream: size),
                               tag: size.id,
                               selection: $selection,
                               label: { IceCreamRow(icecream: size) })
            }
            .navigationTitle("Ice Creams")
            .toolbar {
                Button("Forget") {
                    NSUserActivity.deleteAllSavedUserActivities {
                        print("done!")
                    }
                }
            }
            
        }
        .onContinueUserActivity(aType, perform: { userActivity in
            if let icecreamId = userActivity.userInfo?["sizeId"] as? NSNumber {
                selection = icecreamId.intValue

            }
        })
        .navigationViewStyle(StackNavigationViewStyle())
    }
}

struct IceCreamRow: View {
    let icecream: IceCreamSize
    
    var body: some View {
        HStack {
            Image(icecream.image)
                .resizable()
                .frame(width: 80, height: 80)
            
            VStack(alignment: .leading) {
                Text("\(icecream.name)").font(.title).fontWeight(.bold)
                Text("$ \(String(format: "%0.2f", icecream.price))").font(.subheadline)
                Spacer()
            }
        }
    }
}

struct IceCreamDetail: View {
    let icecream: IceCreamSize
    
    var body: some View {
        VStack {
            Text("\(icecream.name)").font(.title).fontWeight(.bold)
            Text("$ \(String(format: "%0.2f", icecream.price))").font(.subheadline)

            Image(icecream.image)
                .resizable()
                .scaledToFit()
            
            Spacer()
        }
        .userActivity(aType) { userActivity in
            userActivity.isEligibleForSearch = true
            userActivity.title = "\(icecream.name) Ice Cream"
            userActivity.userInfo = ["sizeId": icecream.id]
            
            let attributes = CSSearchableItemAttributeSet(itemContentType: kUTTypeItem as String)
            
            attributes.contentDescription = "Get a delicious ice cream now!"
            attributes.thumbnailData = UIImage(named: icecream.image)?.pngData()
            userActivity.contentAttributeSet = attributes
            
            print("Advertising: \(icecream.name)")
        }
    }
}

---

Handoff

With the methods introduced so far, we can already create a Handoff application. That is an application that can resume work on another device. It can be a single app or a different app. This is usually the case when you distribute two versions of your application: one for iOS and another for macOS.

Note that for Handoff to work, all apps involved need to be signed with the same developer team identifier. Remember to also define your activity type(s) in the NSUserActivityTypes entry of the Info.plist of all participating apps.

More details on Handoff implementation is available in Apple’s website.

The example below implements a simple web browser. The example calls userActivity() to advertise the page the user is viewing and the current scroll position of the page.

If the user switches to another device, the onContinueUserActivity() closure will be called when the app is launched (or resumed). The app can then open the page and scroll to the position the user was at in the other device.

User activities can include payload data, in the form of a userInfo dictionary. This is the place where we store all the specific information about our handoff activity. In this example: scroll position (as a percentage) and the URL of the opened page. The example also includes the bundle id of the app that advertised the activity. This is only for debugging purposes, so we know exactly what is going on.

Also, note that the code of this example works both on iOS and macOS. This will let you build two apps, so you can test handoff between an iOS device and a Mac.

Finally, although not related to NSUserActivity, this example wraps a WKWebView with a Representable view. It is a nice example that showcases how a javascript event (in this case onScroll) can update a binding of your SwiftUI view. The full WebView code is available in the following gist file: WebView.swift

import SwiftUI

// Remember to add this to the NSUserActivityTypes array in the Info.plist file
let activityType = "com.example.openpage"

struct ContentView: View {
    @StateObject var data = WebViewData()
    
    @State private var reload: Bool = false
    
    var body: some View {
        VStack {
            HStack {
                TextField("", text: $data.urlBar, onCommit: { self.loadUrl(data.urlBar) })
                    .textFieldStyle(RoundedBorderTextFieldStyle())
                    .disableAutocorrection(true)
                    .modifier(KeyboardModifier())
                    .frame(maxWidth: .infinity)
                    .overlay(ProgressView().opacity(self.data.loading ? 1 : 0).scaleEffect(0.5), alignment: .trailing)
                
                
                Button(action: {
                    self.data.scrollOnLoad = self.data.scrollPercent
                    self.reload.toggle()
                }, label: { Image(systemName: "arrow.clockwise") })
                
                Button("Go") {
                    self.loadUrl(data.urlBar)
                }
            }
            .padding(.horizontal, 4)

            Text("\(data.scrollPercent)")
            
            WebView(data: data)
                .id(reload)
                .onAppear { loadUrl(data.urlBar) }
        }
        .userActivity(activityType, element: data.url) { url, activity in
            
            let bundleid = Bundle.main.bundleIdentifier ?? ""
            
            activity.addUserInfoEntries(from: ["scrollPercent": data.scrollPercent,
                                               "page": data.url?.absoluteString ?? "",
                                               "setby": bundleid])
            
            logUserActivity(activity, label: "activity")
        }
        .onContinueUserActivity(activityType, perform: { userActivity in
            if let page = userActivity.userInfo?["page"] as? String {
                // Load handoff page
                if self.data.url?.absoluteString != page {
                    self.data.url = URL(string: page)
                }
                
                // Restore handoff scroll position
                if let scrollPercent = userActivity.userInfo?["scrollPercent"] as? Float {
                    self.data.scrollOnLoad = scrollPercent
                }
            }
            
            logUserActivity(userActivity, label: "on activity")
        })
    }
    
    func loadUrl(_ string: String) {
        if string.hasPrefix("http") {
            self.data.url = URL(string: string)
        } else {
            self.data.url = URL(string: "https://" + string)
        }
        
        self.data.urlBar = self.data.url?.absoluteString ?? string
    }
}

func logUserActivity(_ activity: NSUserActivity, label: String = "") {
    print("\(label) TYPE = \(activity.activityType)")
    print("\(label) INFO = \(activity.userInfo ?? [:])")
}

struct KeyboardModifier: ViewModifier {
    func body(content: Content) -> some View {
        #if os(iOS)
            return content
                .keyboardType(.URL)
                .textContentType(.URL)
        #else
            return content
        #endif
    }
}

---

Scene Selection

Introducing handlesExternalEvents()

When the system launches or resumes our app, it must decide which scene gets the user activity (only one will). To help it decide, our app can use the handlesExternalEvents() method. Unfortunately, at the time of this writing (Xcode 12, beta 6), it seems the method is broken. Not only that, although supported on macOS, it is missing from that platform’s definition file.

For the time being, I will just comment on how it is supposed to work. In the future, when it can be successfully used, I will update this post.

There are two versions of this method. One that can be used with WindowGroup scenes:

func handlesExternalEvents(matching conditions: Set<String>) -> some Scene

and another that can be used with views:

func handlesExternalEvents(preferring: Set<String>, allowing: Set<String>) -> some View

In both cases, we specify a Set of strings that the system will compare with the targetContentIdentifier property of the NSUserActivity. If a match is found, that scene is used. If not specified, or no match is found, the behavior is platform dependent. For example, on iPadOS the system will choose an existing scene. On macOS, a new scene will be created.

On systems that only support one scene, this method is ignored.

Note that targetContentIdentifier is also present in UNNotificationContent and UIApplicationShortcutItem, so handlesExternalEvents() will most likely support those events too.

Summary

Apple’s documentation of NSUserActivity is very extensive, and I recommend you check it out. At the moment, it’s lacking in SwiftUI examples. This article aims to fill that void, by providing you with some kickstart code to begin experimenting with NSUserActivity.

Please feel free to comment below, and follow me on twitter if you would like to be notified when new articles come out. Until next time!