What if instead of a CLI, Plugins? Part 2, Start a Build Plugin

January 24, 2024

This article is part of a series.

Intro

To write these plugins I watched the WWDC22 Meet and Create videos for Package Plugins. With respect to build plugins, they showed auto generating code based on data and generating code for image assets respectively. I combined the two to make a fictional FruitStore generator.

The final build script, a version of which can be seen already in the PluginExplorer repo, uses package excluded text files to generate structs and a data store Dictionary that the rest of the code can refer to.

As the name implies, build plugins have to be added to something that builds, so PluginExplorer has its own CLI. In this walkthrough the build plugin is on its own and will be added to both a demo CLI package and a default Xcode project.

This post goes from swift package init --type build-tool-plugin to having a build plugin that successfully looks at text files and creates a .swift file of the same name in the .build folder. The rest will be in Part II of Part 2 - which will just be Part 3.

Getting Started

Like with the command plugin, command line tools have been thoughtfully provided to help kick off a new project.

mkdir $MYPLUGINNAME
cd $MYPLUGINNAME
swift package init --type build-tool-plugin
touch README.md
# git start-repo if available (see part 1) 
git init .
git add .
git commit -m "Initialize repository"

This will create a directory structure that looks like

└── $MYPLUGINNAME
    └── Plugins
    │   └── $MYPLUGINNAME.swift
    └── Package.swift
    └── .git
    └── .gitignore

Again, the default Package.swift will recognize the below layout as well.

└── $MYPLUGINNAME
    └── Plugins
    │   └── $MYPLUGINNAME
    |       └── plugin.swift
    └── Package.swift
    └── .git
    └── .gitignore

The Default Package.swift

// swift-tools-version: 5.9
// The swift-tools-version declares the minimum version of Swift required to build this package.

import PackageDescription

let package = Package(
    name: "$MYPLUGINNAME",
    products: [
        // Products can be used to vend plugins, making them visible to other packages.
        .plugin(
            name: "$MYPLUGINNAME",
            targets: ["$MYPLUGINNAME"]),
    ],
    targets: [
        // Targets are the basic building blocks of a package, defining a module or a test suite.
        // Targets can depend on other targets in this package and products from dependencies.
        .plugin(
            name: "$MYPLUGINNAME",
            capability: .buildTool()
        ),
    ]
)

The Default plugin.swift / $MYPLUGINNAME.swift

This time the struct will conform to BuildToolPlugin. Again, like with the command plugin, there will be a section for package code and project code.

import PackagePlugin

@main
struct $MYPLUGINNAME: BuildToolPlugin {
    /// Entry point for creating build commands for targets in Swift packages.
    func createBuildCommands(context: PluginContext, target: Target) async throws -> [Command] {
        // This plugin only runs for package targets that can have source files.
        guard let sourceFiles = target.sourceModule?.sourceFiles else { return [] }

        // Find the code generator tool to run (replace this with the actual one).
        let generatorTool = try context.tool(named: "my-code-generator")

        // Construct a build command for each source file with a particular suffix.
        return sourceFiles.map(\.path).compactMap {
            createBuildCommand(for: $0, in: context.pluginWorkDirectory, with: generatorTool.path)
        }
    }
}

#if canImport(XcodeProjectPlugin)
import XcodeProjectPlugin

extension $MYPLUGINNAME: XcodeBuildToolPlugin {
    // Entry point for creating build commands for targets in Xcode projects.
    func createBuildCommands(context: XcodePluginContext, target: XcodeTarget) throws -> [Command] {
        // Find the code generator tool to run (replace this with the actual one).
        let generatorTool = try context.tool(named: "my-code-generator")

        // Construct a build command for each source file with a particular suffix.
        return target.inputFiles.map(\.path).compactMap {
            createBuildCommand(for: $0, in: context.pluginWorkDirectory, with: generatorTool.path)
        }
    }
}

#endif

extension $MYPLUGINNAME {
    /// Shared function that returns a configured build command if the input files is one that should be processed.
    func createBuildCommand(for inputPath: Path, in outputDirectoryPath: Path, with generatorToolPath: Path) -> Command? {
        // Skip any file that doesn't have the extension we're looking for (replace this with the actual one).
        guard inputPath.extension == "my-input-suffix" else { return .none }
        
        // Return a command that will run during the build to generate the output file.
        let inputName = inputPath.lastComponent
        let outputName = inputPath.stem + ".swift"
        let outputPath = outputDirectoryPath.appending(outputName)
        return .buildCommand(
            displayName: "Generating \(outputName) from \(inputName)",
            executable: generatorToolPath,
            arguments: ["\(inputPath)", "-o", "\(outputPath)"],
            inputFiles: [inputPath],
            outputFiles: [outputPath]
        )
    }
}

Make Test Package

My build plugin won’t work on just any old project. It will only work on projects with the right files. So lets make one. I’m going to assume general comfort around CLI projects, but here’s some getting started help.

mkdir $DEMOPACKAGE
cd $DEMOPACKAGE
swift package init --type tool
swift run

Change contents of Sources/$DEMOPACKAGE.swift to

//
//  $DEMOPACKAGE.swift
//
//
//  Created by Carlyn Maw on 1/17/24.
//

import ArgumentParser

@main
struct $DEMOPACKAGE: AsyncParsableCommand {
    static let configuration = CommandConfiguration(
        abstract: "For Testing The Plugins", 
        version: "0.0.0", 
        subcommands: [hello.self, fruit_list.self], 
        defaultSubcommand: hello.self)
    
    struct hello: ParsableCommand {
        mutating func run() throws {
            print("Hello, world!")
        }
    }
}

Switching up the structure will allow us to make different tests as separate parsable commands as needed.

Update the Package.swift to Include the Build Plugin

Add the build plugin to the plugins parameter of the executable target. NOT the dependencies. Doing otherwise can cause linker errors.

In this specific case the package containing the plugin and and the plugin itself have the same name and the folder containing it may or may not.

As a general rule, the name of the package and the folder enclosing it should be the same. The name parameter is more of a displayName and in many places the code looks to the enclosing folder to get its identity. When using URLs to include projects this becomes all the more key. Keep this in mind when designing your own packages.

// swift-tools-version: 5.9
// The swift-tools-version declares the minimum version of Swift required to build this package.

import PackageDescription

let package = Package(
    name: "$DEMOPACKAGE",
    dependencies: [
        .package(url: "https://github.com/apple/swift-argument-parser.git", from: "1.2.0"),
        .package(name: "$MYPLUGINNAME", path: "../PATH/TO/PLUGIN/FOLDER/INCLUSIVE")
    ],
    targets: [
        .executableTarget(
            name: "DemoFruitStore",
            dependencies: [
                .product(name: "ArgumentParser", package: "swift-argument-parser")
                //NOT HERE !!! .product(name: , package: "FruitStoreBuild")
            ],
            plugins: [
                //HERE!!
                .plugin(name: "FruitStoreBuild", package: "FruitStoreBuild")
            ] 
        ),
    ]
)

Building a Custom Tool

If you run “$DEMOPACKAGE” now (swift run in the root folder should still work) hopefully you’ll get the following error:

error: Plugin does not have access to a tool named ‘my-code-generator’
error: build stopped due to build-tool plugin failures
error: Plugin does not have access to a tool named ‘my-code-generator’
error: build stopped due to build-tool plugin failures

The lines in the way of a successful run are:

line 11: let generatorTool = try context.tool(named: "my-code-generator")
line 27: let generatorTool = try context.tool(named: "my-code-generator") again
line 42: guard inputPath.extension == "my-input-suffix" else { return .none }
line 51: arguments: ["\(inputPath)", "-o", "\(outputPath)"],

11, 27 and 51 all go hand in hand. Unlike with the command plugin example we’re not going to import an external tool, we’re going to build it.

Make the Tool

From the root directory of the $MYPLUGINNAME project folder:

mkdir Sources
mkdir Sources/MyPluginNameTool
touch Sources/MyPluginNameTool/main.swift

Update Package.swift

// swift-tools-version: 5.9
// The swift-tools-version declares the minimum version of Swift required to build this package.

import PackageDescription

let package = Package(
    name: "$MYPLUGINNAME",
    products: [
        // Products can be used to vend plugins, making them visible to other packages.
        .plugin(
            name: "$MYPLUGINNAME",
            targets: ["$MYPLUGINNAME"]),
    ],
    targets: [
        // Targets are the basic building blocks of a package, defining a module or a test suite.
        // Targets can depend on other targets in this package and products from dependencies.
        .executableTarget(
            name: "my-code-generator",
            path: "Sources/MyPluginNameTool"
        ),
        .plugin(
            name: "$MYPLUGINNAME",
            capability: .buildTool(),
            dependencies: ["my-code-generator"]
        ),
    ]
)

Add content to main.swift

Add code to main.swift that will compliment the arguments from line 52.

import Foundation

let arguments = ProcessInfo().arguments
if arguments.count < 4 {              //<== Number of arguments sample code will send. 
    print("missing arguments")
}

print("ARGUMENTS")

arguments.forEach {
    print($0)
}

Update plugin.swift

Change line 42 from

guard inputPath.extension == "my-input-suffix" else { return .none }

guard inputPath.extension == "swift" else { return .none }

Check the Tool

Run “$DEMOPACKAGE”. You may have to clear the .build folder, but eventually you should get the following print out

Building for debugging...
ARGUMENTS
{$PATH_TO_REPO}/DemoPackage/.build/plugins/tools/debug/my-code-generator
{$PATH_TO_REPO}/DemoPackage/Sources/DemoPackage.swift
-o
{$PATH_TO_REPO}/DemoPackage/.build/plugins/outputs/DemoPackage/DemoPackage/MyPluginName/DemoPackage.swift
error: filename "DemoPackage.swift" used twice: '{$PATH_TO_REPO}/DemoPackage/Sources/DemoPackage.swift' and '{$PATH_TO_REPO}/DemoPackage/.build/plugins/outputs/DemoPackage/DemoPackage/MyPluginName/DemoPackage.swift'
note: filenames are used to distinguish private declarations with the same name

Lets break them down. First we get the print out of the arguments:

0: the tool itself
1: the path to the input file
2: a flag “-o” typically used to indicate what follows is an output file path that should overwrite the defaults.
3: that output path.

Then we get a warning about duplicate files??? What duplicate files?? I didn’t actually do anything?

The in-build plugin infrastructure keep tabs on the inputs and the outputs because in-build plugins only run when those inputs and outputs are stale. Even though we didn’t actually generate the output file the the Command generated in createBuildCommand told the infrastructure it means to, and the infrastructure thinks that’s a bad idea.

Print statements from the build command’s tool end up in the terminal. Print statements from within createBuildCommand DO NOT appear unless “very verbose” mode was chosen, i.e. swift run -vv. Messages to stderr instead of stdout do appear.

What Happens in Xcode?

It takes two steps to load a plugin into an Xcode project:

Adding the package to the project
Navigating to the specific Targetinfo panel and adding it to the build phases.

This process is visible in Meet Swift OpenAPI Generator at minute 8.

I made a default multi-platform app called PluginTesterApp. Here is a screenshot of adding the build plugin to the build phase of the app target.

Build Phase Screenshot

In Xcode the build will FAIL with one message per .swift file in the Target.

Multiple commands produce '/Users/MaybeYou/Library/Developer/Xcode/DerivedData/PluginTesterApp-alhhewcaqznpzedyonljgielnmmv/Build/Intermediates.noindex/PluginTesterApp.build/Debug-iphonesimulator/PluginTesterApp.build/Objects-normal/arm64/ContentView.stringsdata'

Multiple commands produce '/Users/Carlyn/Library/Developer/Xcode/DerivedData/PluginTesterApp-alhhewcaqznpzedyonljgielnmmv/Build/Intermediates.noindex/PluginTesterApp.build/Debug-iphonesimulator/PluginTesterApp.build/Objects-normal/arm64/PluginTesterAppApp.stringsdata'

More details will be visible in the “Report navigator”. There the print statements from within createBuildCommand will be visible, but unlike what happened with the package there will be no indication that the my-code-generator tool ran because Xcode halted everything when confronted with the conflict.

Fix the conflict

Add an apple.txt file to the PluginTesterApp target in PluginTesterApp.
change line 42 in plugin.swift to:
- guard inputPath.extension == "txt" else { return .none }
add print("PROOF OF PLUGIN LIFE from createBuildCommand") right below it.

If those changes were made an external editor, right click on the plugin package in the “Project navigator” and select “Update Package” to see the changes. If that doesn’t clear any problems up, Shift + Cmd + K to clean the build folder.

Xcode will fail to build again, but we can see from the “Report navigator” it isn’t because our script didn’t run this time. Both the print statements from the plugin and the plugin embedded tool are visible in the reports at but in different places. The plugin code runs at the beginning of the build. The tool will run when its appropriate to the build process.

Screenshot of Reports Navigator with a build report selected. It shows the print statements from main.swift

The errors now come from the fact that we promised our code there would be an apple.swift file somewhere and we didn’t actually make one. Every file when it gets complied complains:

Error opening input file '/Users/MaybeYou/Library/Developer/Xcode/DerivedData/PluginTesterApp-alhhewcaqznpzedyonljgielnmmv/SourcePackages/plugins/PluginTesterApp.output/PluginTesterApp/MyPluginName/apple.swift' (No such file or directory)

This file is supposed to be in the build folder of our enclosing App. The tool gets the path to the build folder from context.pluginWorkDirectory in plugin.swift (line 15 for the package and 31 for the project).

        return sourceFiles.map(\.path).compactMap {
            createBuildCommand(for: $0, in: context.pluginWorkDirectory, with: generatorTool.path)
        }

Lets make one.

Update main.swift, run project again.

Update main.swift to actually save that .swift file the Command promised. The contents of every Swift file made will be a single line comment for now.

import Foundation

let arguments = ProcessInfo().arguments
if arguments.count < 4 {
    print("missing arguments")
}

// print("ARGUMENTS")

// arguments.forEach {
//     print($0)
// }

let (input, output) = (arguments[1], arguments[3])

//Added for ease of scanning for our output.
print("FIIIIIIIIIIIIIINNNNNNNNNDDDMMMEEEEEEEEEEEEEEEEE")
print("from MyBuildPluginTool:", input)
print("from MyBuildPluginTool:", output)
var outputURL = URL(fileURLWithPath: output)

let contentsOfFile = "//nothing of importance"

try contentsOfFile.write(to: outputURL, atomically: true, encoding: .utf8)

Clean the build folder and run the project again. It should work without error this time.

Meanwhile back in the package…

add an apple.txt to the package (touch Sources/Data/banana.txt)
swift run again will show the output from the tool but not the plugin code
- `from MyBuildPluginTool:`` print statements will show
- "PROOF OF PLUGIN LIFE from createBuildCommand" will not unless running in –vv mode.

The result will look something like

warning: 'demopackage': found 1 file(s) which are unhandled; explicitly declare them as resources or exclude from the target
    {$PATH_TO_REPO}/DemoPackage/Sources/apple.txt
Building for debugging...
[3/3] Linking my-code-generator
Build complete! (1.29s)
Building for debugging...
FIIIIIIIIIIIIIINNNNNNNNNDDDMMMEEEEEEEEEEEEEEEEE
from MyBuildPluginTool: {$PATH_TO_REPO}/DemoPackage/Sources/apple.txt
from MyBuildPluginTool: {$PATH_TO_REPO}/DemoPackage/.build/plugins/outputs/demopackage/DemoPackage/MyPluginName/apple.swift
[5/5] Linking DemoPackage
Build complete! (2.68s)
Hello, world!

To check if it worked ls the build directory then cat the file:

ls /Users/.../.build/.../MyPluginName/
cat /Users/.../.build/.../MyPluginName/apple.swift

The easiest thing to do is to match up to Xcode’s behavior and call it a Resource. Using the .process descriptor means that the file will be examined and Swift Package Manger will make its best call about whether the file should be optimized or copied as is. If curious, the command plugin from the last post can confirm that Xcode is calling it a resource with a swap in of target.inputFiles.filter({$0.type == .resource}) in the Xcode project source files section.

Update Package.swift

        .executableTarget(
            name: "DemoFruitStore",
            dependencies: [
                .product(name: "ArgumentParser", package: "swift-argument-parser"),
            ],
            resources: [
                .process("apple.txt") //if a directory, `.process("DirName")` works fine.
            ],
            plugins: [
                .plugin(name: "MyPluginName", package: "MyPluginName")
            ] 
        ),

This will silence the warning and our code still works just fine.

HOWEVER this means we will now have a copy of this file in the bundle, which we did not have before.

To verify, from the same working directory as swift run try find .build/ -name "apple.txt" both before and after updating the package file. Do the same on the Xcode project build folder reveals it too hides an apple.txt.

Instead we want to EXCLUDE our text files since they aren’t used by anything other than the build process.

        .executableTarget(
            name: "DemoPackage",
            dependencies: [
                .product(name: "ArgumentParser", package: "swift-argument-parser"),
            ],
            exclude: [
                "apple.txt" //if a directory - `"DirName"` works fine. 
            ],
            plugins: [
                .plugin(name: "MyPluginName", package: "MyPluginName")
            ]
        ),

This time check find .build/ -name "apple.swift" There won’t be one. Our package version of the build plugin checks source code only, and we’ve just excluded “apple.txt” from being source code.

MYPLUGINNAME/plugin.swift line 8 has the relevant code.

        guard let sourceFiles = target.sourceModule?.sourceFiles else { return [] }

This line inside the createBuildCommands function ensures that the target module has source files and then hands them over.

Replace the createBuildCommands function for the package with:

    //Add `import Foundation` to the top of the file because of FileManger

    func createBuildCommands(context: PluginContext, target: Target) async throws -> [Command] {

        // Find the code generator tool to run (This is what we named our actual one.).
        let generatorTool = try context.tool(named: "my-code-generator")
        
        // Still ensures that the target is a source module.
        guard let target = target as? SourceModuleTarget else { return [] }

        // Get the source directory of the target
        let dataDirectory = target.directory
        
        // Get all the files in that directory, source file or not.
        let allSourceFiles = try FileManager.default.contentsOfDirectory(at: dataDirectory).map { fileName in
                dataDirectory.appending([fileName])
        }

        // Construct a build command for each source file with a particular suffix.
        return allSourceFiles.compactMap(\.path).compactMap {
            createBuildCommand(for: $0, in: context.pluginWorkDirectory, with: generatorTool.path)
        }
    }

find .build/ -name "apple.*" will show quite a few files, one of them will be the generated apple.swift, the others will be products generated from it… but there will be NO apple.txt!

Look a Little Deeper

FileManager.default.contentsOfDirectory(atPath: dataDirectory.string) is NOT recursive. To make an optionally recursive directory search one could do something like

func filesFromDirectory(path providedPath:Path, shallow:Bool = true) throws -> [Path] {
    if shallow {
        return try FileManager.default.contentsOfDirectory(atPath: providedPath.string).compactMap { fileName in
            providedPath.appending([fileName])
        }
    } else {
        let dataDirectoryURL = URL(fileURLWithPath: providedPath.string, isDirectory: true)
        var allFiles = [Path?]()
        let enumerator = FileManager.default.enumerator(at: dataDirectoryURL, includingPropertiesForKeys: [.isRegularFileKey], options: [.skipsHiddenFiles, .skipsPackageDescendants])
        
        while let fileURL = enumerator?.nextObject() as? URL {
            if let regularFileCheck = try fileURL.resourceValues(forKeys:[.isRegularFileKey]).isRegularFile, regularFileCheck == true {
                allFiles.append((Path(fileURL.path())))
            }
        }
        return allFiles.compactMap({$0})
    }
}

In production it’d be cleaner to have a non recursive search of the specific directory of relevant files along the lines let dataDirectory = target.directory.appending(["Data"])

What About Excluding in XCode?

As mentioned above Xcode considers files added to it with File > Add as resources by default.

To remove the file from all targets using the GUI:

click on apple.txt in the Navigator,
make sure the Inspector area is on “file Inspector”
and uncheck any boxes under “Target Membership”.

If those words don’t feel familiar check out the documentation on how to configure an Xcode Window.

With the check boxes unchecked, apple.txt disappears from the file list available to the build plugin entirely. Clean the build folder to remove the warnings about the stale apple.swift.

Unfortunately in the XcodeCommandPlugin side of things, target.directory is not a thing. However context.xcodeProject.directory does exist and so does target.displayName and so does inputFiles. Between those three something can be done to get the files we need.

I’ll show this working with the rest of the code in the next post.

So when exactly do the tools run?

In our package the build plugin will now run every time a new file OF ANY TYPE gets added to the Sources directory. That will then kick of the tool running for each text file, regardless if the text file has changed. When a .txt changes, the tool will run for that file alone.

It makes sense that the build plugin would refresh when the file list changes. If the list of relevant files hasn’t changed, I’m not sure why it reruns the tool on the unchanged files.

In Xcode it appears to be running every time. Which is also not what I expected.

That said, I’d rather the things run more than less and it’s questionable to be storing ginormous files unneeded in the final build in with the source anyway.

TO BE CONTINUED

At this point we have a build plugin that can find the wanted files and run a tool over them to create new files for the build.

Doing something actually useful? That’s for another day!

This article is part of a series.

Related Repos

Intro

Getting Started

The Default Package.swift

The Default plugin.swift / $MYPLUGINNAME.swift

Make Test Package

Update the Package.swift to Include the Build Plugin

Building a Custom Tool

Make the Tool

Update Package.swift

Add content to main.swift

Update plugin.swift

Check the Tool

What Happens in Xcode?

Fix the conflict

Update main.swift, run project again.

Meanwhile back in the package…

Look a Little Deeper

What About Excluding in XCode?

So when exactly do the tools run?

TO BE CONTINUED

This article is part of a series.