What if instead of a CLI, Plugins? Part 3, The Actual Code Gen

This article is part of a series.

Allll right. In the last post I created the very basic outline of a minimal viable in-build plugin. I’m going to start from that base code (The pt-1 tag Plugin,CLI), adding the code gen for $DEMOPACKAGE and then loop back and make sure it works in the MacOS/iOS sample app as well. I did not make a repo for the sample App, it’s really is just the “New Project” App, I swear.

But before I start I want to explain why I’ll be doing most of the $DEMOPACKAGE work in Xcode instead of the command line/VScode the way I’d normally do package oriented work.

PROBLEM 1: Since swift run uses process environment variables to help determine if it should run the build plugin tools again, any changes to the process environment triggers a fresh run. Some of those environment variables will be ignored in the future, but not yet. For example TERM_SESSION_ID changes with new shells and that causes a lot of headaches with both VSCode and my multi-Terminal window workflow. The build plugin just CONSTANTLY rebuilds. This one has fixes already down the pipe in 5.10

PROBLEM 2: The VSCode extension doesn’t detect the generated code’s symbols so when you write code by hand which depends on that generated code a ["cannot find $MYNEWTHING in scope" sourcekitd ] error pops up. The code can compile and run just fine, but all the red squiggly line errors grate on my eyeballs. This problem resides on upstream projects (SPM and SourceKit-lsp) not the extension, it appears.

PROBLEM 3: Well, this was more of a me problem. Some of my linker errors came from listing the build plugin in the executable target’s dependencies parameter instead of the plugins parameter in the Package.swift definition. The lack of a warning has been fixed in 5.10. I show the fixed Package.swift entry below.

Meanwhile over in Xcode it all just works. Both for packages and projects. Installing one of the forks of 5.10 could have been another option, but that will be for another day.

Add The Data Files to $DEMOPACKAGE

Start with just two files in an excluded directory.

In the $DEMOPACKAGE root directory (aka BuildPluginExampleTarget) (aka DemoFruitStore):

mkdir Sources/Data
mv Sources/apple.txt Sources/Data/apple.txt
touch Sources/Data/citrus.txt

Update the Package.swift, if haven’t already

         .executableTarget(
            name: "DemoFruitStore",
            dependencies: [
                .product(name: "ArgumentParser", 
                      package: "swift-argument-parser")
                //NOT HERE !!! .product(name: , package: "FruitStoreBuild")
            ],
            exclude: ["Data"], //<<=== what the exclusion should look like
            plugins: [
                 //HERE!!
                .plugin(name: "MyPluginName", package: "MyPluginName")
            ]
        ),

Update the contents of the text files.

apple.txt

//apple.swift
let apple = "macintosh"

citrus.txt

//citrus.swift
let citrus = "calamondin"

Update MyBuildPlugin to point to Data Directory

in $BUILDPLUGINDIR/Plugins/$MYBUILDPLUGIN/plugin.swift

Change the createBuildCommands that’s part of the BuildToolPlugin conformance to only look in the “Data” folder using the function from last post

//OLD: filesFromDirectory(path: target.directory, 
//                        shallow: false)
//The updated location. One known folder, no recursion. 
let dataDirectory = target.directory.appending(["Data"])
let filesToProcess = try filesFromDirectory(path: dataDirectory)

Now the plugin won’t work for any text file outside of the hard coded folder. A plugin designed for sharing or a broader use-case might make a different choice, but this works for this case.

Update Custom Tool

In $BUILDPLUGINDIR/Sources/MyBuildPluginTool/main.swift we now need to get the contents of the text file being inspected and shove it into the swift file.

import Foundation

let arguments = ProcessInfo().arguments
if arguments.count < 4 {
    print("missing arguments")
}

let (input, output) = (arguments[1], arguments[3])
var outputURL = URL(fileURLWithPath: output)

//------------ NEW LINE IS HERE ----------------
let contentsOfInputFile = try String(contentsOf: URL(fileURLWithPath: input))
try contentsOfInputFile.write(to: outputURL, atomically: true, encoding: .utf8)

Update $DEMOPACKAGE and RUN!

Update $DEMOPACKAGEDIR/Sources/$DEMOPACKAGE.swift (or $DEMOPACKAGEDIR/Sources/$DEMOPACKAGE/main.swift depending on your current file structure)

import ArgumentParser

@main
struct DemoFruitStore: ParsableCommand {
    static let configuration = CommandConfiguration(
        abstract: "For Testing The Plugins", 
        version: "0.0.0", 
        subcommands: [hello.self], 
        defaultSubcommand: hello.self)
    
    struct hello: ParsableCommand {
    mutating func run() throws {
        print("Hello, \(apple)!")
        print("Hello, \(citrus)!")
    }
    }
}

The IDE will likely complain that these values do not exits. They should in a minute, so ignore the warning.

If using the command line

## not NEEEDED can be useful to try if getting errors
swift package clean 
## Make the new code, no point in running without it
## Will have linker errors
swift build 
## will build again
swift run 
## will build for debugging without tools firing off
swift run

This should result in something like:

### START HERE
DemoFruitStore % swift build
Building for debugging...
[3/3] Linking my-code-generator
Build complete! (3.05s)
Building for debugging...
FIIIIIIIIIIIIIINNNNNNNNNDDDMMMEEEEEEEEEEEEEEEEE
from MyBuildPluginTool: {$PATH_TO_REPO}/DemoFruitStore/Sources/Data/citrus.txt
from MyBuildPluginTool: {$PATH_TO_REPO}/DemoFruitStore/.build/plugins/outputs/demofruitstore/DemoFruitStore/FruitStoreBuild/citrus.swift
FIIIIIIIIIIIIIINNNNNNNNNDDDMMMEEEEEEEEEEEEEEEEE
from MyBuildPluginTool: {$PATH_TO_REPO}/DemoFruitStore/Sources/Data/apple.txt
from MyBuildPluginTool: {$PATH_TO_REPO}/DemoFruitStore/.build/plugins/outputs/demofruitstore/DemoFruitStore/FruitStoreBuild/apple.swift
ld: Undefined symbols:
  DemoFruitStore.apple.unsafeMutableAddressor : Swift.String, referenced from:
      DemoFruitStore.DemoFruitStore.hello.run() throws -> () in DemoFruitStore.swift.o
  DemoFruitStore.citrus.unsafeMutableAddressor : Swift.String, referenced from:
      DemoFruitStore.DemoFruitStore.hello.run() throws -> () in DemoFruitStore.swift.o
clang: error: linker command failed with exit code 1 (use -v to see invocation)
[55/56] Linking DemoFruitStore
###-------------------- FIRST RUN --------------------
DemoFruitStore % swift run
Building for debugging...
Build complete! (0.18s)
Building for debugging...
FIIIIIIIIIIIIIINNNNNNNNNDDDMMMEEEEEEEEEEEEEEEEE
from MyBuildPluginTool: {$PATH_TO_REPO}/DemoFruitStore/Sources/Data/apple.txt
from MyBuildPluginTool: {$PATH_TO_REPO}/DemoFruitStore/.build/plugins/outputs/demofruitstore/DemoFruitStore/FruitStoreBuild/apple.swift
FIIIIIIIIIIIIIINNNNNNNNNDDDMMMEEEEEEEEEEEEEEEEE
from MyBuildPluginTool: {$PATH_TO_REPO}/DemoFruitStore/Sources/Data/citrus.txt
from MyBuildPluginTool: {$PATH_TO_REPO}/DemoFruitStore/.build/plugins/outputs/demofruitstore/DemoFruitStore/FruitStoreBuild/citrus.swift
[8/8] Linking DemoFruitStore
Build complete! (0.69s)
Hello, macintosh!
Hello, calamondin!
###-------------------- SECOND RUN --------------------
DemoFruitStore % swift run
Building for debugging...
Build complete! (0.12s)
Hello, macintosh!
Hello, calamondin!

w00t! Success! Opening up a new Terminal window or other method of changing the shell environment will trigger the build to run the tools again.

For a silly build plugin trick…

export HELLO='Hello World'
# echo $HELLO #to believe it took, or `env`
swift run

…will trigger a fresh build with tool run.

Switching to Xcode

Xcode now sees the generated symbols fine and Vscode does not. As a result, the rest I did in Xcode. I will still write the command line commands for documentation purposes, but in Xcode I was actually using Schemas. Xcode provides the ability to run Schema to any package or project with an executable in it. Even if that executable is a CLI. Different than typical CLI interactions, when the Schema runs (CMD + R or the “play button”, just like any project) none of the build messages get printed to the Console. It’s all in the Report Inspector. When working with Xcode fiddly-clickers get the reward. Click and right click everywhere. Even blank white space. Secret Buttons Everywhere. (some marked in image)

section of build report showing print outs from the plugin itself section of build report showing print outs from the called tool
Screenshot of Xcode window, Navigator with the current build selected and Console with the output of the successful run also can be seen in addition to the report described in the caption. Screenshot of Xcode window, identical to its pair except the content of the report.

Schema are XML files with build actions described within. Below is the description of the command I’m about to build.

      <CommandLineArguments>
         <CommandLineArgument
            argument = "fruit_list"
            isEnabled = "YES">
         </CommandLineArgument>
         <CommandLineArgument
            argument = "citrus"
            isEnabled = "YES">
         </CommandLineArgument>
      </CommandLineArguments>

To get to the below window to edit a Scheme via the GUI (recommended) choose Product > Scheme > Edit Scheme from the drop down menus to edit the currently selected Scheme. The Arguments page is shown, but on the Options page all the way at the bottom one can switch the Console to be Terminal and Xcode will launch a window instead.

Screenshot of editing a schema for a CLI that takes arguments

Those get shared in the .git repo. The .gitignore generated by the swift package init command works really well for including these helpful files while not sharing private info. (.swiftpm/xcode/xcshareddata/xcschemes is shared while other items in .swiftpm are not)

Referenced .gitignore for posterity:

.DS_Store
/.build
/Packages
xcuserdata/
DerivedData/
.swiftpm/configuration/registries.json
.swiftpm/xcode/package.xcworkspace/contents.xcworkspacedata
.netrc

Adding the Fruit Store

Create a new file called $DEMOPACKAGE/Sources/FruitStoreCommand.swift

This file will hold the fruit_list command. It will let us test what fruits we actually have gotten through from the data.

//
//  FruitStoreCommand.swift
//
//
//  Created by Carlyn Maw on 1/24/24.
//


import ArgumentParser

protocol Fruit {
    var name:String {get}
}

var FruitStore:Dictionary<String,[any Fruit]> = [:]

extension DemoFruitStore {
    struct fruit_list: ParsableCommand {
        @Argument var fruitType:String?
        
        mutating func run() throws {
            if let fruitType {
                if let  fruitList = FruitStore[fruitType] {
                    print("We have \(fruitList.count) \(fruitType)(s).")
                    fruitList.forEach({print("- \($0.name)")})
                } else {
                 print("No fruit of type \(fruitType) in the FruitStore.")
                }
            } else {
                print("full fruit store")
                print("\(FruitStore)")
            }
        }
    }
}

Update the Caller Config & Run

Update the config in $DEMOPACKAGEDIR/Sources/$DEMOPACKAGE.swift to recognize the new command

    static let configuration = CommandConfiguration(
        abstract: "For Testing The Plugins", 
        version: "0.0.0", 
        subcommands: [hello.self, fruit_list.self],  //CHANGED
        defaultSubcommand: hello.self)

Run the Executable

Set up two Schemas,

CMD + R and the Console should show the following outputs respectively. I put what the CLI input would have been as a convenience. Its not actually in the Console. Program ended with exit code: 0, the successful no-errors 0 code announcement, does get printed. I may remove it in future pastes but if anyone reading this had never seen it before. Its okay. It means all good. Heres some numbers that would be less good

## DemoFruitStore % swift run DemoFruitStore fruit_list
full fruit store
[:]
Program ended with exit code: 0
## DemoFruitStore % swift run DemoFruitStore fruit_list citrus
No fruit of type citrus in the FruitStore.
Program ended with exit code: 0

Add FruitStore loading code.

Let’s put items in the FruitStore.

Update citrus.txt with a struct and an attempt to load an instance into FruitStore

//citrus.swift
let citrus = "calamondin"

struct Citrus:Fruit {
    let name:String
}

FruitStore["citrus"] = [ Citrus(name:"\(citrus)")]

Run the code and magic happens. Xcode OPENS the auto generated file and flags the expressions are not allowed at the top level error. Nice. Very cool. The actual file lives deep in the build folder. It’s possible to get there by selecting Product > Show Build Folder in Finder from the menu, navigating up one level then back down to the file (/PluginTesterApp-APPHASH/SourcePackages/plugins/PluginTesterApp.output/PluginTesterApp/MyBuildPlugin)

Screenshot of Xcode being a clever girl and opening the file as described above.

But in the mean time the error needs fixing. Replace the direct assignment with…

func addCitrus() {
    FruitStore["citrus"] = [ Citrus(name:"\(citrus)")]
}

…and add addCitrus() to the fruit_lists run code. I briefly tried adding a convenience init but that screwed up ParsableCommand conformance more than I wanted to wrestle with today. For now this will be fine since we don’t have many fruits yet.

struct fruit_list: ParsableCommand {
        @Argument var fruitType:String?
        
        mutating func run() throws {
            //cannot find add addCitrus in scope warning. 
            //Ignore it. Code will still build and it will go away.
            addCitrus()   

## DemoFruitStore % swift run DemoFruitStore fruit_list
full fruit store
["citrus": [DemoFruitStore.Citrus(name: "calamondin")]]
Program ended with exit code: 0
## DemoFruitStore % swift run DemoFruitStore fruit_list citrus
We have 1 citrus(s).
- calamondin
Program ended with exit code: 0

We can do the same for apple.txt. Add the addApple() under the addCitrus() in fruit_list.run() and it works, too.

% swift run DemoFruitStore fruit_list
full fruit store
## line split mine
["apple": [DemoFruitStore.Apple(name: "macintosh")], \ 
"citrus": [DemoFruitStore.Citrus(name: "calamondin")]]
Program ended with exit code: 0

That step proved that the build system would find the code in the build folder if it was there without having the code be available in the source folder. It also proved that if we could get code that looked like that into the file it would work. Let’s generate it from data, not write it by hand now.

Actual Code Gen in main.swift

Change citrus.txt and apple.txt to just have the single words calamondin and macintosh in them respectively. No quotes. Nothing but a single word each.

in $BUILDPLUGINDIR/Sources/MyBuildPluginTool/main.swift we need to get a little fancier than before. The below Swift code will recreate what I wrote by hand in the text file a minute ago.

import Foundation

let arguments = ProcessInfo().arguments
if arguments.count < 4 {
    print("missing arguments")
}

let (input, output) = (arguments[1], arguments[3])
var outputURL = URL(fileURLWithPath: output)
var inputURL = URL(fileURLWithPath: input)

var generatedCode = generateHeader()

let fileBase = inputURL.deletingPathExtension().lastPathComponent
let structName = fileBase.capitalized
generatedCode.append(generateStruct(structName: structName))

let contentsOfInputFile = try String(contentsOf: URL(fileURLWithPath: input))
                                .trimmingCharacters(in: .whitespacesAndNewlines)

generatedCode.append(generateAddToFruitStore(base:fileBase, 
                                             structName:structName, 
                                             itemToAdd: contentsOfInputFile))

try generatedCode.write(to: outputURL, atomically: true, encoding: .utf8)

func generateHeader() -> String {
        """
        import Foundation
        
        
        """
}

func generateStruct(structName:some StringProtocol) -> String {
    """
    \n
    struct \(structName):Fruit {
        let name:String
    }
    """
}

func generateAddToFruitStore(base:some StringProtocol, 
                       structName:some StringProtocol, 
                        itemToAdd:some StringProtocol) -> String {
    """
    
    let \(base) = "\(itemToAdd)"
    func add\(base.capitalized)() {
        FruitStore["\(base)"] = [ \(structName)(name:"\\(\(base))")]
    }
    """
}

And it works! (The simple-code-gen tag Plugin, CLI)

full fruit store
## line split mine
["citrus": [DemoFruitStore.Citrus(name: "calamondin")], \ 
"apple": [DemoFruitStore.Apple(name: "macintosh")]]
Program ended with exit code: 0

But what if we wanted more items in the text files?


let itemsFromFile = try String(contentsOf: URL(fileURLWithPath: input))
                                                        .split(separator: "\n")
generatedCode.append(generateAddToFruitStore(base:fileBase, 
                                       structName:structName, 
                                       itemsToAdd:itemsFromFile))

//...

func generateAddToFruitStore(base:some StringProtocol, 
                       structName:some StringProtocol, 
                       itemsToAdd:[some StringProtocol]) -> String {
    let initStrings = itemsToAdd.map { "\(structName)(name:\"\($0.capitalized)\")" }
    let fruitArrayCode = "[\(initStrings.joined(separator: ","))]"
    let insertCode = """
        FruitStore["\(base)"] = \(fruitArrayCode)
    """
    return """

    //note: only changes PER RE-BUILD with the plugin.
    let \(base) = "\(itemsToAdd.randomElement() ?? "No items in list")"
    func add\(base.capitalized)() {
         \(insertCode)
    }
    """
}

And now that works, too!

## line splits mine.
full fruit store
["apple": [DemoFruitStore.Apple(name: "Macintosh"), \
    DemoFruitStore.Apple(name: "Rome"),  \
    DemoFruitStore.Apple(name: "Fuji"),  \
    DemoFruitStore.Apple(name: "Cortland")], \
"citrus": [DemoFruitStore.Citrus(name: "Calamondin"), \
    DemoFruitStore.Citrus(name: "Orange"),  \
    DemoFruitStore.Citrus(name: "Yuzu"),  \
    DemoFruitStore.Citrus(name: "Limequat")]]
Program ended with exit code: 0

Note the let \(base) = "\(itemsToAdd.randomElement() ?? "No items in list")" line. This will only generate when the tool rebuilds that Swift file. That means if you change citrus.txt the citrus fruit greeted by hello will likely change, but the apple will not change.

## run hello
Hello, fuji!
Hello, yuzu!
Program ended with exit code: 0
## make a change to citrus.txt
Hello, fuji!
Hello, orange!
Program ended with exit code: 0

Back in the App

Open back up the sample app project and do the following:

import SwiftUI

protocol Fruit {
    var name:String {get}
}

var FruitStore:Dictionary<String,[any Fruit]> = [:]

struct ContentView: View {
    //Even better would be at the init of the App. 
    init() {
        addCitrus()
        addApple()
    }
    
    var body: some View {
        VStack {
            Image(systemName: "globe")
                .imageScale(.large)
                .foregroundStyle(.tint)
            //Displaying them all a an exercise for the reader.
            Text("Hello, \(FruitStore["apple"]?[0].name ?? "nothing here")")
        }
        .padding()
    }
}

Screenshot Xcode window with the project navigator with the Data folder selected, the file inspector open to the Data folders info, and the content area showing ContentView.swift

Back in the plugin the XcodePluginContext needs an update to point to the “Data” folder like the package’s code.

    func createBuildCommands(context: XcodePluginContext, target: XcodeTarget) throws -> [Command] {
        let generatorTool = try context.tool(named: "my-code-generator")
        
        //New directory fetch
        let dataDirectory = context.xcodeProject.directory.appending([target.displayName, "Data"])
        let filesToProcess = try filesFromDirectory(path:dataDirectory)
        
        //Also different. It's already and array of Paths.
        return filesToProcess.compactMap {
            createBuildCommand(for: $0, in: context.pluginWorkDirectory, with: generatorTool.path)
        }
    }

I’m using my function that loads files from the end of the last post, same as the BuildToolPlugin’ conforming function . What’s important, and fragile, is the Path. The context.xcodeProject.directory.appending([target.displayName, "Data"]) relies on the displayName of the Target. I have less than 100% confidence that the displayName will always be the directory name. In a project context we are not given the target.directory as an option. It will frequently be true, so I’m going to go ahead and use it for this. We wouldn’t need to do this at all if having the text files be in the bundle was acceptable.

And that brings it all together. The code generated by the plugin can be used in a package, to support an iOS app, lots of places. As long as the tool the plugin wraps will work on the machine doing the compiling, of course.

Other Approaches to Data Munching

What’s good about the current approach is that when a data file changes, only the code for that data file changes. The tool will only run the once. What’s bad is that we have to add the addCitrus() and addApple() functions by hand and that our data storage variable loads at run time. As a note, adding new files to the source code directory will trigger a rebuild and tool run every time as of Jan 2024.

In PluginExplorer I took a different approach. Each file didn’t get its own build command, the directory got one build command with every file listed as an input. The one and only output file actually has the FruitStore:Dictionary<String,[any Fruit]> declaration which gets generated from all the files working together.

in plugin.swift

func createBuildCommands(context: PackagePlugin.PluginContext, target: PackagePlugin.Target) async throws -> [PackagePlugin.Command] {
    //if target doesn't have source files, don't run the tool. 
    guard let target = target as? SourceModuleTarget else { return [] }
    
    let dataDirectory = target.directory.appending(["Data"]) 
    let dataContents = try FileManager.default.contentsOfDirectory(atPath: dataDirectory.string).map { fileName in dataDirectory.appending([fileName]) }

    let outputFileName = "FruitStore.swift"
    let outputFiles:[Path] = [context.pluginWorkDirectory.appending([outputFileName])]
    
    return [.buildCommand(displayName: "Build the FruitStore",
                                        executable: try context.tool(named: "MyInBuildPluginTool").path,
                                        arguments: [dataDirectory.string, context.pluginWorkDirectory.string, outputFileName],
                                        inputFiles: dataContents,
                                        outputFiles: outputFiles)]
    }

in main.swift

///...

let (inputDir, outputDir, outputFile) = (arguments[1], arguments[2], arguments[3])
var output = URL(fileURLWithPath: outputDir)
output.append(component: outputFile)

///...

try FileManager.default.contentsOfDirectory(atPath: inputDir).forEach { item in
   // Did things to files here
}

///...

try generatedCode.write(to: output, atomically: true, encoding: .utf8)

That could potentially be a massive output file. In the future I’d like to try a hybrid approach where a FruitStoreService() gets re-written for any file changes but the big load in code can happen per file. I have ideas about how to do that (two separate build plugins? One plugin with N+1 commands?), but I think I’ll wait until I have an actual application I care about at this point!

Summary

I’m pretty excited about build plugins because they seem like in the long run they will be much easier to share than all my little bash scripts I have floating around the place. They can be in packages with documentation and snippets and everything. Of course that sounds awfully like a production product… which is dangerous territory for this inveterate note-taker!

And the notes on this topic aren’t over. Next post covers pre-build plugins and using existing tools in more details.

This article is part of a series.