Series on Why No Test Flight

Okay, how bad can getting client HTTPS to work be really?

Mon, 24 Nov 2025 00:00:00 +0000

My chosen deployment method for the server part of my Swift-IoT-Hello project, a Digital Ocean App Server, ONLY does HTTPS connections. Meanwhile my client can only handle HTTP.

I could get around this a few ways. Many of them versions of:

creating a local device gateway of some flavor
configuring a more permissive external server (as the main or as a proxy)

Those types of solutions push the complexity into the project’s architecture in a way that might make it difficult for a student or beginner to spin up. IF I can get HTTPS working the complexity will feel more invisible, perhaps.

So that leaves trying spending at least some time to trying to get HTTPS to work.

For some chips HTTPS would be too much math and too much storage space, but the ESP32-C6 should be able to do it. At this stage, most things that can run Embedded Swift at all can probably do it.

So what does that look like? Here’s my initial todo list and some of resources I’ll need to get it done.

TODO:

watch some videos
do an esp-idf only https client example
(optional) do a pico sdk only https example
study SwiftNIO and it’s TLS library
look at my existing (and other’s) Swift clients
Look at some API packages / type-only packages
choose a C framework
go…

Watch Some Videos

For instruction and reference I prefer written articles, but for orienting myself to a space I like video.

Before creating an implementation I want refresh my memory / understand current best practices on the TLS handshake which distinguishes HTTPS from HTTP.

Here are some videos that help clarify the problems involved. The first one a lone is enough for the already conversant. It’s both specific to the exact topic and an overview. At the end is an “off-topic” but interesting discussion of Rust and Zephyr.

The Linux Foundation
- IoT TLS: Why It’s Hard - David Brown, Linaro (7 y.o.) Great great overview.
- Zephyr TLS implementation uses mbed-tls
Computerphile:
- https://www.youtube.com/@Computerphile/search?query=TLS
- Transport Layer Security (TLS) (5 y.o.)
- TLS Handshake Explained (5 y.o.)
- Ellip
- Secret Key Exchange (Diffie-Hellman) (perhaps a bit extra, 7 y.o.)
- Public Key Cryptography (perhaps a bit extra)
- What are digital signatures (4 y.o.)
- Key Exchange Problems
- Elliptic Curves
- SHA: Secure Hashing Algorithm (perhaps a bit extra)
- Network Stacks and the Internet (if this isn’t familiar, 12 y.o.)
- Secure Web Browsing (if this isn’t familiar, 9 y.o.)
- What happens when you click a link? (most basic DNS/TCP/HTTP recap, 10 y.o.)
Practical Networking
- TLS 1.3
Mongoose Networking Library
- TLS for microcontrollers explained mostly a recap of tls in general until about 5:35
  - https://mongoose.ws
Hitex
- Hitex Webinar Secure communication for IoT devices with Mbed TLS (long, good overview but not example code)
Misc Off Topic
- https://ossna2025.sched.com/event/1zff7/why-rust-and-zephyr-are-a-good-fit-david-brown-linaro

A couple little text items to add because not a lot of decent video on it 1.2 vs 1.3:

Do a couple of hello worlds on the chips I’ll use

The espidf and the pico sdk both have https client examples that I’ll get rolling just to make sure there aren’t hardware/tool chain problems waiting to bite me.

Study SwiftNIO

I’d like the API for my library to feel similar to how swift-nio-ssl works if the ergonomics are right for embedded. This assumes I’ll even get to that level of control for it. Which I probably won’t.

NOTE: An example Swift based HTTP->HTTPS proxy exits in the SwiftNIO examples repo if I decide to fall back from this problem.

Look at my (and other’s) Swift HTTP clients

This might be a good moment to revisit the actual client ergonomics.

https://github.com/carlynorama/APItizer (No dependencies, older)
https://swiftonserver.com/building-swiftnio-clients/
https://github.com/swift-server/async-http-client (async not available, but good to look at the ergonomics)
https://swiftpackageindex.com/search?query=HTTP+client

Look at some API packages / type-only packages

Swift Log, Swift Metrics, Swift Distributed Tracing, Swift Configure all are API definition and/or types only packages. I’d like my client to work similarly. I should spend some time looking at how these get laid out and used.

Choose a base framework

So BoringSSL, which swift-nio-ssl uses, isn’t a great fit for embedded.

I’ve got to say that mbed-tls is winning just because I know that espressif, pico and zephyr all use it. If I push the implementation down low enough then it doesn’t matter as much what future implementers want to use. Here are some links, but I think I’m already pretty focused on mbed-tls.

Summary

This is just an orientation post. In the next couple I’ll get the hello worlds working without Swift, as a base goal. If it’s easy… I’ll do more!

Why would I want a Hummingbird ChildRequestContext?

Mon, 29 Sep 2025 00:00:00 +0000

Last post I made a ForwardingRequestContext and used it to save a value created by middleware.

I’ve got to admit it took me a minute to understand why and how to create a ChildRequestContext.

First the how. I honestly ran in trouble the first time I tried to make one because I just assumed it was like sub-classing and just tried to jump in and make one with out really looking that hard at the documentation. But it isn’t. The “Parent Context” is passed into the initializer as a concrete type that can be used to populate a new set of child defined custom variables. The parent does, however, provide it’s copy of the fundamental storage. That parent gets discarded after the initializer.

Most of the examples seem to have a throwing initializer for a child context. Throwing an error here can kick a client out of the app immediately.

That throwing initializer pattern leads to an answer for WHY have a ChildRequestContext, controlling access to whole groups of api endpoint handlers based on conditions checked by middleware.

To check that behavior out, I wrote RCJunior to block access to certain routes if there’s no number in the URL to seed timeConsumingData.

The code from this post and the last in a project: https://github.com/carlynorama/HummingbirdExamples/tree/main/08_context/customContext

What is the call going to look like?

The RouterGroup initializer provides the chance to insert the child context Type.

I’m going to show doing adding a RouterGroup full written out rather than with the common shorthand since I’ll be adding lots of comments.

//kiddiePoolRoutes will be of type RouterGroup<RCJunior>
let kiddiePoolRoutes = router.group("kiddiePool", context: RCJunior.self)

Like with middleware, the behavior instituted by the child context will only show up for defined api endpoints attached to the group (kiddiePoolRoutes).

The Child RequestContext

RCJunior won’t conform to ForwardingRequestContext, but will receive a MyForwardingContext in the initializer to satisfy the associatedtype ParentContext of a ChildRequestContext

import Hummingbird

struct RCJunior: ChildRequestContext {
    var coreContext: CoreRequestContextStorage
    let magicNumber: Int

    //expects concrete type. 
    init(context parentContext: MyForwardingContext) throws {
        self.coreContext = parentContext.coreContext
        guard parentContext.timeConsumingData != nil else {
            //https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/418
            throw HTTPError.init(.init(code: 418, reasonPhrase: "I am a teapot."))
        }
        self.magicNumber = parentContext.timeConsumingData!
    }
}

Unlike timeConsumingData, magicNumber isn’t an optional. If timeConsumingData isn’t set in the parent this child context will fail to initialize. What does that mean? The context will halt progress and throw an error back up the middleware chain to be turned into the server’s response. All the middleware added to the router before the group gets declared will will run. None of the middleware added to the group will run.

I picked an entertaining to me error message that means “I’m a teapot. Find a coffee maker.” (It’s an old code…).

A ChildContextInfo would still have everything from the parent context’s storage, but none of it’s custom variables. Only it’s own.

struct ChildContextInfo: ResponseEncodable {
  let magicNumber: Int
  let maxUploadSize: Int
  let endpointPath: String?
  let parameters: String
  let request_id: String
  let decoder:String
  let encoder:String
  let logger:String
}

An api endpoint that fails

Adding an api endpoint that is not expected to work:

  let kiddiePoolRoutes = router.group("kiddiePool", context: RCJunior.self) 
  kiddiePoolRoutes.get { _, context -> String in
        print("child route without \"number\"")
        return "CODE: \(context.magicNumber)"
  }

Call it…

curl -i "http://localhost:8080/kiddiePool"

BONK!

HTTP/1.1 418 I'm a teapot
Content-Length: 0
Date: Sun, 28 Sep 2025 18:34:26 GMT
Server: customContext

An api endpoint that works

  kiddiePoolRoutes.get("{number}")  { _, context -> String in
        print("child route with \"number\"")
        return "CODE: \(context.magicNumber)"
  }

curl -i "http://localhost:8080/kiddiePool/354"

HTTP/1.1 200 OK
Content-Type: text/plain; charset=utf-8
Content-Length: 9
Date: Sun, 28 Sep 2025 18:43:29 GMT
Server: customContext

CODE: 366

Other api endpoints that work

Any route added to kiddiePool routes will have to have a /{number}/ in their path somewhere to have a chance of of working.

  kiddiePoolRoutes.get("/magic/{number}/**")  { request, context -> String in
        print("description:\(request.uri.description)")
        //catches the unnamed
        print("parameters:\(context.parameters.getCatchAll())")
        return "CODE: \(context.magicNumber)"
  }

curl -i "http://localhost:8080/kiddiePool/magic/3/rabbit/saw/cards"

HTTP/1.1 200 OK
Content-Type: text/plain; charset=utf-8
Content-Length: 8
Date: Sun, 28 Sep 2025 18:59:31 GMT
Server: customContext

CODE: 15

The log reads:

## printed by first middleware before twoCents in second
## it's the result of number + userAgent.count 
13 is a big number! 
description:/kiddiePool/magic/3/rabbit/saw/cards
parameters:["rabbit", "saw", "cards"]

Don’t forget to test!

Tests

@Test func testKiddiePool() async throws {
  let app = try await buildApplication(TestArguments())
  
  //base, nothing else
  try await app.test(.router) { client in
      let _ = try await client.execute(uri: "/kiddiePool/", method: .get) { response in
          print(response.status)
          #expect(response.status.code == 418)
      }
  }

  //with number
  let number = 65
  try await app.test(.router) { client in
      let _ = try await client.execute(uri: "/kiddiePool/\(number)", method: .get) { response in
          #expect(response.status == .ok)
          let decoded = String(bytes: response.body.readableBytesView, encoding: .utf8 )
          //Hummingbird tests suite does not provide a User Agent.
          let expectedTotal =  2+732+number
          #expect(decoded == "CODE: \(expectedTotal)")
      }
  }

  //something in the path that is not a number
  try await app.test(.router) { client in
      let _ = try await client.execute(uri: "/kiddiePool/notANumber", method: .get) { response in
          print(response.status)
          #expect(response.status.code == 418)
      }
  }

  //undefined route
  try await app.test(.router) { client in
      let _ = try await client.execute(uri: "/kiddiePool/notANumber/and/too/much/else", method: .get) { response in
          print(response.status)
          #expect(response.status == .notFound)
      }
  }

  //magic path success
  let magicNumber = 3
  try await app.test(.router) { client in
      let _ = try await client.execute(uri: "/kiddiePool/magic/\(magicNumber)/rabbit/saw/cards", method: .get) { response in
          print(response.status)
          #expect(response.status == .ok)
          let decoded = String(bytes: response.body.readableBytesView, encoding: .utf8 )
          #expect(decoded != nil)
          let expectedTotal =  2+732+magicNumber
          #expect(decoded == "CODE: \(expectedTotal)")
      }
  }

  //magic path no number
  try await app.test(.router) { client in
      let _ = try await client.execute(uri: "/kiddiePool/magic/notANumber/rabbit/saw/cards", method: .get) { response in
          print(response.status)
          #expect(response.status.code == 418)
      }
  }

  //magic path, number, but no followers
  //** requires something. 
  try await app.test(.router) { client in
      let _ = try await client.execute(uri: "/kiddiePool/magic/\(magicNumber)/", method: .get) { response in
          print(response.status)
          #expect(response.status == .notFound)
      }
  }

}

An extra note on Middleware

Something I was able to confirm while working on getting a child context right is that the order middleware gets added to a group matters, not just to the other middleware, but to the api endpoints as well.

let kiddiePoolRoutes = router.group("kiddiePool", context: RCJunior.self) 
  //applies to everything after, so everything in "kiddiePoolRoutes"
  kiddiePoolRoutes.add(middleware: AlmostEmptyMiddleware(message: "kiddiePool A"))

  //curl -i "http://localhost:8080/kiddiePool/"
  kiddiePoolRoutes.get { _, context -> String in
        print("child route without \"number\"")
        return "CODE: \(context.magicNumber)"
  }

  //curl -i "http://localhost:8080/kiddiePool/354"
  kiddiePoolRoutes.get("{number}")  { _, context -> String in
        print("child route with \"number\"")
        return "CODE: \(context.magicNumber)"
  }

  //this middleware will apply to 
  //kiddiePoolRoutes.get("/magic/{number}/**") only. 
  kiddiePoolRoutes.add(middleware: AlmostEmptyMiddleware(message: "kiddiePool B"))

  
  //curl -i "http://localhost:8080/kiddiePool/magic/3/rabbit/saw/cards"
  kiddiePoolRoutes.get("/magic/{number}/**")  { request, context -> String in
        print("description:\(request.uri.description)")
        print("parameters:\(context.parameters.getCatchAll())")
        return "CODE: \(context.magicNumber)"
  }

Summary

The RCJunior ChildRequestContext uses an optional value in it’s ParentContext to determine if a client will get access to the API endpoints it applies to. Understanding this simple model will hopefully make it easier to sort through the workings of an actually-good auth library, hummingbird-auth.

That probably won’t be the next Hummingbird post, but it’s on the list.

So what even is a Hummingbird RequestContext?

Sun, 28 Sep 2025 00:00:00 +0000

A RequestContext provides a companion data layer to the Hummingbird Request type as it travels through the middleware to the api endpoint handler.

A Hummingbird Request largely echos the HTTPRequest type from HTTPTypes. If a given Hummingbird server wants to track different information that doesn’t normally get sent as part of the request, that’s a job for types conforming to the RequestContext protocol. Some of the things I would have gone to nginx or apache configuration files for can end up handled by a combination of Middlewares and RequestContexts in Hummingbird. (Although many still live at the Server level)

Most basically, use a RequestContext to override the default encoders/decoders and the maximum upload size. Key features of a context include how it can pull information up from the server layer and how it can work with Middleware.

The code in full project form (including the next post in progress) available at: https://github.com/carlynorama/HummingbirdExamples/tree/main/08_context/customContext

Step One: watch a message get passed

First thing I wanted to do is watch how information flows (and doesn’t) via middleware and a RequestContext.

All the RequestContext’s I’ve seen are structs, as is the RequestContextStorage. Information held in a context can only get passed DOWN the Middleware stack and into to the api endpoint handler. Nothing comes back up with the Response. I haven’t done any reading or experiments on storing a reference type inside a RequestContext, which could be an interesting thing to try.

The Context

I wrote both a custom RequestContext and a matching protocol to needed craft the middleware.

NOTE: When writing a middleware to be shared as a library, that protocol will be more important than the concrete type so developers can implement their own Contexts that support all the various Middlewares they need.

There’s only one special item: a noteToPass string.

import Hummingbird

protocol ForwardingRequestContext: RequestContext {
    var noteToPass: String?  { get set }
}

public struct MyForwardingContext: ForwardingRequestContext {
    //protocol requirement
    public var coreContext: CoreRequestContextStorage 

    public var noteToPass: String?

    public init(source: Source) {
        self.coreContext = .init(source: source)
        self.noteToPass = nil
    }
}

The Sending Middleware

The “sending” middleware will be the middleware first listed in the stack. It gets it’s hands on the context first. In this case the middleware needs a promise that the context will have a noteToPass variable, which gets done by providing the ForwardingRequestContext just made, rather than the more generic RequestContext as required by the RouterMiddleware protocol.

The handle function needs its own var copy of the context to make any changes since the parameter comes in as a let.

Nothing done to the context after the call to next matters to subsequent middleware or the ultimate api endpoint handler.

import Hummingbird

struct SendingMiddleware<Context: ForwardingRequestContext>: RouterMiddleware {
    public func handle(_ request: Request, context: Context, next: (Request, Context) async throws -> Response) async throws -> Response {
        print("sending...")

        var contextToPassForward = context
        contextToPassForward.noteToPass = "Howdy future Middleware!"

        let response = try await next(request, contextToPassForward)
        // DON'T TRY TO DO ANYTHING TO A CONTEXT HERE.
        // IT DOESN'T GO BACK "UP"
        return response
    }
}

Receiving Middleware

I called this the “Receiving” middleware, but the noteToPass isn’t private. Any middleware can open it up along the way.

In the example code this will be the last middleware to touch the context before it goes into the final api endpoint handler. Here it swaps in it’s own text for the note.

import Hummingbird

struct ReceivingMiddleware<Context: ForwardingRequestContext>: RouterMiddleware {
    public func handle(_ request: Request, context: Context, next: (Request, Context) async throws -> Response) async throws -> Response {
        print("receiving...")
        
        var contextToPass = context

        print("Receiver got: \(context.noteToPass ?? "no message from sender")")
        contextToPass.noteToPass = "Receiver says hi too!"

        return try await next(request, contextToPass)
    }

}

Back in Application+build

Since most of the examples have been following the Hummingbird suggested model of setting a Typealias for the context it’s super easy to swap in the new custom context up at the top-ish of Application+build.swift

//Set alias to custom RequestContext
typealias AppRequestContext = MyForwardingContext

These new middlewares get added with the sender on top:

  router.addMiddleware {
    SendingMiddleware()
    LogRequestsMiddleware(.info)
    ReceivingMiddleware()
  }

But I also want to see everything that my context knows about the world when it hits the api endpoint handler. I made a type that combines my custom value with the values in the protocol required RequestContextStorage, which itself points to other sources:

struct ContextInfo: ResponseEncodable {
  let noteToPass: String? //custom context
  let maxUploadSize: Int //RequestContext extension 
  let endpointPath: String? //from CoreRequestContextStorage
  let parameters: String //from CoreRequestContextStorage
  let request_id: String  //from Logger in RequestContextSource
  let decoder:String //RequestContext associated type default
  let encoder:String //RequestContext associated type default
  let logger:String //from RequestContextSource protocol
}

Not shown is the context.channel which is available to the application layer, see the IP passing example in the docs for how to get access to it in your own code.

  router.get("contextSpy") { _, context -> ContextInfo in
    ContextInfo(
      noteToPass: context.noteToPass,
      maxUploadSize: context.maxUploadSize,
      endpointPath: context.endpointPath,
      parameters: "\(context.parameters)",
      request_id: context.id,
      decoder: "\(context.requestDecoder)",
      encoder: "\(context.responseEncoder)",
      logger: "\(context.logger)"
      )
  }

Running the server and making the call…

 curl -i "http://localhost:8080/contextSpy"

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Content-Length: 290
Date: Sun, 28 Sep 2025 14:05:22 GMT
Server: customContext

{"parameters":"FlatDictionary<Substring, Substring>(elements: [], hashKeys: [])","logger":"Logger(_storage: Logging.Logger.Storage)","decoder":"Foundation.JSONDecoder","endpointPath":"\/contextSpy","noteToPass":"Receiver says hi too!","encoder":"Foundation.JSONEncoder","maxUploadSize":2097152,"request_id":"53075b9f900ce1b32da2fb96a5c10332"}

The output contains the receiver’s version’s of the note.

Read A Route Parameter

What’s also notable in the output above is the empty parameters dictionary. That’s for storing any information Hummingbird extracted from route variables. Let’s revise the context to store a manipulated value based on an Integer value passed in via the route.

First to put the var in the RequestContext:

import Hummingbird

protocol ForwardingRequestContext: RequestContext {
  var noteToPass: String? { get set }
  var timeConsumingData: Int? { get set }
}

public struct MyForwardingContext: ForwardingRequestContext {
  public var coreContext: CoreRequestContextStorage

  public var noteToPass: String?
  public var timeConsumingData: Int?

  public init(source: Source) {
    self.coreContext = .init(source: source)
    self.noteToPass = nil
    self.timeConsumingData = nil
  }
}

New Sending Middleware

The new sending middleware will conditionally update timeConsumingData based on an “algorithm” that combines the potential number in the url with information from the request.

I called it timeConsumingProcess to emphasize why creating a middleware that’s context dependant might have value. It might seem easier to keep middlewares self reliant, but every repeat process gets repeated for EVERY request that the middleware will touch. That’s a lot of extra work.

import Hummingbird

struct SendingMiddleware<Context: ForwardingRequestContext>: RouterMiddleware {
    public func handle(_ request: Request, context: Context, next: (Request, Context) async throws -> Response) async throws -> Response {
        print("sending...")

        var contextToPassForward = context
        contextToPassForward.noteToPass = "Howdy future Middleware!"
        

        if let pathNumber = Int(context.parameters.get("number") ?? "nope.") {
            let requestNumber = timeConsumingRequestProcessingTask(on: request)
            contextToPassForward.timeConsumingData = pathNumber + requestNumber
        }

        let response = try await next(request, contextToPassForward)
        return response
    }

    private func timeConsumingRequestProcessingTask(on request:Request) -> Int {
        if let agent = request.headers[.userAgent] {
            print("\(agent) wants to play. \(agent.count)")
            return agent.count
        }
        return 732
    }
}

New Receiving

The receiver will check to see if the value in the context has been previously set, and if so add… it’s 2¢.

import Hummingbird

struct ReceivingMiddleware<Context: ForwardingRequestContext>: RouterMiddleware {
    public func handle(_ request: Request, context: Context, next: (Request, Context) async throws -> Response) async throws -> Response {
        print("receiving...")
        
        var contextToPass = context

        print("Receiver got: \(context.noteToPass ?? "no message from sender")")
        contextToPass.noteToPass = "Receiver says hi too!"

        if context.timeConsumingData != nil {
            contextToPass.timeConsumingData = myTwoCents(number: context.timeConsumingData!)
        }
        return try await next(request, contextToPass)
    }

    private func myTwoCents(number: Int) -> Int {
        print("\(number) is a big number!")
        return number + 2
    }
}

New Endpoint

Adding the value to the ContextInfo structure…

struct ContextInfo: ResponseEncodable {
  let timeConsumingData: Int?
  let noteToPass: String?
  let maxUploadSize: Int
  let endpointPath: String?
  let parameters: String
  let request_id: String
  let decoder:String
  let encoder:String
  let logger:String
}

And creating and endpoint that expects a value:

  //http://localhost:8080/contextSpy/anythingHere
  //"parameters": ... elements: [(key: \"number\", value: \"anythingHere\")] ...
  router.get("contextSpy/{number}") { _, context -> ContextInfo in
    ContextInfo(
      timeConsumingData: context.timeConsumingData,
      noteToPass: context.noteToPass,
      maxUploadSize: context.maxUploadSize,
      endpointPath: context.endpointPath,
      parameters: "\(context.parameters)",
      request_id: context.id,
      decoder: "\(context.requestDecoder)",
      encoder: "\(context.responseEncoder)",
      logger: "\(context.logger)"
      )
  }

NOTE: The old endpoint will need updating, too. When it’s called the JSON just won’t have a “timeConsumingData” value and the parameters will still be empty.

Seeing the result:

curl -i "http://localhost:8080/contextSpy/354"

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Content-Length: 377
Date: Sun, 28 Sep 2025 14:26:33 GMT
Server: customContext

{"maxUploadSize":2097152,"timeConsumingData":366,"noteToPass":"Receiver says hi too!","endpointPath":"\/contextSpy\/{number}","decoder":"Foundation.JSONDecoder","request_id":"53075b9f900ce1b32da2fb96a5c10331","encoder":"Foundation.JSONEncoder","logger":"Logger(_storage: Logging.Logger.Storage)","parameters":"FlatDictionary<Substring, Substring>(elements: [(key: \"number\", value: \"354\")], hashKeys: [5772982973116696044])"}

We have a parameter!

"FlatDictionary<Substring, Substring>(elements: [(key: \"number\", value: \"354\")], hashKeys: [-6711111692066375814])"

This parameters array stores route based values only. Code meant to handle a query encoded request like:

curl -i "http://localhost:8080/urlQueryCheck?x=45&y=33"

will accesses the values via the Request:

request.uri.queryParameters

Summary

The RequestContext / Middleware pair can accomplish some pretty powerful work. That power can be sharpened by narrowing down into a ChildRequestContext for a RouteGroup once certain requirements have been met.

That’s the next post!

(changing a terminal prompt)

Sat, 27 Sep 2025 00:00:00 +0000

So name and directory aren’t in the “shot”, where $> is the desired prompt.

## set the prompt for just that window
export PS1="$> "
## clear the window
reset

How about catching errors with Hummingbird Middleware?

Sat, 27 Sep 2025 00:00:00 +0000

This article will show four different error responses based on Middleware:

how to print them to the log
how to return a static HTML based 404 page
how to return dynamic HTML in response to any error
how to choose to send JSON instead.

These can all be seen in the repo along with the examples from the last post.

Order matters in a Middleware stack. All of these examples work by catching the errors relevant to their tasks and returning a Response. Changing a HTTPError with a 404 status into a Response with a 404 status means that middleware which is only looking for unhandled errors will not catch an already packaged Response! This is a feature because it let’s the developer progressively filter out errors when requests or the errors meet certain conditions, like in the JSON example.

It’s also good to note what a not-found Middleware can’t do using the trie router, which is to apply to a whole branch of routes selectively. For groups and collections the trie router applies middleware to the found routes only, but there is a way around that discussed at the end.

Log Errors Middleware

It’s not just Responses that percolate up through middleware, but errors too. From what I can tell most if not all errors thrown by Hummingbird itself will be an HTTPError (docs).

To get a sense of what’s happening LogErrorsMiddleware will print out the errors it encounters to the Log. In projects with a lot of middleware, moving LogErrorsMiddleware’s placement around the stack operates much like moving a stethoscope around, triangulating a problem.

import Hummingbird
import Logging

public struct LogErrorsMiddleware<Context: RequestContext>: RouterMiddleware {
    var level:Logger.Level

    func formatHeaderForLog(headers: HTTPFields) -> String {
                var forLog = ""
        
        for header in headers {
            forLog.append("\t\(header.name):\(header.value)\n")
        }
        forLog.append("------")
        return forLog
    }

    public func handle(_ request: Request, context: Context, next: (Request, Context) async throws -> Response) async throws -> Response {
        do {
            return try await next(request, context)
        } catch let error as HTTPError {
            let status = error.status
            let formattedHeaders = formatHeaderForLog(headers: error.headers)
            context.logger.log(level: level, Logger.Message(stringLiteral: "\nResponding\(status) with:\n\(formattedHeaders)"))
            throw error
        }

    }
}

It’s output looks like:

025-09-27T04:10:53-0700 info inthemiddle: hb.request.id=7c21d9a8dea288aed30d15e3536012ea [inthemiddle] 
Responding 404 Not Found to /other/ call:
------

There are no headers yet. The curl output shows a handful added at the last moment.

## curl -i "http://localhost:8080/other/"
HTTP/1.1 404 Not Found
Content-Length: 0
Date: Sat, 27 Sep 2025 11:10:52 GMT
Server: inthemiddle

These do not appear to be added by the defaultHummingbirdHeaders method called by the default JSONEncoder, but instead added at the Application layer.

I found these lines by searching the Hummingbird repo for the .headerName as determined by the HTTPField names of the HTTPTypes package.

Return Static Page

The original itch to explore middleware came from wanting to get a better “Not Found” error, just a classic 404 page. Hummingbird formulates an HTTPError for this situation in the respond method using the notFoundResponder belonging to the RouterResponder which the application builds from the Router passed to it at startup.

This handler performs it’s task classically by using a redirect. It uses the 303 (see other) redirect, telling the browser that trying this page again in the future might get a different response, or maybe not. As such, a client should never cache it.

Static404Middleware

import Hummingbird 

struct Static404Middleware<Context: RequestContext>: RouterMiddleware {
    
    func handle(
        _ request: Request,
        context: Context,
        next: (
            Request,
            Context
        ) async throws -> Response
    ) async throws -> Response {
        do {
            return try await next(request, context)
        }
        catch let error as HTTPError {
            if error.status == .notFound {
                return Response(
                    status: .seeOther,
                    headers: [
                        .location: "/404.html",
                    ]
                )
            }
            throw error
        }
    }
}

And the static page:

<!DOCTYPE html>
<html>
<head>
  <title>Wrong turn.</title>
  <meta charset="UTF-8">
</head>
<body>
  <H1>Return to the route.</H1>
</body>
</html>

If that’s too uninformative, don’t reach for Javascript’s Document.URL to pass query args and build a dynamic page that way. Hummingbird’s RIGHT HERE to make that dynamic page itself.

Return Dynamic Page

This section leans on the previous post on Mustache and code from the auth-fluent example.

The resulting middleware will serve a 404 without any redirection. If the endpoint won’t ever hold valid content that’s a great choice. But not just 404’s, it will also provide a page for ANY error using an ErrorHTML builder.

ErrorPageMiddleware

import Hummingbird
import Mustache

/// Generate an HTML page for a thrown error
struct ErrorPageMiddleware<Context: RequestContext>: RouterMiddleware {
    let mustacheLibrary: MustacheLibrary

    func handle(
        _ request: Request,
        context: Context,
        next: (Request, Context) async throws -> Response
    ) async throws -> Response {
        do {
            return try await next(request, context)
        } catch {
            if let error = error as? HTTPError {
                return try ErrorHTML(status: error.status, message: error.body ?? "", library: mustacheLibrary).response(from: request, context: context)
            } else {
                return try ErrorHTML(status: HTTPResponse.Status.internalServerError, message: "\(error)", library: mustacheLibrary).response(from: request, context: context)
            }
        }
    }
}

ErrorHTML

import Hummingbird
import Mustache

struct ErrorInfo {
    let title: String
    let status: String
    let message: String
}

struct ErrorHTML:ResponseGenerator  {

    let message: String
    let status: HTTPResponse.Status

    let library: MustacheLibrary

    init(status: HTTPResponse.Status, message: String, library:MustacheLibrary) {
        self.message = message
        self.status = status
        self.library = library
        //should probably check library for the error template. 
    }

    private var html:String {
        if let result = try? render() {
            return result
        } else {
            return "??? oops."
        }
    }

    private func render() throws -> String? {
        let pageContent = ErrorInfo(title: "Error Finding Page", status: "\(status)", message: message)

        let rendered = library.render(
            pageContent,
            withTemplate: "error"
        )
        return rendered
    }

    func response(from request: Request, context: some RequestContext) throws -> Response {
        let buffer = ByteBuffer(string: self.html)
        return .init(status: status, headers: [.contentType: "text/html"], body: .init(byteBuffer: buffer))
    }
}

The Mustache Templates

error.mustache calls head.mustache and foot.mustache, all in the Templates folder included in Package.swift.

{{>head}}
<body>
    <h1>Error</h1>
    <h2>{{  status  }}</h2>
    <p>{{  message  }}</p>
</body>
{{>foot}}

<!DOCTYPE html>
<html>
<head>
    <title>{{  title  }}</title>
    <meta charset="UTF-8">
</head>

</html>

Adding the Middleware and Templates

in the buildRouter function…

  let mustacheLibrary = try await MustacheLibrary(directory: Bundle.module.resourcePath!)

  router.addMiddleware {
    //Mustache based error page.
    ErrorPageMiddleware(mustacheLibrary: mustacheLibrary)
    //...
  }

Return JSON

But what if the app serves a JSON API and serving HTML doesn’t make sense?

While an empty body can absolutely be a valid choice, this example takes an initial crack at returning a Problem Details JSON.

If the client doesn’t specify that “text/html” is an accepted format, it will return JSON. That means clients that accept “/” but don’t call out “text/html” specifically will get the JSON. The accepts list uses specificity and a quality factor to rank types accepted. This middleware doesn’t care, but a future version probably should.

import Hummingbird
import Logging

struct ProblemDetail: Decodable, ResponseEncodable {
  let status: Int?
  let title: String
  let detail: String?

  init(withStatus: HTTPResponse.Status, details: String) {
    self.status = withStatus.code
    self.title = "\(withStatus.code): \(withStatus.reasonPhrase)"
    self.detail = details
  }
}

public struct JSONErrorMiddleware<Context: RequestContext>: RouterMiddleware {

  public func handle(
    _ request: Request, context: Context, next: (Request, Context) async throws -> Response
  ) async throws -> Response {
    do {
      return try await next(request, context)
    } catch let error as HTTPError {

      if let accepts = request.headers[.accept] {
        if !accepts.contains("text/html") {
          let problem = ProblemDetail(
            withStatus: error.status, 
            details: "\(error) from request to \(request.uri)")
          var response = try problem.response(from: request, context: context)
          response.status = error.status
          response.headers[.contentType] = "application/problem+json"
          return response
        }
      }
      throw error
    }
  }
}

Middleware for a whole Route Block

I know I will want to apply Middleware to a whole block of routes. That can be done with the trie router, but only for the happy paths. So, for example if you want to use an authentication middleware for anything filed under /private/ all the routes under /private/ that have a description will call that middleware no problem.

I saw a handful of different ways in the examples, but an addRoutes function seems to work the most reliably (e.g.):

func addRoutes(to router: Router<AppRequestContext>, atPath:RouterPath) {
    router.group(atPath)
        .add(middleware: AlmostEmptyMiddleware<AppRequestContext>(message: "controller method"))
        .get("tryme") { _, _ in
            ExampleCodable(name: "Diane", number: 3)
        }
}

That middleware will only fire for the route "\(atPath)/tryme", not "\(atPath)/nothinghere"

If I wanted something to fire for all of "\(atPath)/" I would have to use the ResultBuilderRouter instead of the trie and I haven’t touched that much yet at all.

Another option would be lots of little apps glued together by a Proxy server. Perhaps tricky to set up, but possibly easier to maintain in the long run since an error in the JSON api wouldn’t bring down the web server, etc. Use a data backend like postgres that can manage multiple client interactions well if that’s needed.

Summary

Middleware is the perfect place to catch and filter errors, but the information gleaned by a Middleware can only be kept around in that Middleware. Unless it gets added to a RequestContext, which will be the next Hummingbird example.

So how exactly does Middleware come into Hummingbird?

Fri, 26 Sep 2025 00:00:00 +0000

So previous examples are already using Hummingbird provided Middleware.

Logging Middleware which determines what gets logged out to the terminal while running, a skin around swift-log
File Middleware which serves up static pages when there’s no route for the same path defined in the Router

Other Middlewares provided include:

CORS Middleware for managing Cross-origin resource sharing
Metrics Middleware The same way the logging middleware wraps swift-log this wraps swift-metrics
Tracing Provides a tracing interface distributed tracing (W3 working group)

Hummingbird also provides some Middleware protocols to help get common tasks started:

AuthenticatorMiddleware
- A whole Authentication framework provided as a library as well.
SessionMiddleware
RequestDecompression and ResponseCompression Middlewares

Not to mention the many examples in the various repos of the Hummingbird Project itself.

But what if you want to write your own?

I’m going to show 3 over the next 3 posts:

a middleware that will log out header information
one that falls back to an error page
one that saves information in the context for later use

Middleware

This middleware isn’t strictly needed for logging request header information because the Logging middleware already handles it in a far more robust and ergonomic way.

        LogRequestsMiddleware(.info, includeHeaders: .some([.contentType]))
        LogRequestsMiddleware(.info, includeHeaders: .all())
        LogRequestsMiddleware(.info, includeHeaders: .all(except: [.cookie]))

What this middleware will show is also logging the response headers, albeit in a much more rudimentary way.

Middleware requires a handler that calls the next middleware on the stack, but since that call happens inside the handler, that method can also operate on the Response bubbling up as well.

public struct EmptyMiddleware<Context: RequestContext>: RouterMiddleware {
    public func handle(_ request: Request, context: Context, next: (Request, Context) async throws -> Response) async throws -> Response {
        return try await next(request, context)
    }
}

So if I wanted to write a middleware that would show the headers for BOTH the request and the response it might look something like:

import Hummingbird
import Logging

public struct LogHeadersMiddleware<Context: RequestContext>: RouterMiddleware {
    var level:Logger.Level

    func formatHeaderForLog(headers: HTTPFields) -> String {
                var forLog = ""
        
        for header in headers {
            forLog.append("\t\(header.name):\(header.value)\n")
        }
        forLog.append("------")
        return forLog
    }

    public func handle(_ request: Request, context: Context, next: (Request, Context) async throws -> Response) async throws -> Response {

        let requestHeaders = formatHeaderForLog(headers: request.headers)
        context.logger.log(level: level, Logger.Message(stringLiteral: "\nHeaders for \(request.uri.path) request:\n\(requestHeaders)"))

        //return Response
        let response = try await next(request, context)

        let responseHeaders = formatHeaderForLog(headers: response.headers)
        context.logger.log(level: level, Logger.Message(stringLiteral: "\nResponding with:\n\(responseHeaders)"))

        return response
    }
}

The output for a curl request to the /hello route in the demo (curl "http://localhost:8080/hello") looks like:

2025-09-26T06:20:35-0700 info inthemiddle: hb.request.id=2f72a050b4afa3a2698b886b64654226 [inthemiddle] 
Headers for /hello request:
        User-Agent:curl/8.7.1
        Accept:*/*
------
2025-09-26T06:20:35-0700 info inthemiddle: hb.request.id=2f72a050b4afa3a2698b886b64654226 [inthemiddle] 
Responding with:
        Content-Type:text/plain; charset=utf-8
        Content-Length:5
------

Order Matters

Let’s see what happens with the middleware called from two different places: right after the normal logger and at the end.

First right after the usual logger.

    router.addMiddleware {
        // logging middleware
        LogRequestsMiddleware(.info)
        //logs the headers of the request and the responses
        LogHeadersMiddleware(level: .info)
        //serves the static files in public folder by default.
        FileMiddleware(searchForIndexHtml: true)
    }

Calling the home page which has a static HTML page available (curl "http://localhost:8080/") the built in Logger prints out first and the custom logger prints out everything second. If the FileMiddleware logged anything, it would have shown up where noted in the added comment.

2025-09-26T06:31:44-0700 info inthemiddle: hb.request.id=94e54b3188d6f6c807d5eed484bca2cb hb.request.method=GET hb.request.path=/ [Hummingbird] Request
2025-09-26T06:31:44-0700 info inthemiddle: hb.request.id=94e54b3188d6f6c807d5eed484bca2cb [inthemiddle] 
Headers for / request:
        User-Agent:curl/8.7.1
        Accept:*/*
------
# FileMiddleware logging would have happened here! 
2025-09-26T06:31:44-0700 info inthemiddle: hb.request.id=94e54b3188d6f6c807d5eed484bca2cb [inthemiddle] 
Responding with:
        Content-Length:364
        Last-Modified:Tue, 16 Sep 2025 16:40:02 GMT
        ETag:W/"0900180b0600303830322e3932373739"
        Content-Type:text/html
        Accept-Ranges:bytes
------

Now move LogHeadersMiddleware AFTER the file middleware.

    router.addMiddleware {
        // logging middleware
        LogRequestsMiddleware(.info)
        //serves the static files in public folder by default.
        FileMiddleware(searchForIndexHtml: true)
        //logs the headers of the request and the responses
        LogHeadersMiddleware(level: .info)
    }

There’s NOTHING for the response because there isn’t one yet. Nothing in the Hummingbird stack has prepped one because there isn’t a route for “/” in the router at all. Not until it goes back up through the FileMiddleware will it have the index page Response in hand.

2025-09-26T06:36:24-0700 info inthemiddle: hb.request.id=eb458259999c78dc5069ba5bc9205f41 [inthemiddle] 
Headers for / request:
        User-Agent:curl/8.7.1
        Accept:*/*
------

The next test is to move the middleware to the top and call a known undefined path (no route, no static page.) (curl "http://localhost:8080/knownempty")

    router.addMiddleware {
        //logs the headers of the request and the responses
        LogHeadersMiddleware(level: .info)
        // logging middleware
        LogRequestsMiddleware(.info)
        //serves the static files in public folder by default.
        FileMiddleware(searchForIndexHtml: true)

    }

It’s exactly the same, request headers print and nothing for response headers.

2025-09-26T06:42:48-0700 info inthemiddle: hb.request.id=eb458259999c78dc5069ba5bc9205f43 [inthemiddle] 
Headers for /knownempty request:
        User-Agent:curl/8.7.1
        Accept:*/*
------

The curl call printed out nothing as well. That will change if change the curl call to (curl -i "http://localhost:8080/knownempty"), in which case the logs will not print out a response, but the curl will be:

HTTP/1.1 404 Not Found
Content-Length: 0
Date: Fri, 26 Sep 2025 15:49:48 GMT
Server: inthemiddle

So when does Hummingbird formulate that Response? There is an HTTPError (docs) type that gets propagated up instead of a Response as part of the try in the try await next(request, context) call. The not found error gets thrown in the respond method by the notFoundResponder of the RouterResponder’s that the application builds from the Router passed to it at startup.

So how to catch it and swap in something else? More middleware!

Summary

LogHeadersMiddleware demonstrates interceding on a path to punt out headers to the logging system. The next demo will show how to catch undefined routes to return appropriate 404 responses.

Can I make the button more pressable?

Thu, 25 Sep 2025 00:00:00 +0000

So this project needs it’s button situation improved. The little boot button that I’m pressing with a bamboo skewer isn’t going to cut it.

Also I want to make the HTTP call “onPress”, not a repeat timer nor “while pressed”. I need to add that feature to MomentaryInput.

For state of the code at the end of these notes see:

https://github.com/carlynorama/swift_esp32c6_hello/tree/f_inputBehaviors

New Parts

So I bought some parts from Adafruit. I generally buy 2 of each thing because accidents happen.

Desired Button:

Fallback Button (only 1x because its a fall back anyway):

Adafruit IoT Button with NeoPixel BFF Add-On for QT Py and Xiao

If decide to take on battery power:

Short USB C cable:

USB C to USB C Cable - USB 3.1 Gen 4 with E-Mark

3 sets of headers went in to assembling the prototype, each a different type.

the low profile pin headers that came with the button BFF
deep socket headers
lower profile socket headers

These all need to be cut down to a length of 7, so if bulk sourcing getting them precut is easier.

Not gonna lie, my husband todbot zipped these on for me since he’s in the middle of a soldering palooza. (Adafruit tutorial on soldering headers and my old play list on soldering in general) I traded firmware-loading duties later in the week. Hero shot for my hero!

To test the new hardware, move the button pin:

//new button on pin 2 instead of on the boot pin (9)
  guard var button = MomentaryInput(2) else {
    fatalError("Difficulty setting up button.")
  }

return it to a simple switch

if button.isActive {
  led.turnOn()
} else {
  led.turnOff()
}
delay(50);

Why the delay? because without it the esp-if monitor gets a little overwhelmed.

Please enable CONFIG_ESP_SYSTEM_USE_FRAME_POINTER option to have a full backtrace.
E (24238) task_wdt: Task watchdog got triggered. The following tasks/users did not reset the watchdog in time:
E (24238) task_wdt:  - IDLE (CPU 0)
E (24238) task_wdt: Tasks currently running:
E (24238) task_wdt: CPU 0: main
E (24238) task_wdt: Print CPU 0 (current core) registers
Core  0 register dump:
MEPC    : 0x4200b03a  RA      : 0x4200b03a  SP      : 0x40811000  GP      : 0x4080bfa4  
--- 0x4200b03a: $e10__idf_main0B0yyF at ??:?
--- 0x4200b03a: $e10__idf_main0B0yyF at ??:?
TP      : 0x40811060  T0      : 0x4200b088  T1      : 0x20000000  T2      : 0xffffffff  
--- 0x4200b088: $es5print_10terminatorys12StaticStringV_ADtF at ??:?
S0/FP   : 0x0000000f  S1      : 0x00000001  A0      : 0x00000000  A1      : 0x00000001  
A2      : 0x00000000  A3      : 0x00000004  A4      : 0x00008000  A5      : 0x60091000  
A6      : 0x00000001  A7      : 0x0000000a  S2      : 0x42020550  S3      : 0x00000000  
S4      : 0x00000000  S5      : 0x00000000  S6      : 0x00000000  S7      : 0x00000000  
S8      : 0x00000000  S9      : 0x00000000  S10     : 0x00000000  S11     : 0x00000000  
T3      : 0x00000000  T4      : 0x00000000  T5      : 0x00000000  T6      : 0x00000000  
MSTATUS : 0x00001889  MTVEC   : 0x40800001  MCAUSE  : 0xdeadc0de  MTVAL   : 0xdeadc0de  
--- 0x40800001: _vector_table at /Users/$USER/esp/esp-idf/components/riscv/vectors_intc.S:54
MHARTID : 0x00000000

The New Code

Because modern buttons are so great, and because I’ve picked one with a real CLICK to it I am not going to worry about a software debounce yet. On this chip I could activate the Espressif provided glitch filter.

Added Inverse to GPIOLevel

extension GPIOLevel {
    var inverse:GPIOLevel {
        (self == .high) ? .high : .low
    }
}

Updates to MomentaryInput

Added a variable to the struct to store the last state, which will need to be initialized using that inverse value just created.

extension MomentaryInput {
    init?(_ pinNum: UInt32, activeLow:Bool = true, useInternalHardware:Bool = true) {
        self.pin = InputPin(pinNumber: pinNum, activeLow:activeLow, useInternalHardware: useInternalHardware)
        self.activeLevel = GPIOLevel(!activeLow)
        self.lastState = self.activeLevel.inverse
    }
}

    var lastState:GPIOLevel

    typealias OnActiveBehavior = () -> Void

    mutating func onActivate(do behavior:OnActiveBehavior) {
        let currentLevel = pin.readLevel()
        if currentLevel != lastState && currentLevel == activeLevel {
            behavior()
        }
        lastState = currentLevel
    }

Since my code will just be using the button for this one behavior it’s fine to do the pin read in the call. If I had a lot of behaviors on different button states I’d have to be careful not to do a new read for every method, but only once per while loop. Future versions of the code will have to decide if the setup will register listeners, if listeners get called explicitly in the loop, if the button itself will register behaviors to do after the poll… TBD.

On a chip like the ESP23C6, it would be even better to hook into their existing button interrupt types.

All of that is research for another day. Today is for the easy win.

back on Main.swift

  while true {
    button.onActivate { print("hello") }
    delay(50); 
  }

TADA! Thats it.

Add the HTTP call back in…

Where I’ve left the code in the branch is with the Wifi and HTTP stuff all un commented again and the new while loop looks like:

  while true {

    button.onActivate { 
      let _ = exampleClient.fetch("/")
    }
    delay(50); //to help out the monitor
  }

Works a treat.

Summary

This was a fairly easy task to knock off the list. Especially with a soldering assist.

More things to do would be to bring the neo-pixel that came on the key cap BFF into the mix to signal different states of the send.

Theres a bit more server work to do, too.

And then both the button and the server have to decide what the secret handshake is going the be for this demo.

I guess this mean what the next post will be is a bit of a surprise… even to me!

Can I Swiftify that ESP32C6 HTTPClient more?

Wed, 24 Sep 2025 00:00:00 +0000

I’m going to straight up lead with my future TODO list, because I’m not satisfied with where the code gets left off.

[] try on MacOS 26 using inline array and span
[] try on MacOS 26 using new C safety features
[] Could UnsafeMutablePointer<addrinfo>? even be a function param? As an inout? Can optionals be inouts?
[] want a concrete HTTPClient type that links to a HTTPClientService protocol instead of a HTTPClient protocol.
[] take a whack at serializing the HTTPRequest type w/o Codable
[] Should I be writing my own ISOLatin1String like what HTTPType has that uses [UInt8] as a backing storage? String implementations in embedded are going to be very varied.
[] this is all still blocking. how to use an interrupt based event loop if the sdk makes one available?

But back to what I was able to do. My current HTTP request gets managed by a whisper thin Swift wrapper around C.

I’d like to make it a little less whisper thin.

The results so far here:

https://github.com/carlynorama/swift_esp32c6_hello/tree/e_ImproveHTTPClient

Setting Up the Protocol

What’s my minimum viable client for the demo? Describe it as a protocol:

protocol HTTPClient {
  init(host: String, port:Int?)
  func fetch(_ path:String)
}

and a the client moves to an implementation.


struct  ServerInfo {
    //scheme is always http
    //let useHTTPS: Bool = true
    let host:String //TODO: will have to split to domain for dns lookup?
    let port:Int
    //let basePath = String //future
}

final class MyClient: HTTPClient {

    var defaultServer: ServerInfo

    init(host: String, port: Int? = nil) {
    if port == nil {
      self.defaultServer = ServerInfo(host: host, port: 80)
    } else {
      self.defaultServer = ServerInfo(host: host, port: port!)
    }
  }

}

In Main.swift using the following instantiation would lock it down so the subsequent code can only use the features of the class that comply with the protocol.

let exampleClient: some HTTPClient = MyClient(host: "example.com")

I don’t love this. As noted in my TODO’s I’d prefer to have a concrete HTTPClient type that links to a HTTPClientService protocol instead, more like the way the momentary button code is two steps away from the underlying hardware architecture.

I am also on the fence about having a client be attached to a specific server.

Resolve the DNS

To resolve the underlying DNS the standard trick is to use the c function getaddrinfo, even SwiftNIO uses it.

It writes the result to a C struct which has another C struct embedded in it passed as a reference:

struct addrinfo {
    int              ai_flags;     // AI_PASSIVE, AI_CANONNAME, etc.
    int              ai_family;    // AF_INET, AF_INET6, AF_UNSPEC
    int              ai_socktype;  // SOCK_STREAM, SOCK_DGRAM
    int              ai_protocol;  // use 0 for "any"
    size_t           ai_addrlen;   // size of ai_addr in bytes
    struct sockaddr *ai_addr;      // struct sockaddr_in or _in6
    char            *ai_canonname; // full canonical hostname

    struct addrinfo *ai_next;      // linked list, next node
};

struct sockaddr {
    unsigned short    sa_family;    // address family, AF_xxx
    char              sa_data[14];  // 14 bytes of protocol address
};

For the time being I chose to hold that result container in a class level variable, making sure to have a deinit for it, just in case. I tried holding the pointee instead of the pointer, which appeared to work, but fell apart when creating the socket.

I also hard coded a retry of 6 times, which a better library would let the developer config.


enum HTTPClientError: Error {
  case hostUnresolved

  var describe: String {
    return switch self {
    case .hostUnresolved: "hostUnresolved"
    }
  }
}

final class MyClient: HTTPClient {
    internal typealias AddressInfo = addrinfo

    var currentInfo: UnsafeMutablePointer<addrinfo>?

    var defaultServer: ServerInfo

    init(host: String, port: Int? = nil) {
        if port == nil {
        self.defaultServer = ServerInfo(host: host, port: 80)
        } else {
        self.defaultServer = ServerInfo(host: host, port: port!)
        }
    }

    //TODO: what if i just nil'd it? 
    deinit {
        if let currentInfo {
            freeaddrinfo(currentInfo)
        }
    }

    func test() {
      print("resolving...")
      do {
        try getAddressInfo(for: server.host, using: server.port)
      } catch let myError {
        print("Error info: \(myError.describe)")
      }
    }

    private func getAddressInfo(for host: String, using port: Int = 80) throws(HTTPClientError) {
        currentInfo = nil
        var retry = 6
        let local_host = host.utf8CString
        while retry > 0 {

        var hints: AddressInfo = addrinfo()
        hints.ai_socktype = SOCK_STREAM  //vs SOCK_DGRAM
        hints.ai_protocol = AF_INET  //IP version 4, vs 6 or unix
        let error = local_host.withContiguousStorageIfAvailable { host_name in
            return getaddrinfo(host_name.baseAddress, "\(port)", &hints, &currentInfo)
        }
        if error == 0 {
            print("got it.")
            retry = 0
            return
        } else {
            if error == nil {
            print("why wasn't contiguous storage available?")
            } else {
            print("getAddressInfo error: \(error!)")
            }
            retry = retry - 1
        }
        }
        currentInfo = nil
        throw HTTPClientError.hostUnresolved
    }
}

NOTE: I am having a problem that requires me to unplug and replug my board while the monitor is still running to get the host to resolve some of the time. I think this might be a esp-idf DNS cache problem? It does not always happen.

Connect the socket

I again created a top level variable to hold an “open socket”. It’s just a CInt that the underlying network code can use to locate which underlying hardware to use. Once a socket has been opened, it’s important to close it. Once a socket has been opened, this top level var should be returned to nil only after close() has been called. This will let my code know if it has done proper clean up.

The swift nio socket type:

final class MyClient: HTTPClient {
  internal typealias AddressInfo = addrinfo
  internal typealias SocketAddress = sockaddr
  internal typealias SocketHandle = CInt  //none of the functions use socklen_t?

  //...

  var currentInfo: UnsafeMutablePointer<AddressInfo>?
  var openSocket: SocketHandle?

  deinit {
    if let openSocket {
      close(openSocket)
    }
    if let currentInfo {
      freeaddrinfo(currentInfo)
    }
  }
  //...
}

    func test() {
      print("resolving...")
      do {
        try getAddressInfo(for: server.host, using: server.port)
        openSocket = try connectSocket()
        freeaddrinfo(currentInfo)  //addrinfo has a freshness value.
        assert(currentInfo == nil) //check to make sure that actually nil'd it too. 
      } catch let myError {
        print("Error info: \(myError.describe)")
      }
    }

If the connection is successful, return the socket handle.

private func connectSocket() throws(HTTPClientError) -> SocketHandle {
    if let currentInfo {
      print("I have AddressInfo")
      let socket: SocketHandle = socket(
        currentInfo.pointee.ai_family, currentInfo.pointee.ai_socktype, 0)
      if socket < 0 {
        throw HTTPClientError.couldNotMakeSocket
      }
      print("socketHandle: \(socket)")
      var retry = 6
      while retry > 0 {
        let connectValue = connect(
          socket, currentInfo.pointee.ai_addr, currentInfo.pointee.ai_addrlen)
        if connectValue == 0 {
          retry = 0
          return socket
        } else {
          print("connect returned \(connectValue)")
        }
        delay(500)
        retry = retry - 1
        print("retry connect \(retry)")
      }
      close(socket)
      throw HTTPClientError.connectionFailed
    }
    throw HTTPClientError.addressUnresolved
  }

Write to the Socket

In a future version of this client I’d like to formulate a HTTPRequest that gets handled to a ClientService that does the serialization, but for now it’s hand coded. Why are those “extra steps” a good idea? Well, because when dealing with protocols it’s REALLY REALLY easy to forget things like required terminating newlines, carriage returns, 0 and other final characters.

It’s helpful to be able to see the request on the receiving side, but since this isn’t a client running on my machine, there are a couple extra steps to troubleshooting with netcat. One has to turn off firewalls and use the ifconfig command (look for en0: ). It’s likely to be an address like 192.168.42.*** and then my client init would look like:

let exampleClient: some HTTPClient = MyClient(host: "192.168.42.***", port:8080)

(For an IP not on the same private network try something like ifconfig.me, but there’s a million of these sites.)

    func test() {
      print("resolving...")
      do {
        try getAddressInfo(for: server.host, using: server.port)
        openSocket = try connectSocket()
        freeaddrinfo(currentInfo)  //addrinfo has a freshness value.
        assert(currentInfo == nil) //check to make sure that actually nil'd it too. 
        try writeRequest(with: openSocket!, to: defaultServer.host, at: "/")
      } catch let myError {
        print("Error info: \(myError.describe)")
        //TODO: close socket if a socket related error? 
      }
    }

Notice my request type is a ContiguousArray<CChar> and I’m still using that withContiguousStorageIfAvailable closure like in the first pass at the client. C Strings are very different than Swift Strings and trying to pass a non-contiguous storage type (Swift String) to a contiguous storage type (C String) will end up with all sorts of bad data reads, etc. Very. Unsafe.

  private func writeRequest(with socket: SocketHandle, to host: String, at path: String)
    throws(HTTPClientError)
  {
    //DON'T FORGET THE CLOSING \r\n
    let request: ContiguousArray<CChar> =
      "GET \(path) HTTP/1.0\r\nHost: \(host)\r\nUser-Agent: \(userAgent)\r\n\r\n".utf8CString
    print("request length: \(request.count)")
    print("\(request)")

    let result = request.withContiguousStorageIfAvailable { request_buffer in
      let writeResponse = write(socket, request_buffer.baseAddress, request_buffer.count)
      print("writeResponse: \(writeResponse)")
      return writeResponse
    }

    if result != nil && result! < 0 {
      close(socket)
      self.openSocket = nil
      throw HTTPClientError.sendFailed
    }

    if result == nil {
      print("what happened?")
      //close socket? 
      throw HTTPClientError.unsendableRequest
    }
  }

Listen

This time I’ll show the function first because the return type is important, its a [UInt8] that DOES NOT null terminate the buffer. If this code was C the buffer would be a char* with null termination. char* becomes [CChar] which comes into Swift as a [Int8]. At the moment, far more String initializers take a [UInt8] than can accommodate the other two, including the initializer to the ISOLatin1String type in HTTPTypes.

If I was going to keep response buffer stored in a [CChar] I would have null terminated it to be consistent with how C stores strings. In fact the line to do so is commented out in the code below.

  private func listenForResponse() throws(HTTPClientError) -> [UInt8] {
    var message: [UInt8] = []
    var buffer: [UInt8] = Array(repeating: 0, count: 512)

    guard let openSocket else {
      throw HTTPClientError.noSocketOpen
    }
    let _ = buffer.withContiguousMutableStorageIfAvailable { buffer in
      var r: CInt
      repeat {
        r = read(openSocket, buffer.baseAddress, buffer.count - 1)
        print("bytes read: \(r)")
        //this buffer could be holding data from a previous read with
        //a longer r. Only copy r bytes. 
        message.append(contentsOf: buffer.prefix(Int(r)))
      } while r > 0
    }

    //done so close the socket. 
    close(openSocket)
    self.openSocket = nil

    //DO NOT NULL TERMINATE IF NOT USING CChar
    //message.append(0)

    return message
  }

Decisions about the byte type is null termination change what in String initializers are available. At the call site I’ve chosen to use String(validating: response, as: UTF8.self), which will return nil if the bytes aren’t all UTF8 codable characters.

Since we have a UInt8 array, another choice would be String(decoding: response, as: UTF8.self) which will replace invalid data with "\u{FFFD}" (official replacement character of the specials), but always return something.

There are MANY String initializers, and MANY of them in Swift 6 are deprecated in favor of the above 2, so those will be the ones I’ll lean on the most in future code.

Neither of the remaining two accept null terminated strings, and it seems like all the CString acceptors have been deprecated, so clipping off the 0 and passing it along seems like the best choice. But only to validating because decoding won’t work with a [Char]. But there is maybe one remaining static function? Anyway, this all seems a bit in flux.

//the one remaining static function. 
let message7 = String.decodeCString(response as! [UTF8.CodeUnit], as: UTF8.self, repairingInvalidCodeUnits: true)

The final fetch function:

  public func fetch(_ path: String) {
    fetch(path, from: defaultServer)
  }

    public func fetch(_ path: String, from server: ServerInfo) {
    do {
      print("resolving...")
      try getAddressInfo(for: server.host, using: server.port)
      print("connecting...")
      openSocket = try connectSocket()
      freeaddrinfo(currentInfo) 
      assert(currentInfo == nil)
      print("currentInfo is nil: \(currentInfo == nil)")
      print("writing...")
      try writeRequest(with: openSocket!, to: defaultServer.host, at: path)

      print("listening...")
      let response = try getResponse()
      //print("response count: \(response.count)")
      //for the future. 
      //let message = ISOLatin1String(response) 

      if let message = String(validating: response, as: UTF8.self) {
        print("message was valid UTF8")
        print(message)
      } else {
        print("message could not be decoded as UTF8")
      }
      
      print("done")

    } catch let myError {
      print("Error info: \(myError.describe)")
      if let openSocket {
        close(openSocket)
      }
      if let currentInfo {
        freeaddrinfo(currentInfo)
      }
    }
  }

Running this code on the Xiao should spit out the full example.com home page!

Summary

So still a lot left to do. Even on top of all I want in that TODO list at the top, a bare minimum is to do enough parsing to tell what the status code is.

That said, before I do that, I’ll probably take a look at the momentary button code so I can do this HTTP call “onPress” instead of on a timer.

Can I make my ESP32C6 HTTP client use HTTPTypes?

Tue, 23 Sep 2025 00:00:00 +0000

So when we left off I had gotten the ESP32C6 to touch the internet using a very thin wrapper over C.

The work for the next two posts can be seen on the e_ImproveHTTPClient branch.

Reminder, in the terminal window, set up the environment if not using the VSCode extension.

. $HOME/esp/esp-idf/export.sh
cd $PROJECT_FOLDER
idf.py set-target esp32c6
# export SOME_PORT=`ls /dev/cu.usbmodem*`
# echo $SOME_PORT
idf.py build

I thought it would be an interesting exercise to see what it would take to bring the code from the HTTPTypes library into an embedded project.

I copied all the files from the core folder into a folder in my project and added them by hand to the CMake sources by hand:

# List of Swift source files to build.
target_sources(${COMPONENT_LIB}
    PRIVATE
    Main.swift
    Bool_Int.swift
    GPIOPin.swift
    DigitalIndicator.swift
    MomentaryInput.swift
    Delay.swift     
    ErrorHandler.swift
    WiFi.swift
    HTTP.swift
    HTTPRevisedTypes/HTTPField.swift
    HTTPRevisedTypes/HTTPFieldName.swift
    HTTPRevisedTypes/HTTPFields.swift
    HTTPRevisedTypes/HTTPParsedFields.swift
    HTTPRevisedTypes/HTTPRequest.swift
    HTTPRevisedTypes/HTTPResponse.swift
    HTTPRevisedTypes/ISOLatin1String.swift
    HTTPRevisedTypes/NIOLock.swift
)

I then got to work fixing all the compiler errors. A lot of these are covered in the Swift Embedded Examples Docs as caveats already.

Starting Easy

I’ve already been incredibly impressed how the esp-idf and Swift Embedded work together giving really nice String and print access with absolutely no work on my part, which is crazy. I’m not sure if people not coming from hardware development get how nice that is.

So it’s not a surprise to me that I needed to strip out the following string mashing related features:

conformances to Codable
conformances to CustomPlaygroundDisplayConvertible
all .description (only reaming one was in CustomDebugStringConvertible)

I read that someone had gotten Codable working maybe? but its pretty slow. I ripped that all out without trying to get them to work.

No KeyPaths

Also an easy fix was changing the map functions that use a KeyPath to actual closures. In HTTPFields.swift

    public subscript(values name: HTTPField.Name) -> [String] {
        get {
            self.fields(for: name).map(\.value)
        }
        set {
            self.setFields(newValue.lazy.map { HTTPField(name: name, value: $0) }, for: name)
        }
    }

becomes

    public subscript(values name: HTTPField.Name) -> [String] {
        get {
            self.fields(for: name).map({$0.value})
        }
        set {
            self.setFields(newValue.lazy.map { HTTPField(name: name, value: $0) }, for: name)
        }
    }

Clarifying Error Types

HTTPParsedFields has a lot of throwing functions. They all need to be given an explicit Error type because any Error, as an Existential is not allowed in embedded.

From:

    private func validateFields() throws {
        guard self.fields[values: .contentLength].allElementsSame else {
            throw ParsingError.multipleContentLength
        }
        guard self.fields[values: .contentDisposition].allElementsSame else {
            throw ParsingError.multipleContentDisposition
        }
        guard self.fields[values: .location].allElementsSame else {
            throw ParsingError.multipleLocation
        }
    }

//...

extension HTTPRequest {
    public init(parsed fields: [HTTPField]) throws {
        var parsedFields = HTTPParsedFields()
        for field in fields {
            try parsedFields.add(field: field)
        }
        self = try parsedFields.request
    }
}

To:

    private func validateFields() throws(ParsingError) {
        guard self.fields[values: .contentLength].allElementsSame else {
            throw ParsingError.multipleContentLength
        }
        guard self.fields[values: .contentDisposition].allElementsSame else {
            throw ParsingError.multipleContentDisposition
        }
        guard self.fields[values: .location].allElementsSame else {
            throw ParsingError.multipleLocation
        }
    }

//...

extension HTTPRequest {
    public init(parsed fields: [HTTPField]) throws(HTTPParsedFields.ParsingError) {
        var parsedFields = HTTPParsedFields()
        for field in fields {
            try parsedFields.add(field: field)
        }
        self = try parsedFields.request
    }
}

The Doozey: HTTPFields underlying storage.

So the biggest fanciest thing in HTTPTypes is the different locks but on the field storage to make them concurrency safe.

There is a base class called _Storage that depending on the compiler information will pick one of two child classes: _StorageWithMutex or _StorageWithNIOLock

Swift Embedded with the esp-idf version I’m using has no Mutex and NIOLock hit the “unable to identify C library” error.

I could not add a child _StorageForEmbedded because the parent a template for the two children to implement their competing lock strategies.

    func withLock<Result>(_ body: () throws -> Result) rethrows -> Result {
            fatalError()
    }

swift:35:14: error: classes cannot have non-final generic functions in embedded Swift
https://docs.swift.org/embedded/documentation/embedded/nonfinalgenericmethods

So options include:

I could potentially write some kind of lock system using pthread and it is entirely possible that the espressif sdk already thought of that and I could just steal it. (They’ve got an event system after all.)
Then I could use it to write a child class of _Storage that uses it…
But then I’d still end up with the non-final generic problem
and there wouldn’t be a great way to pass it in (because for every new piece of hardware new locking storage would have to be provided based on a compiler check of some sort which would make the library pretty hard to maintain.)

So in the mean time I just ripped out all the locking. In my demo that should be fine.

So Much STRIIIIIIINNNNGGGG.

So HTTPTypes is essentially a String handing library: comparing strings, parsing strings, inspecting strings. So even when getting all that done there’s one last library that needs to be explicitly added to prevent errors about missing _swift_stdlib types in during a build.

/Users/$USER/.espressif/tools/riscv32-esp-elf/esp-14.2.0_20241119/riscv32-esp-elf/bin/../lib/gcc/riscv32-esp-elf/14.2.0/../../../../riscv32-esp-elf/bin/ld: main.elf: hidden symbol `_swift_stdlib_getNormData' isn't defined

Looking at the Strings page in the Embedded Swift docs, right there in the list is getNormData.

HTTPRevisedTypes wanted

getNormData
getComposition
getDecompositionEntry
nfd_decompositions

All of those can be found by inspecting the suggested library with strings piped to less

cd `swiftly use --print-location`
# update to be the target arch you're looking for
cd usr/lib/swift/embedded/riscv32-none-none-eabi
strings libswiftUnicodeDataTables.a | less
# inside less type
# /$SEARCH_ITEM
# to find the text, just q to exit.

This is some serious special sauce and it can be added to the project via CMake with

set(COMPILER_TARGET "riscv32-none-none-eabi")

find_program(SWIFTLY "swiftly")
IF(SWIFTLY)
  execute_process(COMMAND swiftly use --print-location OUTPUT_VARIABLE toolchain_path OUTPUT_STRIP_TRAILING_WHITESPACE)
  cmake_path(SET additional_lib_path NORMALIZE "${toolchain_path}/usr/lib/swift/embedded/${COMPILER_TARGET}")
ELSE()
  get_filename_component(compiler_bin_dir ${CMAKE_Swift_COMPILER} DIRECTORY)
  cmake_path(SET additional_lib_path NORMALIZE "${compiler_bin_dir}/../lib/swift/embedded/${COMPILER_TARGET}")
ENDIF()

target_link_directories(${COMPONENT_LIB} PUBLIC "${additional_lib_path}")
target_link_libraries(${COMPONENT_LIB}
    -Wl,--whole-archive
    swiftUnicodeDataTables
    -Wl,--no-whole-archive
    )

## debug message
message(" ----------::: ${additional_lib_path} :::---------- ")

Minimal Test

So… it compiles now, but can it see the types?

Adding a very minimal test

HTTP.swift

final class HTTPClient {

    func test() {
        let request = HTTPRequest(method: .get, scheme: "https", authority: "www.example.com", path: "/")
        print(request.scheme ?? "no scheme")
    }

    //....
}

Main.swift

  //Waiting for wifi to connect...
  delay(2000);
  let client = HTTPClient()
  client.getAndPrint(from: "example.com", route: "/")
  client.test()

And that at least worked?

Can HTTPRevisedTypes be a Library?

I have aspirations to be able to work with both the espressif sdk and the pico-sdk better than I can now. Specifically, I want to be able to compartmentalize code into libraries via CMake since both of those SDKs use it. I can when writing my own simple examples, but not when meshing with C based SDK’s.

As it is I tried a few different ways to change the HTTPRevisedTypes files into a included library, all of which failed for various reasons related to my general CMake weaknesses, not understanding some of what the esp-idf is doing under the hood to piece projects together, and Swift embedded’s relying on wholemodule mode, or maybe even the version of CMake that esp-idf 5.5 uses? I’m a few steps too low on the ladder to reach it yet.

I bought a book, but here is some additional reading to do.

It seems like there is a path forward treating HTTPRevisedTypes as a component, but I’m not confident how that will interact with the Swift requirements and that keeps it chained to the esp-idf. In the future I might give it a shot since it can be done as pure CMake?

In the mean time I want to focus on the client API, which could eventually use the request type under the hood.

How do I make a OpenAPI ClownAPI for Hummingbird?

Mon, 22 Sep 2025 00:00:00 +0000

So now to expand out the full Clown CRUD and the updated tests. This post doesn’t review everything that can be seen in the repo example, but aspects of what it took to get there.

What Response Codes should I use?

When designing an API, follow code conventions! But WHICH code conventions. I’ll swing back around to this topic again perhaps. I’m bringing it up now mostly to say that I didn’t focus on picking the perfect codes for this API. I was focused on understanding how the OpenAPI spec and the generated code work, not writing the perfect API with it. Here are some references for future me when I do care.

Some links

Using the Server Section

Added to my example is a server section:

servers:
  - url: http://localhost:{port}{basePath}
    description: Local Testing
    variables:
      port:
        enum:
          - "8080"
          - "2525"
        default: "8080"
      basePath:
        default: /api/v0/clowns

I haven’t yet configured it to work with multiple servers, but it means that I can query the API for what the endpoint base route should be. In this example I chose to fail gracefully if the API endpoints didn’t register. That makes more sense when the server is handling lots of types of routes.

    let repository = ClownCar()
    let clown_api = ClownAPIHandler(repository: repository)
    do {
        print(try Servers.Server1.url())
        let url = try Servers.Server1.url()
        // like a addRote
        try clown_api.registerHandlers(on: router, serverURL: url) 
  
    } catch {
        //handle this situation. 
        print("hello api failed to register. Those endpoints will not be available.")
    }

I added that same base route as static var to test against in my tests.

  @Suite("ClownAPITests") struct ClownAPITests {
    static let APIBase = "/api/v0/clowns"
    //...
  }

Schema

Open API can really nail down what the acceptable data should be for any type of request or response by specifying a by specifying a schema object. Here are two simple examples with two different schemas, one for a plain text response, and one that shows an extremely basic JSON response.

https://swagger.io/specification/#schema-object

/hello:
    get:
      operationId: greet
      responses:
        '200':
          description: A hello message as a string
          content:
            text/plain:
              schema:
                type: string
  /helloworld:
    get:
      operationId: greetFormally
      responses:
        "200": # When the request is successful
          description: A JSON formatted hello message
          content:
            application/json:
              schema: # The returned JSON object
                type: object
                properties:
                  message:
                    type: string
                required:
                  - message

Once again showing how to write this call the long way:

    func greet(_ input: Operations.Greet.Input) async throws -> Operations.Greet.Output {
        //same as:
        //return .ok(.init(body: .plainText("Hello!")))
        let message = "Hello"
        let httpBody = OpenAPIRuntime.HTTPBody(message.utf8)
        let OkBody = Operations.Greet.Output.Ok.Body.plainText(httpBody)
        let greetOk = Operations.Greet.Output.Ok(body: OkBody)
        let greetOutput = Operations.Greet.Output.ok(greetOk)
        return greetOutput
    }


    func greetFormally(_ input: Operations.GreetFormally.Input) async throws -> Operations.GreetFormally.Output {
        //same as:
        // return .ok(.init(body:
        //    .json(.init(
        //        message: "Hello, world!"
        //    ))
        //))
        let jsonPayload = Operations.GreetFormally.Output.Ok.Body.JsonPayload(message: "Hello, world!")
        let gfOkBody = Operations.GreetFormally.Output.Ok.Body.json(jsonPayload)
        let gfOk = Operations.GreetFormally.Output.Ok(body: gfOkBody)
        let gfOutput = Operations.GreetFormally.Output.ok(gfOk)
        return greetFormallyOutput
    }

Referenced Schema

Schema objects can be referenced, even in external files!

Since I’m likely to be using Clown’s a lot, at the bottom of the YAML I can put:

components:
  schemas:
    #   let id: Int
    #   var name: String
    #   var spareNoses: Int
    Clown:
      type: object
      description: A single clown profile.
      properties:
        id:
          type: integer
        name:
          type: string
        spareNoses:
          type: integer
      required:
        - id
        - spareNoses
        - name
# other schemas...

The schema type lives inside the Types.swift as a Components.Schemas.Clown. I can make my life easier by adding an initializer to it that takes one of my local Clown types.

extension Components.Schemas.Clown {
    /// Maps a `Clown` to a `Components.Schemas.Clown`
    /// This makes it easier to send models to the API
    init(clown: Clown) {
        self.init(id: clown.id, name: clown.name, spareNoses: clown.spareNoses)
    }
}

Then I can make a route that depends on it:

  /test_clown:
    # GET /test_clown
    get:
      operationId: testClown
      responses:
        "200": # When the request is successful
          description: A basic clown profile.
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Clown"

My handler that needs to return that clown has a .json output type that expects an instance of Components.Schemas.Clown at its initialization.

    func testClown(_ input:Operations.TestClown.Input) async throws -> Operations.TestClown.Output {
        let clown = Clown(id: 144214, name:"Polka Dot" , spareNoses: 56)
        //see extension above.
        let myClown = Components.Schemas.Clown(clown: clown)
        return .ok(.init(body: .json(myClown)))
        // same as:
        //return .ok(.init(body: .json(.init(clown:clown))))
    }

More than one response type

Endpoints can have more than one response type, one per HTTP Response code.

A kind of weak example is the current state of the delete function. Based on various conditions it will either return the clown just deleted or a .badRequest type.

    delete:
      operationId: delete
      parameters:
        - name: id
          in: path
          description: Clown ID
          required: true
          schema:
            type: integer
      responses:
        "200":
          description: Successfully deleted the clown returned.
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Clown"
        "400":
          description: Was not able to delete the clown or clown already did not exist.
          content:
            text/plain:
              schema:
                type: string

    func delete(_ input: Operations.Delete.Input) async throws -> Operations.Delete.Output {
        let id = input.path.id

        if let clown = try await repository.get(id: input.path.id) {
            if try await self.repository.delete(id: id) {
                return .ok(.init(body: .json(.init(clown:clown))))
            } else {
                //Not. 
                return .badRequest(.init(body: .plainText("id unable to be deleted.")))
            }
        } else {
            //perhaps a security leak, but okay. 
            // this also may actually debatably be a success, in that this clown isn't in the DB any more. Consider a 204.
            return .badRequest(.init(body: .plainText("id already does not exist.")))
        } 
    }

For 4XX and 5XX responses, consider including a Problem JSON type for actual production schemas.

Undocumented Paths

The generator does give every handler an escape valve with .undocumented outputs.

    func greetFormally(_ input: Operations.GreetFormally.Input) async throws -> Operations.GreetFormally.Output {
        //undocumented(statusCode: Swift.Int, OpenAPIRuntime.UndocumentedPayload)
        let message = "Not right now thanks."
        let httpBody = OpenAPIRuntime.HTTPBody(message.utf8)
        return .undocumented(statusCode: 418, .init(headerFields: .init(dictionaryLiteral: []), body: httpBody))
    }

Use sparingly, if at all.

Using an Input Type

Inputs have generated types as well.

    post:
      operationId: create
      requestBody:
        description: a clown to create
        required: true
        content:
          application/json:
            schema:
              $ref: "#/components/schemas/ClownCreateRequest"
      responses:
        "200":
          description: Successful creation with the clown returned.
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Clown"
        "201":
          description: Successful creation with the id number returned
          content:
            application/json:
              schema: # The returned JSON object
                type: object
                properties:
                  id:
                    type: integer
                required:
                  - id
        "204":
          description: Successful creation with empty body
        "400":
          description: Some bad request.

The clown create request isn’t a Clown, but what’s wanted to create a clown. Only the name is required.

    ClownCreateRequest:
      type: object
      description: Fields to try to create a clown from.
      properties:
        name:
          type: string
        spareNoses:
          type: integer
      required:
        - name

    func create(_ input:Operations.Create.Input) async throws -> Operations.Create.Output {
        print("CREATING!!!")
        //ClownCreateRequest
        
        let (name, suggestedNoses) = switch input.body {
            case .json(let clownInfo):
                (clownInfo.name, clownInfo.spareNoses)
        }
        print(name, suggestedNoses ?? 0)
        let clown = try await repository.create(name: name, spareNoses: suggestedNoses ?? 5)
        return .ok(.init(body: .json(.init(clown: clown))))
    }

Changes To The Tests

Most of the tests remain largely the same, but what to expect on the unhappy paths had to be updated.

My nil == 204 problem can gets “fixed” in OpenAPI because returning a Clown? just isn’t a thing one can do.

    func fetchByID(_ input:Operations.FetchByID.Input) async throws -> Operations.FetchByID.Output {
        if let clown = try await repository.get(id: input.path.id) {
            return .ok(.init(body: .json(.init(clown:clown))))
        } else {
            return .notFound(.init(body: .plainText("Was not able to retrieve that clown.")))
        }
    }

My one real beef with OpenAPI is how it handles Decoding errors.

Decoding errors get thrown as 500’s before they even hit the handler. I would argue that its more likely that the client sent a badly formatted request than there being an error in the decoder code. A 500 encourages the client to send the exact same request later with no changes.

Hopefully sometime later 2025 / early 2026 I’ll be able to be more helpful than just complainy pants about that.

In the mean time, the tests look for that expectation:

        let notAnID = "jieoGEJg"  //change to malformed
        let _ = try await client.execute(uri: "\(Self.APIBase)/\(notAnID)", method: .get) {
          response in
          // TODO: THIS IS BAD
          // EXAMPLE OF 500 PROBLEM!!!
          // #expect(response.status == .badRequest)
          #expect(response.status == .internalServerError)
        }

Summary

So I can see where codifying the API would be invaluable for production servers and for APIs that will have a lot of clients. Having generated code that doesn’t allow for mistakes, fantastic. The ability to share this API in a language agnostic way, also incredibly useful.

On the negative side, I felt like the really long type names from the generator encourages writing hard to read code. I also missed having the ability to intercept more of errors from within the handler.

There’s more to say about the generator and Hummingbird but the next few posts will be back to hardware!

How do I get Hummingbird to work with the OpenAPI plugins?

Fri, 19 Sep 2025 00:00:00 +0000

What is OpenAPI? Originally part of swagger in the their own words from the docs:

The OpenAPI Specification (OAS) provides a consistent means to carry information through each stage of the API lifecycle. It is a specification language for HTTP APIs that defines structure and syntax in a way that is not wedded to the programming language the API is created in. API specifications are typically written in YAML or JSON, allowing for easy sharing and consumption of the specification.

That YAML file might look something like

openapi: '3.1.0'
info:
  title: AppService
  version: 1.0.0
servers:
  - url: https://example.com/api
    description: App service deployment.
paths:
  /:
    get:
      operationId: getHello
      responses:
        '200':
          description: Hello!
          content:
            text/plain:
              schema:
                type: string

That document describes a service that offers one API endpoint “/”. It is expected to return a HTTP response with a status code of “200” and the content type of “text/plain” with the data schema of an unnamed string.

That YAML comes from the starter template for Hummingbird. Running the download script and following the prompts creates a package with all the necessary dependencies for use with Hummingbird already installed.

curl -L https://raw.githubusercontent.com/hummingbird-project/template/main/scripts/download.sh | bash

Some things to notice include that the YAML file I just showed lives in its own library. The premise being that that will change infrequently and as a dependency will be compiled first. This is important because the swift-openapi-generator creates code it puts straight in the build folder.

For example:

ls -al .build/plugins/outputs/helloopenapi/AppAPI/destination/OpenAPIGenerator/GeneratedSources
##yields: Client.swift	Server.swift	Types.swift

That glue code will make it possible to make compiler checked type constrained endpoint handlers.

Resources

Before going any further, it will be easier to follow along after at least skimming some of the resources below:

What’s in The Basic Example

After creating the basic example I changed the name of my server app to “circusServer” and the API to the “ClownAPI”. I change names like that to try to pull apart what’s a defined term and what’s a convention in the generated code.

Package.swift

I moved the tools up to 6.2 and everything still works.

// swift-tools-version:6.2
// The swift-tools-version declares the minimum version of Swift required to build this package.

import PackageDescription

let package = Package(
    name: "circusServer",
    platforms: [.macOS(.v14), .iOS(.v17), .tvOS(.v17)],
    products: [
        .executable(name: "circusServer", targets: ["circusServer"]),
    ],
    dependencies: [
        .package(url: "https://github.com/hummingbird-project/hummingbird.git", from: "2.0.0"),
        .package(url: "https://github.com/apple/swift-argument-parser.git", from: "1.3.0"),
        .package(url: "https://github.com/apple/swift-openapi-generator", from: "1.6.0"),
        .package(url: "https://github.com/apple/swift-openapi-runtime", from: "1.7.0"),
        .package(url: "https://github.com/swift-server/swift-openapi-hummingbird.git", from: "2.0.1"),
    ],
    targets: [
        .executableTarget(name: "circusServer",
            dependencies: [
                .product(name: "ArgumentParser", package: "swift-argument-parser"),
                .product(name: "Hummingbird", package: "hummingbird"),
                .byName(name: "ClownAPI"),
                .product(name: "OpenAPIRuntime", package: "swift-openapi-runtime"),
                .product(name: "OpenAPIHummingbird", package: "swift-openapi-hummingbird"),
            ],
            path: "Sources/Circus"
        ),
        .target(
            name: "ClownAPI",
            dependencies: [
                .product(name: "OpenAPIRuntime", package: "swift-openapi-runtime")
            ],
            path: "Sources/ClownAPI",
            plugins: [.plugin(name: "OpenAPIGenerator", package: "swift-openapi-generator")],
        ),
        .testTarget(name: "AppTests",
            dependencies: [
                .byName(name: "circusServer"),
                .product(name: "HummingbirdTesting", package: "hummingbird")
            ],
            path: "Tests/AppTests"
        )
    ]
)

ClownAPI

The clown API folder has 3 files

ClownAPI.swift: empty file to trigger the package to compile
openapi-generator-config.yaml: the file that tells the generator what types of files to generate
openapi.yaml: the file that says what rules the API should follow

I showed the contents of openapi.yaml above, but the generator config contains

generate:
  - types
  - server
accessModifier: package
namingStrategy: idiomatic

After running a build, cat‘ing out the contents of the Client.swift will reveal it to be an empty file because the config does not request a client.

## new folder after the name changes
ls -al .build/plugins/outputs/helloopenapi/ClownAPI/destination/OpenAPIGenerator/GeneratedSources
##yields: Client.swift	Server.swift	Types.swift
cat .build/plugins/outputs/helloopenapi/ClownAPI/destination/OpenAPIGenerator/GeneratedSources/Client.swift

Defining The Handlers

Back in the server, instead of a ClownController, this time there’s an APIImplementation.swift, which implements the APIProtocol placed by the generator in Types.swift. (The name APIImplementation can be changed and will be ClownAPIHandler in the finished demo.)

import ClownAPI
import OpenAPIRuntime

struct APIImplementation: APIProtocol {
    func getHello(_ input: ClownAPI.Operations.GetHello.Input) async throws -> ClownAPI.Operations.GetHello.Output {
        return .ok(.init(body: .plainText("Hello!")))
    }
}

Excerpt from Types.swift

/// A type that performs HTTP operations defined by the OpenAPI document.
package protocol APIProtocol: Sendable {
    /// - Remark: HTTP `GET /`.
    /// - Remark: Generated from `#/paths///get(getHello)`.
    func getHello(_ input: Operations.GetHello.Input) async throws -> Operations.GetHello.Output
}

But what’s the deal with those crazy input and output types? They get generated too, for each route there will be a Operations.$ROUTE_NAME that will hold the expected input types and allowed output types outlined by the openapi.yaml.

package static let id: Swift.String = "getHello"
        package struct Input: Sendable, Hashable {
            /// - Remark: Generated from `#/paths/GET/header`.
            package struct Headers: Sendable, Hashable {
                package var accept: [OpenAPIRuntime.AcceptHeaderContentType<Operations.GetHello.AcceptableContentType>]
                /// Creates a new `Headers`.
                ///
                /// - Parameters:
                ///   - accept:
                package init(accept: [OpenAPIRuntime.AcceptHeaderContentType<Operations.GetHello.AcceptableContentType>] = .defaultValues()) {
                    self.accept = accept
                }
            }
            package var headers: Operations.GetHello.Input.Headers
            /// Creates a new `Input`.
            ///
            /// - Parameters:
            ///   - headers:
            package init(headers: Operations.GetHello.Input.Headers = .init()) {
                self.headers = headers
            }
        }
        @frozen package enum Output: Sendable, Hashable {
            package struct Ok: Sendable, Hashable {
                /// - Remark: Generated from `#/paths/GET/responses/200/content`.
                @frozen package enum Body: Sendable, Hashable {
                    /// - Remark: Generated from `#/paths/GET/responses/200/content/text\/plain`.
                    case plainText(OpenAPIRuntime.HTTPBody)
                    /// The associated value of the enum case if `self` is `.plainText`.
                    ///
                    /// - Throws: An error if `self` is not `.plainText`.
                    /// - SeeAlso: `.plainText`.
                    package var plainText: OpenAPIRuntime.HTTPBody {
                        get throws {
                            switch self {
                            case let .plainText(body):
                                return body
                            }
                        }
                    }
                }
                /// Received HTTP response body
                package var body: Operations.GetHello.Output.Ok.Body
                /// Creates a new `Ok`.
                ///
                /// - Parameters:
                ///   - body: Received HTTP response body
                package init(body: Operations.GetHello.Output.Ok.Body) {
                    self.body = body
                }
            }
            /// Hello!
            ///
            /// - Remark: Generated from `#/paths///get(getHello)/responses/200`.
            ///
            /// HTTP response code: `200 ok`.
            case ok(Operations.GetHello.Output.Ok)
            /// The associated value of the enum case if `self` is `.ok`.
            ///
            /// - Throws: An error if `self` is not `.ok`.
            /// - SeeAlso: `.ok`.
            package var ok: Operations.GetHello.Output.Ok {
                get throws {
                    switch self {
                    case let .ok(response):
                        return response
                    default:
                        try throwUnexpectedResponseStatus(
                            expectedStatus: "ok",
                            response: self
                        )
                    }
                }
            }
            /// Undocumented response.
            ///
            /// A response with a code that is not documented in the OpenAPI document.
            case undocumented(statusCode: Swift.Int, OpenAPIRuntime.UndocumentedPayload)
        }

These types are enums within enums, so looking at the the getHello example above, we have the Output enum with the case ok taking in a Body enum with its single allowable case of .plainText that has an associated type of OpenAPIRuntime.HTTPBody. OpenAPIRuntime.HTTPBody has several convenience initializers, but none of them actually take a String. What we’re seeing in the example is a StaticString, which behaves differently. A string can be used, it just needs to be transformed into its UInt8 bytes first.

// Works
func getHello(_ input: ClownAPI.Operations.GetHello.Input) async throws -> ClownAPI.Operations.GetHello.Output {
    return .ok(.init(body: .plainText("Hello!")))
}

// Does not work. 
func getHello(_ input: Operations.GetHello.Input) async throws -> Operations.GetHello.Output {
    // 1.
    let message = "Hello!"
    //Cannot convert value of type 'String' to expected argument type 'HTTPBody'
    return .ok(.init(body: .plainText(message)))
}

// Does work
func getHello(_ input: Operations.GetHello.Input) async throws -> Operations.GetHello.Output {
    let message = "Hello"
    return .ok(.init(body: .plainText(.init(message.utf8))))
}

That can be hard to follow with all those .inits. Here is that last call written out the super long way:

func getHello(_ input: Operations.GetHello.Input) async throws -> Operations.GetHello.Output {
    //return .ok(.init(body: .plainText("Hello!")))
    let message = "Hello"
    let bytes:[UInt8] = Array(message.utf8) 
    let httpBody = OpenAPIRuntime.HTTPBody(bytes)
    let okBody = Operations.GetHello.Output.Ok.Body.plainText(httpBody)
    let greetOk = Operations.GetHello.Output.Ok(body: okBody)
    let greetOutput = Operations.GetHello.Output.ok(greetOk)
    return greetOutput
}

Adding the Routes

To get those functions registered with the Hummingbird router, 3 things have to happen in Application+build.swift

The OpenAPIHummingbird glue library needs to be imported
Optionally the OpenAPIRequestContextMiddleware() generated by the template has to be added to the Middleware Group. It’s needed if a Hummingbird context will be used in an OpenAPIRuntime endpoint handler. It does this by storing the Context in TaskLocal (more, reading)
The Routes need to be added! The new packages (generator, runtime, hummingbird-openapi) do some fancy magic under the hood with conjuring up the proper registerHandlers function for the desired server underpinning. Yet another possible post / set of posts…
- https://github.com/apple/swift-openapi-generator/blob/main/Sources/_OpenAPIGeneratorCore/Translator/ServerTranslator/ServerTranslator.swift#L75

//# 1 Add the Glue
import OpenAPIHummingbird

/// Build router
func buildRouter() throws -> Router<AppRequestContext> {
    let router = Router(context: AppRequestContext.self)
    // Add middleware
    router.addMiddleware {
        // logging middleware
        LogRequestsMiddleware(.info)
        //# 2: store request context in TaskLocal
        OpenAPIRequestContextMiddleware()
    }
    // # 3 Add OpenAPI handlers
    let api = APIImplementation()
    try api.registerHandlers(on: router)
    return router
}

Summary

That’s all the pieces in the starter OpenAPI Hummingbird template combo before starting to add in all the CRUD implementation details.

How do I use Hummingbird to make a basic CRUD API?

Thu, 18 Sep 2025 00:00:00 +0000

So my recommendation is to also go through Hummingbird TODO’s example, because that’s what I did to learn how to write this code. The main difference between that tutorial’s early steps and what I’ve done here is that I’ve uses swift-testing and that example still uses XCTests (at date published).

That tutorial goes on to add persistence through a postgres database, which I haven’t here. I’ll mention some options beside postgres in the Repository section

Hummingbird makes it incredibly easy to return JSON for Encodables.

But to do the standard crud functions on a collection of that type, there are some things to do!

Have a type
Make a store/repository to manage the collection of that type.
Make a RouteCollection, or really “Controller” that can offer one up.
WRITE TESTS

This code is available as a swift Package at:

https://github.com/carlynorama/HummingbirdExamples/tree/main/05_helloCRUD

The Type

My type is a Clown, and it conforms to quite a few things.

public struct Clown: Decodable, ResponseEncodable, Sendable, Equatable {
  let id: Int
  var name: String
  var spareNoses: Int
}

Decodable: So the default Hummingbird context make one from JSON that comes in
ResponseEncodable: So the default Hummingbird context can make the JSON and wrap it in a response
Sendable: so the sendable endpoint handlers can work with them and send them both out via JSON and to the actor based repository.
Equatable: I believe I added this for testing
Not Yet: Comparable. But go ahead and add it for sorting list views more conveniently if desired.

The Repository

In Hummingbird examples the data store is generally referred to as a repository (although I called my repository ClownCar).

This type manages the collection of Clowns I’ll be editing. The API controller never touches the underlying storage itself, everything goes through a repository’s interface.

Right now my store is all in memory. Hopefully a future article will cover persistence between server runs, but I’ll list some of the options:

the TODO’s example shows one method using postgres,
the examples folder contain examples for other methods,
and the docs talk about the persistence driver.

actor ClownCar {
  var clowns: [Int: Clown]

  init() {
    clowns = [
      0: Clown(id: 0, name: "Pagliacci", spareNoses: 0),
      1: Clown(id: 1, name: "Joseph Grimaldi", spareNoses: 41417),
      4: Clown(id: 4, name: "Weary Willie", spareNoses: 1),
      //Clown(id: 224111, name:"Bozo" , spareNoses: 4141),
    ]
  }

  // CREATE

  /// create a clown.
  func create(name: String, spareNoses: Int = 10) async throws -> Clown {
    let id = Int.random(in: 1000..<10000)
    let clown = Clown(id: id, name: name, spareNoses: spareNoses)
    self.clowns[id] = clown
    return clown
  }

  // READ

  /// Get clown
  func get(id: Int) async throws -> Clown? {
    return self.clowns[id]
  }
  /// List all clowns
  func list() async throws -> [Clown] {
    return self.clowns.values.map { $0 }
  }

  //UPDATE

  /// Update clown. Returns updated clown if successful
  // note all the optional fields...
  func update(id: Int, name: String?, spareNoses: Int?) async throws -> Clown? {
    if var clown = self.clowns[id] {
      if let name {
        clown.name = name
      }
      if let spareNoses {
        clown.spareNoses = spareNoses
      }
      self.clowns[id] = clown
      return clown
    }
    return nil
  }

  //DELETE

  /// Delete clown. Returns true if successful
  func delete(id: Int) async throws -> Bool {
    if self.clowns[id] != nil {
      self.clowns[id] = nil
      return true
    }
    return false
  }
  /// Delete all clowns
  func deleteAll() async throws {
    self.clowns = [:]
  }

}

The Controller & The Tests

This CRUD needs to be linked up to the API, so lets make a Controller type to glue the model to the data output “views.”

This Controller has three main jobs,

To hold a repository instance to store the data (a ClownCar)
To provide a variable that can serve up a RouteCollection (more docs), specified to the appropriate context. (Also read about RouterGroup)
To house the api endpoint handlers listed in the RouteCollection ((request, context) -> ResponseGenerator types)

NOTE: With so many types being generic with respect to the Apps base context type its easier to change an apps context by providing a type alias.

import Hummingbird

#if canImport(FoundationEssentials)
  import FoundationEssentials
#else
  import Foundation
#endif

public struct ClownController: Sendable {
  // clown repository
  let repository: ClownCar

  // return clown endpoints
  var endpoints: RouteCollection<AppRequestContext> {
    let routes = RouteCollection(context: AppRequestContext.self)
    //routes.group(context: FlexContext.self)
    routes.group()
      .get(":id", use: get)
      .get(use: list)
      .post(use: create)
      .patch(":id", use: update)
      .patch("{id}/honk", use: noseDecrement)
      .delete(":id", use: delete)
      .delete(use: deleteAll)
    return routes
  }

  //implementations here...

}

Some of the handler functions have helper structs modeling the data that a client would be actually expected to include with the request, which may not be the same as the actual base type (Clown) the repository stores.

Each one of these can be tested with curl, but it’s easier in the long run to write the test with it at the same time, so I’ve paired those together below.

Do run through the tutorial’s testing section, but one can use my file as a base to switch to swift-testing. That tutorial is what suggest writing helper static functions at the top which is indeed very helpful.

https://github.com/carlynorama/HummingbirdExamples/blob/main/05_crud/helloCRUD/Tests/AppTests/ClownAPITests.swift

I’ve wrapped the clown API tests in their own Suite, but the tests could be broken up further by tag (create, read, update, delete?) or other traits.

https://developer.apple.com/documentation/testing/addingtags
more about testing traits: https://developer.apple.com/videos/play/wwdc2024/10179?time=360

extension AppTests {
  @Suite("Tests to test the Clown's JSON API") struct ClownAPITests {
      //your tests here
  }
}

Also note in several places the tests look for .noContent instead of .notFound, this is because if a Clown? returns nil, Hummingbird’s default response is to return .noContent. No content is typically a success with no body, which it’s entirely possible for that nil to mean in some contexts. However, in my case that is the wrong response and if I stick with the Hummingbird handlers I will have to do something other than return a raw Clown?. Perfect future use of a Result? TBD.

Create

Controller

  struct CreateRequest: Decodable {
    let name: String
    let spareNoses: Int?
  }

  /// Create clown endpoint
  @Sendable func create(request: Request, context: some RequestContext) async throws
    -> EditedResponse<Clown>
  {
    let request = try await request.decode(as: CreateRequest.self, context: context)

    let clown = try await self.repository.create(
      name: request.name, spareNoses: request.spareNoses ?? 10)
    return EditedResponse(status: .created, response: clown)
  }

Tests

    struct CreateRequest: Encodable {
      let name: String
      let spareNoses: Int?
    }

    static func create(name: String, spareNoses: Int? = nil, client: some TestClientProtocol)
      async throws -> Clown
    {
      let request = CreateRequest(name: name, spareNoses: spareNoses)
      let buffer = try JSONEncoder().encodeAsByteBuffer(request, allocator: ByteBufferAllocator())
      return try await client.execute(uri: "/clowns", method: .post, body: buffer) { response in
        #expect(response.status == .created)
        return try JSONDecoder().decode(Clown.self, from: response.body)
      }
    }

    static func get(id: Int, client: some TestClientProtocol) async throws -> Clown? {
      try await client.execute(uri: "/clowns/\(id)", method: .get) { response in
        // either the get request returned an 200 status or it didn't return a Clown
        #expect(response.status == .ok || response.body.readableBytes == 0)
        if response.body.readableBytes > 0 {
          return try JSONDecoder().decode(Clown.self, from: response.body)
        } else {
          return nil
        }
      }
    }

    @Test func testCreate() async throws {
      let app = try await buildApplication(TestArguments())

      try await app.test(.router) { client in
        let clownName = "Bozo"
        let noseCount = 17
        let clown = try await Self.create(name: clownName, spareNoses: noseCount, client: client)
        #expect(clown.name == clownName)
        #expect(clown.spareNoses == noseCount)

        let retrievedClown = try await Self.get(id: clown.id, client: client)
        #expect(clown == retrievedClown)
      }
    }

Read

Controller

    /// Get clown endpoint
  @Sendable func get(request: Request, context: some RequestContext) async throws -> Clown? {
    let id = try context.parameters.require("id", as: Int.self)
    return try await self.repository.get(id: id)
  }

  /// Get list of clowns endpoint
  @Sendable func list(request: Request, context: some RequestContext) async throws -> [Clown] {
    return try await self.repository.list()
  }

Tests

    // used in Create, too
    static func get(id: Int, client: some TestClientProtocol) async throws -> Clown? {
    //...
    }

    static func list(client: some TestClientProtocol) async throws -> [Clown] {
      try await client.execute(uri: "/clowns", method: .get) { response in
        #expect(response.status == .ok)
        return try JSONDecoder().decode([Clown].self, from: response.body)
      }
    }


    @Test func testClownFetch() async throws {
      let app = try await buildApplication(TestArguments())

      try await app.test(.router) { client in

        let goodID = 1  //start with known good.
        let _ = try await client.execute(uri: "/clowns/\(goodID)", method: .get) { response in
          #expect(response.status == .ok)

          #expect(response.headers.contains(.contentType))
          let contentType = response.headers[.contentType]!
          #expect(contentType == "application/json; charset=utf-8")

          let clown = try JSONDecoder().decode(Clown.self, from: response.body)
          //print(clown)
          //In "real" app would pull this directly from the db using same ID.
          let compareTo = Clown(id: 1, name: "Joseph Grimaldi", spareNoses: 12)
          #expect(clown.name == compareTo.name)
        }

        let badID = 53_253_622  //change to known unused value
        let _ = try await client.execute(uri: "/clowns/\(badID)", method: .get) { response in
          //bad interpretation of a nil
          #expect(response.status == .noContent)
        }

        let notAnID = "jieoGEJg"  //change to malformed
        let _ = try await client.execute(uri: "/clowns/\(notAnID)", method: .get) { response in
          #expect(response.status == .badRequest)
        }
      }
    }

    @Test func testClownsFetch() async throws {
      let app = try await buildApplication(TestArguments())

      try await app.test(.router) { client in
        //this is weak. If the test data, not currently managed by the test suite,
        //changes, these test could fail based on that.
        let goodClown = Clown(id: 1, name: "Joseph Grimaldi", spareNoses: 41417)
        let badClown = Clown(id: -1, name: "Pennywise", spareNoses: 86_428_916_419)
        let _ = try await client.execute(uri: "/clowns/", method: .get) { response in
          #expect(response.status == .ok)

          #expect(response.headers.contains(.contentType))
          let contentType = response.headers[.contentType]!
          #expect(contentType == "application/json; charset=utf-8")

          let clowns = try JSONDecoder().decode([Clown].self, from: response.body)
          #expect(clowns.contains(where: { $0 == goodClown }))
          #expect(!clowns.contains(where: { $0 == badClown }))
        }
      }
    }

Update

This update example highlights that the closure can pull data out of multiple places, the URL and the various fields of the Request.

let id = try context.parameters.require("id", as: Int.self)

pulls the id number out of the URL.

let requestValue = try await request.decode(as: UpdateRequest.self, context: context)

Uses the request’s decode function in conjunction with decoder from the context to try to get the JSON data in the request body to match the specified UpdateRequest

Controller

  struct UpdateRequest: Decodable {
    let name: String?
    let spareNoses: Int?
  }
  /// Update clown endpoint
  @Sendable func update(request: Request, context: some RequestContext) async throws -> Clown? {
    let id = try context.parameters.require("id", as: Int.self)
    let requestValue = try await request.decode(as: UpdateRequest.self, context: context)
    guard
      let clown = try await self.repository.update(
        id: id,
        name: requestValue.name,
        spareNoses: requestValue.spareNoses
      )
    else {
      throw HTTPError(.badRequest)
    }
    return clown
  }

  //custom update function
  @Sendable func noseDecrement(request: Request, context: some RequestContext) async throws
    -> Clown?
  {
    let honkRequestID = try context.parameters.require("id", as: Int.self)
    if let beforeClown = try await self.repository.get(id: honkRequestID) {
      return try await self.repository.update(
        id: honkRequestID, name: nil, spareNoses: beforeClown.spareNoses - 1)
    }
    return nil
  }

Tests


    //also uses the Self.get 

    struct UpdateRequest: Encodable {
      let name: String?
      let spareNoses: Int?
      let completed: Bool?
    }

    static func patch(
      id: Int, name: String? = nil, spareNoses: Int? = nil, completed: Bool? = nil,
      client: some TestClientProtocol
    ) async throws -> Clown? {
      let request = UpdateRequest(name: name, spareNoses: spareNoses, completed: completed)
      let buffer = try JSONEncoder().encodeAsByteBuffer(request, allocator: ByteBufferAllocator())
      return try await client.execute(uri: "/clowns/\(id)", method: .patch, body: buffer) {
        response in
        #expect(response.status == .ok, "\(response.status)")
        if response.body.readableBytes > 0 {
          return try JSONDecoder().decode(Clown.self, from: response.body)
        } else {
          return nil
        }
      }
    }
    //
    //    //for testing uuid is just a string so can pass garbage later.
    static func patchResponseStatus(
      id: String, name: String? = nil, spareNoses: Int? = nil, completed: Bool? = nil,
      client: some TestClientProtocol
    ) async throws -> HTTPResponse.Status {
      let request = UpdateRequest(name: name, spareNoses: spareNoses, completed: completed)
      let buffer = try JSONEncoder().encodeAsByteBuffer(request, allocator: ByteBufferAllocator())
      return try await client.execute(uri: "/clowns/\(id)", method: .patch, body: buffer) {
        response in
        response.status
      }
    }

    @Test func testUpdateBoth() async throws {
      let app = try await buildApplication(TestArguments())

      try await app.test(.router) { client in
        let knownGoodID = 4
        let newName = "Bozo Jr."
        let newNoseCount = 18
        let beforeClown = try await Self.get(id: knownGoodID, client: client)
        let clown = try await Self.patch(
          id: knownGoodID, name: newName, spareNoses: newNoseCount, client: client)
        #expect(clown != nil)
        #expect(clown!.name == newName)
        #expect(clown!.spareNoses == newNoseCount)
        #expect(clown!.name != beforeClown?.name)
        #expect(clown!.spareNoses != beforeClown?.spareNoses)
        #expect(clown!.id == beforeClown?.id)

        let retrievedClown = try await Self.get(id: clown!.id, client: client)
        #expect(clown == retrievedClown)
        let retrievedOrigIDClown = try await Self.get(id: knownGoodID, client: client)
        #expect(retrievedClown == retrievedOrigIDClown)
      }
    }

    @Test func testUpdateBadId() async throws {
      let app = try await buildApplication(TestArguments())

      try await app.test(.router) { client in
        let knownBadID = 890_379_023
        let newName = "Bozo Jr."
        let newNoseCount = 18
        let _ = try await client.execute(uri: "/clowns/\(knownBadID)", method: .get) { response in
          //bad interpretation of a nil
          #expect(response.status == .noContent)
        }
        var updateStatus = try await Self.patchResponseStatus(
          id: "\(knownBadID)", name: newName, spareNoses: newNoseCount, client: client)
        #expect(updateStatus == .badRequest)

        updateStatus = try await Self.patchResponseStatus(
          id: "fgu8ik56gw3R)", name: newName, spareNoses: newNoseCount, client: client)
        #expect(updateStatus == .badRequest)

      }
    }

    @Test func testNoseDecrement() async throws {
        let app = try await buildApplication(TestArguments())

        try await app.test(.router) { client in
          let knownGoodID = 1
          let beforeClown = try await Self.get(id: knownGoodID, client: client)
          #expect(beforeClown != nil)
          let replyClown:Clown? = try await client.execute(uri: "clowns/\(knownGoodID)/honk", method: .patch) { response in 
            if response.body.readableBytes > 0 {
            //print(String(buffer: response.body))
            return try JSONDecoder().decode(Clown.self, from: response.body)
            } else {
            return nil
            }
        }
        
          #expect(replyClown != nil)
          #expect(replyClown!.name == beforeClown?.name )
          #expect(replyClown!.spareNoses != beforeClown?.spareNoses)
          #expect(replyClown!.spareNoses + 1 == beforeClown?.spareNoses)
          #expect(replyClown!.id == beforeClown?.id)
        }
    }

    @Test func testUpdateName() async throws {
      let app = try await buildApplication(TestArguments())

      try await app.test(.router) { client in
        let knownGoodID = 4
        let newName = "Bozo Jr."
        let beforeClown = try await Self.get(id: knownGoodID, client: client)
        let clown = try await Self.patch(id: knownGoodID, name: newName, client: client)
        #expect(clown != nil)
        #expect(clown!.name == newName)
        #expect(clown!.name != beforeClown?.name)
        #expect(clown!.spareNoses == beforeClown?.spareNoses)
        #expect(clown!.id == beforeClown?.id)

        let retrievedClown = try await Self.get(id: clown!.id, client: client)
        #expect(clown == retrievedClown)
        let retrievedOrigIDClown = try await Self.get(id: knownGoodID, client: client)
        #expect(retrievedClown == retrievedOrigIDClown)
      }
    }

Delete

Controller

  /// Delete clown endpoint
  @Sendable func delete(request: Request, context: some RequestContext) async throws
    -> HTTPResponse.Status
  {
    let id = try context.parameters.require("id", as: Int.self)
    if try await self.repository.delete(id: id) {
      return .ok
    } else {
      return .badRequest
    }
  }

  /// Delete all clowns endpoint
  @Sendable func deleteAll(request: Request, context: some RequestContext) async throws
    -> HTTPResponse.Status
  {
    try await self.repository.deleteAll()
    return .ok
  }

Tests

    // also uses Self.get

    static func delete(id: Int, client: some TestClientProtocol) async throws -> HTTPResponse.Status
    {
      try await client.execute(uri: "/clowns/\(id)", method: .delete) { response in
        response.status
      }
    }

    static func deleteAll(client: some TestClientProtocol) async throws -> HTTPResponse.Status {
      try await client.execute(uri: "/clowns", method: .delete) { response in
        response.status
      }
    }

    @Test func testDelete() async throws {
      let app = try await buildApplication(TestArguments())

      try await app.test(.router) { client in
        let knownGoodID = 4
        let beforeClown = try await Self.get(id: knownGoodID, client: client)
        #expect(beforeClown != nil)
        let status = try await Self.delete(id: knownGoodID, client: client)
        #expect(status == .ok)
        try await client.execute(uri: "/clowns/\(knownGoodID)", method: .get) { response in
          //bad interpretation of a nil
          #expect(response.status == .noContent)
        }
        try await client.execute(uri: "/clowns/\(knownGoodID)", method: .delete) { response in
          #expect(response.status == .badRequest)
        }
      }

    }

    @Test func testDeleteAll() async throws {
      let app = try await buildApplication(TestArguments())

      try await app.test(.router) { client in
        let knownGoodID = 4
        let beforeClown = try await Self.get(id: knownGoodID, client: client)
        #expect(beforeClown != nil)
        let status = try await Self.deleteAll(client: client)
        #expect(status == .ok)
        try await client.execute(uri: "/clowns/\(knownGoodID)", method: .get) { response in
          //bad interpretation of a nil
          #expect(response.status == .noContent)
        }
        try await client.execute(uri: "/clowns/\(knownGoodID)", method: .delete) { response in
          #expect(response.status == .badRequest)
        }

        let deletedClowns = try await Self.list(client: client)
        #expect(deletedClowns.isEmpty)
      }
    }

Left to do

I would rate this controller as “needs improvement”, especially on the error handling. Everywhere there’s a

throw HTTPError(.badRequest)

for example, there ought to be a message providing more data.

throw HTTPError(.badRequest, message: "this this you sent didn't make it through")

When a call like

let honkRequestID = try context.parameters.require("id", as: Int.self)

fails, Hummingbird will catch the error and wrap it in it’s own preferred HTTPStatus with an informative response body. I should add my own

When trying to match an API that doesn’t use the same response statuses as the Hummingbird defaults, it may mean catching that error in the closure and implementing your own Error -> HTTPResponse handling.

Also, where is the Authentication and Authorization layer? Can anyone delete anything??!!!

There is definitely more to do.

Summary

It takes a bit of fiddling to implement a CRUD the first time in Hummingbird, but I’ve written a couple of others already mostly by copy-pasting and searching for my Type name. Seems like a potential macro or something could be written, @CRUDable, ha! To do that correctly would pretty much be reimplementing SwiftData so perhaps not this week.

A way to make writing API code that’s very enforceable across projects is to use the OpenAPI plugin for Swift packages.

In the next post we’ll start the process of implementing the Clown JSON API with that.

How do I get Hummingbird to use Mustache templates instead?

Wed, 17 Sep 2025 00:00:00 +0000

As mentioned last post, there are lots of ways to generate HTML with Swift, if that’s how working with HTML works for you.

I’m perhaps a little old school and am more used to working with templates and template engines for web stuff. (Although I suppose there’s no reason not to write a command plugin to generate those templates for you.)

So I was pretty happy that the Hummingbird project already had a solution for that, the swift-mustache library. While it’s part of the Hummingbird Project ecosystem, it doesn’t require Hummingbird to use it.

This example and the hand-written HTML one are available at

https://github.com/carlynorama/HummingbirdExamples/tree/main/04_html_responses

Resources

There are also several examples in the official Hummingbird examples repo.

Package.swift & Directory Structure Changes

To work with swift-mustache, first add the dependency

## from dir with the Package.swift
## add the dependency
swift package add-dependency https://github.com/hummingbird-project/swift-mustache --from 2.0.0
## make the folder
mkdir -p Sources/Templates

The Package.swift will have to be updated by hand to add the dependency to the executable, and to create a Resources folder so the template files will be seen by the build process and be incorporated in the bundle.(Another option includes an absolute path when initializing the MustacheLibrary instance)

// swift-tools-version:6.2
import PackageDescription

let package = Package(
    name: "htmlResponses",
    platforms : [.macOS(.v14)],
    products: [.executable(name: "htmlResponses", targets: ["htmlResponses"])],
    dependencies: [
        .package(url: "https://github.com/hummingbird-project/hummingbird.git", from: "2.0.0"),
        .package(url: "https://github.com/apple/swift-argument-parser.git", from: "1.3.0"),
        .package(url: "https://github.com/hummingbird-project/swift-mustache", from: "2.0.0")
    ],
    targets: [
        .executableTarget(
            name: "htmlResponses",
            dependencies: [
                .product(name: "ArgumentParser", package: "swift-argument-parser"),
                .product(name: "Hummingbird", package: "hummingbird"),
                .product(name: "Mustache", package: "swift-mustache"),
            ],
            resources: [.process("Templates")]

        ),
        .testTarget(name: "addTestingTests",
            dependencies: [
                .byName(name: "htmlResponses"),
                .product(name: "HummingbirdTesting", package: "hummingbird")
            ],
            path: "Tests/AppTests"
        )
    ]
)

The Templates

Next make a couple of simple templates to chuck into that folder.

This example uses two super short template, mostly just to show how to combine two templates.

base.mustache (first) calls body.mustache with the line {{> body }}

body.mustache is referred to as a Partial. A nice little demo is on this mustache playground.

base.mustache

<!DOCTYPE html>
<html>
<head>
    <title>{{  title  }}</title>
    <meta charset="UTF-8">
</head>
{{> body }}
</html>

body.mustache

<body>
<p>__________{{  message  }}__________________________</p>
<p><a href="/">back to home</a>.</p>
</body>

That’s it! Notice the double curlies? {{...}}? hugo uses those too! (see [meta]/tags/meta/), as does the compatible JS library, Handlebars

While what’s in them will be different partially because Mustache is logic-less, the familiarity is nice.

The Type

Once the templates exist, a type that will process them and turn them into a Response comes next.

The main difference between AcrobatHTML and OrganPlayerHTML will be the need to have a reference to a Mustache library and call to the library.render function that pulls all the templates together.

import Hummingbird

#if canImport(FoundationEssentials)
import FoundationEssentials
#else
import Foundation
#endif
import Mustache

//struct with all the needed info for the template. 
//struct parameters must match names in the templates.
struct BasicInfo {
    let title: String
    let message: String
}

struct AcrobatHTML:ResponseGenerator  {
    // load mustache template library

    let message: String
    let library: MustacheLibrary

    init(_ message: String, library:MustacheLibrary) {
        self.message = message
        self.library = library
    }

    private var html:String {
        if let result = try? render() {
            return result
        } else {
            return "??? oops."
        }
    }

    private func render() throws -> String? {
        let messagePost = BasicInfo(title: "I'm a Message!", message: message)

        //func render(_ object: Any, withTemplate name: String, reload: Bool)
        let rendered = library.render(
            messagePost,
            withTemplate: "base"
        )
        return rendered
    }

    func response(from request: Request, context: some RequestContext) throws -> Response {
        let buffer = ByteBuffer(string: self.html)
        return .init(status: .ok, headers: [.contentType: "text/html"], body: .init(byteBuffer: buffer))
    }
}

The object “BasicInfo” will then be passed to base.mustache’s render function, which will recurse through any nested templates, matching up the tokenized information.

It’s a nice implementation of a render that could be fun to go through more completely in a future post! (a bit more on Lexical tokens, which I talked about a little bit [here]({{ relref 20240327-string-scanning }}), but there is much to say.)

Creating the Library Instance & Adding the Route

In this example the Mustache Library will be created in Application+build.swift in the buildRouter function.

As mentioned above, the templates are being put in the bundle, rather than in some absolute path on the local machine.

The suggested docker files all have a line for moving things that are in the bundle’s resource folder

# Copy resources bundled by SPM to staging area
RUN find -L "$(swift build --package-path /build -c release --show-bin-path)/" -regex '.*\.resources$' -exec cp -Ra {} ./ \;

so the transition to a containerized run should go smoothly.

//ADD TO TOP WITH OTHER IMPORTS
#if canImport(FoundationEssentials)
    import FoundationEssentials
#else
    import Foundation
#endif

import Mustache


func buildRouter() async throws -> Router<AppRequestContext> {
    let router = Router(context: AppRequestContext.self)

    //...

    let mustacheLibrary = try await MustacheLibrary(directory: Bundle.module.resourcePath!)
    //let mustacheLibrary = MustacheLibrary()

    router.get("/acrobat") { _, _ in "not here"}
    router.get("acrobat/:message") { _, context in
        guard let message = context.parameters.get("message", as: String.self) else {
            throw HTTPError(.badRequest, "could not read a message from the data provided")
        }
        let decoded = message.removingPercentEncoding ?? "can't-balance-here"
        return AcrobatHTML(decoded, library: mustacheLibrary)
    }

    //...

}

The Call

Back in the public folder, we can add the link to the index.html file.

Don’t forget to percent encode the string!

<!DOCTYPE html>
<html>

<head>
  <title>Test Webpage: Site Index Page</title>
  <meta charset="UTF-8">

</head>

<body>

  <li>These links return generated HTML pages as their response bodies</li>
  <ul>
    <li><a href=/organ/LaTaDadaDadaLaTaDaDah>organ music</a></li>
    <li><a href=/acrobat/not%20gonna%20fall>acrobat</a></li>
  </ul>
  </li>

</body>

</html>

Add some testing

Different routes mean different tests.

These are pretty minimal tests. They just make sure the routes exist with a known happy path.

Unhappy paths, valid HTML headers, valid HTML, that the content is in there as expected are all things one might test in a more comprehensive suite.

struct AppTests {
    struct TestArguments: AppArguments {
        let nameTag: String = "nwtTestServer"
        let hostname = "127.0.0.1"
        let port = 0
        let logLevel: Logger.Level? = .trace
    }

    
    @Test func testCreateApp() async throws {
        let app = try await buildApplication(TestArguments())
        try await app.test(.router) { client in
            let _ = try await client.execute(uri: "/ping", method: .get) { response in
                #expect(response.status == .ok)
            }
        }
    }
    
    @Test func testOrgan() async throws {
    	let app = try await buildApplication(TestArguments())
        try await app.test(.router) { client in
        	let message = "my%20precent%20encoded%20Message"
            let _ = try await client.execute(uri: "/organ/\(message)", method: .get) { response in
                #expect(response.status == .ok)
            }
        }
    }
    
    @Test func testAcrobat() async throws {
        let app = try await buildApplication(TestArguments())
        try await app.test(.router) { client in
        	//http://localhost:8080/acrobat/not%20gonna%20fall
        	let message = "my%20precent%20encoded%20Message"
            let _ = try await client.execute(uri: "/acrobat/\(message)", method: .get) { response in
                #expect(response.status == .ok)
            }
        }
    
    
    }
}

Summary

That’s quick overview of how to get the bare minimum working with Mustache templates. The next post will be about making a CRUD interface for a codable type by hand, and the one after that will compare it to using the OpenAPI plugin for Swift packages.

How do I get Hummingbird to serve html dynamically?

Tue, 16 Sep 2025 00:00:00 +0000

Most the examples so far focus on returning data or simple strings, but what if instead I wanted an actual HTML response generated programmatically?

This post will show simply wrapping a string of known-good HTML and in a response. The next post using a Mustache template. The combined example is available:

https://github.com/carlynorama/HummingbirdExamples/tree/main/04_html_responses

OrganPlayerHTML.swift

I snagged the html string -> http response wrapper code for this from the hummingbird examples repo, but made it a type that would accept a “message” String.

import Hummingbird

struct OrganPlayerHTML: ResponseGenerator {
    let message: String

    init(_ message: String) {
        self.message = message
    }

    private var html:String {
        Self.generateHTMLForMessage(message)
    }

    func response(from request: Request, context: some RequestContext) throws -> Response {
        let buffer = ByteBuffer(string: self.html)
        return .init(status: .ok, headers: [.contentType: "text/html"], body: .init(byteBuffer: buffer))
    }

    static func generateHTMLForMessage(_ message:String) -> String {
        """
        <!DOCTYPE html>
        <html>
        <head>
            <title>Hear the Music!!!</title>
            <meta charset="UTF-8">

        </head>
        <body>
        <p>\(message)</p>
        <p><a href="/">back to index</a>.</p>
        </body>
        </html>
        """
    }
}

The Route

Back in the buildRouter function in Application+build.swift one possible route one could use could be:

    router.get("organ/:message") { _, context in
        guard let message = context.parameters.get("message", as: String.self) else {
            throw HTTPError(.badRequest, message:"Could not parse message.")
        }
        return OrganPlayerHTML(message)
    }

Don’t forget to percent encode any spaces, etc.…

The Call

In the public folders’s index HTML page a call to that route might look like

<body>

  <li>These links return generated HTML pages as their response bodies</li>
  <ul>
    <li><a href=/organ/LaTaDadaDadaLaTaDaDah>organ music</a></li>
  </ul>
  </li>

</body>

Tada!

Not the most snazzy, but I hope it points to how to use a Hummingbird route to trigger building a simple HTML page for more complicated examples.

There are MANY swift packages where people create their favorite way to generate HTML, if the situation is too complex to handle by hand.

For me my preference would be the Mustache templating engine, which conveniently the Hummingbird Project already has covered.

Next post!

How do I test the Hummingbird routes with swift-testing?

Mon, 15 Sep 2025 00:00:00 +0000

It’d be way easier to catch any errors in all those routes if instead of that bash script, we could use swift-testing instead.

AppArguments

02_basic_responses added a file that I didn’t mention: AppArguments.swift

import Hummingbird
import Logging

//Allows for different setups for testing server and production server
public protocol AppArguments {
    var nameTag:String { get }
    var hostname: String { get }
    var port: Int { get }
    var logLevel: Logger.Level? { get }
}

This struct beings the process of of allowing a separate testing server or servers with different features and the live server.

As a demonstration, in the (the todos example)[https://docs.hummingbird.codes/2.0/tutorials/todos#testing-your-application], where I got this data structure from, the testing is done on in-memory data and the live server points to a postgres db.

An instance of AppArguments can be used at that top of every test as needed to spin up a new server unique to that test. For some testing suites it might make sense to make a group of tests that all point to the same App instance, but in this example each test gets its own little serverlette.

#if canImport(FoundationEssentials)
import FoundationEssentials
#else
import Foundation
#endif

import Hummingbird
import HummingbirdTesting
import Logging
import Testing

@testable import nowWithTesting

struct AppTests {
    struct TestArguments: AppArguments {
        let nameTag: String = "nwtTestServer"
        let hostname = "127.0.0.1"
        let port = 0
        let logLevel: Logger.Level? = .trace
    }

    
    @Test func testCreateApp() async throws {
        let app = try await buildApplication(TestArguments())
        try await app.test(.router) { client in
            let _ = try await client.execute(uri: "/ping", method: .get) { response in
                #expect(response.status == .ok)
            }
        }
    }
//more follows...
}

Example Tests

The tests aren’t comprehensive and the “unhappy paths” need improvement on both the server side and the testing side. None the less, these examples provide building blocks for what’s needed:

/goodbye

    @Test func testGoodbye() async throws{
        let app = try await buildApplication(TestArguments())
        try await app.test(.router) { client in
            let _ = try await client.execute(uri: "/goodbye", method: .get) { response in
                #expect(response.status == .ok)
                #expect(String(buffer: response.body) == "Ciao!")

            }
        }
    }

Single wildcard

    @Test func testSingleWild() async throws {
        let app = try await buildApplication(TestArguments())
        try await app.test(.router) { client in

            let _ = try await client.execute(uri: "/files/", method: .get) { response in
                #expect(response.status == .notFound)
            }

            let _ = try await client.execute(uri: "/files/testPhrase", method: .get) { response in
                #expect(response.status == .ok)
                #expect(String(buffer: response.body) == "/files/testPhrase")
            }

            let _ = try await client.execute(uri: "/files/too/many/things", method: .get) { response in
                #expect(response.status == .notFound)
            }
        }
    }

/user route

    @Test func testUser() async throws {
        let app = try await buildApplication(TestArguments())
        try await app.test(.router) { client in

            let _ = try await client.execute(uri: "/user/garbfjfesage", method: .get) { response in
                #expect(response.status == .badRequest)
            }

            var intToTest = 43
            let _ = try await client.execute(uri: "/user/\(intToTest)", method: .get) { response in
                #expect(response.status == .ok)
                #expect(String(buffer: response.body) == "\(intToTest)")
            }

            intToTest = 43131
            let _ = try await client.execute(uri: "/user/\(intToTest)", method: .get) { response in
                #expect(response.status == .ok)
                #expect(String(buffer: response.body) == "\(intToTest)")
            }

            intToTest = -13214
            let name = "some crazy string"
            let _ = try await client.execute(uri: "/user/\(intToTest)/\(name)", method: .get) { response in
                #expect(response.status == .ok)
                #expect(String(buffer: response.body) == "found \(name) for \(intToTest)")
            }
        }
    }

Codable / Decodable


    struct MiniCodable:Decodable, ResponseEncodable {
        let number: Int
        let phrase: String?
    }
    
    @Test func testEncodable() async throws {
        let app = try await buildApplication(TestArguments())
        try await app.test(.router) { client in
            let _ = try await client.execute(uri: "/encodable", method: .get) { response in
                #expect(response.status == .ok)
                //print(String(buffer: response.body))
                #expect(String(buffer: response.body) == "{\"number\":5,\"phrase\":\"hello!\"}")

            }
        }
    }

    @Test func testDecoding() async throws {
        let app = try await buildApplication(TestArguments())
        try await app.test(.router) { client in
        
            let _ = try await client.execute(uri: "/decodable/default", method: .post) { response in
                #expect(response.status == .badRequest, "not the status expected from an empty post")
            }

            let codedData = MiniCodable(number: 34, phrase:"something to say")

            var buffer = try JSONEncoder().encodeAsByteBuffer(codedData, allocator: ByteBufferAllocator())
            // print("coded: \(String(buffer:buffer))")
            // var buffer = ByteBuffer(string: "{\"phrase\":\"something to say\",\"number\":34}" )
            let _ = try await client.execute(uri: "/decodable/default", method: .post, body:buffer) { response in
                #expect(response.status == .ok)
                print("responseBody default: \(String(buffer: response.body))")

                //json shows up in arbitrary order, so check for values individually.

				//responseBody looks like
				//"DECODED: MiniCodable(number: 34, phrase: Optional("something to say"))").contains("\"number\":\(codedData.number)"
                #expect(String(buffer: response.body).contains("number: \(codedData.number)"))
                #expect(String(buffer: response.body).contains("phrase: \"Optional(\(codedData.phrase!))\""))
            }

            
            buffer = ByteBuffer(string: "number=\(codedData.number)&phrase=\(codedData.phrase!)")
            let _ = try await client.execute(uri: "/decodable/form", 
                                            method: .post, 
                                            //need header if detecting based on header, which not yet.
                                            //headers: [.contentType: "application/x-www-form-urlencoded"], 
                                            body:buffer) { response in
                #expect(response.status == .ok)
                print("responseBody form: \(String(buffer: response.body))")
                #expect(String(buffer: response.body).contains("number: \(codedData.number)"))
                #expect(String(buffer: response.body).contains("phrase: \"Optional(\(codedData.phrase!))\""))
            }
        }
    }

Summary

This post just barely scratches the surface of how to test a Hummingbird server. Also all the data being produced is fairly simple text or JSON… what if I wanted to serve a NOT static HTML page? That’s next.

What kinda routes can I use with Hummingbird?

Sat, 13 Sep 2025 00:00:00 +0000

I mean…all of them? So this is all based on walking through the Router Guide, but I wanted one project that served up the sampler.

Here is a buildRouter function with a smorgasbord of examples.

In context at: https://github.com/carlynorama/HummingbirdExamples/tree/main/02_basic_responses

buildRouter

func buildRouter() -> Router<AppRequestContext> {
    let router = Router(context: AppRequestContext.self)

    router.addMiddleware {
        // logging middleware
        LogRequestsMiddleware(.info)
        //serves the static files in public folder by default.
        FileMiddleware(searchForIndexHtml: true)
    }

    // SIMPLE ROUTING

    // These three types have been given ResponseGenerator status by Hummingbird
    // - HTTPResponseStatus
    // - String
    // - Optional

    // The closure takes in a request and a context and returns
    // some ResponseGenerator

    router.get("/ping") { _, _ -> HTTPResponse.Status in
        return .ok
    }

    router.get("/hello") { request, _ -> String in
        return "Well Hello!\n\n\(request)"
    }

    // full call
    router.on("/goodbye", method: .get) { request, context in
        return "Ciao!"
    }

    //---- WILDCARDS
    // https://docs.hummingbird.codes/2.0/documentation/hummingbird/routerguide#Wildcards
    router.get("/files/*") { request, _ in
        return request.uri.description
    }

    //note: ** is 1 or more, not zero or more. (2025/08)
    router.get("/doublewild/**") { _, context in
        return context.parameters.getCatchAll().joined(separator: "/")
    }

    //---- Parameter Capture

    // //https://docs.hummingbird.codes/2.0/documentation/hummingbird/routerguide#Parameter-Capture
    router.get("/user/:id") { _, context in
        guard let id = context.parameters.get("id", as: Int.self) else {
            throw HTTPError(.badRequest)
        }
        return "\(id)"
    }

    router.get("/user/:id/:name") { _, context in
        guard let id = context.parameters.get("id", as: Int.self) else {
            throw HTTPError(.badRequest)
        }
        guard let name = context.parameters.get("name", as: String.self) else {
            throw HTTPError(.badRequest)
        }
        return "found \(name) for \(id)"
    }

    router.get("/img/{image}.jpg") { _, context in
        guard let imageName = context.parameters.get("image") else {
            throw HTTPError(.badRequest)
        }
        switch imageName {
        case "bunny":
            return " \\\n  \\ /\\\n  ( )\n.( o )."
        //    \
        //     \ /\
        //     ( )
        //   .( o ).
        default: return "this is a \(imageName) "
        }
    }

    router.get("/query") { request, _ in
        // extract parameter from URL of form /query?id={userId}&message={message}
        guard let id = request.uri.queryParameters.get("id", as: Double.self) else {
            throw HTTPError(.badRequest, message: "id?")
        }
        guard let message = request.uri.queryParameters.get("message", as: String.self) else {
            throw HTTPError(.badRequest, message: "message?")
        }
        return "tell \(id) I said \"\(message)\""
    }

    // GROUPS

    //if one has a logical group one can batch them together.
    router.group("/circus")
        .get(use: { _, _ in "if I had a list of circus acts I would show a list of circus acts" })

        .get("peanuts") { _, _ in "yummy peanuts" }
        .group("clowns")
        .get { _, _ in "if I had a list of clowns I would show a list of clowns" }
        .get(
            "{id}",
            use: { _, context in
                "that's the clown called \(context.parameters.get("id") ?? "no ID")!"
            })

    struct MiniCodable: Decodable, ResponseEncodable {
        let number: Int
        let phrase: String?
    }

    //the default encoder is JSON
    router.get("/encodable") { _, _ -> MiniCodable in
        return MiniCodable(number: 5, phrase: "hello!")
    }

    router.group("/decodable")
        .get { _, _ in "this was a GET. Please submit some info using POST" }

        //curl -i -X POST localhost:8080/decodable/default -d'{"number":12,"phrase":"are you going to go my way"}'
        .post("default") { request, context in
            let decoded = try await request.decode(as: MiniCodable.self, context: context)
            print("DECODED: \(decoded)")
            return "DECODED: \(decoded)"  //also consider an HTTPResponseStatus
        }
        //curl -i -X POST 'http://localhost:8080/decodable/form' -H 'Content-Type: application/x-www-form-urlencoded' --data-raw 'number=36&phrase=message%20string'
        .post("form") { request, context in
            print(request.headers)
            let decoded = try await URLEncodedFormDecoder().decode(
                MiniCodable.self, from: request, context: context)
            print("DECODED: \(decoded)")
            return "DECODED: \(decoded)"
        }

    return router
}

Testing Script

To test them all here’s a build script with curl calls.

#!/bin/sh

MY_PORT="8080"

separate () {
sleep 2s
echo -e "\n---------------------\n" 
}

curl -i "http://localhost:$MY_PORT/ping"
separate 
curl -i "http://localhost:$MY_PORT/hello"
separate  
curl -i "http://localhost:$MY_PORT/goodbye"

separate 
curl -i "http://localhost:$MY_PORT/files/putonethingafter"
separate 
curl -i "http://localhost:$MY_PORT/doublewild/put/anything/I/want"

separate
curl -i "http://localhost:$MY_PORT/user/8675309"
separate
curl -i "http://localhost:$MY_PORT/user/8675309/Jenny"

separate
curl -i "http://localhost:$MY_PORT/img/non-bunny.jpg"
separate
curl -i "http://localhost:$MY_PORT/img/bunny.jpg"

separate
curl -i "http://localhost:$MY_PORT/query?id=35.53&message=hello%20with%20spaces"

separate
curl -i "http://localhost:$MY_PORT/circus"
separate
curl -i "http://localhost:$MY_PORT/circus/peanuts"
separate
curl -i "http://localhost:$MY_PORT/circus/clowns"
separate
curl -i "http://localhost:$MY_PORT/circus/clowns/anythingcanbeanidforthis"

separate 
curl -i "http://localhost:$MY_PORT/encodable"

separate 
curl -i -X POST localhost:$MY_PORT/decodable/default -d'{"number":12,"phrase":"are you going to go my way"}'
separate 
curl -i -X POST localhost:$MY_PORT/decodable/default -d'{"number":14}'

separate 
curl -i -X POST 'http://localhost:8080/decodable/form' -H 'Content-Type: application/x-www-form-urlencoded' --data-raw 'number=36&phrase=message%20string'
separate 
curl -i -X POST 'http://localhost:8080/decodable/form' -H 'Content-Type: application/x-www-form-urlencoded' --data-raw 'number=36&phrase'

How can I have Hummingbird serve static files?

Fri, 12 Sep 2025 00:00:00 +0000

So I’ve already mentioned Hummingbird a few times. Most recently I showed off the “simplest server” in the how to deploy to an App server post. (It’s based of of the code on the Hummingbird docs home.)

This time I want to show how to set up Hummingbird to host static pages. Hummingbird primarily focuses on serving Swift generated content, but static pages can be incredibly useful for testing APIs.

This example started from a direct copy of https://github.com/adam-fowler/swift-web on 2025, Aug 15, and honestly did not change much.

It’s a utility like python3 -m http.server that will start a servlet based on the contents of a directory. I’ll talk about the install later.

The Code

The code is all of 35 lines. It’s really simple. It depends on Hummingbird and ArgumentParser to make a 1 page app.

FileMiddleware is provided by Hummingbird, but it’s possible to write custom versions. Middleware in this context is kind of like a plugin that a web app developer can insert between the request and the response, altering Hummingbird’s default behaviors. Normally if a route isn’t cataloged in the Router, Hummingbird would send a 404 Response. With FileMiddleware registered, instead Hummingbird will look in the specified directory for a path that matches the route first.

import ArgumentParser
import Hummingbird
import Logging

@main
struct WebServer: AsyncParsableCommand {
    @Option(name: .shortAndLong)
    //will cause issue in containers but this is for
    //running locally.
    var hostname: String = "127.0.0.1"

    @Option(name: .shortAndLong)
    var port: Int = 8080

    @Argument
    var folder: String = "."

    func run() async throws {

        print(folder)

        let logger = Logger(label: "swift-serve")
        let router = Router()
        router.middlewares.add(FileMiddleware(self.folder, searchForIndexHtml: true, logger: logger))
        router.middlewares.add(LogRequestsMiddleware(.info))
        
        let app = Application(
            router: router,
            configuration: .init(
                address: .hostname(self.hostname, port: self.port),
                serverName: "swift-serve"
            ),
            logger: logger
        )
        try await app.runService()
    }
}

The Package.swift

// swift-tools-version:6.1
import PackageDescription

let package = Package(
    name: "swift-serve",
    platforms : [.macOS(.v14)],
    products: [.executable(name: "swift-serve", targets: ["swift-serve"])],
    dependencies: [
        .package(url: "https://github.com/hummingbird-project/hummingbird.git", from: "2.0.0"),
        .package(url: "https://github.com/apple/swift-argument-parser.git", from: "1.3.0")
    ],
    targets: [
        .executableTarget(
            name: "swift-serve",
            dependencies: [
                .product(name: "ArgumentParser", package: "swift-argument-parser"),
                .product(name: "Hummingbird", package: "hummingbird"),
            ]
        ),
    ]
)

Usage

If one goes to the example’s repo, there’s a folder called “public” there with some HTML for testing. The lines

    @Argument
    var folder: String = "."

mean that the server will treat the present working directory as the root folder if nothing else gets specified. The FileMiddleware will actually look for a folder called public in the package root if nothing is specified, but in this example I put the demo directory right outside the root to give an excuse to show passing a path.

# from the directory with the README / 
cd swift-serve
swift run swift-serve ../public

Installing the Tool

It’d hardly be convenient to keep recompiling or go digging in a build folder whenever it’s time to run a server.

Fowler provides instruction on how to install the server tool using mint. It handles where to put and where to find all the compiled binaries really nicely.

But for fun, let’s go through the exercise of installing the hand written debug version of the code in a folder called ~/mybin (Although one could certainly make a release version first: swift build --configuration release). I prefer to do this with tools I’ve written, rather than putting them in /usr/local/bin or /usr/bin.

First step, copy the executable. A symlink would also be an option if you wanted it to stay up to date as you developed. (ln -s /path/to/original /path/to/link)

## make the folder
mkdir ~/mybin
## From Directory with this readme, after having run `swift build` at the very least. 
cd swift-serve/.build/debug
cp swift-serve ~/mybin/swift-serve
cd -
cd public
~/mybin/swift-serve

This should run the server. If you’d like to add the ~/mybin to the path

# see all the folders in your path
echo $PATH 
## then either add it to the beginning to search it first
PATH=~/mybin:$PATH
## OR add it to the end to search it last. 
PATH=$PATH:~/mybin
## check your path again
echo $PATH

This will ONLY work for THIS shell. To update your profile try seeing what you currently have, e.g. cat ~/.zprofile if using zsh or ls -al ~/ to go hunting.

Add something like

if [[ ":$PATH:" != *":~/mybin:"* ]]; then
    export PATH="Users/$USER/mybin:$PATH"
fi

at the bottom of the file to add your custom folder at the front of the PATH’s line every time you open a shell controlled by that profile.

CAUTION: This means programs in ~/mybin always win. If you name your program the same as something else important, that’ll be some serious woe.

Open a new terminal window (or hash -r the current one) and

cd public
swift-serve

should work now!

Can I animate color changes over time with CSS?

Thu, 11 Sep 2025 00:00:00 +0000

So, can color be animated with CSS? Yes. What way will work best for my purposes, TBD.

Hello Animation

Here’s a variation of an example from mdn’s animation page

<style>
    #example1 {
      width: 50vw;
      height: 50vh;
      overflow: hidden;
      background-color: lightblue;
      display: flex;
      justify-content: center;
      align: center;
    }
     
    #example1 .sun {
          background-color: yellow;
          border-radius: 50%;
          /* creates a circular background */
          height: 20vh;
          /* makes the sun the size of the viewport */
          aspect-ratio: 1 / 1;
          animation: 4s linear 0s infinite alternate animating-multiple-properties;
      }

    @keyframes animating-multiple-properties {
      from {
          transform: translateY(calc(50vh - 20vh));
          background-color: red;
          filter: brightness(75%);
      }

      to {
          transform: translateY(0%);
          background-color: orange;
          /* unset properties i.e. 'filter' will revert to default values */
      }
    }
                
</style>
<div id="example1">
  <div class="sun"></div>
<div>

With Variables

CSS has variables! This animation is just the color change from the first, but done with variables.

<style>
  :root {
     --example2-start-color: red;
     --example2-end-color: orange;
  }
  #example2 {  
    width: 50vw;
    height: 50vh;
    overflow: hidden;
    background-color: lightblue;
    display: flex;
    justify-content: center;
    align-items: center;
  }

  #example2 .sun {
      background-color: yellow;
      border-radius: 50%;
      height: 30vh;
      aspect-ratio: 1 / 1;
      animation: 10s linear 0s infinite alternate example-two-frames;
  }

  @keyframes example-two-frames {
    from {
        background-color: var(--example2-start-color);
        filter: brightness(75%);
    }
    to {
        background-color: var(--example2-end-color);
    }
  }
</style>
<div id="example2">
  <div class="sun"></div>
</div>

Control a Variable With Javascript

Once you have a variable, it can be controlled via JavaScript. This sun should be skittering through hue changes.

NOTE: Some blog meta. All the <script> tag enclosed code from the post is consolidated into one script tag at the bottom of the post.

<style>
  :root {
     --example3-color: red;
  }
  #example3 {  
    width: 50vw;
    height: 50vh;
    overflow: hidden;
    background-color: lightblue;
    display: flex;
    justify-content: center;
    align-items: center;
  }

  #example3 .sun {
    background-color: yellow;
    border-radius: 50%;
    height: 30vh;
    aspect-ratio: 1 / 1;
    background: var(--example3-color);
  }
</style>
<div id="example3">
  <div class="sun"></div>
</div>
<script>
let startTime = null;
let hue = 0
let root = document.documentElement;
function animateColor(currentTime) {
  if (!startTime) startTime = currentTime;
  const elapsed = currentTime - startTime;
  hue = elapsed/60;
  root.style.setProperty('--example3-color', `hsl(${hue}, 100%, 50%)`)
  //console.log("hello animateColor.")
  requestAnimationFrame(animateColor);
}
  window.onload = function() {
    animateColor(new Date())
    //console.log("hello from the script.")
  }
</script>

Animation Delays

It’s possible to delay an animation’s start time with animation-delay. Negative numbers work too! It jumps the position of the animation further in as if the animation had been playing for that long before the page had loaded.

This sun should track with the seconds, popping back to red at the flip to a new minute, no matter the precise time the page was opened.

<style>
  :root {
    --example4-start-color: red;
    --example4-end-color: yellow;
    --example4-delay: 0s;
  }
  #example4 {  
    width: 50vw;
    height: 50vh;
    overflow: hidden;
    background-color: lightblue;
    display: flex;
    justify-content: center;
    align-items: center;
  }
  #example4 .sun {
    border-radius: 50%;
    /* creates a circular background */
    height: 30vh;
    /* makes the sun the size of the viewport */
    aspect-ratio: 1 / 1;
    background: var(--example4-start-color);
    animation: example-four-frames 60s none var(--example4-delay) infinite;
  }
  @keyframes example-four-frames {
    from {
        background-color: var(--example4-start-color);
        filter: brightness(75%);
    }
    to {
        background-color: var(--example4-end-color);
    }
  }
</style>
<div id="example4">
  <div class="sun"></div>
</div>
<script>
function setDelay() {
    var date = new Date();
    var seconds = date.getSeconds();
    let root = document.documentElement;
    let delay = seconds * -1
    console.log(seconds, delay);
    console.log("hello from setDelay")
    root.style.setProperty('--example4-delay', `${delay}s`);
}
  window.onload = function() {
    setDelay();
  }
</script>

Resources

I bought a Udemy course on CSS a while ago now that I keep going back to when i need a CSS refresh.

But here are some of the links that helped in making the examples for this post:

General CSS Tools

Animation Docs

Can I make a Swift-wrapped HTTP GET request from the ESP32C6?

Wed, 10 Sep 2025 00:00:00 +0000

So for this I created another component to act as a bridge. Right now this turned out to be mostly C while I figure out how to / whether to bring in things like URL and URL session, HTTPRequest…

I’ve written a client in Swift before, but it relied heavily on Foundation types. The only example in the the Swift Embedded Examples repo that uses Foundation is a beautiful Swift MMIO based example for the stm32 which is making me rethink my chip choices. Although SVD data is available for the esp32c6 even though it’s RISCV, not ARM.

As an alternative to Foundation, Hummingbird brings in SwiftNIO.

Both Foundation and SwiftNIO are very large and I’m not in the mood to wrestle with the CMake file today.

While I overthink this, I just made the proof of concept version with C.

The Swift Files

Main.swift

Right above the while loop but after the delay to wait for the WiFi to connect…

  //Waiting for wifi to connect...
  delay(2000);
  let client = HTTPClient()
  client.getAndPrint(from: "example.com", route: "/")

HTTP.swift

Again with the ugly string passing which I hope is temporary.

THe result code is something I could turn into a thrown error at some point. It’s a really common pattern in C that the result is written at a shared pointer and the error code is the thing that’s returned.

final class HTTPClient {

    func getAndPrint(from host: String, route: String) {
        print("Getting...")
        print(http_bridge_bridge_return_twelve())
        let local_host = host.utf8CString
        let local_route = route.utf8CString

        //TODO: test with span on beta?
        local_route.withContiguousStorageIfAvailable { route_buffer in
            local_host.withContiguousStorageIfAvailable { host_buffer in
                let resultCode = http_bridge_get(host_buffer.baseAddress, route_buffer.baseAddress)
                print("result code: \(resultCode)")
            }
        }

    }
}

The Component Code

The esp-idf has a HTTP Client library, which a more robust version of this code could use. esp-idf example

The C File

Similar code is in the HTTP_Request example.

Both do the following basic steps:

find out if the host name even exists as a valid URL
make a socket
use the socket to connect
send the request over the socket
say done sending
receive the request
… the WHOLE request
close the socket

Making sure to close the socket in case of any failures as well.

int http_bridge_get(const char *host, const char *path) {
    printf("A");
    struct addrinfo hints = {
        .ai_family = AF_INET,
        .ai_socktype = SOCK_STREAM,
    };
    struct addrinfo *res;
    int s;

    printf("B");
    char port_str[] = "80";
    int err = getaddrinfo(host, port_str, &hints, &res);
    //int err = getaddrinfo(host, port_str, &hints, &res);
    if (err != 0 || res == NULL) {
        printf("DNS lookup failed for %s", host);
        return 1;
    }

    printf("C");
    s = socket(res->ai_family, res->ai_socktype, 0);
    if (s < 0) {
        printf("socket failed");
        freeaddrinfo(res);
        return 2;
    }

    printf("D");
    if (connect(s, res->ai_addr, res->ai_addrlen) != 0) {
        printf("connect failed");
        close(s);
        freeaddrinfo(res);
        return 3;
    }

    printf("E");
    freeaddrinfo(res);
    
    printf("F");
    char req[128];
    snprintf(req, sizeof(req),
             "GET %s HTTP/1.0\r\n"
             "Host: %s\r\n"
             "User-Agent: esp-idf/5.5\r\n"
             "\r\n",
             path, host);

    if (write(s, req, strlen(req)) < 0) {
        printf("send failed");
        close(s);
        return 4;
    }

    char recv_buf[512];
    int r;
    while ((r = read(s, recv_buf, sizeof(recv_buf) - 1)) > 0) {
        recv_buf[r] = 0; // null terminate
        printf("%s", recv_buf);
    }
    close(s);
    return 0;
}

The header

#include <string.h>
#include <sys/socket.h>
#include <netdb.h>   


int http_bridge_bridge_return_twelve(void);

int http_bridge_get(const char *host, const char *path);

The CMake

No REQUIRED in this because the includes are part of the the C implementation.

idf_component_register(SRCS "http_bridge.c"
                    INCLUDE_DIRS "include")

Summary

Kinda punted on this part a bit while I figure out how much time I have to really dig in. Need to do some parts ordering, get the real server up, etc.

Can I add Wifi to the ESP32C6 project with Swift?

Tue, 09 Sep 2025 00:00:00 +0000

https://github.com/carlynorama/swift_esp32c6_hello/tree/c_WiFi

I knew this was coming. Having to create a separate component. I tried to get Wifi working in the demo project without creating a bridge library but I couldn’t quite get the mismatch between what C elements Swift will compile inline and the C elements the SDK uses. (looking at you function style macros)

I am going to put out there right now that there could be a better version of this code. I am not using all the C import features in Swift to minimize what’s in the bridging library. Nor am I using the C decorators that make things safer from WWDC25. APINotes make no appearance at all. Those are all niceties for a future version. This is all about Wifi Hello World.

Ultimately my solution is still a whole lot of C wrapped in a thin Swift shell, but it’s somewhere to start. It’s also just a concrete type. No “WifiProvider” protocol for now.

Useful References:

How to write a Test Component

So there’s a whole page on how to write a test component, but I felt it left out some key details (or at least phrased them in away I didn’t grok) that I filled in just by looking at someone elses demo.

Here is how I got my test component working with the Swift base. I did not have to touch the CMakeLists.txt to add the library, although I could have. I relied on the esp-idf searching that in-project components folder for free. Didn’t even have to add the dependency to /main/idf_component.yml.

cd $CURRENT_PROJECT_DIR
mkdir -p components
cd components
idf.py create-component test_cmp

Then I updated the files to the following:

/components/test_cmp/test_cmp.c

#include <stdio.h>
#include "test_cmp.h"


// One of my go to canary functions.
int test_cmp_return_twelve(void)
{
    return 12;
}

/components/test_cmp/include/test_cmp.h

int test_cmp__return_twelve(void);

/components/test_cmp/CMakeLists.txt

left unchanged.

idf_component_register(SRCS "test_cmp.c"
                    INCLUDE_DIRS "include")

/main/BridgingHeader.h

//existing items....

#include "test_cmp.h"

/main/Main.swift

@_cdecl("app_main")
func main() {
  //before...

  print(test_cmp_return_twelve())

  //..etc
  while true {
    //etc.
  }
}

Wifi Component In the Project

So I started with a simple version, but it was having trouble. What didn’t occur to me until much later was that the symptoms I was seeing lined up with the code reacting poorly with the USB startup processes. The esp-idf driven code (sdk and monitor) handles that all so automagically. (print just works! With no Serial.begin or anything!), it just wasn’t on my radar to factor that in until todbot mentioned it.

I made two other working but more complicated versions to try to troubleshoot that can be seen in the git history:

“common works”: One that pulls the protocols_examples_common component out of the example folder (drives the example in the protocol folder.)
“works like station example”: One that pulls the event based functions from the station wifi getting started example.

Those both work, but since this is all for demo code more than a targeting having a robust WiFi library, I really wanted to make the simple iteration happen.

The code current to this post works by adding delays in the Main.swift, it does not recover from a disconnect. It does not retry. It’s not GOOD wifi code, but only essential. (minus a deinit).

@_cdecl("app_main")
func main() {

  guard var led = DigitalIndicator(15) else {
    fatalError("Difficulty setting up pin.")
  }

  guard let button = MomentaryInput(9) else {
    fatalError("Difficulty setting up button.")
  }

  //Waiting for USB...
  delay(2000);

  print("Hello from Swift on ESP32-C6!")
  print(wifi_bridge_return_twelve())

  let wifi = WiFiStation()
  wifi.connect(ssid: "somenetwork", password: "somepassword")

    //Waiting for wifi to connect...
  delay(2000);

  while true {
    if button.isActive {
      led.blink(millis: 500)
    } else {
      led.blink(millis: 2000)
    }
  }
}

It’s an old solution. But it checks out.

Wifi.swift

Getting a cleaner version of the string passing from Swift into C might be more possible on a machine running macOS 26? The C-Interop safety features with spans weren’t quite working correctly. (Could decorate the C, wasn’t being acknowledge by the Swift.)

final class WiFiStation {

    init() {
        checkWithFatal(wifi_bridge_initialize_nvs())
        checkWithFatal(wifi_bridge_initialize_netif())
        checkWithFatal(wifi_bridge_wifi_init_default_config())
    }

    func connect(ssid: String, password: String) {
        //     currentSSID = ssid
        //     currentPassword = password
        let local_ssid = ssid.utf8CString
        let local_pass = password.utf8CString

        //TODO: test with span on beta? 
        // Span was not rendering with 6.2 Sept 6 snapshot and lifetime. 
        local_pass.withContiguousStorageIfAvailable { pass_buffer in
            local_ssid.withContiguousStorageIfAvailable { ssid_buffer in
                checkWithFatal(
                    wifi_bridge_wifi_set_config_and_connect(
                        ssid_buffer.baseAddress, pass_buffer.baseAddress))
            }
        }
        // Compiles, runs, but spits out garbage.
        // checkWithFatal(wifi_bridge_wifi_set_config_and_connect(&local_ssid, &local_pass))
    }
}

ErrorHandler.swift

typealias SDKError = esp_err_t

func checkWithFatal(_ error:SDKError, message:String) {
    guard error == ESP_OK else {
        fatalError(message)
    }
}

func checkWithFatal(_ error:SDKError) {
    guard error == ESP_OK else {
        fatalError(String(cString:esp_err_to_name(error)))
    }
}

func checkWithThrow(_ error:SDKError, throws swiftError: some Error) throws {
    guard error == ESP_OK else {
        throw(swiftError)
    }
}

func checkWithThrow(_ error:SDKError) throws {
    guard error == ESP_OK else {
        throw(SDKErrorWrapper.passMessage(String(cString:esp_err_to_name(error))))
    }
}

enum SDKErrorWrapper:Error {
    case passMessage(String)
}

Component Internals

This is what the project layout looks like with the component now.

|- components
|  |- wifi_bridge
|  |  |- include
|  |  |  |- wifi_bridge.h
|  |  |- CMakeLists.txt
|  |  |- common_config.txt (vestigial from previous version)
|  |  |- idf_component.yml (needs to be updated)
|  |  |- wifi_bridge.c 
|- main
|  |- Bool_Int.swift
|  |- BridgingHeader.swift
|  |- CMakeLists.txt
|  |- Delay.swift
|  |- DigitalIndicator.swift
|  |- ErrorHandler.swift
|  |- GPIOPin.swift
|  |- Main.swift
|  |- MomentaryInput.swift
|  |- Wifi.swift
|  |- idf_component.yml
|
|- CMakeLists.txt
|- .gitignore

wifi_bridge.h

#include <string.h>
#include "esp_wifi.h"
#include "esp_event.h"
#include "esp_log.h"
#include "nvs_flash.h"

int wifi_bridge_return_twelve(void);

void test_string_passing(const char *bridge_first, const char *bridge_last);

esp_err_t wifi_bridge_initialize_nvs(void);

esp_err_t wifi_bridge_initialize_netif(void);

esp_err_t wifi_bridge_wifi_init_default_config(void);

esp_err_t wifi_bridge_wifi_set_config_and_connect(const char *wifi_ssid, const char *wifi_pass);

wifi_bridge.c

#include "wifi_bridge.h"


int wifi_bridge_return_twelve(void)
{
    return 12;
}

esp_err_t wifi_bridge_initialize_nvs(void) {
    esp_err_t ret = nvs_flash_init();
    if (ret == ESP_ERR_NVS_NO_FREE_PAGES || ret == ESP_ERR_NVS_NEW_VERSION_FOUND) {
        esp_err_t flash_ret = nvs_flash_erase();
        if (flash_ret == ESP_OK) {
             ret = nvs_flash_init();
        } else {
            ret = flash_ret;
        }
    }
    return ret;
}

//TODO: what if needed that pointer later? 
esp_err_t wifi_bridge_initialize_netif(void) {
    esp_err_t ret = esp_netif_init();
    if (ret == ESP_OK) {
        ret = esp_event_loop_create_default();
    }
    esp_netif_t *sta_netif = esp_netif_create_default_wifi_sta();
    assert(sta_netif);
    return ret ;
}

esp_err_t wifi_bridge_wifi_init_default_config(void) {
    wifi_init_config_t cfg = WIFI_INIT_CONFIG_DEFAULT();
    return esp_wifi_init(&cfg);
}

// #define TMP_SSID "example"
// #define TMP_PASS "example"


//strings don't actually work
esp_err_t wifi_bridge_wifi_set_config_and_connect(const char *wifi_ssid, const char *wifi_pass) {
    //uint8_t ssid[32]
    //uint8_t password[64]

    // wifi_config_t wifi_config = {
    //      .sta = { 
    //          .ssid = TMP_SSID, 
    //          .password = TMP_PASS, 
    //      }, 
    //  }; 

    printf("ssid: %s, pass: %s\n", wifi_ssid, wifi_pass);

    wifi_config_t wifi_config = {0}; // empty config
    // //count is the maximum 
    strncpy((char *)wifi_config.sta.ssid, wifi_ssid, sizeof(wifi_config.sta.ssid));
    strncpy((char *)wifi_config.sta.password, wifi_pass, sizeof(wifi_config.sta.password));

    esp_err_t ret = esp_wifi_set_mode(WIFI_MODE_STA);
    if (ret == ESP_OK) {
        ret = esp_wifi_set_config(WIFI_IF_STA, &wifi_config);
    } else {
        return ret;
    }

    if (ret == ESP_OK) {
        ret = esp_wifi_start();
    } else {
        return ret;
    }

    if (ret == ESP_OK) {
        ret = esp_wifi_connect();
    } else {
        return ret;
    }

    return ret;
}

CMakeLists.txt

idf_component_register(SRCS "wifi_bridge.c"
                    PRIV_REQUIRES esp_wifi nvs_flash esp_netif
                    INCLUDE_DIRS "include")

Summary

Went through some dead end investigations that I think will have served me well in the future. An HTTP get request is next!

Can I combine an LED and a button on the ESP32C6 with Swift?

Mon, 08 Sep 2025 00:00:00 +0000

So I wasn’t satisfied with just adding a switch to the existing demo… I had to start making it fancier.

For some reason right about when I switched to writing my own Swift code my esp-idf install stopped working? It took me awhile to realize to was the tooling, not my code. The idf was handing out error messages about the creation of project_description.json on ALL projects and idf.py menuconfig crashed hard in multi-colored glory when called, even on examples that worked previously.

I reinstalled the whole esp-idf again and everything worked fine. I’m not sure if it was a software update overnight on my computer or if my spelunking around in that folder did something. Thank goodness for high speed internet because that was another big download.

After all that I ended up with two more steps of the project done:

rewriting the base: https://github.com/carlynorama/swift_esp32c6_hello/tree/a_start_with_LED
adding a switch: https://github.com/carlynorama/swift_esp32c6_hello/tree/b_add_a_switch

Compare to:

PicoW Version from Last Year: https://github.com/carlynorama/swift-pico-w-hello/tree/main/01-OnboardLED
PicoW Version from Last Year: https://github.com/carlynorama/swift-pico-w-hello/tree/main/02-Switch
Base project: https://github.com/swiftlang/swift-embedded-examples/tree/main/esp32-led-blink-sdk

Project Anatomy

I’m going to focus on recapping the project with both the LED and the button working, which has the following files:

|- main
|  |- Bool_Int.swift
|  |- BridgingHeader.swift
|  |- CMakeLists.txt
|  |- Delay.swift
|  |- DigitalIndicator.swift
|  |- MomentaryInput.swift
|  |- GPIOPin.swift
|  |- Main.swift
|  |- idf_component.yml
|
|- CMakeLists.txt
|- .gitignore

All of these files were added by hand to main/CMakeLists.txt

# List of Swift source files to build.
target_sources(${COMPONENT_LIB}
    PRIVATE
    Main.swift
    Bool_Int.swift
    GPIOPin.swift
    DigitalIndicator.swift
    MomentaryInput.swift
    Delay.swift
)

Main.swift

Notice that the only call directly backed by the esp32C6 sdk is print, many SDK’s implement print but it isn’t guaranteed. ESP_LOG and print do not behave identically. That’s its own post! What we do have is a DigitalIndicator type and a MomentaryInput type. I think it’s possible those would be better as protocols (DigitallyIndicating and ``DigitalReceiving`?), but for now they’re concrete types.

@_cdecl("app_main")
func main() {
  print("Hello from Swift on ESP32-C6!")

  guard var led = DigitalIndicator(15) else {
    fatalError("Difficulty setting up pin.")
  }

  guard let button = MomentaryInput(9) else {
    fatalError("Difficulty setting up button.")
  }

  while true {
    if button.isActive {
      led.blink(millis: 500)
    } else {
      led.blink(millis: 2000)
    }
  }
}

DigitalIndicator.swift

Also NO esp32c6 specific code

A DigitalIndicator has some more things defined

OutputPin: currently concrete implementation of the GPIOPin protocol, GPIOPin.swift
GPIOLevel a binary enum with cases (.high, .low), GPIOPin.swift
delay(_:millis:UInt32), Delay.swift

struct DigitalIndicator {
    let pin:OutputPin 
    let activeLevel: GPIOLevel
    var expectedState: Bool

    func set(on:Bool) {
        let level = on ? activeLevel.asBool : !activeLevel.asBool
        pin.setLevel(levelHigh:level)
    }

    mutating func turnOn() {
        self.set(on: true)
        expectedState = true
    }

        mutating func turnOff() {
        self.set(on: false)
        expectedState = false
    }

    mutating func toggle() {
        let level: Bool = expectedState ? false : true
        self.set(on: level)
        expectedState = level
    }

    mutating func blink(millis:UInt32) {
        self.toggle()  // Toggle the boolean value
        delay(millis)
    }

    mutating func blink(onMillis:UInt32, offMillis:UInt32) {
        self.turnOn()
        delay(onMillis)
        self.turnOff()
        delay(offMillis)
    }
}

extension DigitalIndicator {
    init?(_ pinNum: UInt32, activeLow:Bool = true) {
        self.pin = OutputPin(pinNumber: pinNum, activeLow:activeLow)
        self.activeLevel = GPIOLevel(!activeLow)
        self.expectedState = false
    }
}

Delay.swift

A ESP32C6 specific implementation of a delay function. TODO: why did the demo code use configTICK_RATE_HZ instead of portTICK_PERIOD_MS?

func delay(_ millis:UInt32) {
    vTaskDelay(millis / (1000 / UInt32(configTICK_RATE_HZ)))
}

GPIOPin.swift:GPIOLevel and Bool_Int.swift

No esp32c6 specific code. But maybe it could. Considering creating a typealias for archInt and archUInt.

These two work to clarify that 1, true and .high all mean the same thing, and to translate C’s 1/0 returns to true/false easily as needed.

//TODO: raw value? Bool or UInt32? 
enum GPIOLevel:Equatable {
    case high
    case low
}

extension GPIOLevel {
    init(_ v:Int32) {
        if v == 0 {
            self = .low
        } else if v == 1 {
            self = .high
        } else {
            fatalError("Expected 0 or 1, got \(v)")
        }
    }

    init(_ v:UInt32) {
        if v == 0 {
            self = .low
        } else if v == 1 {
            self = .high
        } else {
            fatalError("Expected 0 or 1, got \(v)")
        }
    }

    init(_ v:Bool) {
        if v == false {
            self = .low
        } else {
            self = .high
        }
    }

    var asUInt32:UInt32 {
        (self == .high) ? 1 : 0
    }

    var asInt32:Int32 {
        (self == .high) ? 1 : 0
    }

    var asBool:Bool {
        (self == .high) ? true : false
    }
}

extension Bool {
    init(_ v:UInt32) {
        if v == 0 {
            self = false
        } else if v == 1 {
            self = true
        } else {
            fatalError("Expected 0 or 1, got \(v)")
        }
    }

    init(_ v:Int32) {
        if v == 0 {
            self = false
        } else if v == 1 {
            self = true
        } else {
            fatalError("Expected 0 or 1, got \(v)")
        }
    }

    var asUInt32:UInt32 {
        self ? 1 : 0
    }

}

GPIOPin.swift:GPIODirection

This has a esp32C6 dynamic var, could be an extension in sub package.

enum GPIODirection  {
    case input
    case output
}

//esp32C6 implementation
extension GPIODirection {
    var esp32C6:gpio_mode_t {
        switch(self) {
            case .input: 
                return GPIO_MODE_INPUT
            case .output: 
                return GPIO_MODE_OUTPUT
        }
    }
}

GPIOPin.swift:GPIOPin

GPIOPin is a protocol

protocol GPIOPin {
    var pinNumber:UInt32 { get }
    var direction: GPIODirection { get }
}

GPIOPin.swift:OutputPin

I decided to make OutputPin a Struct (esp32c6 specific) rather than a Protocol for now because I didn’t want to deal with something like DigitalIndicator<ESP32C6OutputPin>(15) for now. This could also be handled by typaliasing, but I have some questions around how to best use Swift embedded with existentials and generics that I’d like to temporarily side step.

This being a struct is a tad deceiving, It’s backed by C functions that point to specific hardware registers. In a Swift-all-the-way-down implementation I would maybe choose a final Class or even an Actor if that becomes a thing one could do?

//MARK: OutputPin
struct OutputPin:GPIOPin {
    let pinNumber:UInt32
    let direction:GPIODirection = .output

    func setLevel(levelHigh:Bool) {
        gpio_set_level(gpio_num_t(Int32(pinNumber)), levelHigh.asUInt32)
    }

    func setLevel(level:GPIOLevel) {
        gpio_set_level(gpio_num_t(Int32(pinNumber)), level.asUInt32)
    }
}


//esp32C6 init
extension OutputPin {
    init(pinNumber:UInt32, activeLow:Bool = true) {
        //validate is GPIO output pin.
        let ledPin = gpio_num_t(Int32(pinNumber))
        self.pinNumber = pinNumber

        guard gpio_reset_pin(ledPin) == ESP_OK else {
            fatalError("cannot reset output pin \(pinNumber)")
        }
        guard gpio_set_direction(ledPin, self.direction.esp32C6) == ESP_OK else {
            fatalError("cannot reset output pin \(pinNumber)")
        }
    }
}

GPIOPin.swift:InputPin

ESP32C6 specific implementation, see notes above with OutputPin

struct InputPin:GPIOPin {
    let pinNumber:UInt32
    let direction:GPIODirection = .input

    //If the pad is not configured for input (or input and output) the returned value is always 0.
    func readLevel() -> GPIOLevel {
        GPIOLevel(gpio_get_level(gpio_num_t(Int32(pinNumber))))
    }
}

extension InputPin {
    init(pinNumber:UInt32, activeLow:Bool = true, useInternalHardware:Bool = true) {
        //validate is GPIO output pin.
        let ledPin = gpio_num_t(Int32(pinNumber))
        self.pinNumber = pinNumber

        guard gpio_reset_pin(ledPin) == ESP_OK else {
            fatalError("cannot reset output pin \(pinNumber)")
        }

        guard gpio_set_direction(ledPin, self.direction.esp32C6) == ESP_OK else {
            fatalError("cannot reset output pin \(pinNumber)")
        }

        //validate pin has hardware 
        if activeLow && useInternalHardware {
            //use gpio_pull_mode_t
            gpio_set_pull_mode(ledPin, GPIO_PULLUP_ONLY)
        } else if !activeLow && useInternalHardware {
            gpio_set_pull_mode(ledPin, GPIO_PULLDOWN_ONLY)
        } else {
            //could set with GPIO_FLOATING, but for now want to leave it as default
        }
    }
}

Summary

All of this was pretty easy once I got the the tooling issues sorted. The next batch, getting the WiFi working will require adding a C bridging component. I tried without but the WiFi library uses function like macros, and while there maybe a fancy work around, i’m just going to add a lib.

How does the espressif SDK handle wifi?

Sun, 07 Sep 2025 00:00:00 +0000

Next step for me was to see if I could get any of the WiFi examples working. Which I did. This post is definately more notes than a how-to.

Here are the docs I used:

Here are the code examples I mainly looked at:

What are the different modes?

- Station mode (aka STA mode or Wi-Fi client mode). ESP32 connects to an access point. 
- AP mode (aka Soft-AP mode or Access Point mode). Stations connect to the ESP32.
- Station/AP-coexistence mode (ESP32 is concurrently an access point and a station connected to another access point).

What was it like running the different examples.

I followed the same pattern from the getting started, copying the folders and then running them.

. $HOME/esp/esp-idf/export.sh
SOME_PORT=/dev/$(ls /dev/ | grep cu.usbmodem)
cd ~/esp
cp -r $IDF_PATH/examples/wifi/scan wifi_scan
cd wifi_scan
idf.py set-target esp32c6
idf.py menuconfig  # especially for the secon two
idf.py build # just to catch any build errors first
idf.py -p $SOME_PORT flash monitor

WiFi Scan

https://github.com/espressif/esp-idf/tree/v5.5.1/examples/wifi/scan

The scan sketch failed to run the first time I tried to build and flash it with the following error"

File "/Users/$MY_USER/.espressif/python_env/idf5.5_py3.13_env/lib/python3.13/site-packages/serial/serialposix.py", line 517, in _reconfigure_port termios.tcsetattr( ~~~~~~~~~~~~~~~~~^ self.fd, ^^^^^^^^ termios.TCSANOW, ^^^^^^^^^^^^^^^^ [iflag, oflag, cflag, lflag, ispeed, ospeed, cc]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ termios.error: (6, 'Device not configured')

But it worked again just fine when I re-ran it. No extra work on my part.

I was not able to see many WiFI networks because I have not attached an external antenna (search on page for antenna), but I could see my home one just fine.

Also noted the introduction of ESP_ERROR_CHECK(someFunction); to catch esp_err_t errors. These depend on an esp_check.h include I don’t actually see in the header, so some other library must include it publicly, or it may even be in the base project include some how, although I did not find an explicit reference to it in project.cmake, target.cmake or idf.cmake.

New libraries

“esp_wifi.h”: Simple Wifi
“esp_event.h”: Event Loop Library “The event loop library allows components to declare events so that other components can register handlers – codes that executes when those events occur. This allows loosely-coupled components to attach desired behavior to state changes of other components without application involvement… One common case is, if a high-level library is using the Wi-Fi library: it may subscribe to ESP32 Wi-Fi Programming Model directly and act on those events.”
“freertos/event_groups.h”: Event Groups API
“nvs_flash.h”: Non-Volatile Storage Library Handles remembering the WiFi Access Points found.
“regex.h”: C regex lib (current working directory over system headers) (Might be vestigial? Seems to build without it.)

Libraries Seen Before

“esp_log.h”: Logging Library
<string.h> : C String Library (system)
“freertos/FreeRTOS.h” : FreeRTOS main

Not explicitly imported:

WiFi Getting Started Station

https://github.com/espressif/esp-idf/tree/v5.5.1/examples/wifi/getting_started/station

Similar to scan, but also handles a login. Also sets up for other “event types” going forward.

Station Example also ran fine after menuconfig (not before). It includes teh LwIP libraries, which I think are being used by the IP_EVENT handler?

New Includes

“freertos/task.h” : FreeRTOS TaskHandle
“esp_system.h” : ESP System
“lwip/err.h” : Lightweight TCP/IP stack, Errors
“lwip/sys.h” : Lightweight TCP/IP stack, Errors System

Seen Before

<string.h>
“freertos/FreeRTOS.h”
“freertos/event_groups.h”
“esp_wifi.h”
“esp_event.h”
“esp_log.h”
“nvs_flash.h”

The two main functions. An event handler, and an initializer.

The event_handler is the handler given to the esp_event_handler_instance_register in the start up.

I’m unclear if Swift will play nice with this event handler.

static void event_handler(void* arg, esp_event_base_t event_base,
                                int32_t event_id, void* event_data)
{
    if (event_base == WIFI_EVENT && event_id == WIFI_EVENT_STA_START) {
        esp_wifi_connect();
    } else if (event_base == WIFI_EVENT && event_id == WIFI_EVENT_STA_DISCONNECTED) {
        if (s_retry_num < EXAMPLE_ESP_MAXIMUM_RETRY) {
            esp_wifi_connect();
            s_retry_num++;
            ESP_LOGI(TAG, "retry to connect to the AP");
        } else {
            xEventGroupSetBits(s_wifi_event_group, WIFI_FAIL_BIT);
        }
        ESP_LOGI(TAG,"connect to the AP fail");
    } else if (event_base == IP_EVENT && event_id == IP_EVENT_STA_GOT_IP) {
        ip_event_got_ip_t* event = (ip_event_got_ip_t*) event_data;
        ESP_LOGI(TAG, "got ip:" IPSTR, IP2STR(&event->ip_info.ip));
        s_retry_num = 0;
        xEventGroupSetBits(s_wifi_event_group, WIFI_CONNECTED_BIT);
    }
}

```C
void wifi_init_sta(void)
{
    s_wifi_event_group = xEventGroupCreate();

    ESP_ERROR_CHECK(esp_netif_init());

    ESP_ERROR_CHECK(esp_event_loop_create_default());
    esp_netif_create_default_wifi_sta();

    wifi_init_config_t cfg = WIFI_INIT_CONFIG_DEFAULT();
    ESP_ERROR_CHECK(esp_wifi_init(&cfg));

    esp_event_handler_instance_t instance_any_id;
    esp_event_handler_instance_t instance_got_ip;
    ESP_ERROR_CHECK(esp_event_handler_instance_register(WIFI_EVENT,
                                                        ESP_EVENT_ANY_ID,
                                                        &event_handler,
                                                        NULL,
                                                        &instance_any_id));
    ESP_ERROR_CHECK(esp_event_handler_instance_register(IP_EVENT,
                                                        IP_EVENT_STA_GOT_IP,
                                                        &event_handler,
                                                        NULL,
                                                        &instance_got_ip));

    wifi_config_t wifi_config = {
        .sta = {
            .ssid = EXAMPLE_ESP_WIFI_SSID,
            .password = EXAMPLE_ESP_WIFI_PASS,
            /* Authmode threshold resets to WPA2 as default if password matches WPA2 standards (password len => 8).
             * If you want to connect the device to deprecated WEP/WPA networks, Please set the threshold value
             * to WIFI_AUTH_WEP/WIFI_AUTH_WPA_PSK and set the password with length and format matching to
             * WIFI_AUTH_WEP/WIFI_AUTH_WPA_PSK standards.
             */
            .threshold.authmode = ESP_WIFI_SCAN_AUTH_MODE_THRESHOLD,
            .sae_pwe_h2e = ESP_WIFI_SAE_MODE,
            .sae_h2e_identifier = EXAMPLE_H2E_IDENTIFIER,
        },
    };
    ESP_ERROR_CHECK(esp_wifi_set_mode(WIFI_MODE_STA) );
    ESP_ERROR_CHECK(esp_wifi_set_config(WIFI_IF_STA, &wifi_config) );
    ESP_ERROR_CHECK(esp_wifi_start() );

    ESP_LOGI(TAG, "wifi_init_sta finished.");

    /* Waiting until either the connection is established (WIFI_CONNECTED_BIT) or connection failed for the maximum
     * number of re-tries (WIFI_FAIL_BIT). The bits are set by event_handler() (see above) */
    EventBits_t bits = xEventGroupWaitBits(s_wifi_event_group,
            WIFI_CONNECTED_BIT | WIFI_FAIL_BIT,
            pdFALSE,
            pdFALSE,
            portMAX_DELAY);

    /* xEventGroupWaitBits() returns the bits before the call returned, hence we can test which event actually
     * happened. */
    if (bits & WIFI_CONNECTED_BIT) {
        ESP_LOGI(TAG, "connected to ap SSID:%s password:%s",
                 EXAMPLE_ESP_WIFI_SSID, EXAMPLE_ESP_WIFI_PASS);
    } else if (bits & WIFI_FAIL_BIT) {
        ESP_LOGI(TAG, "Failed to connect to SSID:%s, password:%s",
                 EXAMPLE_ESP_WIFI_SSID, EXAMPLE_ESP_WIFI_PASS);
    } else {
        ESP_LOGE(TAG, "UNEXPECTED EVENT");
    }
}

WiFi HTTP Request

https://github.com/espressif/esp-idf/tree/v5.5.1/examples/protocols/http_request

New Libraries

This minimal example just uses lwip directly.

“lwip/err.h”
“lwip/sockets.h”
“lwip/sys.h”
“lwip/netdb.h”
“lwip/dns.h”

Note there is also a more complete http client example that uses an esp written http client library

These examples use a dependency on protocol_examples_common found in the idf_component.yml

${IDF_PATH}/examples/common_components/protocol_examples_common

The example itself says its’ not a good WiFi implementation.

/**
 * @brief Configure Wi-Fi or Ethernet, connect, wait for IP
 *
 * This all-in-one helper function is used in protocols examples to
 * reduce the amount of boilerplate in the example.
 *
 * It is not intended to be used in real world applications.
 * See examples under examples/wifi/getting_started/ and examples/ethernet/
 * for more complete Wi-Fi or Ethernet initialization code.
 *
 * Read "Establishing Wi-Fi or Ethernet Connection" section in
 * examples/protocols/README.md for more information about this function.
 *
 * @return ESP_OK on successful connection
 */
esp_err_t example_connect(void);

/**
 * Counterpart to example_connect, de-initializes Wi-Fi or Ethernet
 */
esp_err_t example_disconnect(void);

Did look briefly into trying to make my own Wifi wrapper component:

But decided to wait and see if I needed to do that to make WiFi in Swift work or not. I suspect I might, but not today.

Summary

I have now seen the board do everything I plan for the Swift demo for Sketching this year using just the esp-idf tools. Next step is to transfer this knowledge to Swift!

How does the espressif SDK handle inputs?

Sat, 06 Sep 2025 00:00:00 +0000

So even though my target is writing as much as possible in Swift, my process will be to make things work in C first.

The next simple step is to check how to add a digital input pin.

Here are the docs I used:

Here are the examples I mainly looked at (not to be confused with the sleep examples in OpenThread directory). Most of these inputs were interrupt enabled, which I will likely use later but did not in this example.

This was the resulting project:

https://github.com/carlynorama/esp32c6_tests/tree/add_button

Peculiarities of Using the Boot Button

Adding a basic input button has pretty much the same steps no matter the chip.

the micro has to know the pin will be an input
the micro has to know if internal hardware (pullups, pull downs) are being used

Given chips and given pins on those chips might have some specialty behaviors, but those are the two biggies.

On the ESP32-C6 GPIO 9 is one of the so-called “strapping” pins, pins that are associated with telling the micro what boot mode to be in.

From the technical reference manual (9.2.1) the default configuration of GPIO9 will be to have it’s internal pull up enabled.

By default, GPIO9 is connected to the chip’s internal pull-up resistor. If GPIO9 is not connected or is connected to an external high-impedance circuit, the internal weak pull-up determines the default input level of this strapping pin (see Table 9.2-1).

The manual continues in section 9.2.2:

The values of GPIO8 and GPIO9 at reset determine the boot mode after the reset is released. Table 9.2-2 shows the strapping pin values of GPIO8 and GPIO9, and the associated boot modes. (x: this value is ignored.)

Boot Mode	GPIO9	GPIO8
SPI Boot	1	x
Download Boot	0	1

We can see on the Xiao’s schematic (labeled net BOOT)that Seeed has attached the appropriate active low switch so that GPIO9 stays high by default.

This all means that when using pin 9 failing to set the internal pull up resistor won’t be a big deal because it will be set for us. It also means that the boot switch must be active low if we don’t want to change the expected default boot behaviors.

The ESP32-C6 SDK offers a handy command gpio_dump_io_configuration to print out the current pin, as discussed in the API reference.

It looks like:

================IO DUMP Start================
IO[9] -
  Pullup: 1, Pulldown: 0, DriveCap: 2
  InputEn: 1, OutputEn: 0, OpenDrain: 0
  FuncSel: 0 (IOMUX)
  SleepSelEn: 1

IO[15] -
  Pullup: 1, Pulldown: 0, DriveCap: 2
  InputEn: 0, OutputEn: 1, OpenDrain: 0
  FuncSel: 1 (GPIO)
  GPIO Matrix SigOut ID: 128 (simple GPIO output)
  SleepSelEn: 1

=================IO DUMP End=================

Example Code

// Needed for logging to serial monitor
// with TAG prefix
#include <stdio.h>
#include "esp_log.h"

// Needed for portTICK_PERIOD_MS
#include "freertos/FreeRTOS.h"
#include "freertos/task.h"
// Needed for io pins
#include "driver/gpio.h"

// Blinking related defines
#define BLINK_GPIO          15
#define BLINK_PERIOD_SLOW   2000
#define BLINK_PERIOD_FAST   500

// Button related defines
#define BOOT_BUTTON_NUM     9
#define MY_BUTTON_NUM BOOT_BUTTON_NUM
// active low
#define MY_BUTTON_PRESSED_STATE 0

// logging prefix
static const char *TAG = "simple_button";

// current state of the LED
static uint8_t s_led_state = 0;

//current blink period
static uint32_t current_blink_period = BLINK_PERIOD_SLOW;


//----------- LED
static void configure_led(void)
{
    ESP_LOGI(TAG, "Example configured to blink GPIO LED!");
    gpio_reset_pin(BLINK_GPIO);
    gpio_set_direction(BLINK_GPIO, GPIO_MODE_OUTPUT);
}

static void blink_led(void)
{
    /* Set the GPIO level according to the state (LOW or HIGH)*/
    gpio_set_level(BLINK_GPIO, s_led_state);
}


//----------- Button

static void configure_button(void)
{
    gpio_set_direction(MY_BUTTON_NUM, GPIO_MODE_INPUT);
    //??? No need to set internal pulls? 
}

// will return 1 when the button read matches the value expected for a 
// pressed button. 
static uint8_t read_button(){
    return (gpio_get_level(MY_BUTTON_NUM) == MY_BUTTON_PRESSED_STATE);
}

//----------- Configure Info 
static void reveal_configuration(void) {
    //shows all
    //gpio_dump_io_configuration(stdout, SOC_GPIO_VALID_GPIO_MASK);
    gpio_dump_io_configuration(stdout, (1ULL << BLINK_GPIO) | (1ULL << BOOT_BUTTON_NUM));
}

void app_main(void)
{
    // Configure the peripherals
    configure_led();
    configure_button();
    //print out the information
    reveal_configuration();


    while (1) {
        // pin HIGH is __LED OFF__ for the XIAO built in LED. 
        ESP_LOGI(TAG, "Turning the LED %s!", s_led_state == true ? "OFF":"ON");
        blink_led();
        /* Toggle the LED state */
        s_led_state = !s_led_state;

        // if button is true, blink fast, else blink slow
        if (read_button() == 1) {
            current_blink_period = BLINK_PERIOD_FAST;
        } else {
            current_blink_period = BLINK_PERIOD_SLOW;
        }

        vTaskDelay(current_blink_period / portTICK_PERIOD_MS);
    }
}

Summary

That’s enough for one round. Next step is to confirm I can make the WiFi work with just the SDK as well.

Can I program an ESP32C6 with the Swift example code?

Fri, 05 Sep 2025 00:00:00 +0000

So the crazy thing is how much easier this process was than last year getting Swift up and running on the picow.

Between the improved SwiftLang embedded docs and example repo, Swiftly being available on the Mac, espressif support… a comparative cake walk. Seeed also gives a shout out on how to use Swift with this exact board, but its docs are little behind. Not because they’re slow to update, just because things have moved that fast.

Run an Example

It is still recommended to use a development chain which is now easy to get with Swiftly.

The below are the commands only from a article from espressif with only a couple of tweaks, it assumes there is a working esp-idf installed, but that hasn’t already been loaded into the shell.

Also, provided in that article is a Wokwi demo, which is a very very nice touch!

## Locate the esp-idf install, if haven't added alias to shell
## assumes it lives in user root. 
. $HOME/esp/esp-idf/export.sh
# brew install swiftly
## Install Swift 6.2 development snapshot and switch to it when done
swiftly install --use 6.2-snapshot 
## put this where you want it:
git clone https://github.com/swiftlang/swift-embedded-examples.git --single-branch --branch main
## change into example folder
cd swift-embedded-examples/esp32-led-strip-sdk

## set the target for the project
idf.py set-target esp32c6
## build and run
idf.py build flash monitor
# Use Ctrl+] to quit the monitor application.

For me it just worked! Whew! That was nice!

Optional Environment Improvement, add alias for esp-idf

Consider adding an alias to make loading in the location of the esp-idf tools easier. from the docs:

Copy and paste the following command to your shell’s profile (.profile, .bashrc, .zprofile, etc.)

alias get_idf='. $HOME/esp/esp-idf/export.sh'

Refresh the configuration by restarting the terminal session or by running source [path to profile], for example, source ~/.bashrc. Now you can run get_idf to set up or refresh the esp-idf environment in any terminal session.


## Summary

So there are two things on deck for me next

A. get a switch working
B. reorganize the code

And I'm not sure which one I'm going to prioritize so I'll save it for the next post.

Can I program a an ESP32C6 with the esp-idf?

Thu, 04 Sep 2025 00:00:00 +0000

So to go along with the server I want to have an object that talks to it. Which means one with a stable WiFi chip. Very popular right now is espressif’s ESP32-C6

The ESP32-C6 SoC (System on Chip) supports Wi-Fi 6 in 2.4 GHz band, Bluetooth 5, Zigbee 3.0 and Thread1.3. It consists of a high-performance (HP) 32-bit RISC-V processor, an low-power (LP) 32-bit RISC-V processor,wireless baseband and MAC (Wi-Fi, Bluetooth LE, and 802.15.4), RF module, and numerous peripherals. Wi-Fi,Bluetooth and 802.15.4 coexist with each other and share the same antenna.

I was interested in trying out a RISC-V chip since I hadn’t played with one of those before.

Seeed Studio’s XIAO ESP32C6 won out as the dev board, honestly because todbot collects boards that have the same form factor as Adafruit’s QT Py and he had an extra. Also its small size is perfect for my target project.

Other ESP32-C6 dev boards include (info links, not purchase links):

About the Board

The XIAO ESP3232C6 board is tiny, but it packs A LOT in.

https://wiki.seeedstudio.com/xiao_esp32c6_getting_started/

Of the two led’s on the board one is the classic “built in led” to blink hello world to, and the other is hooked into the battery management system. Hopefully I’ll get back to that later.

The red light behavior for the XIAO ESP32C6 is as follows:

When no battery is connected:
The red light turns on when the Type-C cable is connected and turns off after 30 seconds.
When a battery is connected and the Type-C cable is plugged in for charging:
The red light flashes.
When the battery is fully charged via the Type-C connection:
The red light turns off.

The two buttons are the reset button and the boot button.

While the boot button is tiny I’m hoping later in the project I should be able to find a toothpick or something to temporarily hijack it for hello worlding a button.

Board Proof of Life

It’s fairly rare these days to get a dud board, but it’s best to make sure the board actually works with familiar tooling before launching into unfamiliar tooling.

Seeed actually provides step by step instructions with screen shots on how to get things set up in the Arduino IDE, an environment I know well.

Quick link to the board profile if you know your way around: https://github.com/espressif/arduino-esp32

Change the blink sketch to uneven

One of the things I do with the example blink sketch is change the the blink patter to be uneven.

// the setup function runs once when you press reset or power the board
void setup() {
  // initialize digital pin LED_BUILTIN as an output.
  pinMode(LED_BUILTIN, OUTPUT);
}

// the loop function runs over and over again forever
void loop() {
  digitalWrite(LED_BUILTIN, HIGH);  // turn the LED on (HIGH is the voltage level)
  delay(500);                      // wait for a second
  digitalWrite(LED_BUILTIN, LOW);   // turn the LED off by making the voltage LOW
  delay(2000);                      // wait for a second
}

On the Xiao, this code will have the LED be ON for longer than OFF because the LED’s current is being sinked by the pin, rather than sourced from the pin. This is more the norm in production electronics vs hobby electronics more.

See the The “GPIO15” labeled net on the schematic (big red X means not populated): https://files.seeedstudio.com/wiki/SeeedStudio-XIAO-ESP32C6/XIAO-ESP32-C6_v1.0_SCH_PDF_24028.pdf
Where pin 15 is declared as the builtin in the code: https://github.com/espressif/arduino-esp32/blob/561136870b4308d9deb942461e63e42c788546f3/variants/XIAO_ESP32C6/pins_arduino.h#L13

Install Official Tools

Espressif wraps its SDK in a pretty robust set of tools (Integrated Development Framework), for which they’ve written very explicit install instructions:

https://docs.espressif.com/projects/esp-idf/en/v5.5.1/esp32c6/get-started/linux-macos-setup.html

Here’s a summary, but I suggest reading the full page, making sure its actually about the target you’re looking for. (i.e. the url says esp32c6 NOT simply esp32. esp32 is actually a different family than esp32c6 and its tooling won’t work!)

## required
brew install cmake ninja dfu-util
## recommended
brew install ccache

Had:

New:

https://dfu-util.sourceforge.net
- dfu-util is a host side implementation of the DFU 1.0 and DFU 1.1 specifications of the USB forum. DFU is intended to download and upload firmware to/from devices connected over USB.
https://ccache.dev
- Ccache is a compiler cache. It speeds up recompilation by caching previous compilations and detecting when the same compilation is being done again.

Note, ccache posted a fairly ominous warning… maybe I will start using containers more…

To install symlinks for compilers that will automatically use
ccache, prepend this directory to your PATH:
  /opt/homebrew/opt/ccache/libexec

If this is an upgrade and you have previously added the symlinks to
your PATH, you may need to modify it to the path specified above so
it points to the current version.

NOTE: ccache can prevent some software from compiling.
ALSO NOTE: The brew command, by design, will never use ccache.

So espressif wants you to put their tooling at the root of the user, but does provide instruction on how to map to somewhere else.

Also, there is a VSCode plugin that will do this all for you, but I find VSCode tooling very aggressive and prefer custom scripts that don’t start automagically doing things I didn’t ask to be done. I also don’t enjoy vendor lock in. I also find doing these things myself helps with trouble shooting later. YMMV.

mkdir -p ~/esp 
cd ~/esp
git clone -b v5.5.1 --recursive https://github.com/espressif/esp-idf.git
## Install the tools
cd ~/esp/esp-idf
./install.sh esp32c6 #note NOT esp32, esp32c6 <---- !!!!!

Hello World with Official Tools

https://docs.espressif.com/projects/esp-idf/en/stable/esp32/get-started/linux-macos-setup.html#step-4-set-up-the-environment-variables

In the terminal window that will be the shell doing the building and running:

## Set up the environment variables
. $HOME/esp/esp-idf/export.sh
## Go to the dev folder
cd ~/esp
## pull out the example
cp -r $IDF_PATH/examples/get-started/hello_world .
## linux
# ls /dev/ | grep tty
## macOS 
ls /dev/ | grep cu. ## or ls /dev/cu.*

Mine was called /dev/cu.usbmodem2101, plug into a different port, it will change.

## SPECIFIC TO MINE
SOME_PORT=/dev/cu.usbmodem2101
cd ~/esp/hello_world
idf.py set-target esp32c6
# this creates sdkconfig, look at it
cat sdkconfig
## optional/not needed for hello world. THe below edits sdkconfig 
## with verification to help prevent errors
# idf.py menuconfig 
idf.py build
idf.py -p $SOME_PORT flash

# assumes 40 MHz, use idf.py menuconfig to change if needed
idf.py -p $SOME_PORT monitor
# CNTRL+] to leave

blink with official tools

The hello world example doesn’t touch the GPIO which, so on to blink.

https://github.com/espressif/esp-idf/blob/v5.5.1/examples/get-started/blink/README.md

If in a totally new Terminal window

. $HOME/esp/esp-idf/export.sh
ls /dev/ | grep cu. ## or ls /dev/cu.*
SOME_PORT= dev.cu.someusbmodem

Pull out the blink example code and go…

## or where ever you want to store your files
cd ~/esp
cp -r $IDF_PATH/examples/get-started/blink .
## Set the GPIO_BLINK pin to 15 here
cd blink
# this is PER PROJECT, do it in the project folder. 
idf.py set-target esp32c6
## MUST change Example Configuration. It's not an LED strip, GPIO pin (needs to be 15), etc. 
idf.py menuconfig
idf.py build
## do it on one this time
idf.py -p $SOME_PORT flash monitor

Also went fairly smoothly and easily. Yet I don’t know that I’ve actually learned anything really about the SDK, about RISC-V…

Pulling it Apart

One thing I will say about the VSCode plugin is that it did help making a new project off of a base template and that is nice. I will look to see if that’s a feature of the idf.

Starting from the blink example I tore out a lot of code and deleted a lot of files.

Tore out the specialized config stuff, left in the logging

https://github.com/carlynorama/esp32c6_tests/tree/simple_blink

// Needed for logging to serial monitor
#include <stdio.h>
#include "esp_log.h"

// Needed for portTICK_PERIOD_MS
#include "freertos/FreeRTOS.h"
#include "freertos/task.h"
// Needed for io pins
#include "driver/gpio.h"

#define BLINK_GPIO 15
#define BLINK_PERIOD 2000

// logging prefix
static const char *TAG = "simple_blink";

static uint8_t s_led_state = 0;

static void blink_led(void)
{
    /* Set the GPIO level according to the state (LOW or HIGH)*/
    gpio_set_level(BLINK_GPIO, s_led_state);
}

static void configure_led(void)
{
    ESP_LOGI(TAG, "Example configured to blink GPIO LED!");
    gpio_reset_pin(BLINK_GPIO);
    /* Set the GPIO as a push/pull output */
    gpio_set_direction(BLINK_GPIO, GPIO_MODE_OUTPUT);
}

void app_main(void)
{
    /* Configure the peripheral according to the LED type */
    configure_led();

    while (1) {
        // pin HIGH is __LED OFF__ for the XIAO built in LED. 
        ESP_LOGI(TAG, "Turning the LED %s!", s_led_state == true ? "OFF":"ON");
        blink_led();
        /* Toggle the LED state */
        s_led_state = !s_led_state;
        // calculating the millis by hand based on the crystal speed is 
        // ANNNNNOOOOOOYYYYING. Keeping FreeRTOS. 
        vTaskDelay(BLINK_PERIOD / portTICK_PERIOD_MS);
    }
}

Links to the functions that are left:

gpio_set_level
gpio_reset_pin
gpio_set_direction
- more on push-pull outputs
ESP_LOGI
- how it differs from printf
vTaskDelay and portTICK_PERIOD_MS
- also available: pdMS_TO_TICKS

Blink without the SDK?

To do this without the SDK, one has to do things like the make a vector table, etc. I found one example that did that for an older esp32 chip that was not a RISC-V.

https://github.com/aykevl/esp32-minimal

I think I want to save that energy for once I’ve started writing is Swift, if at all this project.

Summary

Pretty smooth sailing. Next post, blinking with Swift.

Swift Embedded Sept 2025 Link Dump

Wed, 03 Sep 2025 00:00:00 +0000

I have a bunch of notes in unpublished posts from 2024, which some day I’ll go back and publish. In the mean time I will just pull out a bunch of links from then and new ones that will help with the current project.

Official

Official Manufacturer

Specific Forum Posts

Misc Repos And Notes

Podman: How do I write a container that will do a build?

Tue, 02 Sep 2025 00:00:00 +0000

In yesterday’s notes I used a locally compiled binary with a very slim docker file. This can work out fine, but what comes recommended by Vapor and Hummingbird? Can I understand those files yet?

What I’ve figure out by now is that those files have a build section that will spin up a container to do the compile, no cross compiling SDK needed. Potentially that means the app service will do that compile for me on the hardware that the binary will run on? So I don’t even have to know what the underlying architecture is? Interesting… And then also how does that get worked in to the fee?

NOTE: (One can use the static-sdk in a Containerfile compile, it’s just that one doesn’t have to.)

But first steps first. Currently the project layout looks like:

|- Container
|  |- binary
|  |  |- SmallestServer (binary)
|  |- Containerfile
|  
|- Sources
|  |- main.swift
|
|- buildLocalNRun.sh
|- Package.swift

This is going to simplified to:

|- Sources
|  |- main.swift
|
|- buildContainerNRun.sh
|- Containerfile
|- Package.swift

The Container folder can be deleted because there is no intermediate build product to store in the repo now.

Update shell script

Update the script to remove any reference to building or compiling the package that way. The only thing that will build is the container.

#!/bin/sh

## change in script if want to change.
DEFAULT_PORT="8080"
APP_NAME="SmallestServer"
REPOSITORY="smallserver"

## can change at runtime 
## 'zsh ./buildLocalNRun.sh debug' will run debug build instead.
DEFAULT_CONFIG_VALUE="release"
CONFIGURATION="${1:-$DEFAULT_CONFIG_VALUE}"

## creates a tag based on the day and local time, e.g. mon104642
TAG=`date +"%a%H%M%S" | tr '[:upper:]' '[:lower:]'`

podman build -t $REPOSITORY:$TAG .
# no -d because want to see errors inline
# this one will keep podman a little tidier. 
# podman run --rm --rmi -p $DEFAULT_PORT:8080 $REPOSITORY
podman run -p $DEFAULT_PORT:8080 $REPOSITORY:$TAG

A different Containerfile

So the two example Dockerfiles do a lot more than the tiny one. They change up the memory allocator (although how long it will be to jemalloc tbd.), they make sure backtracing works, move in resource files.

What they are still doing is ending up with a mostly statically linked executable so the final container does not have to have Swift on it. It appears to still need a C library? Something else the ubuntu base image has that alpine doesn’t?

Let’s go through the Vapor example section by section.

The Build Section

Difference #1, there is a named build section based on an image that has Swift installed.

# ================================
# Build image
### CHECK OUT THAT NAME after the AS!! 
FROM swift:6.1-noble AS build

Then the base image gets updated, jemalloc gets installed, apt/lists gets cleaned out

# Install OS updates
RUN export DEBIAN_FRONTEND=noninteractive DEBCONF_NONINTERACTIVE_SEEN=true \
    && apt-get -q update \
    && apt-get -q dist-upgrade -y \
    && apt-get install -y libjemalloc-dev \
    && rm -rf /var/lib/apt/lists/*

Create a build area and move the project files into it, with a good reason to do the dependencies first (caching the dependencies).

# Set up a build area
WORKDIR /build

# First just resolve dependencies.
# This creates a cached layer that can be reused
# as long as your Package.swift/Package.resolved
# files do not change.
COPY ./Package.* ./
RUN swift package resolve \
        $([ -f ./Package.resolved ] && echo "--force-resolved-versions" || true)

# Copy entire repo into container
COPY . .

Then, the build gets run and the products of that get put in a new directory called “staging”. Additionally any extra resources get moved, too. The Vapor and Hummingbird Dockerfiles do the almost exactly the same commands, but the Hummingbird template does each command as a separate RUN. Each run gets cached separately, so I can see the pros and cons to both.

The --mount=type=cache,target=/build/.build line allows for incremental builds, it’s a docker directive
It’s a release build.
The {{name}} is a product of the GitHub template… FIND THEM ALL!
--static-swift-stdlib flag to statically link the Swift runtime libraries for this platform. It still requires the same platform dependencies. For fully static there used to be a --static-executable, but that’s gone now?
tell the linker to use a different malloc (see links above about memory issues)
move the binary
move the resources Swift Package Manager knows about.

## Skipped in hummingbird version because moves come after
## WORKDIR /staging
RUN mkdir /staging

# Build the application, with optimizations, with static linking, and using jemalloc
# N.B.: The static version of jemalloc is incompatible with the static Swift runtime.
RUN --mount=type=cache,target=/build/.build \
    swift build -c release \
        --product {{name}} \
        --static-swift-stdlib \
        -Xlinker -ljemalloc && \
    # Copy main executable to staging area
    cp "$(swift build -c release --show-bin-path)/{{name}}" /staging && \
    # Copy resources bundled by SPM to staging area
    find -L "$(swift build -c release --show-bin-path)" -regex '.*\.resources$' -exec cp -Ra {} /staging \;

Change the pwd to staging and copy over the swift-backtrace-static binary to it, to enable the backtracing library to work. That will be enabled in the second part. This would not be necessary if the base image for the deployment had swift installed, but then it would be HUGE.

# Switch to the staging area
WORKDIR /staging

# Copy static swift backtracer binary to staging area
RUN cp "/usr/libexec/swift/linux/swift-backtrace-static" ./

Move files from the default Vapor folders. In Hummingbird for example, the “Public” folder is “public”

# Copy any resources from the public directory and views directory if the directories exist
# Ensure that by default, neither the directory nor any of its contents are writable.
RUN [ -d /build/Public ] && { mv /build/Public ./Public && chmod -R a-w ./Public; } || true
RUN [ -d /build/Resources ] && { mv /build/Resources ./Resources && chmod -R a-w ./Resources; } || true

The Deploy section

About halfway down the file is another FROM call, so a second image is being made. It can refer to the first with --from= labels.

# ================================
# Run image
# ================================
FROM ubuntu:noble

This is a common base, “Noble Numbat” is the most recent canonical release. It’s much bigger than scratch or alpine.

Then again run updates and clear the cache

# Make sure all system packages are up to date, and install only essential packages.
RUN export DEBIAN_FRONTEND=noninteractive DEBCONF_NONINTERACTIVE_SEEN=true \
    && apt-get -q update \
    && apt-get -q dist-upgrade -y \
    && apt-get -q install -y \
      libjemalloc2 \
      ca-certificates \
      tzdata \
# If your app or its dependencies import FoundationNetworking, also install `libcurl4`.
      # libcurl4 \
# If your app or its dependencies import FoundationXML, also install `libxml2`.
      # libxml2 \
    && rm -r /var/lib/apt/lists/*

Add a user and move everything from staging into their home directory

# Create a vapor user and group with /app as its home directory
RUN useradd --user-group --create-home --system --skel /dev/null --home-dir /app vapor

# Switch to the new home directory
WORKDIR /app

# Copy built executable and any staged resources from builder
COPY --from=build --chown=vapor:vapor /staging /app

Enable backtracing and put the tool’s location in the environment

# Provide configuration needed by the built-in crash reporter and some sensible default behaviors.
ENV SWIFT_BACKTRACE=enable=yes,sanitize=yes,threads=all,images=all,interactive=no,swift-backtrace=./swift-backtrace-static

Switch to the user and run the program

# Ensure all further commands run as the vapor user
USER vapor:vapor

# Let Docker bind to port 8080
EXPOSE 8080

# Start the Vapor service when the image is run, default to listening on 8080 in production environment
ENTRYPOINT ["./{{name}}"]
CMD ["serve", "--env", "production", "--hostname", "0.0.0.0", "--port", "8080"]

The Redux

SOOOOOO…. How much of this can be stripped out?

What I left out

I wouldn’t leave these out in production of for a template, but for this specific example seemed fine:

app is tiny so improved linux allocator seemed worth trying to leave out.
I also didn’t update… because isn’t that what using a :latest is for?
Copying resource files, because this example doesn’t have any

Leaving out the allocator and updating was partially because I was going to try to use scratch or alpine for the deployment container. Scratch doesn’t have a package manager and alpine uses apk, so I would have had to translate the command. Turns out I had to use FROM ubuntu:noble anyway to prevent a dynamic library loading error? (See below)

What’s new

What’s new from the previous section’s container file:

The whole build section, obviously!
FROM ubuntu:noble. Originally I thought I’d be able to use alpine or scratch instead, but the deployment container complained a missing dynamic library. Switching to ubuntu:noble made it go away… TODO… WHAT dynamic library was missing?
because there’s an OS in the container re-added adding a user.
Backtrace support.
- copying the backtrace tool
- setting the env

I had to run it and re-run it a few times to shake out all the errors so I got a good look at all the caching features which help A LOT.

A Containerfile that Builds!


# ================================
# Build image
# ================================
FROM swift:6.1-noble AS build

# Install OS updates
RUN export DEBIAN_FRONTEND=noninteractive DEBCONF_NONINTERACTIVE_SEEN=true \
    && apt-get -q update \
    && apt-get -q dist-upgrade -y

# Set up a build area
WORKDIR /build

# First just resolve dependencies.
# This creates a cached layer that can be reused
# as long as your Package.swift/Package.resolved
# files do not change.
COPY ./Package.* ./
RUN swift package resolve \
        $([ -f ./Package.resolved ] && echo "--force-resolved-versions" || true)

# Copy entire repo into container
COPY . .

RUN mkdir /staging

# Build the application, with optimizations, with static linking, and using jemalloc
# N.B.: The static version of jemalloc is incompatible with the static Swift runtime.
RUN --mount=type=cache,target=/build/.build \
    swift build -c release \
        --product SmallestServer \
        --static-swift-stdlib && \
    cp "$(swift build -c release --show-bin-path)/SmallestServer" /staging

# Switch to the staging area
WORKDIR /staging

# Copy static swift backtracer binary to staging area
RUN cp "/usr/libexec/swift/linux/swift-backtrace-static" ./

# ================================
# Run image
# ================================
# Most swift examples use Ubuntu. 
FROM ubuntu:noble

RUN useradd \
    --user-group \
    --create-home \
    --system \
    --skel /dev/null \
     --home-dir /app \
     hummingbird

# Switch to the new home directory
WORKDIR /app

# give the binary to the hummingbird user
# COPY --from=build --chown=hbUser:hbGroup /staging /app/
COPY --from=build --chown=hummingbird:hummingbird /staging /app

# Provide configuration needed by the built-in crash reporter and some sensible default behaviors.
ENV SWIFT_BACKTRACE=enable=yes,sanitize=yes,threads=all,images=all,interactive=no,swift-backtrace=./swift-backtrace-static


# Ensure all further commands run as the hummingbird user
# USER hbUser:hbGroup
USER hummingbird:hummingbird

# Let Docker bind to port 8080
EXPOSE 8080


# Start the Hummingbird service when the image is run, default to listening on 8080 in production environment
RUN ls
ENTRYPOINT ["./SmallestServer"]
CMD [""]

Summary

There’s still a lot to learn about working with container. I’ll be switching to Swift Embedded for a bit and it’s possible I’ll use them for that. But I’ll keep linking to the series when something new comes up!

Podman: How do I deploy a Hummingbird App Server?

Mon, 01 Sep 2025 00:00:00 +0000

What is Hummingbird?

Hummingbird is a very modular Swift based server framework build on top of SwiftNIO. It’s easy to use and it has a great community around it. (Why Hummingbird)

Vapor would also have been a great choice, but it comes fully bundled with more than I needed.

What’s it take to build a starter server for a container?

Not much! This is what it took for me to make the 00_smallest_server example

Create A Start

mkdir -p 00_smallest_server
swift package init --type executable --name SmallestServer
swift package add-dependency https://github.com/hummingbird-project/hummingbird --from 2.16.0

Update main.swift

In main.swift add

import Hummingbird

let hostname: String = "0.0.0.0"
let port: Int = 8080
let configuration: ApplicationConfiguration = .init(
    address: .hostname(hostname, port: port),
    serverName: "SmallestServer"
)

let router = Router()
    .get("hello") { request, _ -> String in
        return "Hello"
    }

// create application using router
let app = Application(router: router, configuration:configuration)
// run hummingbird application
try await app.runService()

The code slightly modifies from code from the docs page to make it work in a container. If not working in a container leaving the default in place 127.0.0.1, the local loop back is the better choice.

let app = Application(router: router)

Old internet heads will balk at passing 0.0.0.0 because it means “listen on everything!” which could be a security risk. In a container that is apparently considered okay because it’s isolated, and these days will unlikely to have been given access to root. The container will be given an address once running which can be discovered with inspect

podman inspect $CONTAINER_ID_OR_NAME | grep IPAddress

On linux one can instead bind to the specific IP a container is given by listening on host.docker.internal (docs) or the more generic podman equivalent

let hostname: String = "host.docker.internal"
let hostname: String = "host.containers.internal"

I have not seen this working myself. Something about Containers and that Mac’s run them in VMs and that Linux has cgroups? On Linux there may be other options like adding --network=host to the run command. (Interesting convo on the Hummingbird discord! link at bottom of page) (Built fine, core-dumped when running ‘Address not available (errno: 99)’)

Update Package.swift

Update Package.swift to add the dependency to the executable and require at least MacOS 14.0:

import PackageDescription

let package = Package(
    name: "SmallestServer",
    platforms : [.macOS(.v14)],
    dependencies: [
        .package(url: "https://github.com/hummingbird-project/hummingbird", from: "2.16.0"),
    ],
    targets: [
        // Targets are the basic building blocks of a package, defining a module or a test suite.
        // Targets can depend on other targets in this package and products from dependencies.
        .executableTarget(
            name: "SmallestServer",
            dependencies: [.product(name: "Hummingbird", package: "hummingbird")]),
    ]
)

Build and run in the background

Slightly fancy version of building the package, running the server and going to the route in the browser.

swift build
./.build/arm64-apple-macosx/debug/SmallestServer & disown
open -a Safari http://127.0.0.1:8080/hello
## also
curl "http://127.0.0.1:8080/hello"

Usually one chooses to build and run a swift package with swift run and that works with Hummingbird apps too!

However, I wanted to show launching the binary in background and disowning it to highlight a problem with hand deploying apps on a server. They need to be run in the background (the &) and not quit when the terminal gets shut (the disown). One might also see solutions with nohup, tmux, screen…

If one runs ps, though, one will see that the process isn’t listed there anymore. To get the pid to kill it one will need to

ps aux | grep SmallestServer
kill $PID # returned by ps aux
## or
pKill SmallestServer

Derived from a good stack exchange answer (but also available from reading man ps) the aux after the ps corresponds to:

a = show processes for all users
u = display the process's user/owner with a name not a #
x = also show processes not attached to a terminal

So we’re looking trough all the processes being run by anyone and killing the one that has SmallestServer in the name.

The thing is, a good sys admin will have systemd kill user processes on logout, even if the process had been disown’d. And even if they agree not to kill it this sys admin will have to agree to configure the server to expose your experimental process to the internet. I mean, if it’s my server, I’m saying no.

And if it’s just your server, and you just have this one little app… who wants to knock about alone in a big echo-y machine all alone like that?

Get the Server into a Container

An easier way to get a simple app up on the internet is through a provider that offers some kind of Platform as a Service offering, which is a step up from the Infrastructure as a Service model that a VPS might be a component of.

https://www.freecodecamp.org/news/vps-vs-paas-how-to-choose-a-hosting-solution/

This business model probably only became viable because Containers exist!

There are several ways to get the app into a container that can be hosted on an app service, but this article will focus on what we’ve done before, cross compiling a statically linked binary that will be as small as possible.

Cross Compile Locally

For this, first cross compile just a hello world.

This section is essentially an evolution from the more general Wrapping a Binary section from a couple days ago, but the Containerfile will be a little bit different.

Shell Script

So let’s take the steps from that last post and make them a buildLocalNRun.sh script. The script below generates a tag based on the day and time and will do a release build by default.

#!/bin/sh

## change in script if want to change.
DEFAULT_PORT="8080"

## can change at runtime 
## 'zsh ./buildLocalNRun.sh debug' will run debug build instead.
DEFAULT_CONFIG_VALUE="release"
CONFIGURATION="${1:-$DEFAULT_CONFIG_VALUE}"

## make it if it doesn't exist. The enclosing folder will exist because
## The Containerfile is already there. 
mkdir -p Container/binary/

## Do the statically linked build
swift build -c $CONFIGURATION --swift-sdk x86_64-swift-linux-musl

## Move it to the binary folder where the Containerfile thinks it is
cp .build/x86_64-swift-linux-musl/$CONFIGURATION/SmallestServer Container/binary/SmallestServer

## creates a tag based on the day and local time, e.g. mon104642
TAG=`date +"%a%H%M%S" | tr '[:upper:]' '[:lower:]'`

podman build -f Container/Containerfile -t smallserver:$TAG Container/
# no -d because want to see errors inline
# this one will keep podman a little tidier. 
# podman run --rm --rmi -p $DEFAULT_PORT:8080 smallserver
podman run -p $DEFAULT_PORT:8080 smallserver:$TAG

Containerfile

And for this example an even more stripped down Containerfile:

FROM scratch

# Make and change into app directory
WORKDIR /app

# The binary directory contains the compiled app 
COPY ./binary/ /app/

# Let Docker bind to port 8080
EXPOSE 8080

# Start the Hummingbird service when the image is run, default to listening on 8080 in production environment
ENTRYPOINT ["./SmallestServer"]

Notice the different base image. This time it’s scratch instead of alpine. The scratch container provides NOTHING to the app’s context while the alpine container still provides some basic operating system tools. Our project needs nothing from an OS, so it’s safer not to give it anything that can be exploited. That said, scratch containers can’t be ssh’d into to troubleshoot and other ENV problems can crop up. It isn’t the easiest base image to work with if size isn’t an issue. It’s not that I didn’t make a new user in the Containerfile because I was lazy. I didn’t make a new user in the Containerfile because I couldn’t.

Linker directive in Package.swift (Optional)

Like in the static sdk example, to make our resulting release build image even smaller optionally add a linker directive to strip symbols from the object file to Package.swift.

    targets: [
        .executableTarget(
            name: "hello-world",
            dependencies: [
                .product(name: "Hummingbird", package: "hummingbird"),
                .product(name: "ArgumentParser", package: "swift-argument-parser"),
            ],
            linkerSettings: [
                // STRIP_STYLE = all
				.unsafeFlags(["-Xlinker", "-s"], .when(configuration: .release)), 
			]
        )
    ]

Push to Registry

The current build scripts will launch the Hummingbird app on the local computer. To hand it off to an App Server, it goes to a registry first.

Previously I pushed an image to Digital Ocean and Docker Hub.

Let’s add the push to Docker Hub to the bottom of the script:

## COMMENT OUT THE RUN!!
# podman run -p $DEFAULT_PORT:8080 $REPOSITORY:$TAG

## DON'T FORGET TO LOGIN to docker either using
## `docker login` or `podman login`
DOCKER_USER="someusername"
REMOTE_REPOSITORY="smalltest"
REMOTE_TAG="v1"

podman tag localhost/smallserver:$TAG docker.io/$DOCKER_USER/$REMOTE_REPOSITORY:$REMOTE_TAG
podman push docker.io/$DOCKER_USER/$REMOTE_REPOSITORY:$REMOTE_TAG

See the result here: https://hub.docker.com/repository/docker/carlynorama/smalltest/general

Make it ALIVE!

The next step is to sign up for a Digital Ocean App Server and link it to a repository in a registry.

Honestly, this was the least time consuming part of this whole process.

Changed nothing about the defaults except to move my service tier down from the 12/mo to the 5/mo service. And it worked!

Although have a different hummingbird server running there at the moment…

Next Up…

Next is switching from the most minimal set up to one based on the Vapor and Hummingbird templates, which are more robust.

How do I cross compile a Swift statically linked binary?

Sun, 31 Aug 2025 00:00:00 +0000

Resources

https://www.swift.org/documentation/articles/static-linux-getting-started.html
Swift to the cloud in a single step - Euan Harris: https://www.youtube.com/watch?v=9AaINsCfZzw
WWDC24: What’s new in Swift | Apple - (7:23) https://www.youtube.com/watch?v=17fZZ1v_N2U&t=457s
https://theswiftdev.com/building-static-and-dynamic-swift-libraries-using-the-swift-compiler/
https://github.com/swiftlang/swift-evolution/blob/main/proposals/0387-cross-compilation-destinations.md
https://github.com/swiftlang/swift-sdk-generator

In the post about podman and sharing images I referenced having a linux static binary available to use in my container image.

How’d I get it?

The first thing I did was simply follow the tutorial with a demo hello app provided by Swift.org.

To complete it I needed two things, an NON-Xcode open source version of Swift and a Swift Linux SDK.

Non-Xcode Swift

In the shell where one intends to compile it’s important to run

which swift
swift --version

to make sure the non-Xcode version of Swift will be used. I had trouble with VSCode’s integrated terminal defaulting to the XCode toolchain. I haven’t bothered to troubleshoot too deeply, but it seems to be tied to the xcode-select version being preferred by default. This may have JUST been addressed, but for now I just used Terminal instead.

Swiftly

If you haven’t installed Swiftly on your Mac, it’s a good choice for switching between different version of Swift. It’s now the recommended path on the swift/install/macos page

I used

# https://formulae.brew.sh/formula/swiftly
brew install swiftly

but it’s doable by hand:

curl -O https://download.swift.org/swiftly/darwin/swiftly.pkg && \
installer -pkg swiftly.pkg -target CurrentUserHomeDirectory && \
~/.swiftly/bin/swiftly init --quiet-shell-followup && \
. "${SWIFTLY_HOME_DIR:-$HOME/.swiftly}/env.sh" && \
hash -r

Example usage:

## install and switch
swiftly install --use latest
which swift
# /Users/$USER/.swiftly/bin/swift
swift --version
# Apple Swift version 6.1.2 (swift-6.1.2-RELEASE)
# Target: arm64-apple-macosx15.0

## to see other toolchains visible to Swiftly
swiftly list 
## to switch
swiftly use $SOME_TOOLCHAIN_IN_LIST

Linux SDK

Since I’m compiling on a Mac I need the software dev kit that will work for my target version of Linux.

I found the reference I needed halfway down the main install page.

The dev kit needs to match the version of Swift one is compiling with, so since I am using the latest release (6.1.2), that’s the dev kit I need.

swift sdk install https://download.swift.org/swift-6.1.2-release/static-sdk/swift-6.1.2-RELEASE/swift-6.1.2-RELEASE_static-linux-0.0.1.artifactbundle.tar.gz
 --checksum df0b40b9b582598e7e3d70c82ab503fd6fbfdff71fd17e7f1ab37115a0665b3b

To check that its there

swift sdk list

The installer put the sdk at:

Real and the same:
- /System/Volumes/Data/Users/$USER/Library/org.swift.swiftpm/swift-sdks/
- /Users/$USER/Library/org.swift.swiftpm/swift-sdks/
Symlink:
- /Users/$USER/.swiftpm/swift-sdks

Building An Example Package

These are all the steps to run and test the the example, assuming one has a remote machine to port it to. If not check out some of the podman series to run it in a container!

cd Place/For/Code
mkdir hello
cd hello
swift package init --type executable
swift build --swift-sdk x86_64-swift-linux-musl
file .build/x86_64-swift-linux-musl/debug/hello
scp .build/x86_64-swift-linux-musl/debug/hello username@remote.example.com:~/hello
ssh username@remote.example.com ~/hello

Applying To an Existing Project

With the SDK’s installed one can do this for any Swift package with an executable†.

cd /location/of/MyPackage
swift build -configuration release --swift-sdk x86_64-swift-linux-musl
file .build/x86_64-swift-linux-musl/release/$EXECUTABLE_NAME

Once moving into release mode do consider having the linker perform a full strip. It makes the binaries much smaller, and one could argue its a good idea to do for binaries going on internet exposed machines in general. I had forgotten about this until reminded!

    targets: [
        .executableTarget(
            name: "hello-world",
            dependencies: [
                .product(name: "Hummingbird", package: "hummingbird"),
                .product(name: "ArgumentParser", package: "swift-argument-parser"),
            ],
            linkerSettings: [
				.unsafeFlags(["-Xlinker", "-s"], .when(configuration: .release)), // STRIP_STYLE = all
			]
        )
    ]

About strip in general:
- https://unix.stackexchange.com/questions/2969/what-are-stripped-and-not-stripped-executables-in-unix
- https://linuxtldr.com/strip-command/
Its companion strings : https://www.howtogeek.com/427805/how-to-use-the-strings-command-on-linux/

† written with frameworks that will run on Linux

There is another way to cross compile that involves containers that will be the next post in the podman series!

Podman: How do I make a new image from a base?

Sat, 30 Aug 2025 00:00:00 +0000

So the example image based on httpd that I made is actually a pretty bad image. I manually changed the index.html file in htdocs/ and committed it directly instead of creating a new project with just the changed files.

Thankfully the folks offering up the httpd container also offer pretty easy advice on how to extend it.

Building on that advice, here’s a demo:

cd ~/to/place/to/store/projects
mkdir simple_site
mkdir simple_site/public_html
touch simple_site/public_html/index.html
echo -e '<html><body>
    <h1>Hello!</h1>
    <p>This is from a containerized project.</p>
</body></html>' > simple_site/public_html/index.html
touch simple_site/Containerfile
echo -e 'FROM httpd:2.4
COPY public_html /usr/local/apache2/htdocs/' > simple_site/Containerfile
podman build -f simple_site/Containerfile -t my-apache2-demo simple_site
podman run -dit -p 8080:80 my-apache2-demo
curl localhost:8080

The Containerfile

podman and docker will now both recognize Containerfile as well as Dockerfile so in my demo I chose to use Containerfile.

FROM httpd:2.4
COPY ./public-html/ /usr/local/apache2/htdocs/

That’s crazy simple, but it’s on top of a much more elaborate file from the base container which includes a call to start running the apache server, i.e. the required for most cases CMD call.

EXPOSE 80
CMD ["httpd-foreground"]

For complete Containerfile/Dockerfile novices here is some video help:

docker crash course by TechWorld with Nana starting at 49:10 https://youtu.be/pg19Z8LL06w?si=hYbiGuqQzW0Zj1KT&t=2950
the podman zero to hero video, starting at 22:32 https://youtu.be/YXfA5O5Mr18?si=sWsvxCabGD_EkBnf&t=1352

podman build

A new command in that script is podman build, which creates an image and launches a container.

podman build 
# where is the Containerfile relative to the pwd 
-f simple_site/Containerfile 
# tag (--tag) for the IMAGE that will be created by this build
# `podman images` will show a localhost/my-apache2-dem
-t my-apache2-demo 
## context, what should the working directory be for the commands
## run in the container file.
simple_site

If the script above had cd’d into simple_site to run the build command it would have looked like:

podman build -t my-apache2-demo .

Podman would have found the Containerfile and the context would be the pwd.

Another alternative would have been to tag it with our local registry and then the new image could be pushed there!

## can tag it even if registry is not available. 
## Registry isn't verified at the "tag" step.
podman build -t localhost:5050/my-apache2-demo .
## make sure its running
# podman ps -a
# podman start my_registry # or use ID number
podman push localhost:5050/my-apache2-demo

If one didn’t care about having an image of the build around, from the directory with the Containerfile:

podman build .

And a nameless image would be created that could be easily removed with prune once the container is wound down or by using --rm and --rmi in the run command. (also see podman container cleanup)

The Volume approach

The httpd docs point out that one doesn’t HAVE to have a Containerfile for such a simple situation. Instead it’s possible to just map the files using the –volume (-v) explored in the previous post.

podman run -dit --name my-apache-app -p 8080:80 -v "$PWD":/usr/local/apache2/htdocs/ httpd:2.4

There would be no image to push.

Wrapping a Binary

All of this is trying to move me towards understanding the swift-container-plugin so let’s do one more simplistic example, that honestly could have a lot of problems that I’m not aware of, but it works.

Let’s say one has a statically compiled binary that one knows works on most basic Linux machines.

I have one that came in at about 197MB (debug build) based on a simple Hummingbird server. The base code is only slightly edited from an example in the swift-container-plugin examples folder.

Using the swift-container-plugin I got an image that clocked in at almost 500MB, pressing the limit of my free Digital Ocean account. I want SMALLER.

So I tried this one and it worked to get an image closer to 200MB. It borrowed heavily from examples in the Hummingbird examples repo, but uses Alpine as the base instead of Ubuntu. Alpine will take the musl based binary I built. I also stripped out the build steps and a lot of the niceties. Including some

# Most swift examples use Ubuntu. 
FROM alpine

# Create a hummingbird user and group with /app as its home directory
RUN addgroup \
    -S \
    hbGroup \
&& adduser \
    -S \
    hbUser \
    -h /app/ \
    -k /dev/null \
    -G hbGroup

# Switch to the new home directory
WORKDIR /app

# give the binary to the hummingbird user
COPY --chown=hbUser:hbGroup ./binary /app/

# Ensure all further commands run as the hummingbird user
USER hbUser:hbGroup

# Let Docker bind to port 8080
EXPOSE 8080

# Start the Hummingbird service when the image is run, default to listening on 8080 in production environment
ENTRYPOINT ["./hello-world"]
CMD ["--hostname", "0.0.0.0", "--port", "8080"]

The following script assumes the binary for HelloServer has already been cross-compiled, which I’ll cover in a different post.

cd ~/to/place/to/store/projects
mkdir -p HelloWrapper
mkdir -p HelloWrapper/binary
cp HelloServer/.build/x86_64-swift-linux-musl/debug/hello-world HelloWrapper/binary/hello-world
touch HelloWrapper/Containerfile
# add contents to containerfile
cd HelloWrapper
podman build -t hellowrapper .
podman run -d -p 8080:8080 localhost/hellowrapper
curl localhost:8080
podman images
podman image inspect localhost/hellowrapper

That last “inspect” command is new, podman inspect provides a number of detains about the entities podman manages: containers, images, volumes, network, pods. One can limit it by providing a subtype. In this case images with podman image inspect

Summary

These were two VERY basic Containerfiles that would need improvement before going into production… but they work! Improving them and actually getting the site up on Digital Ocean is part of the next few posts. One avenue to investigate is starting from the base scratch (Thanks jheck!)

Appendix: The Swift Code

For posterity the swift code. Again this code it largely identical to:

https://github.com/apple/swift-container-plugin/tree/main/Examples/HelloWorldHummingbird

Which is Apache-2.0’d to Apple Inc. and the SwiftContainerPlugin project authors.

Package.swift

import PackageDescription

let package = Package(
    name: "hello-world",
    platforms: [.macOS(.v14)],
    dependencies: [
        .package(url: "https://github.com/hummingbird-project/hummingbird.git", from: "2.1.0"),
        // This was here from when the example was just a main.swift?
        // .package(path: "../.."),
        .package(url: "https://github.com/apple/swift-argument-parser", from: "1.3.0"),
        .package(url: "https://github.com/apple/swift-container-plugin", from: "1.0.0"),
    ],
    targets: [
        .executableTarget(
            name: "hello-world",
            dependencies: [
                .product(name: "Hummingbird", package: "hummingbird"),
                .product(name: "ArgumentParser", package: "swift-argument-parser"),
            ]
        )
    ]
)

App.swift

import ArgumentParser

@main
struct Hello: AsyncParsableCommand {
    @Option(name: .shortAndLong)
    var hostname: String = "0.0.0.0"

    @Option(name: .shortAndLong)
    var port: Int = 8080

    func run() async throws {
        let app = buildApplication(
            configuration: .init(
                address: .hostname(hostname, port: port),
                serverName: "HelloServer"
            )
        )
        try await app.runService()
    }
}

Application+build.swift

import Foundation
import Hummingbird
import Logging

let myos = ProcessInfo.processInfo.operatingSystemVersionString

func buildApplication(configuration: ApplicationConfiguration) -> some ApplicationProtocol {
    let router = Router()
    router.addMiddleware { LogRequestsMiddleware(.info) }
    router.get("/") { _, _ in
        "Hello World, from Hummingbird on \(myos)\n"
    }

    let app = Application(
        router: router,
        configuration: configuration,
        logger: Logger(label: "HelloWorldHummingbird")
    )

    return app
}

Podman: How do I preserve or share my image?

Fri, 29 Aug 2025 00:00:00 +0000

Vocabulary Overview

When talking about moving things it helps to be clear about what is being moved and to what kind of place.

registry: a service that stores images
repository: a collection of related images (same name, different tags)
image: a recipe and resources to build a container
container: an instance of an image, fully assembled and prepared
ID: both containers and images get giant hashes to identify themselves, like git commits.
name: a human readable name for a container or image. the word on the street is that images are okay to name, but containers shouldn’t be because there may be lots of them in a shared environment and if everyone goes around trying to use the label “my_container”, it will get ugly.
artifact: supplemental material/non-container material it makes sense to try to store in a registry along with images
tag: like a git tag, a label for a version. they can be mutable or immutable

Tags were what I needed clarified the most. One already showed up when doing a pull. It’s the :latest. latest is the default when nothing is specified as well.

podman pull docker.io/library/httpd:latest

I suspected that like with git tags there’d be conventions around their usage so here is some reading:

Make Sure There’s an Image to Work With

So imagine needing to reinit the Mac’s podman VM.

podman machine stop
podman machine rm
podman machine init
podman machine start
podman images

On the start up there’d be no images! Including the new template! How to keep it around even if the podman VM gets wiped? How to share one once it’s ready?

First, recreate it:

# docker login #if needed
podman pull docker.io/library/httpd:latest # login required!
podman run -dt -p 8080:80/tcp docker.io/library/httpd
podman ps
podman exec -it {$CONTAINER_ID} bash
:/usr/local/apache2# > echo '<html><body><h1>Still works!</h1></body></html>' > htdocs/index.html
:/usr/local/apache2# > exit
curl localhost:8080

podman commit {$CONTAINER_ID} my-httpd-template
podman images
# podman image rm {$CONTAINER_ID} ## to delete

Option 1: Save to an archive

First option to preserve the image even if we delete podman machine again is to use the save command that takes an output on the host mac.

podman save --output httpd_template_image.tar localhost/my-httpd-template

saves a copy of the image (or images if desired) to an archive file called httpd_template_image.tar in the pwd using save

One could load it back into podman with load

podman load --input httpd_template_image.tar

Option 2: Export is for Containers

Export is similar to save, but for use on containers, not images. Using import takes that export archive of a container and will bring it in as an image.

So if we really wanted to we could have used it on our changed httpd container rather than commit the container to an new image.

## will create tar file in pwd
podman export --output changed_httpd_container.tar {$CONTAINER_ID}
podman import changed_httpd_container.tar imported-httpd-image

Option 3: podman push :dir / podman cp

If one just wanted to move the folder around the podman machine one could pretty easily get a copy using push just to another directory. push :dir is only for moving items within the same machine.

podman machine ssh
core@localhost:~ mkdir -p ~/container_tmp
# podman-machine-default has podman installed too! 
core@localhost:~ podman push my-httpd-template dir:/Users/$USER/container_tmp/my-httpd-template
core@localhost:~ ls container_tmp/

This DOES NOT get the files out of podman-machine-default. /Users/$USER/container_tmp/ is still on the VM. To move the files to the host machine use podman machine cp from the host mac.

# this moves the folder we just created with the push. 
podman machine cp podman-machine-default:/var/home/core/container_tmp/my-httpd-template ~/saved_containers/my-httpd-template

This is different than podman cp, which is for copying to, from and between containers. That page also provides alternatives to itself for moving files around containers.

Option 4: Using push to a remote registry.

Rather than just keeping a hand-tooled local backup, the more usual would be to push useful images up to a registry. I am going to show two. Docker Hub and Digital Ocean.

Docker Hub

If you’ve followed along, you have a login solution because it was required to get the image, but see the end of this section for more info.

One can create a new repository Docker Hub’s repository home page:

https://hub.docker.com/repository/create?namespace=$DOCKER_USERNAME

Notice I am saying repository. The registry is docker hub. Each project gets a repository for all its versions.

Instead of creating one from the web-gui, I pulled out the CLI directions.

docker tag local-image:tagname new-repo:tagname
docker push new-repo:tagname

This translates to podman as:

podman tag localhost/my-httpd-template docker.io/$DOCKER_USERNAME/my-httpd-template:v1
podman images #optional. see the new image (it's still only local)
podman push docker.io/$DOCKER_USERNAME/my-httpd-template:v1

This exact repository can be seen for now at:

https://hub.docker.com/repository/docker/carlynorama/my-httpd-template/

A reminder about logging in:

Option A (docker login):

docker login

Option B (podman login):

podman login docker.io
Username: $DOCKER_USERNAME
Password:

Note: if the expected password for a service doesn’t work, try an API key.

Digital Ocean

Create a container registry. There is a free version that allows for 1 and only 1 repository, but that’s okay for this. (affiliate link with some free credits if useful.)

The steps walk through the process of installing THEIR cli and link to how to create an API token as needed. DO NOT LOOSE IT. (Passwords, BitWarden, LastPass, 1Password, Encrypted Note, Keychain Secure Note… something.)

As a note on Digital Ocean, I used to unreservedly recommend them because they put so much work into high quality documentation that covered topics in general, not just how to do things on their platform. It was a real differentiator. Made me willing to pay a premium to support the public good they provided. Back in 2023 they laid off a bunch of those folks, changed their docs home page and blog topics to be more DO specific and leaned into AI… so ¯\_ (ツ)_/¯. At least they still seems to have straight forward pricing models that prevent surprises.

Logging in

One can either login through the digital ocean CLI, registering the API key with Docker directly, or the podman login process.

## Option A
doctl registry login
## Option B
docker login registry.digitalocean.com
#> Username: <registered-email>
#> Password: <paste-api-token>
## Option C
podman login registry.digitalocean.com
#> Username: <registered-email>
#> Password: <paste-api-token>

push to Digital Ocean

# LOGIN FIRST
podman tag localhost/my-httpd-template registry.digitalocean.com/$DO_REGISTRY_NAME/my-httpd-template:v1
podman images #optional. see the new image (it's still only local)
podman push registry.digitalocean.com/$DO_REGISTRY_NAME/my-httpd-template:v1

This one is PRIVATE, so I can’t provide a link. It uses 45.65 MB / 500 MB of my space. YIKES. That’s a lot for one wee little web page. But its storing the whole server profile, so it won’t get much bigger than that as more content gets added.

Option 5: A local registry

Sometimes one wants a registry on ones own machine or for the local network. For testing, for privacy, for control, for bandwidth. Many reasons. The examples I found were for Linux machines, and there are a couple of gotcha’s porting them to Mac.

The TL;DR

## Make the storage folder
mkdir -p ~/web_tools/registry
## pull and run the registry image from docker using that folder
## configured to use the folder we want and the port we want. 
podman run -d --name my_registry -p 5050:5000 -v /Users/$USER/web_tools/registry:/var/lib/registry --restart=always registry:latest
## tag the image over to the new registry port
podman tag localhost/my-httpd-template localhost:5050/my-httpd-template
## see that no files are there yet
ls ~/web_tools/registry
## push to the registry
podman push localhost:5050/my-httpd-template --tls-verify=false 
## see the files are there now.
ls ~/web_tools/registry/docker/registry/v2/repositories/my-httpd-template

## Remove all the local versions
## See all the current images, some of mine had the same ID
## b/c of how I pushed to the remotes
podman images
## remove all the images with my-httpd-template in the name
podman image rm localhost:5050/my-httpd-template
podman image rm localhost/my-httpd-template
podman image rm -f $MY_DUPLICATED_ID
## should see no my-httpd-template
podman images

## make is so don't have to use --tls-verify=false every time and so its contents will appear in searches
## see the search without the registry
podman search my-httpd-template
podman machine ssh
## On the VM change the registries.conf in the local folder. 
## WARNING: this command will overwrite anything that is currently in it. 
core@localhost:~ echo -e 'unqualified-search-registries = ["localhost:5050", "docker.io", "ghrc.io", "quay.io", "registry.fedoraproject.org"]\n\n[[registry]]\nlocation="localhost:5050"\ninsecure=true' > $HOME/.config/containers/registries.conf
core@localhost:~ exit

## see the search with our local registry in the results
podman search my-httpd-template

## and pull from the local!
podman pull localhost:5050/my-httpd-template
## the image is back
podman images

When trying to figure out what all those commands should be there are four questions have to be answered:

where will the files be stored?
where is it going to be served to?
what tool is going to serve it?
how will podman know to look in it?

Where will the files be stored?

The the files question can be easy to answer. I’m going to create a folder on the mac at /Users/$USER to start.

mkdir -p ~/web_tools/registry

The expected location for registry files is /var/lib/registry, and one could create that inside podman machine and set up a volume to use that instead, but for me the point is to have the registry saved to my user on the host mac.

Where is it going to be served to?

All the examples use port 5000 on both the host and the container. But on a Mac that’s typically in use (along with 7000) by the Control Center. To free it up one needs to turn off Continuity, AirPlay… things I like using.

## See whose on your port
lsof -n -P -i :5000
## Dig further:
lsof -i
sudo skywalkctl flow -n -P $PID_CONTROL_CENTER
sudo skywalkctl netstat -a

So I am going to use a different port in this example. The nice thing is the container will still be able to use the default port (5000). This does mean when using the registry we will sometimes need to specify the port for software that would otherwise look for the default.

What tool is going to serve it?

The answer is to have podman run an existing registry image.

But how to give podman access to the folder and the port? We’re going to need to use some flags.

volume

[[SOURCE-VOLUME|HOST-DIR:]CONTAINER-DIR[:OPTIONS]]

How will podman know about the folder? Because it will be mapped to a volume.

One can create a volume in a separate step or at the time of container creation. At the time of container creation means the run command will handle all the appropriate configuration.

If one calls the command to run (and also pull if not already on the machine) a busybox image in one step:

podman run -dit --volume src:/dest busybox

and no volume called src already exists podman will create it on the podman vm at

/var/home/core/.local/share/containers/storage/volumes/src

The /dest part is where it maps to on the container’s file structure.

podman ps #to get the container id if didn't nab it. 
podman exec $BUSYBOX_CONTAINER_ID ls -ld /dest
podman exec $BUSYBOX_CONTAINER_ID touch hello.txt
podman machine ssh
core@localhost:~$ ls -la /var/home/core/.local/share/containers/storage/volumes/

NOTE: busybox can be used in the run command without docker.io/busybox because docker.io is one of the unqualified hosts in both the default registries.conf and the one updated in the last post. more

To do something a little more useful, where the source folder is accessible by the mac user, one can either set up a volume ahead of time or just pass in an actual path to an existing directory. While the folder for the SOURCE side must exist before hand, the one on the DESTINATION side does not. If it doesn’t exist it will be created as part of the container’s start up.

NOTE: the following commands will open an interactive shell immediately to help with testing file creation (-it without the d). The pwd will be /, not /var/home/core/ like with podman machine ssh.

mkdir -p ~/my_demo_volume_a
touch ~/my_demo_volume_a/waiting_to_be_seen.txt
podman run -it --rm --volume ~/my_demo_volume_a:/dest busybox
# / # cd /dest
# / # ls
# / # touch hello_a.txt
# / # exit

Or the long way. See discussion on this github issue

mkdir -p ~/my_demo_volume_b
## Look at how much has to be specified!!!
podman volume create --driver local --opt 'type=none' --opt "device=/Users/$USER/my_demo_volume_b" --opt 'o=bind' my_fancy_volume
podman run -it --rm --volume my_fancy_volume:/dest busybox

The volumes created by run where the source is a directory path do not show up in podman volume ls, only named ones do.

There is an alternate syntax for options with a colon after the destination parameter followed by comma separated options. A common one to see is z or Z. They seem to largely handle permission issues. Try letting run handle your situation before mucking about with them.

publish

[[ip:][hostPort]:]containerPort[/protocol]

In a run command, -p isn’t actually short for “port” but publish.

The registry will need to be available on the network since that’s what all the tooling expects. The publish option can map a single port or a range of ports. If a container has a lot of ports to expose, --publish-all, -P will map random host ports as needed. One can see the ports being used with podman ports.

my-httpd-template needed this option itself!

podman run -dt -p 8080:80/tcp localhost/my-httpd-template

Alternatives might have looked like:

# will only respond on 127.0.0.1, not all IPs (double check etc/hosts)
podman run -dt -p localhost:8080:80/tcp localhost/my-httpd-template
# to get a random available port. 
podman run -dt -p localhost::80/tcp localhost/my-httpd-template
# uses tcp by default
podman run -dt -p 8080:80 localhost/my-httpd-template
# use udp, scpt is the other option. To use more than one, -p for each.
podman run -dt -p 8080:80/udp localhost/my-httpd-template

restart

--restart=policy

https://docs.podman.io/en/latest/markdown/podman-run.1.html#restart-policy

From the docs, valid policy values are:

no : Do not restart containers on exit
never : Synonym for no; do not restart containers on exit
on-failure[:max_retries] : Restart containers when they exit with a - non-zero exit code, retrying indefinitely or until the optional max_retries count is hit
always : Restart containers when they exit, regardless of status, retrying indefinitely
unless-stopped : Identical to always

Our registry will use “always”, but keep in mind “Restart policy does not take effect if a container is stopped via the podman kill or podman stop commands.” (also from doc link.)

privileged

Older examples for creating a registry use --privileged. I am just going to pull the explanation from the run documentation:

Give extended privileges to this container. The default is false.

By default, Podman containers are unprivileged (=false) and cannot, for example, modify parts of the operating system. This is because by default a container is only allowed limited access to devices. A “privileged” container is given the same access to devices as the user launching the container, with the exception of virtual consoles (/dev/tty\d+) when running in systemd mode (–systemd=always).

A privileged container turns off the security features that isolate the container from the host. Dropped Capabilities, limited devices, read-only mount points, Apparmor/SELinux separation, and Seccomp filters are all disabled. Due to the disabled security features, the privileged field should almost never be set as containers can easily break out of confinement.

Containers running in a user namespace (e.g., rootless containers) cannot have more privileges than the user that launched them.

Newer examples leave this off, I suspect because narrower privileging around volumes seems to have improved in the intervening years. I am going to leave it off too, but wanted to mention it in case someone finds there way here after seeing an older example.

How will podman know how to find our registry?

To let podman know about the registry, update the $HOME/.config/containers/registries.conf from the last post to add it to the unqualified-search-registries and to indicate that it isn’t secure.

unqualified-search-registries = ["localhost:5050", "docker.io", "ghrc.io", "quay.io", "registry.fedoraproject.org"]

[[registry]]
location="localhost:5050"
insecure=true'

Summary

So these are 5 options on how to get images out of the podman VM on the mac and out to somewhere that will survive if it’s nuked.

The next post will be about improving our image, and maybe writing one from scratch.

Podman: How do I get and edit an image on MacOS?

Thu, 28 Aug 2025 00:00:00 +0000

This may all be obsolete when using MacOS 26 because of the containers announced at WWDC25, but there is an open source version of docker called podman that complies with the same Open Container Initiative specs as the Apple containers.

Why use podman now when I’ve avoided containers like the plague? I was looking to deploy a swift server and came across the swift container plugin thanks to the Hummingbird folks, and was interested in seeing how it worked. What could it make easier about sharing swift server examples?

The plugin requires a registry, which I made on Digital Ocean (affiliate link, why not), but I had troubles with it (the DO error code left something to be desired) and needed to explore containers in general to do a bit more to troubleshooting. Including figure out how to run a registry locally on my own computer.

And because Docker irritated. me. so. much. the last time I used it, enter Podman.

Podman “is an open source Open Container Initiative (OCI)-compliant container management tool from Red Hat used for handling containers, images, volumes, and pods” (https://en.wikipedia.org/wiki/Podman)

Some tutorials are difficult to follow because they try to leverage a knowledge of Docker, and everything I learned about Docker a few years a go I’ve tried to actively forget. But these resources provided a nice base.

https://www.redhat.com/en/topics/containers/what-is-podman
https://docs.podman.io/en/v5.2.4/Introduction.html
QUICK & EASY START HERE -> podman.io get started
Podman Tutorial Zero to Hero | Full 1 Hour Course

Install Podman

There are two options for installing podman on the mac. CLI only (first) and CLI with GUI (second).

I don’t find the GUI that intuitive, so while I do have it installed, I haven’t been using it. The install process may have tweaked some settings around docker and it installed a couple extra tools. I will be keeping a look out for commands that might work differently because I installed it, but nothing I’ve done so far appears to have changed from before and after.

(As a note, one can’t stop the podman server from running if the GUI is open.)

## CLI Only
brew install podman
## install GUI that will also install CLI
brew install --cask podman-desktop

Note: to use docker.io later to retrieve the base project one option is to install it and login that way. I think you could leave it out and choose a different base project, but I was just following along. At least it made ME do that. YMMV.

brew install docker
## will lead you through the steps to create a login and that shell will be able
## to access docker.io just fine at the least. 
docker login 
## it doesn't change .zprofile but it does create a config file. For more info:
# cat ~/.docker/config.json

Afterwards I realized there was a podman login command, but I have yet to try it.

Set up and launch the podman VM

Podman is a linux only tool, so on the Mac podman runs in a linux VM that will need to be created and then ssh’d into to configure.

podman machine init
podman machine start

If for any reason you want to nuke it and start again:

podman machine stop
podman machine rm
podman machine init
podman machine start

You can find out more information about your set up by running info and version

podman info
podman version

Tell it what registries you prefer it to search

Updating the registries.conf file to control what registries podman will search isn’t in the getting started tutorial, but it is in the video.

What the video skips, is that on the mac, in order to update the VM running the podman tool, you’ll need to ssh into it and then follow tutorial instructions.

# https://docs.podman.io/en/latest/markdown/podman-machine-ssh.1.html
podman machine ssh

Once in, add the registries to a config file in the VM user’s directory so you don’t have to change the main one for the whole VM, but on the VM it’s less of a big deal to change the main one than when running directly on a linux box, I would suspect.

Look at the VM’s main file:

cat /etc/containers/registries.conf

Make a local user one if desired (this one matches the video tutorial):

echo -e 'unqualified-search-registries = ["docker.io", "ghrc.io", "quay.io", "registry.fedoraproject.org"]' > $HOME/.config/containers/registries.conf

Get an Existing Container Image

The getting started tutorial picks a really simple http server which is perfect to kick off with.

The tutorial of course starts with showing how to search for it.

# all the things with that name
podman search httpd
# flagged official
podman search httpd --filter=is-official

From the documentation, supported filters are:

stars (int - number of stars the image has)
is-automated (boolean - true | false) - is the image automated or not
is-official (boolean - true | false) - is the image official or not

Pull the desired container image onto your machine and run it.

podman pull docker.io/library/httpd:latest 
podman run -dt -p 8080:80/tcp docker.io/library/httpd

Run is used for when the container doesn’t exist yet. Some folks add a --rm to the command so a container being created that one doesn’t intend to edit will get zotted when it’s wound down. --rmi will delete the image!

This example command uses -d for detached. That will let the process run in the background. The -t also adds a pseudo-tty to run arbitrary commands in an interactive shell.

The string of numbers you’ll see after executing is the full ID string of the container that has been made from the image.

If you forget that string you can get it again by running ps which shows the running containers. The shortened string works just as well as an id later.

# currently running
podman ps
# even ones that have been stopped.
podman ps -a

To end a specific process use stop using that ID number. If you didn’t add -rm, it can be restarted with start.

podman stop {$CONTAINER_ID}
podman start {$CONTAINER_ID}

With this image up and running you should be able to hit the demo web page:

curl http://localhost:8080
#<html><body><h1>Still works!</h1></body></html>

Make changes to it…

Let’s say you wanted to change that web page. With the container still running, launch its bash with exec -i -t, where -i is interactive and -t is that tty again. Don’t forget to add the tool you want to launch at the end!

podman exec -it {$CONTAINER_ID} bash

Once in the interactive shell one can change the web page.

ls -al
echo '<html><body><h1>Still works!</h1></body></html>' > htdocs/index.html

In a different Terminal, try the curl again. It should reflect the new page.

curl http://localhost:8080
# <html><body><h1>Still works!</h1></body></html>

Stop, start and curl again… The changes are preserved!

Make a new image from an existing container

Let’s say you want to make a new image from this changed container. That’s what the commit command is for.

## see containers
podman ps -a
## see images
podman images 
## will provide default info (best for shared environments)
# podman commit {$CONTAINER_ID} 
podman commit {$CONTAINER_ID} {$SOME_HUMAN_READABLE_LABEL_FOR_NEW}
## look for new image
podman images

I chose “my-httpd-template” for my new localhost/my-httpd-template now lives in my list of images, and I can make new containers from it instead of the one from the registry.

podman run -dt -p 8080:80/tcp localhost/my-httpd-template
## see it running
podman ps

Until next post

This is a handy place to stop this section of notes because we now have a our own new copy of an image that we made changes to, and can make changes to.

The next set of notes all covers how to share an image once you have it, i.e. various ways of copying it out of the VM, pushing it to a remote registry, pushing it to a local registry, etc.

DocC: notes on getting started

Sat, 05 Jul 2025 00:00:00 +0000

DocC solves two different problems. It helps automate API documentation, and it also provides a system for writing support articles and tutorials.

This article is mostly rough notes on the meta of how to view DocC docs, how generate the output and handle the output outside of the XCode ecosystem.

Additional Resources

There are additional resources listed inline with the notes, but here are some not otherwise mentioned but helpful .

WWDC

2021
- year released as open source
- https://developer.apple.com/news/?id=xa4ak3qr
- Meet: https://developer.apple.com/videos/play/wwdc2021/10166
- Host and Automate: https://developer.apple.com/videos/play/wwdc2021/10236
- Elevate: https://developer.apple.com/videos/play/wwdc2021/10167
- Build Tutorials: https://developer.apple.com/videos/play/wwdc2021/10235
2022
- What’s new in: https://developer.apple.com/videos/play/wwdc2022/110368
- Improve discoverability: https://developer.apple.com/videos/play/wwdc2022/110369
2023
- Create rich: https://developer.apple.com/videos/play/wwdc2023/10244

Misc Others

Getting Help via the Command line

These are the different commands I found:

## stand alone
docc --help
## xcode project related
xcrun docc --help
xcodebuild docbuild -- help
## package
swift package plugin generate-documentation --help

Resources

How to View Your Work

XCode: Open the Documentation Preview

The fastest way to see the results of your DocC work is to use Xcode and open the documentation window (Window > Developer Documentation (Shft-Cmd-0)).

But the most convenient while working on it is to have the Documentation Preview open.

You can access Documentation Preview by first activating the assistant editor via Editor > Assistant, and then selecting “Documentation Preview” in the assistant editor’s jump bar.

https://developer.apple.com/documentation/xcode-release-notes/xcode-15-release-notes#Documentation

To actually build the docs Product > Build Documentation

VSCode

Purportedly when using 6.2 or higher there’s a docc preview available. I haven’t fully checked that out. I could not get it to work when using a swiftly .swift-version file to drive using 6.2

i.e.:

swiftly use 6.2-snapshot-2025-06-12
swiftly run swift package plugin generate-documentation

View a side-by-side live preview of your documentation as you edit it with the Swift extension for VS Code. Access this feature using the Preview Swift Documentation button at the top right of an editor, or be invoking Swift: Preview Documentation in the command palette. This opens up a new editor pane with your rendered documentation:

https://docs.swift.org/vscode/documentation/userdocs/docc-live-preview/

TODO: check it out with the system’s default toolchain set to 6.2

Package: commandline and a browser.

Using the Package Plugin Directly

Add it to your Package dependencies:

    ],
        dependencies: [
        .package(url: "https://github.com/apple/swift-docc-plugin", from: "1.0.0"),
    ],

generate and launch preview:

swift package plugin generate-documentation
swift package --disable-sandbox preview-documentation --target $TARGET_NAME
## or
swift package --disable-sandbox preview-documentation --product $PRODUCT_NAME

https://swiftlang.github.io/swift-docc-plugin/documentation/swiftdoccplugin/

Command Line

The docc command for generating an archive is convert. Highly recommend checking out the help to see what’s possible:

which docc
docc convert -- help

The example convert provided is:

docc convert MyNewPackage.docc \
  --fallback-display-name MyNewPackage \
  --fallback-bundle-identifier com.example.MyNewPackage \
  --fallback-bundle-version 1 \
  --additional-symbol-graph-dir .build \
  --output-dir MyNewPackage.doccarchive

So from the package root of my example project:

DocCExplorer % docc convert Sources/DocCExplorer/DocCExplorer.docc \
--fallback-display-name DocCExplorer \
--additional-symbol-graph-dir .build \
--output-dir ~/Desktop/DocCExplorer.doccarchive

Working with an archive

Finding an archive built fo a package

## from the package root folder
## will return both preview builds and output builds
find . -type d -name "*.doccarchive"
## poke around inside
cd .build/plugins/Swift-DocC\ Preview/outputs/$TARGET_NAME.doccarchive
## OR move it
cp -r .build/plugins/Swift-DocC/outputs/$TARGET_NAME.doccarchive $NEW_LOCATION/$TARGET_NAME.doccarchive

What is in an archive

css				
data				
developer-og-twitter.jpg	
developer-og.jpg		
documentation			
downloads			
favicon.ico			
favicon.svg
images
img
index
index.html
js
metadata.json
videos

cd’ing into that dir and running python3 -m http.server $PORT, works, but one will have to navigate by hand down into a subfolder. (that subfolder can be reset see below)

http://localhost:$PORT/documentation/$TARGET_NAME/

What’s interesting is the data folder which will contain json files for all the inline documentation.

The files come out pretty bulky, and seem to use Vue.js to run as an app and webpack to minfy.

One recommended script to “get it ready for static hosting”:

docc process-archive \
transform-for-static-hosting $PATH_TO_DOCCARCHIVE_FILE \
--output-path $PATH_TO_SAVE_FILES \
--hosting-base-path $URL_BASE_PATH

So from the same directory as an example doccarchive file, the below command changed it in place and didn’t change the root directory and got 34 changed files from running:

## in the folder with the doccarchive
docc process-archive \
transform-for-static-hosting ./DocCExplorer.doccarchive \

In the HTML files, which essentially are just light frames for javascript, new js files get swapped in. Because the js files come out all minimized it was irritating to figure out what was different exactly, so I didn’t.

The actual function run is the TransformForStaticHosting command.

Looking at that code it looks like it’s more or less a straight copy when using a --output-path if not also changing the hosting-base-path.

Astro: how to write an integration as a package

Mon, 30 Jun 2025 00:00:00 +0000

So I decided to look into Astro (yet another static site generator). Lots of reasons that I won’t get into for the sake of time, but two major ones:

it’s a Typescript/Node set up
seems pretty extendable because of that.

The official introductory tutorial does a fantastic job walking one through using it, although it is probably easier for people who already have some familiarly with HTML/css/Javascript/Node.

Since I’m thinking about this for sites other than mine, I wanted to understand how to write features that could be used across more than one website, generally referred to as “Integrations” in the Astro-verse.

I found that name mildly confusing because many of the packages listed on Astro’s Integration page don’t actually take advantage of the Integration API which gives a package hooks into various stages of the build process, but merely take advantage of using Astro as a peerDependency. There doesn’t appear to be a separate names for node packages with those two different levels of enmeshment.

Ultimately the best resources for me were:

They can be found at the VEEEEERRRRRRYYYY bottom of the API page linked above, but they were also mentioned by the very helpful folks on the Astro discord.

The first link had an example of an integration implementation that was very easy to follow along with. The second link is a utility written by one of the maintainers of Astro that generates a monorepo with an install of Astro in it to help with testing. It’s a fantastic resource to get rolling on developing an integration if you already know what you’re doing.

I kinda needed something in the middle: an example of a simple integration in an stand alone package, but not a package full of bundlers and extra noise. I did not find one, so I made one.

https://github.com/carlynorama/astro-integration-hello

It reproduces the in-package example in the first link as a very simple separate package.

Below is a copy of the STEPS_TO_CREATE.md from that repo.

The next step for this investigation will probably be to try to recreate the YouTube shortcode I did for this website as an Astro integration. There are plenty of those already to crib from so seems like it should be a really reasonable exercise.

Repo Setup Information

Commands to Start Up

npm init -y
npm install typescript @types/node --save-dev
npx tsc --init
npm install astro-integration-kit
npm install astro -D

touch README.md
touch LICENSE
touch .gitignore
## UPDATE GITIGNORE FIRST!!!
git init; git add .; git commit -m "Initialize repository"
mkdir src
touch src/index.ts
touch integration/index.ts
mkdir scripts
touch src/scripts/support_script.ts
npm install kleur #this is to make the example code work exactly like the original demo.

.gitignore

copied from:

https://github.com/florian-lefebvre/astro-integration-template/tree/main/templates/manual-kit

# build output
dist/

# generated types
.astro/

# dependencies
node_modules/

# logs
npm-debug.log*
yarn-debug.log*
yarn-error.log*
pnpm-debug.log*

# environment variables
.env
.env.production

# macOS-specific files
.DS_Store

tsconfig.json

starting from the generated tsconfig, matched it to strictest by hand, but had to change some of the setting to better match what was reflected in the tsup settings of the example.

{
  "compilerOptions": {
    /* Visit https://aka.ms/tsconfig to read more about this file */

    /* Projects */

    /* Language and Environment */
    "target": "ESNext",                                  /* Set the JavaScript language version for emitted JavaScript and include compatible library declarations. */
    "jsx": "preserve",                                   /* Honestly N/A for this lib, Specify what JSX code is generated. */

    /* Modules */
    "module": "esnext",                                  /* Specify what module code is generated. */
    //"rootDir": "./src",                                /* Specify the root folder within your source files. */
    "moduleResolution": "bundler",                       /* Specify how TypeScript looks up a file from a given module specifier. */
    "allowImportingTsExtensions": false,                 /* DIFFERENT FROM BASE Allow imports to include TypeScript file extensions. Requires '--moduleResolution bundler' and either '--noEmit' or '--emitDeclarationOnly' to be set. */
    
  
    /* JavaScript Support */
    "allowJs": true,                                     /* Allow JavaScript files to be a part of your program. Use the 'checkJS' option to get errors from these files. */

    /* Emit */
    "declaration": true,                                 /* Generate .d.ts files from TypeScript and JavaScript files in your project. */
    "sourceMap": true,                                   /* Create source map files for emitted JavaScript files. */
    "noEmit": false,                                     /* DIFFERENT FROM BASEDisable emitting files from a compilation. */
    "outDir": "./dist",                                  /* Specify an output folder for all emitted files. */
   
    /* Interop Constraints */
    "isolatedModules": true,                             /* Ensure that each file can be safely transpiled without relying on other imports. */
    "verbatimModuleSyntax": true,                        /* Do not transform or elide any imports or exports not marked as type-only, ensuring they are written in the output file's format based on the 'module' setting. */
    "esModuleInterop": true,                             /* Emit additional JavaScript to ease support for importing CommonJS modules. This enables 'allowSyntheticDefaultImports' for type compatibility. */
    "forceConsistentCasingInFileNames": true,            /* Ensure that casing is correct in imports. */

    /* Type Checking */
    "strict": true,                                      /* Enable all strict type-checking options. */
    "noUnusedLocals": true,                              /* Enable error reporting when local variables aren't read. */
    "noUnusedParameters": true,                          /* Raise an error when a function parameter isn't read. */
    "exactOptionalPropertyTypes": true,                  /* Interpret optional property types as written, rather than adding 'undefined'. */
    "noImplicitReturns": true,                           /* Enable error reporting for codepaths that do not explicitly return in a function. */
    "noFallthroughCasesInSwitch": true,                  /* Enable error reporting for fallthrough cases in switch statements. */
    "noUncheckedIndexedAccess": true,                    /* Add 'undefined' to a type when accessed using an index. */
    "noImplicitOverride": true,                          /* Ensure overriding members in derived classes are marked with an override modifier. */
    "allowUnusedLabels": false,                          /* Disable error reporting for unused labels. */
    "allowUnreachableCode": false,                       /* Disable error reporting for unreachable code. */

    /* Completeness */
    "skipLibCheck": true                                 /* Skip type checking all .d.ts files. */
  },
  "include": [
    "src/"
  ],
  "exclude": [
    "dist/"
  ]
}

package.json

Some of the needed changes were made by the install scripts, but we’ll need a few more. Also there would be a lot more needed in order to publish correctly.

{
  "name": "astro-integration-hello",
  "version": "0.0.0",
  "description": "",
  "keywords": [],
  "author": "",
  "license": "Apache-2.0",
  "type": "module",
  "exports": {
    ".": {
      "types": "./dist/index.d.ts",
      "default": "./dist/index.js"
    },
    "./globalLog": {
      "types": "./dist/scripts/support_script.d.js",
      "default": "./dist/scripts/support_script.js"
    }
  },
  "files": [
    "dist"
  ],
  "scripts": {
    "build": "tsc -b --verbose",
    "clean": "rm -rf ./dist"
  },
  "peerDependencies": {
    "astro": "^5.0.0"
  },
  "dependencies": {
    "astro-integration-kit": "^0.18.0",
    "kleur": "^4.1.5"
  },
  "devDependencies": {
    "@types/node": "^24.0.4",
    "astro": "^5.10.1",
    "typescript": "^5.8.3"
  }
}

type

“type”: “commonjs”, => “type”: “module”,

main -> exports/files

Uses the more modern exports syntax. Also set up an entry point for the reference to the logging script.

https://nodejs.org/api/packages.html#exports

  "exports": {
    ".": {
      "types": "./dist/index.d.ts",
      "default": "./dist/index.js"
    },
    "./globalLog": {
      "types": "./dist/scripts/support_script.d.js",
      "default": "./dist/scripts/support_script.js"
    }
  },
  "files": [
    "dist"
  ],

scripts

Add them.

  "scripts": {
    "build": "tsc -b --verbose",
    "clean": "rm -rf ./dist"
  },

peerDependency

Since this isn’t a mono repo where the plugin could be easily checked inside a running Astro instance. went ahead and installed astro as a dev dependency. Still need to add it as peerDependency by hand for other environments.

  "peerDependencies": {
    "astro": "^5.10.1"
  },

index.ts

exports the integration as the default export.

Note the js in the import statement. To import ./integration.ts, 'allowImportingTsExtensions' would need to be enabled in the tsconfig. That option requires noEmit to also be enabled, and we need to emit.

https://github.com/florian-lefebvre/astro-integration-template/blob/5944471673762c71672666b13217d89b7020583e/templates/manual-kit/package/src/index.ts

This uses the word integration in the file name and object name but that isn’t required.

import { integration } from "./integration.js";

export default integration;

integration.ts

This is the file that emits the js file referred to in the index.ts

It contains an almost exact dupe of src/integrations/astro-hello.ts from Understanding Astro: Integrations, except we import the entry point that was set up in the package.json instead of a local file reference.

import { defineIntegration } from "astro-integration-kit";
import kleur from "kleur";

const dateTimeFormat = new Intl.DateTimeFormat([], {
  hour: "2-digit",
  minute: "2-digit",
  second: "2-digit",
});

const logServerMessage = (message:String) => {
  const date = dateTimeFormat.format(new Date());
  console.log(`${kleur.gray(date)} ${kleur
    .bold()
    .cyan("[astro-hello-integration]")} ${message}
    `);
};

export const integration = defineIntegration({
	name: "astro-integration-hello",
	setup() {
		return {
			hooks: {
				"astro:config:setup": (options) => {
					options.injectScript(
						"page",
            //THIS IS WHAT'S DIFFERENT!!! 
						`import "astro-integration-hello/globalLog";`
					)
					logServerMessage("Integration Happened");
				}
				
			},
		};
	},
});

Aside: one does a full import instead of just passing in a file name because that second parameter is a raw code string.

export const integration = defineIntegration({
	name: "astro-integration-hello",
	setup() {
		return {
			hooks: {
				"astro:config:setup": (options) => {
					options.injectScript(
						"page",
						`console.log("test this!");`
					)
					logServerMessage("Integration Happened");
				}
				
			},
		};
	},
});

scripts/support_script.ts

This file is specifically not named the same as the exported entry point to make it clear that the entry point is how the client package is accessing the code.

started with vanilla javascript from the Understanding Astro: Integrations example.

const logger = () => {
  const msg = "Hello Integrations";
  console.log(`%c ${msg}`, "background: black;  color: yellow");
};

logger();

Checking typed errors with swift-testing

Mon, 26 May 2025 00:00:00 +0000

It seems reasonable at this point to put all my file-wrangling knowledge in a reusable package so I’m not reinventing the wheel every time. Every project needs something a little bit different, so even if I pirate files or functions out, at least there will be one place to look.

Since in one of my projects I already had some XCTests, I’m using this as a chance to migrate over to swift-testing as well. When watching the WWDC video it mentioned that one should also be writing tests for"unhappy paths". An obviously good thing to do which I have not been doing!

So I started playing around with watching for the right error messages with swift-testing

Resources

General Error Catching

First I have my errors that I want to try to catch. They have to conform to equitable in order for me to test for them using the #expect macro the way I want to.

enum ExampleError:Error {
    case unknownError(_ message: String)
    case codedError(_ code:Int)
    case noExtras
}

extension ExampleError:Equatable {}

Then I have the struct to test. I am using async functions since that is the harder case, but this all works with synchronous functions as well.

struct TestMe {
    
    func throwsMessage(_ message:String) async throws {
        throw ExampleError.unknownError(message)
    }
    
    func throwsCode(_ codeToThrow:Int) async throws {
        throw ExampleError.codedError(codeToThrow)
    }
    
    func throwsSimple() async throws {
        throw ExampleError.noExtras
    }
}

No associated type

The very first test function, one where it catches an error with no associated value is pretty easy:

    //use in all the tests as part of a suite
    let itemToTest:TestMe = TestMe()

    @Test func simpleErrorTest() async throws {
        //required explicit equatable
        await #expect(throws: ExampleError.noExtras) {
            try await itemToTest.throwsSimple()
        }
    }

Associated values with known values to test against.

Things get a little messier when I want to test against the Error’s subtype.

When the value is known that is also easy, but to retrieve the value and test requires finding a way to pull it out with a case statement.

    @Test func testForExpectedCode() async throws {
        let expectedCode = 42
        
        //when you know the exact code, everything is fine.
        await #expect(throws: ExampleError.codedError(expectedCode)) {
            try await itemToTest.throwsCode(expectedCode)
        }
        
        let allowedRange = (20...49)
        await #expect {
            try await itemToTest.throwsCode(expectedCode)
        } throws: { error in
            guard let error = error as? ExampleError else {
                return false
            }
            switch error {
            case .codedError(let code):
                return allowedRange.contains(code)
            default:
                return false
            }
        }
    }

That seemed long and like I was missing something, so asked on the forums for some help. I used the advice to formulate:

    @Test func testForExpectedCodeV2() async throws {
        let expectedCode = 42
        //demoing require keyword, not strictly necessary here. 
        let error = try await #require(throws: ExampleError.self) {
            try await itemToTest.throwsCode(expectedCode)
        }
        #expect(error == .codedError(expectedCode))
        
        #expect(await errorContainedIn(range: 20...49, 
                                       with: Int.random(in: 20...49), 
                                       from: itemToTest.throwsCode))
        
        func errorContainedIn(range:ClosedRange<Int>, with:Int,
            from function:(Int) async throws -> Void) async -> Bool {
            //but cannot use require here because has to be failable for guard-case 
            guard case .codedError(let code) = (await #expect(throws: ExampleError.self) {
                try await function(with)
            }) else { return false }
            return range.contains(code)
        }
    }

Nice reusable function that I can lift to the top of the suite.

Trying to ignore the associated value

Ignoring the associated type’s value was still pretty verbose my first pass.

    @Test func testForAnyCode() async throws {
         //Can't find the right way to frame this? Is it possible?
         // ExampleError.codedError is not a _Type_
         // #expect(throws: ExampleError.codedError(_)) {
         //     try await throwsCode(Int.random(in: 0...5))
         // }
        await #expect {
            try await itemToTest.throwsCode(Int.random(in: 0...5))
        } throws: { error in
            guard let error = error as? ExampleError else {
                return false
            }
            switch error {
            case .codedError(_):
                return true
            default:
                return false
            }
        }
    }

First thing I did was add a function to the enum that honestly would have been annoying to maintain because it would need to be done for every case with an associated value.

    func isCodedError() -> Bool {
        switch self {
        case .codedError(_):
            return true
        default:
            return false
        }
    }

Which leads to a much shorter test, but a lot of down the road hassle on the error enum.

    @Test func testForAnyCodeV2() async throws {
        
        let error = await #expect(throws: ExampleError.self) {
            try await itemToTest.throwsCode(Int.random(in: 0...5))
        }
        #expect(error?.isCodedError() == true)
    }

On a different forum thread someone else posted something along the lines of…

//MARK: Error Extension
import Foundation
extension Error {
    var discriminator:Int {
        (self as NSError).code
    }
}

@Test func testForAnyCodeV3() async throws {
    
    let error = await #expect(throws: ExampleError.self) {
        try await itemToTest.throwsCode(Int.random(in: 0...5))
    }
    
    //the presence of an actual number could be confusing if it is arbitrary
    //let typeCodeForCodedError = ExampleError.codedError(3213).discriminator
    let typeCodeForCodedError = 1 //better never change the order of your enum if use this though. 
    #expect(error?.discriminator == typeCodeForCodedError)
}

… which I have mixed feeling about because it requires Foundation and to jump through some minor hoops to get the right value for the error (see code comments.)

See the demo repo for all the tests.

Catching errors that don’t belong to me

In the actual FileWrangler code I want all my missing files to throw the same error, a FileServiceError.noFileAtLocation(_ path: String).

So the test for that might look like:

        @Test(.tags(.existing)) func tryToDeleteNonExistentFile() async throws {
            let fileToTestPath = touchPractice
            let fileToTest = FWFile(path: fileToTestPath)
            
            let expectedErrorMessage = "FileID not associated with deletable file."
            
            let error = try await #require(throws: FileServiceError.self) {
                try await fileServiceToTest.delete(fileToTest)
            }
            
            print(error)
            
            #expect(allowedError(error))

            
            let postExists = try await fileServiceToTest.locationExists(for: fileToTest)
            #expect(!postExists)
            
            func allowedError(_ error:FileServiceError) -> Bool {
                switch error {
                case .noFileAtLocation(_): return true
                case .unknownError(let message):
                    return message == expectedErrorMessage
                default: return false
                }
            }
        }

But the problem is that my fileServiceToTest.delete(fileToTest) doesn’t throw MY error, it throws an error from the system.

I found a great article on catching CocoaErrors so now my delete function re-skins that error:

    public func delete(_ id: FWFile) async throws {
        let path = id.locationPath
            //let url = id.locationURL
        
        //note returns true if file does not exist. 
        //appears to be essentially a permissions check.  
        guard fileManager.isDeletableFile(atPath: path) else {
            throw FileServiceError("FileID not associated with deletable file.")
        }
        
        do {
            try fileManager.removeItem(atPath: path)
        } catch let error as CocoaError where error.code == .fileNoSuchFile {
            throw FileServiceError.noFileAtLocation(path)
        } catch {
            throw FileServiceError.caughtError(error)
        }
    }

SwiftData, my first look

Mon, 05 May 2025 00:00:00 +0000

SwiftData seems to be a very good fit for simple projects. When using defaults developers get a lot of features right out of the gate, like change history and undo. Default settings also syncs across devices easily with CloudKit, with a few caveats.

In 2024 Apple added a DataStore protocol so you can swap in your own preferred persistence style, rather than the very fancy wrapper around SQLite that it comes with out of the box.

Resources

https://www.hackingwithswift.com/quick-start/swiftdata
https://developer.apple.com/videos/play/wwdc2023/10187/ (2023 - Meet SwiftData)
https://developer.apple.com/videos/play/wwdc2024/10137/ (2024 - What’s new in SwiftData)
https://developer.apple.com/videos/play/wwdc2024/10138 (2024 - Create a custom data store with SwiftData)
https://developer.apple.com/videos/play/wwdc2023/10196 (2023 - Dive deeper into SwiftData)
https://developer.apple.com/xcode/swiftdata/
https://www.hackingwithswift.com/quick-start/swiftdata/whats-the-difference-between-modelcontainer-modelcontext-and-modelconfiguration

The pieces

After going through some basic hello world tutorials/intro WWDC videos, definitely do watch the 2023 Dive Deeper video to understand how the parts work together.

Schema

If this word is unfamiliar to you: https://en.wikipedia.org/wiki/Database_schema

@Model‘ing classes in your code is how SwiftData adds what it needs to use those classes to build a schema. There are a couple of ways to set up a Schema type. The fast way when there is only one class is just to give that @Model decorated class to the simple ModelContainer view modifier style init.

@Model
final class MyModelType {
    var item:Item?
    
    init(item:Item) {
        self.item = item
    }
}

SomeView().modelContainer(for: MyModelType.self, inMemory: true)

Expanding that macro adds the following code:

import Foundation
import SwiftData



final class MyModelType {
    @_PersistedProperty var item:Item?
    
    init(textInfo: String, timestamp: Date) {
        self.item = Item(textInfo: textInfo, timestamp: timestamp)
    }
    
    @Transient
    private var _$backingData: any SwiftData.BackingData<MyModelType> = MyModelType.createBackingData()
    
    public var persistentBackingData: any SwiftData.BackingData<MyModelType> {
        get {
            return _$backingData
        }
        set {
            _$backingData = newValue
        }
    }
    
    static var schemaMetadata: [SwiftData.Schema.PropertyMetadata] {
        return [
            SwiftData.Schema.PropertyMetadata(name: "item", keypath: \MyModelType.item, defaultValue: nil, metadata: nil)
        ]
    }
    
    init(backingData: any SwiftData.BackingData<MyModelType>) {
        _item = _SwiftDataNoType()
        self.persistentBackingData = backingData
    }
    
    @Transient private let _$observationRegistrar = Observation.ObservationRegistrar()
    
    internal nonisolated func access<_M>(
        keyPath: KeyPath<MyModelType, _M>
    ) {
        _$observationRegistrar.access(self, keyPath: keyPath)
    }
    
    internal nonisolated func withMutation<_M, _MR>(
        keyPath: KeyPath<MyModelType, _M>,
        _ mutation: () throws -> _MR
    ) rethrows -> _MR {
        try _$observationRegistrar.withMutation(of: self, keyPath: keyPath, mutation)
    }
    
    struct _SwiftDataNoType {
    }
}

extension MyModelType: SwiftData.PersistentModel {
}

extension MyModelType: Observation.Observable {
}

Note the conformance to Observation.Observable at the end there, which is part of how SwiftData works so seamlessly with SwiftUI.

If there will be more than one class as part of the schema one can declare a Schema type explicitly by giving it an array of all the relevant macro’ed types.

let schema = Schema([
            MyModelType.self, MyHelperType.self
        ])
let modelConfiguration = ModelConfiguration(schema: schema, isStoredInMemoryOnly: false)

Glue Layer Between Models and Persistence

So whole point of using SwiftData is to persist the data associated with the models across app launches. There needs to be code that moves the information between active use and persistent storage and vice versa. In SwiftData the ModelContainer hold the description of how that data will be saved and retrieved.

ModelContainer has a configuration type ModelConfiguration with several initializers like the two below:

ModelConfiguration(_ name: String?, 
                    schema: Schema?, 
                    isStoredInMemoryOnly: Bool, 
                    allowsSave: Bool, 
                    groupContainer: ModelConfiguration.GroupContainer, 
                    cloudKitDatabase: ModelConfiguration.CloudKitDatabase)


ModelConfiguration(<name: String?, 
                    schema: Schema?, 
                    url: URL, 
                    allowsSave: Bool, 
                    cloudKitDatabase: ModelConfiguration.CloudKitDatabase)

One can instantiate a model container using a Schema and an array of configurations:

return try ModelContainer(for: schema, configurations: [modelConfiguration])

This initializer can fail because some aspect of what you’re telling the ModelContainer to believe about the persistence layer could end up being wrong. The file location could not exist. The code could not have permission to change the file, etc.

That’s in contrast to the convenience initializer that uses the default data store as the basis of its assumed configuration. (recall: SomeView().modelContainer(for: MyModelType.self, inMemory: true))

Once the ModelContainer gets created it gets passed into an App’s ModelContext. The ModelContext does the actually pretty darn fancy footwork of moving information between the active processes and the data storage. For example, if with every little change of information the program was constantly writing to disk the code would end up being very slow. The ModelContext batches the changes, balancing speed and safety.

Make a struct and a matching model

The easiest way to check out SwiftData would be be to create a new project with SwiftData the CloudKit options selected. But following the HWS example listed in the references up top, Paul Hudson takes you through the process from the beginning. That tutorial is way more complete than these notes. Follow that. As are the WWDC videos. Watch those.

When looking at tutorials what one will see is that everyone makes their model a class when using SwiftData. And when that class is going to be used with CloudKit, each parameter has to be optional or have a default value. This helps the code know how to resolve when connections fail.

However that’s kind of annoying if one wants use structs for your data type. When writing a project without SwiftData I have typically used structs for my models and an Actor for my DataStore and protocol based PersistenceService to save the data.

I will have to change how I work to use SwiftData, but I wanted to see how keeping structs in the mix might still work.

I tried three different approaches.

Approach 1: Class wrapper for the struct

One approach uses the SwiftData model class as a thin wrapper around a struct. “Business logic” code interacts with the struct, and the UI code selects a record via the class’s code and then pulls out the underlying struct to render, update or access business logic.

Since my interest is working with CloudKit, this example shows having the struct be an Optional in the class.

import Foundation
import SwiftData


import Foundation


struct Item:Sendable, Codable, Hashable, Equatable {
    var textInfo:String
    var timestamp:Date

    var displayMe:String {
        "\(textInfo) at \(timestamp.formatted(date: .long, time: .shortened))"
    }
}

extension Item {
    init(_ text:String) {
        self.init(textInfo: text, timestamp: .now)
    }
    
    init() {
        self.init(textInfo: "Very important information!!!", timestamp: .now)
    }
}

@Model
final class ItemSDMC {
    var item:Item?
    
    init(textInfo: String, timestamp: Date) {
        self.item = Item(textInfo: textInfo, timestamp: timestamp)
    }
}

extension ItemSDMC {
    convenience init(_ text:String) {
        self.init(textInfo: text, timestamp: .now)
    }
    
    convenience init() {
        self.init(textInfo: "Very important information!!!", timestamp: .now)
    }
}

//
//  EditItemView.swift
//  DataFling
//
//

import SwiftUI

struct EditItemView: View {
    
    @Bindable var sdModelItem:ItemSDMC
    @State var nonOptionalItemHolder:Item
    
    
    init(_ item: ItemSDMC) {
        self.sdModelItem = item
        if let wrapped = item.item {
            nonOptionalItemHolder = wrapped
        } else {
            nonOptionalItemHolder = Item(textInfo: "Empty Item", timestamp: Date())
        }
    }
    
    var body: some View {
        var boundItem = Binding(
            get: {
                _nonOptionalItemHolder.wrappedValue
            },
            set: {
                //Note this should never happen in this view because
                //is ItemSDMC somehow returns a nil Item, no edit view will
                //be displayed.
                if $0.textInfo != "EmptyItem" {
                    nonOptionalItemHolder = $0
                    sdModelItem.item = $0
                } else {
                    fatalError("How did an Empty Item get sent to be updated?")
                }
            }
        )

        VStack {
            if sdModelItem.item != nil {
                Text(sdModelItem.item!.displayMe)
                Form {
                    TextField("text", text:boundItem.textInfo)
                    DatePicker("Date", selection: boundItem.timestamp)
                }
                
            } else {
                Text("There has been an mistake. This item has no data")
            }
        }
#if os(iOS)
        .navigationTitle("Edit Item")
        .navigationBarTitleDisplayMode(.inline)
#endif
    }
}

#Preview {
    @Previewable var myItem = ItemSDMC("Example Item for Preview")
    EditItemView(myItem)
}

Discussions on how to handle bindings:

Approach 2: Keep the struct dumb

This approach differs because both the class and the struct have all the fields, and also because the class would contain the business logic and the struct would be a fairly dumb data structure.

Since my interest is working with CloudKit, this example shows having each parameter having a default value, the other choice being to use optionals like in the previous approach.

One of the advantages of this approach is that the newish Observation.Observable protocol has more granularity than what you could do before, i.e. an update to one parameter won’t trigger updates unless that parameter is in fact being observed by something active in the program.

import Foundation
import SwiftData


struct ThingLite:Sendable, Codable, Hashable, Equatable {
    let textInfo:String
    let timestamp:Date
}



@Model
final class Thing {
    //using this default value requires writing some clean up logic looking for empty text info.
    var textInfo:String = ""
    //using this default value would require writing some data clean up functions looking for out of bound dates.
    var timestamp:Date = Date.distantPast
    
    init(textInfo: String, timestamp: Date) {
        self.textInfo = textInfo
        self.timestamp = timestamp
    }
    
    var displayMe:String {
        "\(textInfo) at \(timestamp.formatted(date: .long, time: .shortened))"
    }
    
    var liteCopy:ThingLite {
        ThingLite(textInfo: textInfo, timestamp: timestamp)
    }
    
    var liteCopyWithID:(data:ThingLite, id:PersistentIdentifier) {
        //(sendableCopy, id) also works
        (sendableCopy, persistentModelID)
    }
}

//
//  EditThingView.swift
//  DataFling
//
//

import SwiftUI

struct EditThingView: View {
    @Bindable var thing:Thing
    
    
    var body: some View {
        VStack {
            Text(thing.displayMe)
            Form {
                TextField("text", text:$thing.textInfo)
                DatePicker("Date", selection: $thing.timestamp)
            }
        }.task {
            await doSomeAsyncThing(thing:thing)
        }
#if os(iOS)
        .navigationTitle("Edit Item")
        .navigationBarTitleDisplayMode(.inline)
#endif
    }
    
    func doSomeAsyncThing(thing:Thing) async {
        print(thing.liteCopy)
    }
}

#Preview {
    @Previewable var myThing = Thing(textInfo:"Example for Preview", timestamp: Date())
    EditThingView(thing: myThing)
}

https://fatbobman.com/en/posts/nsmanagedobjectid-and-persistentidentifier/

Third Approach: Protocol

The third approach is to use a protocol as the main basic for all the business logic. I won’t get into that too much now because its the one I’m going to be trying out for a bit.

It might look a little something like the below, although there is still things to work out about how ID would work.

import Foundation
import SwiftData


protocol Thingable:Identifiable {
    var textInfo:String { get }
    var timestamp:Date { get }
}

extension Thingable {
    var thingDisplay:String {
        "\(textInfo) with \(id) at \(timestamp.formatted(date: .long, time: .shortened))"
    }
}

extension Thingable where Self:PersistentModel {
    var thingDisplayWithID:String {
        "\(textInfo) with modelID \(self.persistentModelID.id) in \(String(describing: self.persistentModelID.storeIdentifier)) at \(timestamp.formatted(date: .long, time: .shortened))"
    }
}

struct ThingLite:Thingable, Codable, Sendable {
    var textInfo: String
    var timestamp: Date
    var id: Int
}


@Model
final class Thing:Thingable {
    //using this default value requires writng some clean up logic looking for empty text info.
    var textInfo:String = ""
    //using this default value would require writing some data clean up functions looking for out of bound dates.
    var timestamp:Date = Date.distantPast
    
    init(textInfo: String, timestamp: Date) {
        self.textInfo = textInfo
        self.timestamp = timestamp
    }
}

extension Thing {
    var LiteThing:ThingLite {
        ThingLite(textInfo: textInfo, timestamp: timestamp, id: persistentModelID.hashValue)
    }
}

If one goes that route, at least as of May 2025, it is ill advised to try to make a generic view that takes in some Thingable because Thinable’s that come from an @State deceleration will want to work with @Binding as opposed to @Model objects which, since they conform to Observation.Observable and are owned by the context, will be passed via @Bindable. Those play very poorly together since @State provides a binding to the enclosing object and @Bindable only provides bindings to the properties. .

Post By Me: https://developer.apple.com/forums/thread/782729
Post By Me: https://developer.apple.com/forums/thread/783142
https://developer.apple.com/videos/play/wwdc2023/10149/ (Discover Observation in SwiftUI)
https://www.hackingwithswift.com/quick-start/swiftdata/whats-the-difference-between-bindable-and-binding

Conclusion

Leaning into using SwiftData` requires different methods working with information than previous approaches. I imagine I’ll keep changing my mind as to what patterns I want to try out project by project for some time.

Neat, switching to the ItsyBitsy just... works

Tue, 30 Apr 2024 00:00:00 +0000

The ItsyBitsy

Switching to the ItsyBitsy from the Trinket means means I’ll able to do a breadboard circuit without worrying about smooshing the fragile little solder joints on the bottom for the programmer. (With normal Trinket usage that wouldn’t be a problem because normal sensible people use the UF2 bootloader capabilities.) The ItsyBitsy uses the SAMD21G18A instead of the SAMD21E18A, requiring some minor tweaks to the code. todbot already had one all soldered up with the headers and I snicked it.

Differences between E and G

Screenshot of the pinout diagram of the SAMD21G18A (5.2.1 in family datasheet)

The G has more pins than the E. That doesn’t matter much to this project except the vector table gets two new peripherals added back in: SERCOM4, SERCOM5. See the new startup.s [TODO LINK]

To figure out what peripherals changed, I looked at the Configuration Summary (Section 2-1, page 14 in the Family Datasheet).

Making the Circuit

Unlike the bare ATTiny, I don’t have to make the voltage regulator part of the circuit because the ItsyBitsy already has one. I just pulled the 3.3V and the GND into the breadboard buses. Otherwise it is pretty much the same, with extended wires out to a pseudo-header for the J-LINK.

(The red and black wire up on the top right are there as another pseudo-header for providing non-USB power. I left them off the schematic )

switch pin: ItsyBitsy A4 / Chip PA05
led pin: ItsyBitsy A3 / Chip PA04

Updating the Code

Not really much to do.

The Makefile

line 7

CHIP ?= SAMD21G18A

startup.s

lines 41, 42

    .word   SERCOM4_Handler
    .word   SERCOM5_Handler

lines 88, 89

.weakref SERCOM4_Handler, default_handler
.weakref SERCOM5_Handler, default_handler

main.s

step 1

lines 17, 18

.equ ledPinOffset, 4    //PA04, "A3" on ItsyBitsy
.equ switchPinOffset, 5  //PA05, "A4" on ItsyBitsy

This works but the logic is inverted now, LED is on, unless the button is pressed, so lines 73 and 77 also need to be swapped, resulting in:

  ledOff:  //default path
    STR R3, [R5] //R3==ledMask, R5==portA_OUTSET
    B loop

  ledOn:
    STR R3, [R6] //R3==ledMask, R6==portA_OUTCLR
    B loop

and… done! It works!

Summary

So these two chips come from the same family so it’s not too much of surprise the transition went smoothly. The real test of the Arm promise will be sharing code between one of these M0+ and a chip with a different Coretex-M rating. To do that it would make sense to be working up in the C so the compiler can handle rendering the different appropriate assembly code. A mixed C/Assembly project’s the next step!

It'd make sense to do some toolchain clean up

Sat, 27 Apr 2024 00:00:00 +0000

My Makefile was a bit of a “get it done” hack. It still will be after this session, but a better one!

Goals

clean up how makefile finds its sources in prep for switching chips
clean up assembler options for .s vs .S
break apart OpenOCD tasks
make launching GDB easier
automate some of the gdb tasks
add the .PHONY
clean up the comments (make headers, trailing whitespace matters)
small misc improvements

Conditional Find Files

I’m going to switch from the SAMD21E18A to the SAMD21G18A the next post because todbot has an ItsyBitsy M0 Express ready to swipe and it’s bugging me that the LED and the switch don’t match the AVR example.

This means my code may end up being able to support two chips if I get fancy with my Makefile. To start, proof of concept with the original files.

New directory layout

└── Project
    └── SAMD21E18A
    │   └── linker.ld
    |   └── startup.s 
    └── main.s
    └── Makefile

New additions to the Makefile

I had switched to hard coding the files when I split into main.s and startup.s, now it’s time to switch back to a more automated search.

CHIP ?= SAMD21E18A
MCU := cortex-m0plus

LINKER_FILE ?= $(CHIP)/linker.ld

SOURCES = $(wildcard *.s) # only this dir
SOURCES += $(wildcard *.S) # only this dir
SOURCES += startup.s # all dir in VPATH

VPATH=$(CHIP) #will look in this folder for missing files in sources.

OBJECTS = $(addsuffix .o, $(basename $(SOURCES))) #replaces any suffix

This portion sets up the Makefile to look both in the current directory and anywhere added to the VPATH for the files it needs later on. VPATH is a built in variable. The new custom variable CHIP sets the string indicating which SAMD21 chip is currently on deck. The string gets added to the VPATH as a directory name.

Note: The linker script called with the -T flag needs its full path. To see the default linker script that would be used without that flag add -Wl,--verbose to the linker flags.
Note: to get arm-gcc to spit out the list of mcpu options use arm-none-eabi-gcc --target-help -mcpu=nonsense where “nonsense” can be anything invalid. from stackoverflow

Longer aside: I’ve been switching between :=, = and =? for my assignments. Simplistically

:=: “calculate this here, this is final”
= : “store the calculation and run it at every call site”.
=? : “if there isn’t an environment variable, use this”

::= and :::= are assignment operators too. (More precisely := assigns “simply expanded variables” and = assigns “recursively expanded” variables, see the list of variable flavors)

Sorting out .s vs .S

If you’ve never already, go to a working directory without a makefile in it and type make -p to see all the implicit variables and commands make uses under the hood.

In many setups, adding steps like %.o:%.s or %.o:%.S or %.o:%.c isn’t needed because make already has sensible defaults for “transform every x file into needed y file”

Also, in many setups the assembler will never see human written pure assembly files. All pure assembly will have been written by the compiler.

What’s the general difference between the .S and .s suffix? Well, we can see in the default rules that:

.s files are expected to be ready as is to be made into an object
.S files get extra preprocessing.

# default
PREPROCESS.S = $(CC) -E $(CPPFLAGS)

# default
COMPILE.S = $(CC) $(ASFLAGS) $(CPPFLAGS) $(TARGET_MACH) -c

# default
LINK.S = $(CC) $(ASFLAGS) $(CPPFLAGS) $(LDFLAGS) $(TARGET_MACH)

# default
COMPILE.s = $(AS) $(ASFLAGS) $(TARGET_MACH)

# default
LINK.s = $(CC) $(ASFLAGS) $(LDFLAGS) $(TARGET_MACH)

%: %.s
#  commands to execute (built-in):
	$(LINK.s) $^ $(LOADLIBES) $(LDLIBS) -o $@

%.o: %.s
#  commands to execute (built-in):
	$(COMPILE.s) -o $@ $<

%: %.S
#  commands to execute (built-in):
	$(LINK.S) $^ $(LOADLIBES) $(LDLIBS) -o $@

%.o: %.S
#  commands to execute (built-in):
	$(COMPILE.S) -o $@ $<

%.s: %.S
#  commands to execute (built-in):
	$(PREPROCESS.S) $< > $@

S files “might include C preprocessing statements (#define, #ifdef, and etc.), but they should not contain C statements” (microchip online docs: brief FAQ answer , output files list)

I’ve been personally sloppy on whether my hand rolled assembly should get the .s or .S suffix because of all the .___ compiler directives (.thumb, .syntax unified) and C style comments in them threw me for a loop. In a related sloppiness, I’ve been using the -x assembler-with-cpp option without questioning it for the .s files, even though they shouldn’t need it.

Back when working on separating the startup file, I did confirm the actual assembler works just fine with this flavor of assembly, so I’m going to get a little less sloppy and use the assembler for .s and the complier with -x assembler-with-cpp for .S files.

My main tool variables and build rules now look like

TOOL_BASE := arm-none-eabi-
CC = $(TOOL_BASE)gcc
AS = $(TOOL_BASE)as

## Default
all: $(OBJECTS) $(TARGET).elf
	
## Assembler
%.o:%.s
	$(AS) $(ASFLAGS) -c $< -o $@

%.o:%.S
	$(CC) -x assembler-with-cpp -c $(CFLAGS) $< >$@

## Linker
$(TARGET).elf: $(OBJECTS)
	 $(CC) $(LDFLAGS) $(OBJECTS) $(LINKONLYOBJECTS)  -T./$(LINKER_FILE) -o $@

Improving OpenOCD shortcuts

OpenOCD can be way way fancier than anything I’m doing, but I’ve tired to modularize the project a bit. The previous Makefile only had the one “load-program and launch servers” command. I’ve made some more rules and pushed the asscoiated files into their own directory. I’ve tried to show both stowing options in files and calling commands directly from the Makefile.

New Files

└── Project
    └── openocd
    │   └── segger.cfg
    |   └── SAMD21E18A.cfg 
    |   └── flash_program_prep.cfg 
    └── SAMD21E18A
    │   └── linker.ld
    |   └── startup.s 
    └── main.s
    └── Makefile

segger.cfg

# Segger J-Link EDU, SWD modere
adapter driver jlink
transport select swd

SAMD21E18A.cfg

same name as used for $(CHIP)

set CHIPNAME at91samd21e18
source [find target/at91samdXX.cfg]

flash_program_prep.cfg

init
reset halt
at91samd bootloader 0  #clear bootloader

New variables

PROGRAMMER := openocd
PRGMR_FILES_DIR := openocd/
PROGRAMMER_COMMON = -f $(PRGMR_FILES_DIR)segger.cfg  #load hardware
PROGRAMMER_COMMON += -f $(PRGMR_FILES_DIR)$(CHIP).cfg #load chip description

FLASH_FLAGS = $(PROGRAMMER_COMMON)
FLASH_FLAGS += -f $(PRGMR_FILES_DIR)flash_program_prep.cfg
FLASH_FLAGS += -c "program $(TARGET).elf verify reset exit"

New rules

# Program the device.  
flash: $(TARGET).elf
	$(PROGRAMMER) $(FLASH_FLAGS)

#fns == "flash and serve"
fns: $(TARGET).elf
	$(PROGRAMMER) $(PROGRAMMER_COMMON) \
	-f $(PRGMR_FILES_DIR)flash_program_prep.cfg \
	-c "program $(TARGET).elf verify"

serve:
	$(PROGRAMMER) $(PROGRAMMER_COMMON)

talk: 
	telnet localhost 4444

.PHONY: talk serve

Improving GDB shortcuts

Like with OpenOCD, GDB encompasses huge capabilities, which I’m only lightly touching on. Again, did some modularization and some convenience instructions. Again trying to show performing some tasks inline in the Makefile and others in external files.

GDB has a whole auto-loading schema of rules and extensions that I’m not touching AT ALL, but I’ve put some links in the references.

New Files

└── Project
    └── gdb
    │   └── main_breakpoints.txt
    |   └── mem_locs.txt  
    └── openocd
    │   └── segger.cfg
    |   └── SAMD21E18A.cfg 
    |   └── flash_program_prep.cfg 
    └── SAMD21E18A
    │   └── linker.ld
    |   └── startup.s 
    └── main.s
    └── Makefile

main_breakpoints.txt

break loop
break _start
break multiPinPullup

One way to generate this file would be to quickly add all functions with rbreak . and del [break point number] or clear [filename:linenum] unwanted ones (see current with info break or just i b). Once the breakpoints look the way that works use save breakpoints [filename] to generate the file.

For an example usage see the new Makefile rule below.

mem_locs.txt

Comments on own line only.

# for the direction set
x/1x 0x41004400 
# for the current output values
x/1x 0x41004410
# for the PINCFG settings, PA07 is top
x/1x 0x41004444

Example usage, witch gdb session open try source gdb/mem_locs.txt and the responses should load.

Tool Def & Variables

DEBUGGER = $(TOOL_BASE)gdb

DEBUGGER_FILES_DIR = gdb/
DEBUG_FLAGS = -iex "target extended-remote localhost:3333"
DEBUG_FLAGS += -ex "layout regs"
DEBUG_FLAGS += -ex "source $(DEBUGGER_FILES_DIR)main_breakpoints.txt"

New Rule

debug: $(TARGET).elf
	$(DEBUGGER) $^ $(DEBUG_FLAGS) #$^ inserts ALL the listed input files.

Adding in PHONY

.PHONY chiefly protect against a rule/file name collision preventing a rule from running. I hadn’t been having that trouble, but it’s the right thing to do. (There are other more specific reasons to use it, read the linked doc)

## Phony List 
.PHONY: all talk debug res_check clean serve fnl flash symb disamb symb_o disamb_o

Helpful References

Makefiles

gcc & as

GDB

Summary

clean up how makefile finds its sources in prep for switching chips
clean up assembler options for .s vs .S
break apart openocd tasks
make launching gdb easier
automate some of the gdb tasks
add the .PHONY
clean up the comments (make headers, trailing whitespace matters)
small misc improvements

Next to update the actual circuit!

Full Makefile

###############################################################################
# Makefile for a simple SAMD21 Assembly project
###############################################################################

TARGET ?= combined

CHIP ?= SAMD21E18A
MCU := cortex-m0plus

###############################################################################
## Resoruces
LINKER_FILE ?= $(CHIP)/linker.ld

SOURCES = $(wildcard *.s)# only this dir
SOURCES += $(wildcard *.S)# only this dir
SOURCES += startup.s# all dir in VPATH

VPATH=$(CHIP)#will look in this folder for missing files in sources.

OBJECTS = $(addsuffix .o, $(basename $(SOURCES)))#replaces any suffix

###############################################################################
## Tools
TOOL_BASE := arm-none-eabi-
CC := $(TOOL_BASE)gcc
AS := $(TOOL_BASE)as
OBJD := $(TOOL_BASE)objdump
DEBUGGER := $(TOOL_BASE)gdb
NM := $(TOOL_BASE)nm

PROGRAMMER := openocd

###############################################################################
## Flags

## Options common to compile, link and assembly rules
COMMON = -mcpu=$(MCU)
COMMON += -mthumb

## Compile options common for all C compilation units.
CFLAGS = $(COMMON)
CFLAGS += -Wall
CFLAGS += -g
#CFLAGS += -Os #compression

ASFLAGS = $(CFLAGS)

## Linker flags
LDFLAGS = $(COMMON)
LDFLAGS += --specs=nosys.specs -nostdlib -lgcc
LDFLAGS += -Wl,-Map=$(TARGET).map

# Programming & Debugging 
PRGMR_FILES_DIR = openocd/
PROGRAMMER_COMMON = -f $(PRGMR_FILES_DIR)segger.cfg#load hardware
PROGRAMMER_COMMON += -f $(PRGMR_FILES_DIR)$(CHIP).cfg#load chip description

FLASH_FLAGS = $(PROGRAMMER_COMMON)
FLASH_FLAGS += -f $(PRGMR_FILES_DIR)flash_program_prep.cfg
FLASH_FLAGS += -c "program $(TARGET).elf verify reset exit"

DEBUGGER_FILES_DIR = gdb/
DEBUG_FLAGS = -iex "target extended-remote localhost:3333"
DEBUG_FLAGS += -ex "layout regs"
DEBUG_FLAGS += -ex "source $(DEBUGGER_FILES_DIR)main_breakpoints.txt"

###############################################################################
## Rules

all: $(OBJECTS) $(TARGET).elf

## Assembler
%.o:%.s
	$(AS) $(ASFLAGS) $< -o $@

%.o:%.S
	$(CC) -x assembler-with-cpp -c $(CFLAGS) $< >$@

## Linker
$(TARGET).elf: $(OBJECTS)
	 $(CC) $(LDFLAGS) $(OBJECTS) $(LINKONLYOBJECTS)  -T./$(LINKER_FILE) -o $@
	
## Simple Checking 
%.lss: $(TARGET).elf
	$(OBJD) -h -S $< >$@

disamb: $(TARGET).elf
	$(OBJD) --disassemble $<

disamb_o: $(OBJECTS)
	$(OBJD) --disassemble $^

symb: $(TARGET).elf
	$(NM) --numeric-sort $<

symb_o: $(OBJECTS)
	$(NM) --numeric-sort $^

## Programming & Debugging  
flash: $(TARGET).elf
	$(PROGRAMMER) $(FLASH_FLAGS)


fnl: $(TARGET).elf
	$(PROGRAMMER) $(PROGRAMMER_COMMON) \
	-f $(PRGMR_FILES_DIR)flash_program_prep.cfg \
	-c "program $(TARGET).elf verify"

serve:
	$(PROGRAMMER) $(PROGRAMMER_COMMON)

talk: 
	telnet localhost 4444

debug: $(TARGET).elf
	$(DEBUGGER) $(TARGET).elf $(DEBUG_FLAGS)

## Utilities
clean:
	-rm -rf $(OBJECTS) $(TARGET).elf $(TARGET).map

# "resource check" - to get list of items make expects to find
res_check:
	@echo $(SOURCES)
	@echo $(OBJECTS)

## Phony List 
.PHONY: all talk debug res_check clean serve fnl flash symb disamb symb_o disamb_o

And now for 3 ways to set an internal pullup

Tue, 23 Apr 2024 00:00:00 +0000

Related Repo: https://github.com/carlynorama/StrippedDownChipRosetta/blob/main/ARM/SAMD21E18/04_AssemblySwtich/

Slowly working towards parity with the AVR example, this post adds the switch with an internal pullup resistor. One of the nice things about internal pullups? The part of a switch can comfortably be played by some 22 AWG yellow wire from the Dish of Scrap Miscellany.

Circuit Warning

This circuit doesn’t match the AVR circuit. The AVR Circuit sank the LED and pulled up the switch, following classic recommend practice. This circuit:

different: sources the LED, because that’s the Arduino built-in LED convention. (Off==On is kinda hard for intro classes)
same: pulls up the switch

Since the switch and led don’t match electrically, the code will handle the inversion to get the same result as the AVR circuit. (On when closed, off when open)

“Bit Hacking”

Checking a switch and conditionally turning on an LED requires some fancier moves than setting an LED alone. I made an emulator script with some examples. The two-step instructions like MOV R1, R0, LSL #6 need to be broken up and MOV changed to MOVS on the actual chip (MOV is reserved for specific registers on CM0+), but the concepts translate.

References

YouTube: 3: Bit Hacks (8:22) as part of MIT 6.172 Performance Engineering of Software Systems, Fall 2018 by MIT OpenCourseWare
Bit Twiddling Hacks by Sean Eron Anderson at Stanford Computer Graphics Laboratory
Knuth TAOCP, 4A section 7.1.3

Example Code

emulator link

.global _start
_start:

  //-------------------------------------------------
  //Adding new 1 bits
  
  MOV R0, #1
  ORR R0, R0, LSL #6 //R0 will now have 1 bit at 0 and 5
  
  //-------------------------------------------------
  //Using TST
  //TST compares two values with an AND, discarding result, 
  //but sets zero flag if the result of the operation was 0
  //So if the two values have ANY bits the same ZF == 0
  
  //R0 is 0x41
  TST R0, #1  //result of AND is 1, ZF == 0
  TST R0, #0x40 //result of AND is 0x40, ZF == 0
  
  MOV R1, #0
  TST R1, #1  //result of AND is 0, ZF == 1
  TST R1, #64 //result of AND is 0, ZF == 1
  
  MOV R1, R0, LSL #6 //R1 will have a bit at 5 and 11 
  TST R1, R0 //have bits in common, ZF 0
  
  MOV R0, R1
  TST R1, R0
  
  //EQ and NE are named for the CMP results (which does a SUB)
  //So results confusing. 
  //BEQ to branch when ZF == 1  //has no things in common
  //BNE to branch when ZF == 0  //has some things in common
  
  //-------------------------------------------------
  //Set specific bits to zero
  LDR R0, =0xAAAAAAAA
  MOV R1, #10  //put an a in the 0 position
  LSLS R1, #12 //move the a to the 3rd position (4*3)
  
  //Performs XOR operation
  //on bits of Rn with bits of Rm.
  EORS R2, R0, R1
  
  //Performs AND operation
  //on bits of Rn with ~bits of Rm.
  BICS R3, R0, R1
  
  //Looks the same? Not quite!
  
  //0xA == 0b1010, 0x2 = 0b0010
  LDR R4, =0x22222222
  ORR R5, R4, R1   //Adds missing bits => 0b1010
  EORS R5, R4, R1  //removes where duplicated => 0b1000
  BICS R5, R4, R1  //removes where mask (R1) is 1 => 0b0000
  
  //-------------------------------------------------
  //Retrieve selective chunk with mask
  MOV R0, #0xFF
  LSL R0, #16 //(4*N)
  AND R5, R4, R0
  LSR R5, #16  //if need it down in lowest part

    //-------------------------------------------------
  //Isolate selective chunk no mask
  LDR R0, =#0x22FF2222
  LSL R0, #8 //(4*N)
  LSR R0, 24
  
  //-------------------------------------------------
  //Set selective chunk
  
    LSL R0, 24 //mask to clear target area
  BICS R4, R0 //clean out the bit area in target
    MOV R1, #0xAA  //put desired value in a register
  ORR R4, R1, LSL #16 //orr clean target with shifted new value

Conditional Branching with the Thumb-1 Instructions Set

What’s another thing you won’t see in any of the code for the SAMD21E18A? Most of the compound conditional instructions. (cc place holder for conditional code)

BLcc (Bcc okay though)
MOVcc Rd Rm
IT cc…

Cortex-M0/Cortex-M0+ chips use Arm v6, which means the thumb v1 instruction set, which means NONE of the conditional niceties available to their bigger siblings. I’m not used to assembly having them anyway, but much of the example code out there seems to target M3 or higher. It just won’t compile with the -mcpu=cortex-m0plus flag set.

References

See also the links in the above ¶
armv6 vs armv7 overview from Arm community
Instruction Set Assembly Guide for Armv7 and earlier Arm architectures from Arm documentation service
Arm Cortex-M0+ DataSheet from Arm contains image below. Figure 5, page 6.

Example Code

What does that difference look like in practice? In full Arm syntax, BLNE (branch with link if Z!=0) can be called with no fuss.

//full syntax
//.syntax unified
//.arm //might also see as .code 32
  TST R0, R4
  BLNE doIfAnySwitch
  //continue...

In thumb2 BLNE can be called inside an IT block.

//on armv7, thumb2 capable chips
//.syntax unified
//.thumb //might also see as .code 16
  TST R0, R4
  IT NE
  BLNE doIfAnySwitch
  //continue...

In thumb1 BLNE doesn’t exist at all but BEQ and BNE do. The best call will be to look for the negative branch condition instead and jump past the branch instead.

//------ thumb1 style only
//on armv6, thumb 1 only
//.syntax divided 
//.thumb //might also see as .code 16
  TST R0, R4
  BEQ skipBranch
  BL doIfAnySwitch
  skipBranch:
  //continue...

So if you’re wondering why my switch code doesn’t use any of the fancy branching syntax you’ve seen in other places, that’s why.

Calling Conventions

It’s finally time to touch on proper sub routine calling conventions. The TL;DR

Don’t trust R0-R3 to stay the way you left them (“callee clobbered”)
Put R4-R11 and the stack pointer (“SP”) need to be put back the way th callee found them (“callee preserved”).
R12-R15 the system can alter at any time
Subroutines with subroutines have to deal with the original LR being “clobbered”, but can handle that by putting their desired “LR” back into “PC” themselves.

It’s up to the programmer to play nice with expectations, and each manufacturer has slight variations on top of the Arm standard to be aware of.

Register Conventions for SAM MCUs

Register	Label	Role
R0		argument/result/scratch register 1
R1		argument/result/scratch register 2
R1		argument/scratch register 3
R3		argument/scratch register 4
R4		variable register 1
R5		variable register 2
R6		variable register 3
R7		variable register 4
R8		variable register 5*
R9		variable register 6*
R10		variable register 7*
R11		variable register 8*
R12	IP	Intra-procedure call scratch register*‡
R13	SP	Stack pointer
R14	LR	Link Register
R15	PC	Program Counter

* In the M0 and M0+ spec, registers above R7 have limitations. Only R0-R7, the low registers, can uniformly be accessed by all instructions. The higher registers can only be accessed by add, blx, bx, cmp, mov (not movs), msr, mrs. This is indicated in the Cortex-M0+ Technical Reference Manual (Revision: r0p1) section 3.3 as some functions only being able to use with “Lo” (R0-R7) vs “Any”, See also A4.1 in the ARMv6-M Architecture Reference Manual for a brief mention

References

StackOverflow discussion on what registers to save for Arm
Microchip Register Conventions Guide for Pic32C/SAM MCUS
Procedure Call Standard for the ARM® Architecture v2.08 (Older doc, linked to from UMich EECS 373 website.)
Example of push/pop in the wild from FFMPEG code base
Relationship to C function calls from TI
“The Rust Calling Convention We Deserve” mcyoung

Code Example

It doesn’t do the full “prologue” and “epilogue” one might see in complicated programs, but it’s a start.

emulator

.syntax unified
.global _start
_start:
  
  MOV R4,  #0xAA
  MOV R5,  #0xAB
  MOV R6,  #0xAC
  MOV R7,  #0xAD //NOTE: regs above 7 have limits in
  MOV R8,  #0xAE //CM0/CM0+ chips. can only use with
  MOV R9,  #0xAF //add, blx, bx, cmp, mov, msr, mrs
  MOV R10, #0xBA 
  MOV R11, #0xBC
  
  loop:
    MOV R0, #1
    MOV R1, #2
    MOV R2, #3
    MOV R3, #4

    BL subCallWithSubCall
    //R4-R11 will look exactly the same
    //R0-R3 will be entirely different
    //R0 will have possible return value
    //sp is back to zero
  B loop
  
subCallWithSubCall:
  PUSH {R4-R11, LR}
  PUSH {R0-R3} //move R0-R3 
  POP {R4-R7}  //into R4-R7

  ADD R1, R5, R6
  ADD R2, R7, R8
  ADD R3, R8, R9

  PUSH {R1-R3} //save my calculations
  BL leafCall
  POP {R1-R3} //retrieve them.
  //R0 has "result of leafCall"

  CMP R0, R10
  MOVLT R0, R10 //wouldn't be possible in thumb1
  //R0 has result of this routine
POP {R4-R11, PC} //no BX LR b/c return was handled by 
         //direct load of saved LR into PC

leafCall:
  PUSH {R4-R11} //J.I.C. would remove really. 

  ADD R1, R5, R6
  ADD R2, R7, R5
  ADD R0, R1, R2

  POP {R4-R11}
BX LR

Setting pullups

Using the data gleaned in the last post, I wrote three different versions of setting the pullup resistor for my switch on PA07.

V1 - inline, hard coded

I gave this code a label so I could set a break point, but it lives inline before the loop as you can see in the repo (link below).

The biggest gotcha was picking the address to put in switchPinCNFGOffset because to use LDR that address has to be divisible by 4.

So instead of portA_PINCNFG+0x07, it’s portA_PINCNFG+0x04

I show loading a specific byte in the later examples, but I thought this was a good lesson. To use word aligned data I fetched the nearest but lower aligned location and shifted the needed bits up to the last byte to match the 7 total offset.

in repo

setPullup:
  //---- For using internal pullup only
  //switchPinCNFGOffset == portA_PINCNFG+0x04 
 LDR R5, =switchPinCNFGOffset //pinConfig closest word location
  LDR R0, [R5] //load current settings into R0
  MOVS R1, #6 //create value for INEN 1 (bit 1) //set PULLEN 1 (bit 2)
  LSLS R1, #24 //(8*(7-4)) //move it from 4 to 7
  ORRS R0, R1 //apply mask
  STR R0, [R5] //put the updated word back into the config.
  LDR R5, =portA_OUTSET
  STR R4, [R5] //set the out of the switch high
  //--- END setting internal pullup

V2 - one pin per branch

This code takes the pin number in R0 and sets the config for that pin. I don’t bother pushing R4-R7 because I carefully don’t touch them.

in repo

//Set R0 to contain the pin number
//function uses R0-R3 
.word setPullup
.thumb_func
setPullup:
  //Get base address
  LDR R1, =portA_PINCNFG
  //LDRB will allow byte call
  LDRB R2, [R1, R0] // CFG base, with immediate offset (R0==pin #)

  MOVS R3, #6 // INEN 1 (bit 1) //set PULLEN 1 (bit 2)
  ORRS R2, R3 // update retrieved settings with new settings

  //Must save with STRB
  STRB R2, [R1, R0] 
  //Make pull UP by setting high 
  LDR R1, =portA_OUTSET
  MOVS R2, #1
  LSLS R2, R0 //move the bit to the pin pos
  STR R2, [R1] //set the out of the switch pin high
  BX lr

V3 - batch set

This code uses the WRCONFIG address to batch set multiple pins. See page 383, Section 23.8.11 in the Family Datasheet. I don’t need it for this code, but since they provide the address I wanted to show it!

Using the WRCONFIG address can set up to 16 pins in the same half of a pin group that should share the same configuration all the same time.

Bit	Name	Acronym	Usage

0-15	Pin mask	PINMASK	1: set values for this pin, 0: ignore
16	Peripheral Multiplexing Enable	PMUXEN	1: enabled , 0: disabled (0 in PINCFG)
17	Input Enable	INEN	1: pin is input, 0: pin is output (1 in PINCFG)
18	Pull Enable	PULLEN	1: pull enabled , 0: pull disabled (2 in PINCFG)
19	—	—	No setting in SAMD21
20	—	—	No setting in SAMD21
21	—	—	No setting in SAMD21
22	Output Driver Strength Selection	DRVSTR	0 normal 1 strong (6 in PINCFG)
23	—	—	No setting in SAMD21
24-27	Peripheral Multiplexing	PMUX	see 23.8.12 and Sec. 7
28	-N–	WRPMUX	Should update PMUX (1, yes, 0 will ignore)
29	—	—	No setting in SAMD21
30	Write PINCFG	WRPINCNFG	Should update PINCFGs (1: yes, 0: will ignore)
31	Half-Word Select	HWSEL	Indicates if 0-15 are the lower (0) or higher (1) pins

This code takes in the whole 32 pin group and splits it into the necessary two batches.

Again this code carefully avoids spilling over past R3. It doesn’t use a conditional linked branch to a sub routine that does the repeated tasks of pairing a setting with a half-pin-group and loading it to the register. Little austerities that probably wouldn’t make sense to a functional programming mindset without knowing the thumb1 assembly context.

mock version in emulator
in repo

.word multiPinPullup
.thumb_func
multiPinPullup:
  //Go ahead and set the ups
  LDR R1, =portA_OUTSET
  STR R0, [R1]

  MOVS R2, #1
  MOVS R3, #2
  RORS R2, R3  // Bit 30 -> 1 to enable PINCFG set
  MOVS R3, #6  // INEN 1 (bit 1) //set PULLEN 1 (bit 2)
  LSLS R3, #16 // move up to 3rd byte
  ORRS R3, R2  // Add bit 30 to R3

  LDR R2, =#0xFFFF
  TST  R0, R2
  BEQ  mpp_upperHalf
  MOVS R1, R0  // copy R0 for editing
  AND R1, R2   //isolate bottom half
  ORRS R1, R3  // update R1 with settings 
  LDR R2, =portA_WRCONFIG
  STR R1, [R2] // send WRCONFIG bottom half info

  mpp_upperHalf:
  LDR R1, =#0xFFFF0000
  TST R0, R1
  BEQ mpp_exit
  LSRS R0, #16 // isolate top half
  ORRS R0, R3  // add settings
  LSLS R1, #15 // make R1 0x80000000
  ORRS R0, R1  // Add top half flag in bit 31
  LDR R2, =portA_WRCONFIG //might be this value already. 
  STR R0, [R2] // send WRCONFIG top half info
  mpp_exit:
  BX LR

Reacting to the Switch

I wrote two ways to react to the switch values. The first requires knowing the distance between the two pins on the PA register, but doesn’t depend on branching. The second uses a TST and BEQ to send the ledMask to OUTCLR or OUTSET.

V1 - Use the Switch Pin Directly

This code never “knows” what the state of the switch pin is. It sends whatever that state is to the LED’s OUTCLR position and it’s inverse to OUTSET every time. (see hardware note)

in repo

  MOVS R4, #1  
  LSLS R4, #switchPinOffset
  LDR R5, =portA_OUTSET
  LDR R6, =portA_OUTCLR
  LDR R7, =portA_IN

  loop:
    //get the value of the IN register (address in R7)
    LDR R0, [R7]

    //Isolate the switch reading 
    ANDS R0, R4 //should typically be 1, due to pullup resistor
    MOVS R1, R0
    EORS R1, R4 //R1 will 1 at the switch bit iff the read was 0

    LSLS R0, #3  //move bit 7 (switch) to 10 (led)
    LSLS R1, #3  //move bit 7 (switch) to 10 (led)

    //Sending 0s the to set and clear registers has no effect.
    //The state of R0 always matches what should go to CLEAR
    //The state of R1 matches what should be in SET 
    STR R0, [R6]
    STR R1, [R5]
  B loop

V2 - Test the Switch Pin

Uses separate masks for the led pin and switch pins, but requires navigating conditionals in the thumb 1 instruction set!

in repo

  MOVS R4, #1
  LSLS R3, R4, #ledPinOffset 
  LSLS R4, #switchPinOffset
  LDR R5, =portA_OUTSET
  LDR R6, =portA_OUTCLR
  LDR R7, =portA_IN

  loop:
    //get the value of the IN register
    LDR R0, [R7]
        //TST the reading against the mask. 
                //R0 should read 1 at switch location as default
                //due to pullup resistor so if the AND result
    TST R0, R4  //EQUALS 0 (no match), the switch is _closed_ 
    BEQ ledOn   //so jump to on

  ledOff:  //default path
    STR R3, [R6] //R3==ledMask, R6==portA_OUTCLR
    B loop

    ledOn:
    STR R3, [R5] //R3==ledMask, R5==portA_OUTSET
    B loop

Complete main.s examples

Version 1

.syntax unified
.cpu cortex-m0
.fpu softvfp 
.thumb  

.section .text.program_code

.equ portA_address, 0x41004400
.equ portA_DIRCLR, 0x41004400+0x04
.equ portA_DIRSET, 0x41004400+0x08
.equ portA_OUTCLR, 0x41004400+0x14
.equ portA_OUTSET, 0x41004400+0x18
//.equ portA_OUTTGL, 0x41004400+0x1C
.equ portA_IN, 0x41004400+0x20
.equ portA_PINCNFG, 0x41004400+0x40



.equ ledPinOffset, 10    //PA10
.equ switchPinOffset, 7  //PA07, 
//this value is NOT 7 because use LDR below
//which requires a word alignment
.equ switchPinCNFGOffset, portA_PINCNFG+0x04 
//NOTE: a CLOSED (true) switch will read LOW

.thumb_func
.global _start
_start:

	MOVS R3, #1
	LSLS R3, #ledPinOffset  //R3 is now LED mask (1 at led pos)
  MOVS R4, #1  
	LSLS R4, #switchPinOffset //R4 is now switch mask (1 at switch pos)

  //The DIRSET should have one for every LED and 0 for everys switch.
  //The 0 is the default, but an explicit set will be safe. 

  LDR R5, =portA_DIRSET
  STR R3, [R5]   //move the 1s in for the LEDs
  LDR R5, =portA_DIRCLR
  STR R4, [R5]   //clear all the switches

setPullup:
  //---- For using internal pullup only
  LDR R5, =switchPinCNFGOffset //pinConfig closest word location
  LDR R0, [R5] //load current settings into R0
  MOVS R1, #6 //create value for INEN 1 (bit 1) //set PULLEN 1 (bit 2)
  LSLS R1, #24 //(8*(7-4)) //move it from 4 to 7
  ORRS R0, R1 //apply mask
  STR R0, [R5] //put the updated word back into the config.
  LDR R5, =portA_OUTSET
  STR R4, [R5] //set the out of the switch high
  //--- END setting internal pullup

  //if remove pullup code
  //set R5 to hold portA_OUTSET here
  LDR R6, =portA_OUTCLR
  LDR R7, =portA_IN

  loop:
    //get the value of the IN register
    LDR R0, [R7]

    //OPTION 1
    //Isolate the switch reading 
    ANDS R0, R4 //should typically be 1, due to pullup resistor
    MOVS R1, R0
    EORS R1, R4 //R1 will 1 at the switch bit iff the read was 0

    LSLS R0, #3  //move bit 7 (switch) to 10 (led)
    LSLS R1, #3  //move bit 7 (switch) to 10 (led)

    //Sending 0s the to set and clear registers has no effect.
    //The state of R0 always matches what should go to CLEAR
    //The state of R1 matches what should be in SET 
    STR R0, [R6]
    STR R1, [R5]
  B loop

Version 2

.syntax unified
.cpu cortex-m0
.fpu softvfp 
.thumb  

.section .text.program_code

.equ portA_address, 0x41004400
.equ portA_DIRCLR, 0x41004400+0x04
.equ portA_DIRSET, 0x41004400+0x08
.equ portA_OUTCLR, 0x41004400+0x14
.equ portA_OUTSET, 0x41004400+0x18
//.equ portA_OUTTGL, 0x41004400+0x1C
.equ portA_IN, 0x41004400+0x20
.equ portA_WRCONFIG, 0x41004400+0x28
.equ portA_PINCNFG, 0x41004400+0x40

.equ ledPinOffset, 10    //PA10
.equ switchPinOffset, 7  //PA07, 
//NOTE: a CLOSED switch will read LOW

//since pull a byte later, need the raw value to be a byte


.thumb_func
.global _start
_start:


  MOVS R4, #1 
  LSLS R3, R4, #ledPinOffset //R3 is now LED mask (1 at led pos)
  LSLS R4, #switchPinOffset  //R4 is now switch mask (1 at switch pos)

  //The DIRSET should have one for every LED and 0 for everys switch.
  //The 0 is the default, but an explicit set will be safe. 

  LDR R5, =portA_DIRSET
  STR R3, [R5]   //move the 1s in for the LEDs
  LDR R5, =portA_DIRCLR
  STR R4, [R5]   //clear all the switches

  @ OPTION 1: Set 1 pin
  @ MOVS R0, #switchPinOffset  //put switch # into R0 
  @ PUSH {R0-R3} // stash in stack
  @ BL setPullup
  @ POP {R0-R3}  // restore

  @ OPTION 2: Set multiple pins
  MOVS R0, R4  //put switch mask into R0
  PUSH {R0-R3} //stash in stack
  BL multiPinPullup
  POP {R0-R3}  //restore 

  LDR R5, =portA_OUTSET
  LDR R6, =portA_OUTCLR
  LDR R7, =portA_IN

    loop:
    //get the value of the IN register
    LDR R0, [R7]
				        //TST the reading against the mask. 
                //R0 should read 1 at switch location as default
                //due to pullup resistor so if the AND result
    TST R0, R4  //EQUALS 0 (no match), the switch is _closed_ 
    BEQ ledOn   //so jump to on

	ledOff:  //default path
    STR R3, [R6] //R3==ledMask, R6==portA_OUTCLR
    B loop

    ledOn:
    STR R3, [R5] //R3==ledMask, R5==portA_OUTSET
    B loop

//end _start


//Set R0 to contain the 32 pin mask of pins that should have pullups
.word multiPinPullup
.thumb_func
multiPinPullup:
  //Go ahead and set the ups
  LDR R1, =portA_OUTSET
  STR R0, [R1]

  MOVS R2, #1
  MOVS R3, #2
  RORS R2, R3  // Bit 30 -> 1 to enable PINCFG set
  MOVS R3, #6  // INEN 1 (bit 1) //set PULLEN 1 (bit 2)
  LSLS R3, #16 // move up to 3rd byte
  ORRS R3, R2  // Add bit 30 to R3

  LDR R2, =#0xFFFF
  TST  R0, R2
  BEQ  mpp_upperHalf
  MOVS R1, R0  // copy R0 for editing
  ANDS R1, R2   //isolate bottom half
  ORRS R1, R3  // update R1 with settings 
  LDR R2, =portA_WRCONFIG
  STR R1, [R2] // send WRCONFIG bottom half info

  mpp_upperHalf:
  LDR R1, =#0xFFFF0000
  TST R0, R1
  BEQ mpp_exit
  LSRS R0, #16 // isolate top half
  ORRS R0, R3  // add settings
  LSLS R1, #15 // make R1 0x80000000
  ORRS R0, R1  // Add top half flag in bit 31
  LDR R2, =portA_WRCONFIG //might be this value already. 
  STR R0, [R2] // send WRCONFIG top half info
  mpp_exit:
  BX LR

Summary

Choosing the smallest of the Arm family makes writing the assembly extra fidly. To the rescue… switching to C and letting the compiler sort it out! That better matches the AVR example anyway. I’m going to do same GDB/makefile tidy-up in prep.

Uhg, How do I post a blinking LED image again?

Tue, 23 Apr 2024 00:00:00 +0000

Every time I have to look this up… I have a longer page of notes to go through, but at least I can post the links.

Link Dump

https://stackoverflow.com/questions/42980663/ffmpeg-high-quality-animated-gif
https://stackoverflow.com/questions/3211595/renaming-files-in-a-folder-to-sequential-numbers
https://stackoverflow.com/questions/3688870/create-animated-gif-from-a-set-of-jpeg-images
https://superuser.com/questions/556029/how-do-i-convert-a-video-to-gif-using-ffmpeg-with-reasonable-quality/556031#556031
http://blog.pkh.me/p/21-high-quality-gif-with-ffmpeg.html
https://cassidy.codes/blog/2017/04/25/ffmpeg-frames-to-gif-optimization/
https://mattj.io/posts/2021-02-27-create-animated-gif-and-webp-from-videos-using-ffmpeg/
https://gist.github.com/devadvance/f2ad3cfe38afe3eeef64c72c46692158
https://www.paulirish.com/2021/video-stabilization-with-ffmpeg-and-vidstab/
https://onlinegiftools.com/analyze-gif
https://www.baeldung.com/linux/convert-videos-gifs-ffmpeg
https://shotstack.io/learn/convert-video-gif-ffmpeg/
https://ffmpeg.org/ffmpeg-filters.html#paletteuse
framerate: Set the frame rate for the video stream. It defaults to 25.
loop: If set to 1, loop over the input. Default value is 0.
start_number: Set the index of the file matched by the image file pattern to start to read from. Default value is 0.
start_number_range: Set the index interval range to check when looking for the first image file in the sequence, starting from start_number. Default value is 5.
video_size: Set the video size of the images to read. If not specified the video size is guessed from the first image file in the sequence
-r[:<stream_spec>] <rate>: override input framerate/convert to given output framerate (Hz value, fraction or abbreviation)
-vf <filter_graph>: alias for -filter:v (apply filters to video streams)

get ffmpeg

brew install ffmpeg

Use ffmpeg for creating a video from the images in the file sequence img-001.jpeg, img-002.jpeg, …, assuming an input frame rate of 10 frames per second:

ffmpeg -framerate 10 -i 'img-%03d.jpeg' out.mkv

As above, but start by reading from a file with index 100 in the sequence:

ffmpeg -framerate 10 -start_number 100 -i 'img-%03d.jpeg' out.mkv

Read images matching the “*.png” glob pattern , that is all the files terminating with the “.png” suffix:

ffmpeg -framerate 10 -pattern_type glob -i "*.png" out.mkv 
#-video_size 320x240

## destination folder must already exist.
ffmpeg -i $filename.mov -r 1 "$filename_frames/$filename%03d.png"
## edit frames...
ffmpeg -f image2 -i "$filename_frames/$filename%03d.png" -loop 0 $filename.gif

ffmpeg -i IMG_1703.mov -r 12 "all_frames2/switch_press%03d.png"

ffmpeg -i $filename.mov -r 1 "$filename_frames/$filename%03d.png"
ffmpeg -f image2 -i "$filename_frames/$filename%03d.png" -loop 0 $filename.gif

# Change these placeholders:
# * $INPUT_START_TIME - number of seconds in the input video to start from.
# * $LENGTH - number of seconds to convert from the input video.
# * $INPUT_FILENAME - path to the input video.
# * $OUTPUT_FPS - ouput frames per second. Start with `10`.
# * $OUTPUT_WIDTH - output width in pixels. Aspect ratio is maintained.
# * $NUMBER_OF_LOOPS - use `0` to loop forever, or a specific number of loops.
# * $OUTPUT_FILENAME - the name of the output animated GIF.

ffmpeg -ss $INPUT_START_TIME -t $LENGTH -i $INPUT_FILENAME \
-vf "fps=$OUTPUT_FPS,scale=$OUTPUT_WIDTH:-1:flags=lanczos,split[s0][s1];[s0]palettegen[p];[s1][p]paletteuse" \
-loop $NUMBER_OF_LOOPS $OUTPUT_FILENAME

ffmpeg -f image2 -framerate 12 -start_number 085 -i these/switch_press%003d.png -vf "fps=12,scale=240:-1:flags=lanczos,split[s0][s1];[s0]palettegen[p];[s1][p]paletteuse" -loop 0 touch_12fps.gif

num=298; for i in *; do mv "$i" "$(printf 'switch_press%003d' $num).${i#*.}"; ((num++)); done
ffmpeg -f image2 -i "these5/switch_press%003d.png" -loop 0 touchc.gif

ffmpeg -f image2 -framerate 12 -i these/switch_press%003d.png -vf "fps=10,scale=320:-1:flags=lanczos,split[s0][s1];[s0]palettegen[p];[s1][p]paletteuse" -loop 0 toucha_12to10fps.gif

ffmpeg -i IMG_1706.mov -vf "palettegen" palette.png
ffmpeg -i IMG_1706.mov -loop 0 -filter_complex "fps=10, scale=-1:240[s]; [s]split[a][b]; [a]palettegen[palette]; [b][palette]paletteuse" final_a.gif

It's ALIVE! (SAMD21E18A, Assembly, No SDK)

Mon, 15 Apr 2024 00:00:00 +0000

More Refs

YouTube: CMU Computer Architecture (18-447), Spring 2015 by Carnegie Mellon Computer Architecture If one wants to go off in the weeds? Taught by Prof. Onur Mutlu
YouTube: Lecture 5: Memory Mapped I/O (8:22) as part of Short Lectures by Embedded Systems and Deep Learning Playlist for book by Yifeng Zhu
- Chapter 14: GPIO PDF of slides from above video by Dr. Yifeng Zhu for his 2018 class at the University of Maine
YouTube: Raspberry Pi Pico Bare Metal Programming by Life with David I've only watched the first one but it looks excellent. Based on the RP2040
GitHub: The samd port folder (5114f2c) in the micropython repo

Finding the GPIO Information in the Datasheet

Still this one

What Pin do we care about?

D13 == PA10

We aren’t trying to figure out how to lay out a circuit, we’re trying to figure out how to talk to the circuit we have. The PyRuler like many arduino compatible boards has a built in LED on “pin 13”. That means NOTHING to the SAMD21E18A. Going to the Adafruit docs on the Trinket M0, we can see that D13 matches to PA10.

What pins does the chip have?

When we write our code, what will be there to listen? What pins might have more than one purpose so if whe set a port up to do one thing it won’t do another?

(Notice the number skipping compared the SAMD21J (5.1.1) , not all locations get a pin to the outside in this smaller package.)

What will this documentation call the GPIO peripheral?

PORT

Chip documentation uses a lot of abbreviations, steeped in history. Even though so many software packages call twiddling pins their GPIO library, an acronym, which you’d think would be perfect, don’t look for it in the datasheet. If you didn’t know about the history, how might you find out?

Section 6, Signal Descriptions List, gives us our list of acronyms for the type of electrical information the chip can take in and give out. Towards the end of the list we see “General Purpose I/O - PORT” Gotcha.

Generally speaking where will these values be in memory?

lowest value: 0x40000000
highest value: 0x42FFFFFF

We know from Figure 9-1, the memory map (shown last post) where the peripherals will be in memory memory map peripherals

Yeah but where are the PINS in there?

PORT 0x41004400

Table 12-1. Peripherals Configuration Summary Called PORT 0x41004400 (NOT GPIO) That puts it in the AHB-APB Bridge B section. This is where the “memory map” happens.

So thats where the pins are, Where and what do I call specifically to set them?

PA group offset 0x00 => 0x41004400
PB group offset 0x80 => 0x41004480

(from 23.6.2.2 Operation )

Everything we need is in section “23. PORT - I/O Pin Controller”, especially how to interact with the Memory Mapped I/O. As mentioned in the video above (Lecture 5: Memory Mapped I/O) there aren’t special port instructions at an assembly level on Arm chips, it’s all just memory instructions. Typically the chip manufacturers will provide special addresses to make certain operations faster. We can find them in

Table 23-7.

Offset	Name	Acronym	Usage

0x00	Data Direction Values	DIR	0 is input, 1 output
0x04	Data Direction Clear	DIRCLR	set to input with a 1 at bit, 0 no change
0x08	Data Direction Set	DIRSET	set to output with a 1 at bit, 0 no change
0x08	Data Direction Toggle	DIRTGL	toggled with a 1 at bit, 0 no change
0x10	Data Output Value	OUT	0 is low, 1 is high
0x14	Data Output Clear	OUTCLR	set to low with a 1 at bit, 0 no change
0x18	Data Output Set	OUTSET	set to high with a 1 at bit, 0 no change
0x1C	Data Output Toggle	OUTTGL	toggled with a 1 at bit, 0 no change
0x20	Data Input Value	IN	0 is a low reading, 1 is a high reading
0x24	Input Sampling Mode	CTRL	0 on demand sampling, 1 continuous sampling
0x28	Write Configuration	WRCONFIG	various meanings see 23.8.11
0x30	Peripheral Multiplexing	PMUX	see 23.8.12 and Sec. 7
0x40	Pin Configuration	PINCFG	various see 23.8.13, AND Table 23-2

What about when I want a to add a switch?

Remember for the ATtiny45 I had to set the pull up? Same for the SAMD21E18A. The configuration information lives in Table 23-2 (Pin Configurations Summary) and section 23.8.13 (PINCFG register details).

See 23.8.13
offset to pin settings == n*0x01 [n=0..31]

7	6	5	4	3	2	1	0
	DRVSTR				PULLEN	INEN	PMUXEN

They’re a little tricky to use so lets do an example. Say I want a switch with logic low and a pull up resistor on PA05:

Byte Address	Name	Value for Pin	Byte Address Calc	Which Bit	Meaning
0x41004400	DIR	0	PORT + DIR	5	Pin 5 is an output
0x41004445	DPINCFG05.INEN	1	PORTA + PINCONFIG + pin settings	INEN (bit 1)	Turn on the input buffer
0x41004445	DPINCFG05.PULLEN	1	PORTA + PINCONFIG + pin settings	PULLEN (bit 2)	Use a pull
0x41004410	OUT	1	PORT + OUT	5	That pull should be a pull up

Delay Time

GCLK 0x40000C00
Default: 1MHz clock - OSC8M/8

Another thing I did with the ATtiny45 was check the clock settings. In Section 8.3.1, Starting of Clocks, the datasheet says - “the device will use a 1MHz clock. This clock is derived from the 8MHz Internal Oscillator (OSC8M), which is divided by eight.” Since that matches how I left the the ATtiny project, I’ll leave this chip alone.

Arm’s Cortex-M0+, the smallest of the family, offers more than one clock to it’s peripherals. That’s impressive. All of this is in Section 15 on GCLK, We’d have picked up from table 12-1 that all the offsets in this section are from GCLK 0x40000C00

What we want is in 15.7 the register summary and the follow up detailed pages.

Offset	Name	Acronym	Usage

0x00	Control	CNTRL	Only bit 0, SWRST, software reset
0x01	Status	STATUS	Only bit 7, SYNCBUSY, Synchronization Busy Bus
0x02	Generic Clock Control	CLKCNTRL	WRTLOCK, CLKEN, GEN and ID
0x04	Generic Clock Generator Control	GENCNTRL	RUNSTDBY, DIVSEL, OE, OOV, IDC, GENEN, SRC, ID
0x08	Generic Clock Generator Division	GENDIV	0 is low, 1 is high

The key bits for inspecting that 1MHz clock:

GENCNTRL
Bit 20 – DIVSEL Bits 12:8 - SRC
Bits 3:0 - ID

GENDIV
Bits 23:8 – DIV Bits 3:0 - ID

Once I start loading code on the chip, I will use GDB to inspect the proper location.

Options for Blinking in Assembly

Assembly has different strategies than C. Try this fake-led blinking demo example first if you didn’t already.

I have two strategies I want to show. Both of these behave as if we only had the one address to use, the Data Output Value. SLOOOOW. The code on chip will be faster.

Explicit shifting and setting

Example Link

.equ led_reg_addr, 0xff200000  // some archs have separate read and write addr

.equ onValue, 0x1     //will matter later because in real circuit 0 is frequently on
.equ offValue, 0x0
.equ pinLocation, 3

.equ delayTime, 3  // low number because I only step through on emulator

.global _start
_start:
	
	loop:
		MOV R3, #delayTime
		MOV R1, #onValue
		LSL R1, #pinLocation  //create the value
		LDR R0, =led_reg_addr
		STR R1, [R0]
		BL delay

		MOV R3, #delayTime
		MOV R1, #offValue
		LSL R1, #pinLocation
		LDR R0, =LED
		STR R1, [R0]
		BL delay
	
	BAL loop
	
delay:
	SUBS R3, R3, #1
	BNE delay
	BX LR

XOR Toggle

Example Link
discussion

.equ led_add_reg, 0xff200000 //some archs have a 
							 //different read and write 
							 //address
.equ maskValue, 8
.equ delayTime, 3

.global _start
_start:

	ldr R5, =led_add_reg
	mov R6, #maskValue
	
	loop:
		LDR R4, [R5]  //if no one else will change the port or memory register (R4)
                      //and have no other reason to check port, potentially 
                      //could skip the read and go with a desired initial value
                      //set before the loop and not do a full reload here. (just xor)
		EORS R4, R6 //XOR
		STR R4, [R5]
		LDR R3, =delayTime //instead of mov for future value
		BL delay
		B loop

delay:
	SUBS R3, R3, #1
	BNE delay
	BX LR

Now try it for real!

Finally! Whew.

Only one file has changed. The whole project reduces to an 11kb elf file.

.syntax unified
.cpu cortex-m0
.fpu softvfp 
.thumb  

.section .text.program_code

.equ portA_address, 0x41004400
.equ portA_DIRSET, 0x41004400+0x08
.equ portA_OUTTGL, 0x41004400+0x1C

.equ delayTime, 50000     //same clock speed as avr, so same value
.equ pinOffset, 10

.thumb_func
.global _start
_start:

	MOVS R4, #1
	LSLS R4, R4, #pinOffset  //R4 now contains the 1 at the pin offset bit. 
                           //It should stay that way for the rest of the program

  //set the output pins
  LDR R5, =portA_DIRSET
  STR R4, [R5]

 
  LDR R5, =portA_OUTTGL  //R5 now points to the toggle-address
                          //It should stay that way for the rest of the program

  loop:
    STR R4, [R5]
    LDR R3, =delayTime
    BL delay

  B loop

.thumb_func
delay:
	SUBS R3, R3, #1
	BNE delay
	BX LR

Summary

Now was that so hard?

(how can I get an aside?)

Fri, 12 Apr 2024 00:00:00 +0000

I’ve been avoiding using hugo shortcodes as much as possible because I’m not committed to always using hugo.

But since if I switch I will already have things to find and swap…
And it’s easier than learning asciidoc right now…

References:

hugo doc pages:
- https://gohugo.io/templates/shortcode-templates/
- https://gohugo.io/content-management/shortcodes/
github search: path:shortcodes path:aside

{{- $type := default "note" (.Get "type") -}}
<aside class="DocsMarkdown--aside" role="note" data-type="{{ $type }}">
  {{- with .Get "header" }}
    <div class="DocsMarkdown--aside-header">{{- safeHTML . -}}</div>
  {{- end -}}

  {{- .Inner | .Page.RenderString (dict "display" "block") -}}
</aside>

via CloudFlare

{{/*
Asides show up as a colored box. Most things (headings, figures, iframes etc.)
can be placed inside a box and display well, but make sure to check the final result
especially on mobile.
class = notice/success/warning
will give blue/green/red box
*/}}
{{ $_hugo_config := `{ "version": 1 }` }}
{{- $class := .Get 0 | default "notice" -}}
{{- $markdown := .Inner | markdownify -}}
<aside class="{{ $class }}">
  {{ if not ( findRE "<[h|p][^>]*>" $markdown ) }}
    <p>{{ $markdown }}</p>
  {{ else }}
    {{ $markdown }}
  {{ end }}
</aside>

via discoverthree-js

But what I really want to start is just something like:

<aside class="irt-aside">
  {{ .Inner | markdownify }}
</aside>

via Ottowa civic tech (their css)

Test of {{% aside %}} some multiline content with mark down {{% /aside %}}:

It was!

But I have no styling for an aside tag.

There are fancier ways to

conditionally add css depending on shortcodes,
use the config file to make a custom list of css to include and update the head.html template

Or maybe your theme, like the one I use (etch), already gathers resources from the assets folder and you can replace the head.html partial

Relevant section of original file:

    {{ $resources := slice -}}
    {{ $resources = $resources | append (resources.Get "css/main.css") -}}
    {{ $resources = $resources | append (resources.Get "css/min770px.css") -}}
    {{ $dark := .Site.Params.dark | default "auto" -}}
    {{ if not (eq $dark "off") -}}
        {{ $resources = $resources | append (resources.Get "css/dark.css" | resources.ExecuteAsTemplate "dark.css" .) -}}
    {{ end -}}

    {{ if .Site.Params.highlight -}}
        {{ $resources = $resources | append (resources.Get "css/syntax.css") -}}
    {{ end -}}

    {{ $resources = $resources | append (resources.Get "css/aside.css") -}}
    {{ $css := $resources | resources.Concat "css/style.css" | minify }}
    {{ printf `<link rel="stylesheet" href="%s">` $css.RelPermalink | safeHTML }}

So I moved a copy of the head.html to my local partials folder and right above the call to concat and minify added:

    {{ $resources = $resources | append (resources.Get "css/aside.css") -}}

If you’ve already customized your main css, consider just throwing the new aside spec in there. Fewer changed files to track after an update. I don’t think the theme I chose is getting updates anymore, so I’m not worried about that.

I don’t really have time to swap in CSS brain so for now I mostly cribbed off of this theme’s code fence CSS. Unlike the example I did not add a class to my shortcode partial.

main#content aside {
    /* VSCode auto completed this, sure! */
    font-family: system-ui, -apple-system, 
    BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, 
    Cantarell, 'Open Sans', 'Helvetica Neue', sans-serif;
}

main#content aside {
    font-size: 0.96em;
    padding: 0 5px;
}

main#content aside {
    display: block;
    overflow-x: auto;
    /*font-size: 14px;
    font-size: 1.4rem;*/
    margin: 0px 20px;
    padding: 1.5rem 1.5rem;
    /*line-height: 1.4;*/
    border: 1px solid #b1b1b1;
    border-radius: 1px;
}

Not quite ready for production, but progress.

(how did my hugo public folder get to be a mess?)

Fri, 12 Apr 2024 00:00:00 +0000

I’m thinking changing folder names while hugo serve -D is running isn’t such a good idea? I somehow ended up with a public folder that was a mess. With multiple folders that clearly were iterations of names I was work-shopping. If I had thought that through, oh course it would do that.

And since I never looked, my rsync script was uploading huge amounts that it would never delete.

I did two things to fix it

1 - Cleaned the public folder

hugo --cleanDestinationDir

2 - Updated my upload script

The original line excluded .DS_Store but nothing else. Never deleted because what if I wanted to put non-hugo items up there?.

rsync -zaP --exclude='.DS_Store' $SOURCE $DESTINATION

The new one, shown in context below, deletes everything that isn’t in the local public folder, but has an exceptions file. If the destination is space restrained, the slower exclude-before might be a better fit.

SOURCE="$HUGO_ROOT/public/"
DESTINATION="$USER@$HOST:/home/$USER/$SITENAME"

#echo "#rsync -zaP $SOURCE $DESTINATION"
# z: compress
# a: archive mode (preserve permissions)
# P: show progress, keep partial (I watch every time)
rsync -zaP --delete --exclude='.DS_Store' --exclude-from='exclude-rsync.txt' $SOURCE $DESTINATION

example exceptions file:

the difference between dir/* and dir, is that the first refers to the contents and the later the directory itself as well.

some_special_file.txt
pre* 
dir1/*
dir

Might have things like

favi*
robots.txt
.well-known

Note: some things you might manually add can be done via hugo instead of this file

via the static folder - i.e. favicon
or configuration - i.e. robots.txt (poor robots.txt)
404

Running the script

I run this script by hand from variables in a .env file, but NOT THE SERVER PASSWORD. If remembering the password every time isn’t an option, use a ssh key.

How can I get this SAMD21E18 startup code a little sturdier?

Thu, 11 Apr 2024 00:00:00 +0000

At the end of the last post I mentioned that I wasn’t going to comment on the code or the linker script, but that I didn’t recommend editing them yet. The code from the last post worked because it had workable hooks where they needed to be for what it was trying to do. Many minimal examples work because they do nothing. Adding more code in the wrong place can break things with no error messages to help.

When programming for a machine with an operating system, generally folks can start with the apps entry point, “main”, and go from there. On a microcontroller the software has to be both application and “operating system”. Thankfully IDEs and frameworks hide a lot of that so folks can get right to work on their idea and only dig deeper if they have a special concern.

When one does need to go under the hood, the process of setting up a new chip has similarities to learning how to implement a new Protocol. There’s a contract to deliver on and you have to learn what it is.

The first step involves providing a start up script that at least:

Defines and places a critical list of function pointers called a “vector table”. (On Arm® Cortex®-M0+ Processor Core MCU chips the official name of the interrupt handler is the Nested Vectored Interrupt Controller.)
Initializes two memory areas (.data and .bss)
Starts the program!

This post is less tutorial and more record of what I did, what resources I needed and what resources I think others might find helpful to fill in the gaps

References

Tutorials and repos here. Topic specific links in their sections.

Official

Writing start up scripts and linker files correctly requires intimate knowledge of the specific microcontroller’s physical layout. Smart people focused on concrete goals use manufacture provided versions of these files because the manufacturer in theory knows their chip best (and any errata). To find the all the information needed can take a little digging. These are all the sources I looked at. I will state which one contributed to what as I go.

Most Important Link For Vector Table. Do. Not. Skip.
- https://developerhelp.microchip.com/xwiki/bin/view/products/mcu-mpu/32bit-mcu/sam/samd21-mcu-overview/samd21-processor-overview/samd21-nvic-overview/
Product Reference Page
- https://www.microchip.com/en-us/product/ATSAMD21E18
- Reachable from here:
  - arm-gcc compiler tools, (come with Atmel Studio if you have that) https://www.microchip.com/en-us/tools-resources/develop/microchip-studio/gcc-compilers
  - http://packs.download.atmel.com/ (rename .atpack files to .zip) (packs on github)
  - https://www.keil.arm.com/devices/ for CMSIS? Link provided by Microchip. hunh.
Full Family Data Sheet
- https://ww1.microchip.com/downloads/aemDocuments/documents/MCU32/ProductDocuments/DataSheets/SAM-D21-DA1-Family-Data-Sheet-DS40001882H.pdf
Cortex-M0+ Technical Reference Manual (Revision: r0p1)
- https://developer.arm.com/documentation/ddi0484/c/?lang=en
ARMv6-M Architecture Reference Manual
- PDF https://documentation-service.arm.com/static/5f8ff05ef86e16515cdbf826
- Online Viewer https://developer.arm.com/documentation/ddi0419/latest/
Demo project repo, SAMD21N Getting Started. Needs MPLAB software to run, uses C, but I wanted to look at the list of hardware defs and linker scripts to double check my work.
- https://github.com/Microchip-MPLAB-Harmony/reference_apps/tree/143e5438a43c50660b893980bcb41b930dcf9821/apps/sam_d21_cnano/samd21n_getting_started/firmware/src/packs
https://github.com/Microchip-MPLAB-Harmony/dev_packs/tree/345dc12d42a4fdec72117b64a8c3527023bdeca9/Microchip/SAMD21_DFP/3.6.144/samd21a/armcc/armcc

More Instructional

Carry Over: STM32 Based Fastbit Embedded Brain Academy“Bare metal embedded” playlist. Especially videos 2 & 3
Continuation: “Bare Metal” STM32 Programming (Part 2)
- https://vivonomicon.com/2018/04/20/bare-metal-stm32-programming-part-2-making-it-to-main/
If you’re using a Pico (RP2040), stop reading this and go
- read https://blog.smittytone.net/2022/06/19/get-started-with-arm-assembly-on-the-pi-pico/
  - https://github.com/smittytone/pico-asm/blob/main/source/asm_led_sdk/main.S
- watch (Life With David) https://www.youtube.com/playlist?list=PLiRALtgGsxmYTF6ZqDSg4DRWDLGVer-Kb
  - https://github.com/LifeWithDavid/RaspberryPiPico-BareMetalAdventures/tree/main
https://forum.digikey.com/t/getting-started-with-the-sam-d21-xplained-pro-without-asf/13176
https://jacobmossberg.se/posts/2017/01/10/minimalistic-assembler-program-on-stm32-h103.html
- armasm source: https://pygmy.utoh.org/riscy/cortex/led-stm32.html

How to search github

When looking for start up scripts to verify my work wasn’t totally off base, github searches of:

SAMD21E18 path:*.s
path:*SAMD21* path:*.s
path:*SAMD21* path:*.s path:*startup*
#ifndef _SAMD21E18A_

Turned up some of the most useful results.

To go a little more scatter shot, the following phrases (plus your Microchip part number) are an indicator that file started from the official source. I’m sure you can come up with more “Statistically Improbable Phrases” from the demo projects, etc above. - “\brief Header file for” - “Microchip Technology Inc. and its subsidiaries” - “Copyright (wild card) Atmel Corporation” - “\brief This module contains”

How To Compare Assembly Start Up Files

This will not be a course in assembly, although I’ve tried to provide alternate examples and some explanations.

I mentioned searching github, but assembly start up scripts can look wildly different even for the same chip:

for tool chain reasons
for syntax preference reasons
for what parts of the chip’s functionality is actually needed reasons
for compatibility with other code reasons (will there be C?, is this just for one project or the basis of a library?)
for stylistic decisions about how to ensure code blocks end up where they’re supposed to be reasons (physical placement, naming conventions, manually setting the counter value)
for stylistic decisions about how much gets handled in the code files vs the linker file reasons (where do you see “align”)

It can be reaaaaly hard to tell what’s a meaningful difference between two files and what is not. Hopefully the links below will help when my comments aren’t good enough.

https://en.wikipedia.org/wiki/X86_assembly_language#Syntax
Nice syntax explanation in a relevant example of a start up (Not M0 though)
- https://community.arm.com/arm-community-blogs/b/architectures-and-processors-blog/posts/writing-your-own-startup-code-for-cortex-m
- same code? https://github.com/koson/blinky/blob/6029c2ca627b582f7f7b17fc89c888f67f15f7b4/startup/startup_LPC1768.S
https://www.chibialiens.com/arm/thumb.php
http://www.coranac.com/tonc/text/asm.htm#sec-thumb
Intro to Assembly for Arm programming for playlist from last post.
- https://www.youtube.com/watch?v=in-UY_EyI14&list=PL2EF13wm-hWAlQe87UB2HV0SVhBXFpXbn
- https://cpulator.01xz.net
MIT Open Courseware 4. Assembly Language & Computer Architecture from MIT 6.172 Performance Engineering of Software Systems, Fall 2018

A - A mildly better version

Finding what I needed to make a vector table

The “vector table” lists all the addresses of the critical low level functions the chip will need to know what to do when “something happens”. They must live at exactly the location defined by the architecture and the manufacturer. They are incredibly device dependent, although the nice thing about Arm chips is that they share some values across cores and architectures.

Some of the types of things that get answered by the vector table are:

what’s the first thing I should do?
what do I do when I fail?
what do I do with this messages?
its been 27 ticks - I check where after 27 ticks?

Misc Overviews

Program, Interrupted - Computerphile https://www.youtube.com/watch?v=54BrU82ANww (6:40)
https://en.wikipedia.org/wiki/Vectored_interrupt
18.2.6 Strong Priority Systems https://www.youtube.com/watch?v=PmOq8G_hs4o from Computation Structures Uses keyboards and printers, but micros have other “peripherals”

How does that all work on the SAMD21 specifically

READ THIS!!!!! —-> https://developerhelp.microchip.com/xwiki/bin/view/products/mcu-mpu/32bit-mcu/sam/samd21-mcu-overview/samd21-processor-overview/samd21-nvic-overview/ <———

A vector table for a SAMD21 has to deliver on the Arm® Cortex®-M0+ Processor Core MCU requirements as well as support its own peripherals. The above link is by far the best information I found for this chip.

Images below from that page:

But this just tells you about the Cortex-M0+ values. The chip specific information can be tracked down in the Datasheet. Unfortunately I couldn’t the find a comprehensive list in one easy place like for the Atmega328 (not Arm, different beast, table 11-1 in 328 datasheet) or the STM32F0 (Arm Coretex-M0 (not M0p) Section 11, Table 36 in the reference manual). If it wasn’t for this one page I’m not sure how one would have pulled together the full list except by using sample code.

Product Mapping

Datasheet 9 Product Mapping Figure 9-1. SAM D21 Product Mapping

Chip Specific Hardware List

The 0 next to PM - Power Manager lines up with the the 16 IRQ 0 in the very first image, and the 0 PM_Manager (addresses 0x00000040)

Datasheet 11.2 Nested Vector Interrupt Controller 11.2.2 Interrupt Line Mapping

!!!WARNING!!! FAMILY Datasheet. Cross reference the below with section 2 Configuration Summary (not shown) for which ones your chip specifically implements.

Updating Hello Code With Minimal Vector Table & Handlers

Between your start up function and the linker file the idea is to defend the expected amount of space at the expected location and fill it with either useful or guaranteed harmless information.

My first example will only specify Cortex-M0+ needed values. Since embedded chips are small it’s not uncommon to not leave space for the features of the chip you know your code (AND CIRCUIT!) won’t use.

.syntax unified
.cpu cortex-m0
.fpu softvfp
.thumb 

@ vector_table will be at the top of text, because it is at the top of text
@ and we have no other files. 
.text
vector_table:
    .word   sp_initial_value //edge of stack, set in linker
    .word   reset_handler   //reset handler
    .word   nmi_handler
    .word   hard_fault_handler
    .word   0//not in M0+
    .word   0//not in M0+
    .word   0//not in M0+
    .word   0//not in M0+
    .word   0//not in M0+
    .word   0//not in M0+
    .word   0//not in M0+
    .word   svc_handler
    .word   0//not in M0+
    .word   0//not in M0+
    .word   pendsv_handler
    .word   systick_handler

  .weakref nmi_handler, default_handler
  .weakref hard_fault_handler, default_handler
  .weakref svc_handler, default_handler
  .weakref pendsv_handler, default_handler
  .weakref systick_handler, default_handler


.word default_handler
.thumb_func
default_handler:
   ldr r3, =#0x8BADF00D
   @ hangs the program.
   b .


.global reset_handler
.thumb_func
reset_handler:
  LDR  r0, =sp_initial_value
  MOV  sp, r0

  LDR  r7, =0xF0CACC1A
  MOVS r0, #0
  main_loop:
    @ Add 1 to register 'r0'.
    ADDS r0, r0, #1
    @ Loop back.
    B    main_loop

I’m using a different syntax than the the vivonomicon version. This version of the code WILL NOT WORK for non-thumb mode arm processors. I’ll revert to that more core flexible syntax, using the gcc assembler directive .type, in code where I care about portability.

Each interrupt gets a weakref (search on this gnu docs page) to a default handler function. This lets a future user of this code override their values in another file. I could have just put default_handler everywhere I put a unique function name in the vector table. You might see this weak ref creation in other files as:

A two step process

    .weak       SysTick_Handler
    .thumb_set  SysTick_Handler, Default_Handler

A different flavor of assembly (armasm, not the now recommended by Arm gccasm) in some of the official docs

SysTick_Handler           PROC
                          EXPORT SysTick_Handler           [WEAK] 
                          B       .
                          ENDP

Handled by a macro as suggested in Arm provided code (set vs thumb_set depending on the instruction set)

.macro IRQ handler
    .weak  \handler
    .set   \handler, Default_Handler
.endm

IRQ SysTick_Handler

Or even good old

void SysTick_Handler(void) __attribute__ ((weak, alias ("Default_Handler")));

These all serve the same purpose.

Another thing you might see in thumb mode vector tables is + 1 to the function address or some .align N where N is odd before a function declaration later in the code or something somewhere that shifts the handler addresses so the least significant bit of the address will be 1. I put some links in the assembly link section above and yesterday that cover how the assembler can tell if it should be using 16 or 32 bit instructions by the lsb. It comes up a lot.

Thankfully .thumb_func should be taking care of that for me. The =mthumb in the gcc calls should do that too when using the .type directive. (.type name, %function), but I haven’t confirmed that yet.

Mildly Improving the Linker

This program relies on the vector table being defined in the one source file immediately after the .textcompiler directive is invoked. I’ll make a more resilient set up next. The compiler already has opinions about what to do with .text, so you could leave it out in other circumstances.

How does this differ from the previous file?

changes the entry point officially ENTRY
uses the SECTIONS command with some of the linker software special-treatment section names (text, data, bss) [todo links]
sweeps in those sections from ALL files.
places those sections with > AT based on the defined memory locations

ENTRY(reset_handler);
MEMORY {
    flash(rx): ORIGIN = 0x00000000, LENGTH = 0x00040000 /*256K*/
    sram(rwx): ORIGIN = 0x20000000, LENGTH = 0x00008000 /*32K*/
}
sp_initial_value = ORIGIN(sram) + LENGTH(sram);  /* 0x20008000 */

SECTIONS {
    .text   : { *(.text)  } > flash
    .data : { *(.data) } > sram AT> flash
    .bss    : { *(.bss)    } > sram
}

B - Separating the Startup from The Program Code

This code works (use the makefile), but it’s still fragile. What the chip needs and our wants are all mushed together and everything relies on putting things in the right place in the file. And we haven’t even done some of the important startup basics.

1 - Pull out the Main Loop, Still one File

First I changed hello_improved to hello_main, pulling out the loop in the same file via branching. I did not following all the proper calling conventions.

hello_main also has new section markers, which were tested by adding them in the linker an running the shiny new disassemble command in the makefile.

This is all in the repo here

2- Doing more for the start up

repo folder

When looking for the right assembly syntax for the reset handler I was influenced by the following scripts:

TODO: Benchmark them. I’m not doing anything fancy enough with this code to really make a difference.

I haven’t commented every line. Some assistance (also last post’s links…):

thumb instruction quick reference card by arm
Fancier branching:
- https://azeria-labs.com/arm-conditional-execution-and-branching-part-6/
Multi register handling:
- https://developer.arm.com/documentation/dui0170/b/thumb-instruction-reference/thumb-memory-access-instructions/ldmia-and-stmia?lang=en
- https://azeria-labs.com/load-and-store-multiple-part-5/
What’s the deal with _sidata
- since data is relocatable (unlike text), we need the “real” address once available.
- https://en.wikipedia.org/wiki/Data_segment
- LOADADDR in the docs: “Return the absolute load address of the named section. This is normally the same as ADDR, but it may be different if the AT keyword is used in the section definition”
NOLOAD starts showing up next to bss in fancier linkers - discussion
more about linkers:
working with variable in gcc assembly (to test this worked)
- https://azeria-labs.com/memory-instructions-load-and-store-part-4/
ALIGN (right under LOADADDR) and why memory alignment is important.

The new reset handler code

.section .text.reset_handler
@ The Reset Handler 
.global reset_handler
.thumb_func
reset_handler:

  /* reset handler started */
  /* r7 will hold my notes to self  */
  ldr  r7, =0x0000AAAA
 
  /* set stack pointer */
  ldr  r1, =_end_stack
  mov  sp, r1

  /* set  .bss section to 0 (SRAM) */
  /* uses the jump to exit style loop */
  zero_bss:
    /* get 0 handy in two registers */
    movs r0, #0
    movs r1, #0
    /* get the bounds */
    ldr  r2, = _start_bss
    ldr  r3, = _end_bss
    loop_start_bss_zero:
      /* compare, first one is that there is a bss.  */
      cmp r2, r3
      /* compare: unsigned higher or same (is end higher than current )  */
      bhs loop_end_bss_zero
      /* will load 2 and increase the pointer after */
      @ stmia r2!, {r0,r1}
      stmia r2!, {r0} /* decided against moving bss to after data in linker  */
      b loop_start_bss_zero
    loop_end_bss_zero:

  /* copy .data section to SRAM */
  /* uses the jump to entry style loop */
  move_data_to_ram:
    /* get the bounds */
    ldr r0, = _start_data   
    ldr r1, = _end_data
    /* read head location */
    ldr r2, = _sidata
    b loop_enter_data_copy

    loop_action_data_copy:
      ldmia r2!, {r3}
      stmia r0!, {r3}
    loop_enter_data_copy:
      cmp r0, r1
      bcc loop_action_data_copy

  /* call the clock system initialization function.*/
  /* not done */

  /* call init of libraries (stdlib when loop in C) that need it.*/
  /* not done */

  /* note to self handler finished */
  LDR  r7, =0x0000BBBB

  /* Call the application's entry point.*/
  bl my_main

And the top of the file with lots of variables to test

.bss 
/* note, bss is marked noload in the linker */
/* no matter how long, shouldn't effect size of elf */
/* assembler wont let you store values that aren't 0 */
make: .word 0
something: .word 0
be: .word 0
here: .word 0

.data
/* like define, but tricksy. */
.equ pelargonium, 0x600dCA7
.equ rosebud, 0xF0CACC1A

/* variables with values */
ennie:  .word  124
meenie: .word  125
mineie: .word  126
moe:    .word  127

@ this is NOT how you create unallocated memory
@ these all point to the same place in memory. 
breakfast:  .word
lunch:      .word
dinner:     .word
midnight_snack: .word 27 

adr_ennie: .word ennie
adr_meenie: .word meenie
adr_mineie: .word mineie 
adr_moe: .word moe
adr_breakfast: .word breakfast
adr_lunch: .word lunch
adr_dinner: .word dinner
adr_midnight_snack: .word midnight_snack

adr_make: .word make

increment: .word 1

.section .text.program_code

.word my_main
.thumb_func
my_main:
  /* note to self main started finished */
  LDR  r7, =0x0000CCCC

  @ values of the unallocated should match
  @ ennie-moe after done with this. 
  LDR r0, =ennie
  ldmia r0!, {r1-r4}
  LDR r0, =make
  stmia r0!, {r1-r4}
  
  /* start with 0 */
  movs r0, #0
  loop_start:
    // Add 1 to register 'r0'.
    ADDS r0, r0, #1
  b loop_start

The Matching Linker

ENTRY(reset_handler);
MEMORY {
    flash(rx): ORIGIN = 0x00000000, LENGTH = 0x00040000 /*256K*/
    sram(rwx): ORIGIN = 0x20000000, LENGTH = 0x00008000 /*32K*/
}

_end_stack = ORIGIN(sram) + LENGTH(sram);  /* 0x20008000 */

SECTIONS {
    .text : { 
        hello_startup.o(.vectors)  
        KEEP(*(.vectors .vectors.*))
        *(.text.reset_handler)
        *(.text.default_handler)
        /* the rest of the .text and .text sub sections */
        *(.text .text.*)
        /* all files read only data and read only data sub sections */
        *(.rodata, .rodata*) /* read only data */
       
    } > flash /* nothing relocatable */
    . = ALIGN(4);
    _end_text = .; /* store current location counter value in _end_text */

    /* Used by the startup to initialize data with the "originating" address */
    /* not so relevant on the M0+, more significant diff on beefier cores */
    _sidata = LOADADDR(.data);

    .data : { 
        . = ALIGN(4);
        _start_data = .; /* store current location in _end_data t*/
        *(.data) 
        . = ALIGN(4);
        _end_data = .; /* store current location in _end_data t*/
    } > sram AT> flash

    .bss(NOLOAD) : { 
        . = ALIGN(4);
        _start_bss = .;
        *(.bss) 
         *(COMMON) 
     } > sram
     /* todo, inside or outside */
     . = ALIGN(4);
     _end_bss = .;
}

C - Splitting The Code Into Separate Files

Honestly. It just worked.

split the files
change the name of hard include in the linker
updated the Makefile to pull in two files (vey unpolished. I’ll fix it later.)

Direct link

The really change is finally adding in all the peripherals! (see Table 2-1 (Configuration Summary). Some of those are 0s b/c not on the E version of the chip)

.section  .vectors
.global vector_table
vector_table:
    //-------------- arm core list
    .word   _end_stack //edge of stack, set in linker
    .word   reset_handler //reset handler
    .word   nmi_handler
    .word   hard_fault_handler
    .word   0//not in M0+
    .word   0//not in M0+
    .word   0//not in M0+
    .word   0//not in M0+
    .word   0//not in M0+
    .word   0//not in M0+
    .word   0//not in M0+
    .word   svc_handler
    .word   0//not in M0+
    .word   0//not in M0+
    .word   pendsv_handler
    .word   systick_handler
    //-------------- peripherals list
    .word   PM_Handler
    .word   SYSCTRL_Handler
    .word   WDT_Handler
    .word   RTC_Handler
    .word   EIC_Handler
    .word   NVMCTRL_Handler
    .word   DMAC_Handler
    .word   USB_Handler
    .word   EVSYS_Handler
    .word   SERCOM0_Handler
    .word   SERCOM1_Handler
    .word   SERCOM2_Handler
    .word   SERCOM3_Handler
    .word   0
    .word   0
    .word   TCC0_Handler
    .word   TCC1_Handler
    .word   TCC2_Handler
    .word   TC3_Handler
    .word   TC4_Handler
    .word   TC5_Handler
    .word   0
    .word   0
    .word   ADC_Handler
    .word   AC_Handler
    .word   DAC_Handler
    .word   PTC_Handler
    .word   I2S_Handler
    .word   0
    .word   0

.text

.section .text.default_handler
.word default_handler
.thumb_func
default_handler:
   ldr r3, =#0x8BADF00D
   //hangs the program.
   b .

.weakref nmi_handler, default_handler
.weakref hard_fault_handler, default_handler
.weakref svc_handler, default_handler
.weakref pendsv_handler, default_handler
.weakref systick_handler, default_handler

.weakref PM_Handler, default_handler
.weakref SYSCTRL_Handler, default_handler
.weakref WDT_Handler, default_handler
.weakref RTC_Handler, default_handler
.weakref EIC_Handler, default_handler
.weakref NVMCTRL_Handler, default_handler
.weakref DMAC_Handler, default_handler
.weakref USB_Handler, default_handler
.weakref EVSYS_Handler, default_handler
.weakref SERCOM0_Handler, default_handler
.weakref SERCOM1_Handler, default_handler
.weakref SERCOM2_Handler, default_handler
.weakref SERCOM3_Handler, default_handler
//.weakref SERCOM4_Handler, default_handler
//.weakref SERCOM5_Handler, default_handler
.weakref TCC0_Handler, default_handler
.weakref TCC1_Handler, default_handler
.weakref TCC2_Handler, default_handler
.weakref TC3_Handler, default_handler
.weakref TC4_Handler, default_handler
.weakref TC5_Handler, default_handler
//.weakref TC6_Handler, default_handler
//.weakref TC7_Handler, default_handler
.weakref ADC_Handler, default_handler
.weakref AC_Handler, default_handler
.weakref DAC_Handler, default_handler
.weakref PTC_Handler, default_handler
.weakref I2S_Handler, default_handler
.weakref AC1_Handler, default_handler
//.weakref TCC3_Handler, default_handler

Summary

I already have a WIP with all the peripherals commented, a better default handler, etc. But I think this post is quite long enough!

Hopefully, fingers crossed, next time, blinking?

How can I make programming an ARM chip as hard as possible?

Sat, 06 Apr 2024 00:00:00 +0000

Historically chip companies have designed and manufactured their own chips. Switching to a different manufacturer meant changing everything, software to board.

Arm changed that by licensing their designs and instruction sets to more than one manufacturer.

44 min video overview of the architecture and ecosystem put out by Arm
13 min Interview with Prof. Furber by Computerphile

With so many to choose from, I’m going to go with the Arm chip that reminds me the most of the ATtiny - Microchip’s (Atmel) SAMD21 family. (Also - We have A LOT of Trinket M0’s in the house)

Product Reference Page (has application notes)
Full Data Sheet
- Especially Section 10: Memories pg 41
Summary Data Sheet
- Especially Figure 9-1. SAM D21 Product Mapping
About Core M0 chips (ARMv6-M)
- https://www.arm.com/products/silicon-ip-cpu/cortex-m/cortex-m0-plus
- https://en.wikipedia.org/wiki/ARM_Cortex-M
Peripheral Comparison Chart for Microchip 32-Bit chips

What’s the First Step?

Embedded Swift isn’t an Arduino replacement. It’s gunning to be pretty low level. Documenting how to load an assembly program on to the chip with the GNU tool chain provides a solid reference base for then showing what it takes to switch over to the C from the assembly. Adding Embedded Swift to the mix will hopefully feel like just another think to add to a Makefile…errr… Cmake…file.

I don’t love it, but “bare metal” has become a trendy term in programming these days. Like calling a product “green”, there’s no precise definition as to what it means. While environments without an operating system may have more of claim than dockerless servers, even within that context people differ in how they use it. In general it means the author will be trading expedience for control, so it can be a useful search term. That said, when known, use:

actual part numbers
software package names
chip family names
language names
code snippets

Using these I still got results with “bare metal” in the title! Ha! My top instructional resources that contributed to this page and will help it make more sense:

This post essentially implements the following tutorial using a different chip (SAMD21E18 vs STM32-F031K6) and a different programming toolchain (OpenOCD vs STLink).
- article: https://vivonomicon.com/2018/04/02/bare-metal-stm32-programming-part-1-hello-arm/
- repo: https://github.com/WRansohoff/STM32F0_minimal
STM32 Based Fastbit Embedded Brain Academy “Bare metal embedded” playlist. No-library C based. Incredibly thorough review of how the software interacts with the hardware. A++. (Windows, but since GNU command line tools doesn’t matter.)
- playlist: https://www.youtube.com/watch?v=qWqlkCLmZoE&list=PLERTijJOmYrDiiWd10iRHY0VRHdJwUH4g
- repo: https://github.com/niekiran/baremetalembedded
I stole his programmer, he told me about OpenOCD, recently updated: https://github.com/todbot/samd21-programming-notes
The product I’m using, but the Metro M4 would be a better choice if shopping. Shows the CircuitPython path: https://learn.adafruit.com/adafruit-pyruler/downloads
Has a a good chip overview and overall a much more reasonable Arduino based start up process with a better board than my example: https://learn.sparkfun.com/tutorials/samd21-minidev-breakout-hookup-guide/all#samd21-overview
Epic linker script with a ton of further reading, other SAMD21 information on the blog
- https://blog.thea.codes/the-most-thoroughly-commented-linker-script/
- https://github.com/wntrblm/Castor_and_Pollux/blob/main/firmware/scripts/samd21g18a.ld
Using OpenOCD: https://omzlo.com/articles/programming-the-samd21-using-atmel-ice-with-openocd-(updated)
Introduction to Assembly Programming with Arm - Nice overview but NOT .thumb mode needed for Coretex M0+. Uses an emulator for the first part but switches to a VM for the last part. Handy.
- https://www.youtube.com/watch?v=in-UY_EyI14&list=PL2EF13wm-hWAlQe87UB2HV0SVhBXFpXbn
- https://cpulator.01xz.net

TODO: I’ve got a giant notes page with even more links that needs to be organized.

Which Programmer?

I love the UF2 (USB Flashing Format) programming method which allows many boards to be programmed via USB, no programmer needed.

2017 makecode blog post
USB Flashing Format (UF2) explanation from microsoft.github.io

This post will not use that feature.

I would like the first couple of examples to be as close as possible to the AVR process which means programming via a programmer, not USB.

The role of AVRDude will be played by OpenOCD software. OCD stands for “On Chip Debugger” so it’s not exactly a 1 to 1 swap. Also install the gnu gcc arm toolchain because it has the version of GDB that we need.

To install on MacOS

## this is the cross compiler version
## installs from https://developer.arm.com/Tools%20and%20Software/GNU%20Toolchain
## see https://formulae.brew.sh/cask/gcc-arm-embedded#default
## the below command worked for me in 4/2024
## if it doesn't work for you try "brew install --cask gcc-arm-embedded"
brew install gcc-arm-embedded  
arm-none-eabi-gcc --help

brew install openocd
## confirm install see options
openocd --help

## Optional
## if want to use openocd in telnet mode
brew install telnet

There are MANY programmers that will work with OpenOCD. Anything implementing JTAG (Joint Test Action Group), really.

Since the SAMD21D has so few pins, the Trinket exposes the SWD (Serial Wire Debug) interface instead of JTAG. The following programmers can work with it (~INT = ball park price in US dollars). The first -three- four I have seen in action personally.

Segger JLink EDU edition (this post, ~150, pro is ~1k comparison guide)
Atmel ICE (~200, CMSIS-DAP)
ST-Link (V2 around ~23, V2-ISOL ~90)
New! todbot just got a DapLink compatible programmer from Amazon and confirmed it worked. (specifically this one, requires soldering ~8) (affiliate links heads up. I made him.) (PLANS)
Segger JLink EDU mini (~60)
Raspberry pi debug probe (~12)
Black Magic Probe (~75)
SAMD11C SWD Programmer Stick PLANS ONLY! (repo)
ESP32 Wifi Debugger PLANS ONLY (repo)
MIT Machines That Make open source design PLANS ONLY

Prep the Board & Programmer

I’ve appropriated a PyRuler I’ve had on the shelf for awhile. It has a lot going on around the Trinket. They are super fun. If I was going to actually buy something for this task, I would buy something with a SWD or JTAG connection that was ready to go. When shopping you might see boards with a place for the tag-connect, but you’ll need a special cable for your debugger.

Trinkets expose the pads, but you have to solder to them or design a jig. (Looking forward to putting the UF2 bootloader back on…)

Reset pin notes: The PyRuler does not overload the SWD pads (PA31, PA31) so setting up the reset pin may not be needed or useful for your set up.

Programmer	Trinket	Wire Color
1 VTref	3V	brown
7 SWDIO	pad further from usb edge	blue
9 SWCLCK	pad closer to usb edge	yellow
15 RESET	RESET	green
19 5V-Supply*	BAT	red
18 GND	GND	black

The 5V-Supply/red wire in theory could be used to power the board. But I haven’t sorted the configuration yet, so I’m using USB for power. It’s attached in the pictures so it isn’t flapping in the breeze.

The official pin out is from the point of view of looking down at a board’s connector, not AT the programmer’s connector. I’ve flipped this to make it look more like the picture.

Talk to the Board

Config File and Start Up

OpenOCD requires .cfg files for programmers, boards and chips. Since we have a single chip board we only need two config files, but we write one config file that can find them both for us.

The below is the minium viable config file. No commands. Nothing but the programmer and board info. Call it openocd.cfg

# Segger J-Link EDU, SWD mode
## adapter line same as 
# #source [find interface/jlink.cfg]
adapter driver jlink
transport select swd

# Chip info (pyruler trinket)
set CHIPNAME at91samd21e18
source [find target/at91samdXX.cfg]

From the directory with this file running the bare openocd command will start up the various local servers that openocd will manage for you (gdbserver, telnet and tcl).

If one has more than one config file, you might want to call a specific one directly. Then openocd will want some direction about what you want to do with it.:

## init - necessary start up
## targets - lists all attached chips
## reset halt - puts chip in reset mode (i.e. in order to program, etc.)
openocd -f ./YOUR_FILE_NAME.cfg -c 'init; targets; reset halt;'

more commands: https://openocd.org/doc/html/General-Commands.html

Those commands could be in the .cfg file instead, and then you don’t need the -c part of the above command.

# Segger J-Link EDU, SWD modere
## adapter line same as 
# #source [find interface/jlink.cfg]
adapter driver jlink
transport select swd

# Chip info (pyruler trinket)
set CHIPNAME at91samd21e18
source [find target/at91samdXX.cfg]

init
targets
reset halt

The output with the first set up should look something like the below, but keep in mind my chip already has code on it. If you have trouble getting here, try putting your Trinket in reset mode by hand to start (double click the reset button).

Open On-Chip Debugger 0.12.0
Licensed under GNU GPL v2
For bug reports, read
        http://openocd.org/doc/doxygen/bugs.html
Info : Listening on port 6666 for tcl connections
Info : Listening on port 4444 for telnet connections
Info : J-Link V9 compiled Dec 13 2019 11:14:50
Info : Hardware version: 9.20
Info : VTarget = 3.325 V
Info : clock speed 400 kHz
Info : SWD DPIDR 0x0bc11477 # <-- common place to fail
Info : [at91samd21e18.cpu] Cortex-M0+ r0p1 processor detected
Info : [at91samd21e18.cpu] target has 4 breakpoints, 2 watchpoints
Info : starting gdb server for at91samd21e18.cpu on 3333
Info : Listening on port 3333 for gdb connections

Telnet Mode

Notice that OpenOCD is Listening on port 4444 for telnet connections that’s for people. If one wants to write automation that’s what the tcl option on port 6666 is for. Those port numbers can be changed via the config file.

In a separate shell try:

telnet localhost 4444
# ... once connected
> reset
> targets
> reg

reset gets the board out of reset mode. targets was covered above and the reg command will print out all the values of the registers.

> targets
    TargetName         Type       Endian TapName            State       
--  ------------------ ---------- ------ ------------------ ------------
 0* at91samd21e18.cpu  cortex_m   little at91samd21e18.cpu  running

> reg
## If no values, try again when in debug mode
===== arm v7m registers
(0) r0 (/32): 0x00003200
(1) r1 (/32): 0x00000000
(2) r2 (/32): 0x00000000
(3) r3 (/32): 0x0000131a
(4) r4 (/32): 0x00400040
(5) r5 (/32): 0xe000ed00
(6) r6 (/32): 0x00000005
(7) r7 (/32): 0x20002db0
(8) r8 (/32): 0x7afff4ef
(9) r9 (/32): 0xdde5fa1f
(10) r10 (/32): 0x27f9bfba
(11) r11 (/32): 0xefedfdff
(12) r12 (/32): 0xb9e9f4ff
(13) sp (/32): 0x20002dd8
(14) lr (/32): 0x00000527
(15) pc (/32): 0x00000288
(16) xPSR (/32): 0x21000000
(17) msp (/32): 0x20002dd8
(18) psp (/32): 0xf63efff0
(20) primask (/1): 0x00
(21) basepri (/8): 0x00
(22) faultmask (/1): 0x00
(23) control (/3): 0x00
===== Cortex-M DWT registers

Debugger Mode

In yet another shell window, start up gdb, but tell it to run from the server that’s already going.

## iex - Execute GDB command before loading the inferior.
arm-none-eabi-gdb -iex "target extended-remote localhost:3333"

This command will open gdb but push it straight over to the openocd host. Since in my case there’s no compiled file on the local machine I can give it to cross check (yet) many of the usual commands won’t work.

In the mean time these should:

layout asm to see the assembly
show architecture current target architecture.
help target

To be honest, I haven’t used gdb before this past week. It wasn’t a thing I thought I needed. So wrong. So. So. Wrong. I love it. (Even Arduino IDE 2 has it now! I did not know!)

I’ve got a lot to learn - reading list:

YouTube: Last video in assembly playlist
YouTube: Chris Bourke UNebraska 9.2 - Debugging - GDB Tutorial
https://sourceware.org/gdb/documentation/
https://www.cs.cmu.edu/~gilpin/tutorial/
https://jacobmossberg.se/posts/2017/01/17/use-gdb-on-arm-assembly-program.html
https://azeria-labs.com/debugging-with-gdb-introduction/
https://www.usna.edu/Users/cs/lmcdowel/courses/ic220/S22/labs/armIntro/assembly.html
http://cs107e.github.io/guides/gdb/
https://beej.us/guide/bggdb/ from tutorial below

core_hello.S and linker_hello.ld

The below code is pretty much taken from the excellent Hello Arm example, but with the linker script edited and commented for the SAMD21E18.

I’m not going to comment on them until the next post with a more complete example. I don’t recommend editing them piecemeal. They work because there’s almost nothing in them. Adding more with out adding ALL the needed more could end up being more trouble than just doing the full linker and chip specific start script. Next post!

If you’ve never seen assembly or a linker script before I’d say do the following in this order:

watch the assembly playlist (keep in mind, it’s not quite the same version, see below. )
watch the C based playlist because he covers a lot of using the compiler to learn
Read the Vivonomicon tutorial
read stargirl’s commented linker script.

If coming from desktop Arm, the Coretex-M0+ class of chip uses the 16-bit Thumb instruction set (video 7min | wikipedia) It’s a little different.

// These instructions define attributes of our chip and
// the assembly language we'll use:
.syntax unified
.cpu cortex-m0
.fpu softvfp //The Cortex-M0 line has no floating-point hardware
.thumb  //name for the ARM Cortex-M instruction set

// Global memory locations.
//don't put anything above the fake vector table. 
.global vector_table 
.global on_reset


/*
 * interupt vector table
 * Only the RAM boundry and reset handler are
 * included, for simplicity.
 */
.type vector_table, %object
vector_table:
    .word _estack
    .word on_reset
.size vector_table, .-vector_table

/*
 * The Reset Handler. 
 */
.type on_reset, %function
on_reset:
  // Set the stack pointer to the end of the stack.
  // The '_estack' value is defined in the linker script.
  LDR  r0, =_estack
  MOV  sp, r0

  // Set some dummy values. When we see these values
  // in our debugger, we'll know that our program
  // is loaded on the chip and working.
  LDR  r7, =0xDEADBEEF
  MOVS r0, #0
  main_loop:
    // Add 1 to register 'r0'.
    ADDS r0, r0, #1
    // Loop back.
    B    main_loop
.size on_reset, .-on_reset

And the linker


MEMORY {
    /* Syntax: name(attr): ORIGIN = int, LENGTH = int */
    /* attr is kindof like chmod */

    /* get values from chip data sheet: SAMD21 section 10.2 */
    /* only need to mentions the one you'll use */

    /* also called CODE_MEMORY or ROM */
    /* NOT LEAVING ROOM FOR BOOTLOADER */
    flash(rx): ORIGIN = 0x00000000, LENGTH = 0x00040000 /*256K*/

    /* also called just RAM */
    sram(rwx): ORIGIN = 0x20000000, LENGTH = 0x00008000 /*32K*/
}

/* end of the stack  See 9.1 in Summary Data sheet */ 
_estack     = ORIGIN(sram) + LENGTH(sram);    /* stack points to end of SRAM */

Compile, Upload, Debug

First we use the tools to compile core_hello.S to core.o to main.elf in a way that should feel very familiar from the [AVR post]/excuses/hello-led-on-an-avr-attiny45-in-c/.

WARNING: Don’t forget the -g flag! (need it for gdb later)

arm-none-eabi-gcc -x assembler-with-cpp -c -O0 -mcpu=cortex-m0 -mthumb -Wall -g core_hello.S -o core.o
arm-none-eabi-gcc core.o -mcpu=cortex-m0 -mthumb -Wall --specs=nosys.specs -nostdlib -lgcc -T./linker_hello.ld -o main.elf
arm-none-eabi-nm main.elf

Once we have the elf we can use openocd to load it on the the chip. I put the commands in another cfg file.

# Segger J-Link EDU, SWD mode
adapter driver jlink
transport select swd

# Chip info (pyruler trinket)
set CHIPNAME at91samd21e18
source [find target/at91samdXX.cfg]

init
reset halt
at91samd bootloader 0
program main.elf verify

openocd -f program_and_serve.cfg

The resulting output:

Open On-Chip Debugger 0.12.0
Licensed under GNU GPL v2
For bug reports, read
        http://openocd.org/doc/doxygen/bugs.html
Info : J-Link V9 compiled Dec 13 2019 11:14:50
Info : Hardware version: 9.20
Info : VTarget = 3.327 V
Info : clock speed 400 kHz
Info : SWD DPIDR 0x0bc11477
Info : [at91samd21e18.cpu] Cortex-M0+ r0p1 processor detected
Info : [at91samd21e18.cpu] target has 4 breakpoints, 2 watchpoints
Info : starting gdb server for at91samd21e18.cpu on 3333
Info : Listening on port 3333 for gdb connections
[at91samd21e18.cpu] halted due to debug-request, current mode: Thread 
xPSR: 0x61000000 pc: 0x00000008 msp: 0x20008000
[at91samd21e18.cpu] halted due to debug-request, current mode: Thread 
xPSR: 0x61000000 pc: 0x00000008 msp: 0x20008000
** Programming Started **
Info : SAMD MCU: SAMD21E18A (256KB Flash, 32KB RAM)
** Programming Finished **
** Verify Started **
** Verified OK **
Info : Listening on port 6666 for tcl connections
Info : Listening on port 4444 for telnet connections

In a new shell window, launch gdb with elf file specified and pushed into openocd host. OpenOCD and gdb work together to match up what’s on the chip with the expected symbol table. The example commands below will show the constant and the loop.

cd $YOUR_PROJECT_FOLDER
arm-none-eabi-gdb main.elf -iex "target extended-remote localhost:3333"

## Optional Layout Changes
(gdb) layout asm
(gdb) layout regs  
## ^x o to switch between
## ^x a to close

(gdb) info registers
(gdb) info register r0
(gdb) info register r7
(gdb) info break
(gdb) break main_loop
(gdb) info break
(gdb) continue # ^c to escape
(gdb) info register r0
(gdb) info register r7
(gdb) continue # ^c to escape
(gdb) info register r0
(gdb) stepi
(gdb) info register r0
# etc...

Makefile

Based on the AVR Makefile and the commands ran above. Fewer options for now. Critical for this gdb focused example because missing the -g flag makes it kind of useless.

An important difference

MCU = attiny
-mmcu=$(MCU)

MACH = cortex-m0
-mcpu=$(MACH)

Different hardware specific version of GCC have different flag sets. Always check!

###############################################################################
# Makefile for a simple SAMD21 Assembly project
###############################################################################

## General Flags
TARGET ?= hello#set if not defined 
LINKER_FILE ?= $(TARGET).ld
MACH = cortex-m0

TOOL_BASE = arm-none-eabi-
CC = $(TOOL_BASE)gcc

## Options common to compile, link and assembly rules
COMMON = -mcpu=$(MACH)
COMMON += -mthumb

## Compile options common for all C compilation units.
CFLAGS = $(COMMON)
CFLAGS += -Wall
CFLAGS += -g
CFLAGS += -Os


## Linker flags
LDFLAGS = $(COMMON)
LDFLAGS += --specs=nosys.specs -nostdlib -lgcc

## Objects that must be built in order to link
OBJECTS = $(TARGET).o 

## Objects explicitly added by the user
LINKONLYOBJECTS = 

# Programming support using avrdude. Settings and variables.
PROGRAMMER = openocd
PROGRAMMER_FLAGS = -f program_and_serve.cfg

## Build
all: $(TARGET).elf

## Compile
%.o : %.S
	$(CC) -x assembler-with-cpp -c $(CFLAGS) $<

##Link
%.elf: $(OBJECTS)
	 $(CC) $(LDFLAGS) $(OBJECTS) $(LINKONLYOBJECTS)  -T./$(LINKER_FILE) -o $(TARGET).elf

# Program the device.  
program: $(TARGET).elf
	$(PROGRAMMER) $(PROGRAMMER_FLAGS)

## Clean target
#.PHONY: clean
clean:
	-rm -rf $(OBJECTS) $(TARGET).elf $(TARGET).hex $(TARGET).lss $(TARGET).map

Summary

This assembly example works because there’s nothing in it. To do anything else it would be better to have a real start up script for the chip… a Makefile… A repo so much to improve! And that’s the next post!

Errata

I made a mistake when I first posted this (since corrected). So that big table of peripherals is for all SADM21*, not just for the SAMD21E18A [TODO LINK]. I had properly left out the ones that didn’t belong in my original original vector table code, but in a fit of insecurity I put them all back in. I’ve now taken them out again. The source of my insecurity was all this code on the internet! When looking at code on github, make sure both projects face the same tradeoffs. Notice the difference between the two files below. One has ALL of peripherals, the other does not. One is called startup_SAMD21.s the other is called startup_samd21e18a.s The first file supports the whole family at once. Presumably their default handlers can handle it. The second is the one that matches JUST this chip. There is a lot of SADM21.s and SADM21XXX.h code in the world. I was worried I was missing something… turns out I was. I was missing that it wasn’t solving the same problem.

Hello LED on an AVR (ATtiny45) in C

Tue, 02 Apr 2024 00:00:00 +0000

I want to start using official Embedded Swift, but I was thinking I should have some reference code on some different board families in C handy first.

AVR (ATtiny this post)
Probably skip PIC
SAM which are all 32 bit Arm Cortex
probably straight to STM32 Also 32-bit Arm Cortex MCU
Maybe ESP32

Turns out there’s a great tutorial series for AVR already done. Mitch Davis has a YouTube playlist:Fundamentals of Microcontrollers - Arduino bare-metal breakdown

The 9 part video series goes from blinking an LED on an Arduino board to blinking an LED with an ATTiny85 on a breadboard via linkers and optimizers and setting fuses and more. Assumes basic Arduino knowledge, but no C or other microcontroller experience. Because this series did such a good job covering what I would have wanted to cover I can just hop on down to where he leaves off by making an example with a switch and a Makefile.

One note: please unplug your boards when moving around wires.

More links below for supplemental reference.

Arduino 101

https://docs.arduino.cc/learn/starting-guide/getting-started-arduino/
https://itp.nyu.edu/physcomp/lessons/
ICs & Micros general info playlist. (General topics. It’s old.) https://www.youtube.com/watch?v=1qQE5Xwe7fs&list=PLfbQcOtLPyIFqfXwXdpnFJ2EN2RuLUpUl

Arduino on a breadboard

DataSheets

Misc Other

A note about the ATtiny85 and doing things on a breadboard.

I used to teach in person so I have A LOT of ATtinyX5 dip package chips lying around. Most people these days would buy a dev board (e.g. Arduino Pro mini) that has a surface mount AVR. Probably one that runs at 3.3V. I still like the DIP socket boards for prototyping because the DIP chips are sturdier and easily replaceable. I have many products from both companies below.

The chip I’m actually using is the ATtiny45. It has half the memory of the 85. For the ATTiny the number after the “y” is the flash memory size in k (it could be “16”). The last number is an indicator of the number of pins, but but not literally. The ATtiny44 has more than the ATtiny45. There’s a whole new set of ATTiny’s now, too. Here’s a guide.

AVR® Microcontrollers Peripheral Integration, Quick Reference Guide (previous version with ATTiny85(V) on it)

Check computer readiness before starting

All this presumes a machine that has compiled C and Swift, so it may be set up for clang but not avr-gcc. None of this will use an Arduino IDE’s install of these tools so it doesn’t matter if that’s on the box or not. Nor will any of this require software or libraries from ATMEL (now Microchip). Nor will the code import AVR LibC.

which avr-gcc

# points to 3rd party repo
# https://github.com/osx-cross/homebrew-avr
brew tap osx-cross/avr 

brew install avr-gcc
brew install avrdude

## Check install by printing out the compatible mcu's
## using zsh not bash escape the ?
avrdude -p \?

No need for avrdude’s -C (config file location) option. My 2024 homebrew install (version 7.3) doesn’t seem to need it.

Confirm compilation

Before I wired up the board, I checked that the code compiled and I got reasonable looking hex/assembly. My hello-world code doesn’t duplicate the tutorial exactly, but it’s pretty close.

This code will successfully blink a pin on PB3 (tested.)

//TODO: (blog meta) chroma treats #define as a comment bug report

//blink.c

//typically handled by AVR LibC
//10.4.2 PORTB – Port B Data Register
#define PORTB *((volatile unsigned char*) 0x38) //(0x20 + 0x18)
//10.4.3 DDRB – Port B Data Direction Register
#define DDRB *((volatile unsigned char*) 0x37)  //(0x20 + 0x17)

#define LED_PIN 3

//delay for the number of clock cycles
//approximately micros with default clock
//(version in AVR LibC)
void delay(volatile long time) {
    while (time != 0) {
        time--;
    }
}

int main() {
    DDRB |= (1<<LED_PIN); 
    while (1) {
        PORTB |= (1<<LED_PIN); // pin high
        delay(50000); //half a second at a 1MHz clock
        PORTB &= ~(1<<LED_PIN); // pin low
        delay(50000);
    }
}

How to compile, similar to video, with bonus how to get the assembly.

## (-g turns on debugging information, -Os optimizes)
avr-gcc -g -Os -mmcu=attiny45 blink.c

## leads to a.out which can be inspected in a hex editor 
## or save the interleaved disassembly and source with
avr-objdump -h -S a.out > dump.txt

## Make the ihex file for avrdude
avr-objcopy -O ihex -j.text -j.data a.out a.hex

blink_dump.txt

Wire Up the Board

The videos cover a basic version that will mostly work with the code above (LED inverted and on a different pin). The Arduino on a breadboard links at the top also cover a similar task.

Here’s a photo of my final set up because it’s a bit different. Links to a schematic and an annotated BOM.

Things to note:

LED Pin is on PB3 not PB1
for the LED HIGH is OFF (pin output in sink mode, not source mode)
Input pin uses internal pull up resistor (so none on board), switch closed to GROUND is turning on the LED.
In the picture you’ll see some extra wires “to nowhere.” I make “psuedo headers” in my breadboards sometimes so I don’t have to keep checking the pin diagrams.

schematic: kicad sch file

Bill of Materials

Annotated parts list.

Designator	Description	Qty	Use
SW1, SW2	6mm Tact Switch with 5 mm Height	2	Reset, Digital Input
R2, R4*	10kΩ axial resistor	2	external pull up
R1, R3†	330Ω axial resistor	2	current limiter
J2	wires to external power.	1	external power source (7-16V)
J1	wires to AVR-ISP programmer	1	chip programming
D2	5.0mm LED (Red)	1	Power Indicator
D4	5.0mm LED (Yellow)	1	Digital Output
D1**	1N4004	1	protection diode
C2	10uF polarized electrolytic	1	input (higher voltage) side
C1	1uF polarized electrolytic	1	output (regulated voltage) side
U2	DIP-8 ATtinyX5	1	Microcontroller
U1	LM7805 TO-220 or T-92 fine	1	Voltage Regulator

* R4 not in picture. Using chip’s internal pull up in real circuit, but its important not to forget to set it
† R3 is actually 1k in the picture. Yellow LEDs get bright.
** D1 not in picture. It’s in my power supply.
Why caps in the power circuit
7805 Datasheet

Program the Board

The new code:

//10.4.2 PORTB – Port B Data Register
#define PORTB *((volatile unsigned char*) 0x38) //(0x20 + 0x18)
//10.4.3 DDRB – Port B Data Direction Register
#define DDRB *((volatile unsigned char*) 0x37)  //(0x20 + 0x17)
//10.4.4 PINB – Port B Input Pins Address
#define PINB *((volatile unsigned char *)0x36)  //(0x20 + 0x16)

#define LED_PIN 3
#define SWITCH_PIN 4

int main() {
    //set led pin bit to output
    DDRB |= (1<<LED_PIN);  
    
    //set switch pin bit to input
    DDRB &= ~(1<<SWITCH_PIN); 
     //for voltage high == no press config, sets pull up.
    PORTB |= (1<<SWITCH_PIN);
    //give the pull up a moment.
    
    //internal pull ups have high R and pin may be slow to reach logic high threshold. If problem, chip has to have settle time out of reset (SUT) or code does. 

    while (1) {
        if (!(PINB & (1<<SWITCH_PIN))) {  //if pressed
        //above if compares (0x000?0000 & 0x00010000) => 
        // - if true, seeing pull up. button not pressed
        // - if false, seeing ground via closed switch, button pressed
            PORTB &= ~(1<<LED_PIN);   // pin low, LED ON
        } else {
            PORTB |= (1<<LED_PIN);    // pin high, LED OFF 
        }
    }
}

switch_control_dump.txt

avrdude -p t45 -c usbtiny  #to confirm board is ready

## REDO compile with new code, & upload
avr-gcc -g -Os -mmcu=attiny45 switch_control.c
avr-objcopy -O ihex -j.text -j.data a.out a.hex
avrdude -p t45 -c usbtiny -U flash:w:a.hex:i

More on Fuses & Clock Settings

Video 5 of the series(“AVRDude and fuses”) talks about setting the fuses for the chip’s clock settings.

# memtype:operation:value:format
# m means direct value, as opposed to i for intel hex
avrdude -v -p t45 -c usbtiny -U lfuse:w:0xF2:m

I skipped that, but now I want to see the settings for the chip I grabbed.

# h is hexadecimal mode for the read.
# "each value will get the string 0x prepended."
avrdude -v -p t45 -c usbtiny -U hfuse:r:-:h -U lfuse:r:-:h

In the output will be:

avrdude: using SCK period of 10 usec

avrdude: processing -U hfuse:r:-:h
avrdude: reading hfuse memory ...
avrdude: writing output file <stdout>
0xdf 

avrdude: processing -U lfuse:r:-:h
avrdude: reading lfuse memory ...
avrdude: writing output file <stdout>
0x62

That means the registers look like:

 low fuse: 0110 0010
high fuse: 1101 1111

Both of these values are the defaults, and the clock information is discussed in section 6.2 of the datasheet. ( 1 is unset and 0 is set.)

	7	6	5	4	3	2	1	0
low	CKDIV8	CKOUT	SUT1	SUT0	CKSEL3	CKSEL2	CKSEL1	CKSEL0
high	RSTDISBL	DWEN	SPIEN	WDTON	EESAVE	BODLEVEL2	BODLEVEL1	BODLEVEL0

This default setting means the chip is running at 1MHz (8MHz calibrated internal RC oscillator (6.2.3) with the CKDIV8 bit set).

The 0xF2 setting from the video clears the dividing bit (sets CKDIV8 to 1) and the SUT0 bit as well since it only matters with CKDIV8 is enabled.

Clock settings matter a lot. Here’s some tips.

Why use an external clock? Synchronize with another IC
Why use a crystal? Higher reliability for real time behavior
If staying internal, why stick with the 1MHz clock? (or go lower)
- Lower power
- Easy math (1MHz = once per 0.000001 seconds = every microsecond)
Why not? Something happens faster. Communication protocols, audio, expensive math…

My little demo program isn’t doing anything fancy so 1MHz works for me.

More Info on Fuses

Switching to a Make file

Running those compile and upload commands by hand gets old. Enter Makefiles, which I reviewed before.

For embedded projects I tend to cheat off of todbot.

But here are some demos from AVR LibC

Today I’ll roll one to better match the commands already shown.

Some differences?

uses the elf file format instead of the .out for the intermediary
uses the target name rather than “a”
makes a lss (listing file, I called it dump.txt earlier) file
makes a map file (debugging map)
can switch which target with make TARGET=switch_control
can compile and run in one step with which target with make TARGET=switch_control program
I left out everything about the fuses for now, but there is an erase function.

I tend to have the Makefile to put build products in a separate folder, but since I’m undecided on how I’m going to organize these demo files I’m skipping that for now.

###############################################################################
# Makefile for a simple ATtiny project
###############################################################################

## General Flags
TARGET ?= blink #set if not defined 
MCU = attiny45

# for when using AVR LibC
#F_CPU = 16000000   # 16MHz on Arduino board
#F_CPU = 8000000    # internal osc
#F_CPU = 1000000    # internal osc w/ clkdiv8
#F_CPU  =  500000   # internal osc w/ clkdiv16
#F_CPU =  250000    # internal osc w/ clkdiv32

# Theses values are currently the defaults. 
LFUSE=0x62
HFUSE=0xDF
EFUSE=0xFF


CC = avr-gcc
CPP = avr-g++

## Options common to compile, link and assembly rules
COMMON = -mmcu=$(MCU)

## Compile options common for all C compilation units.
CFLAGS = $(COMMON)
CFLAGS += -Wall
CFLAGS += -g
CFLAGS += -Os

# for when using AVR LibC // i.e. #if F_CPU == 500000 type code
#Flags that start with -D are a define 
#equivalent to #define F_CPU = the variable value
#CFLAGS += -DF_CPU=$(F_CPU) #creates the F_CPU var for the compiler. 

## Linker flags
LDFLAGS = $(COMMON)
LDFLAGS +=  -Wl,-Map=$(TARGET).map

## Objects that must be built in order to link
OBJECTS = $(TARGET).o 

## Intel Hex file production flags
HEX_FLASH_FLAGS = -j.text -j.data 

## Objects explicitly added by the user
LINKONLYOBJECTS = 

# Programming support using avrdude. Settings and variables.
AVRDUDE = avrdude
AVRDUDE_PROGRAMMER = usbtiny
#AVRDUDE_PRODUCT = t45

AVRDUDE_WRITE_FLASH = -U flash:w:$(TARGET).hex:i

AVRDUDE_FLAGS = -p $(MCU) -c $(AVRDUDE_PROGRAMMER)

## Build
all: $(TARGET).elf $(TARGET).hex $(TARGET).lss size

## Compile
%.o : %.c
	$(CC) $(INCLUDES) $(CFLAGS) -c  $<

##Link
%.elf: $(OBJECTS)
	 $(CC) $(LDFLAGS) $(OBJECTS) $(LINKONLYOBJECTS) -o $(TARGET).elf

%.hex: $(TARGET).elf
	avr-objcopy -O ihex $(HEX_FLASH_FLAGS)  $< $@

%.lss: $(TARGET).elf
	avr-objdump -h -S $< > $@

size: ${TARGET}.elf
	@echo
	@avr-size -C --mcu=${MCU} ${TARGET}.elf

erase:
	$(AVRDUDE) $(AVRDUDE_FLAGS) -e -U lfuse:w:$(LFUSE):m -U hfuse:w:$(HFUSE):m -U efuse:w:$(EFUSE):m

# Program the device.  
program: $(TARGET).hex
	$(AVRDUDE) $(AVRDUDE_FLAGS) $(AVRDUDE_WRITE_FLASH)

## Clean target
#.PHONY: clean
clean:
	-rm -rf $(OBJECTS) $(TARGET).elf $(TARGET).hex $(TARGET).lss $(TARGET).map

The Swift/clang ecosystem prefers CMake, which I don’t know very well at all (CMake hello world post).

I guess I’ll be learning!

What would be a very simple working Decoder?

Sun, 31 Mar 2024 00:00:00 +0000

After finding a couple of great public examples of simple Encoders (swift talk | stack overflow), I did not find the same for a Decoder. Here’s my shot at it. Like the very good but behind a paywall swift talk example, it has limited use. Both were designed with a single type in mind. This one can decode a single serialized struct with Strings, Ints and Dates as properties.

property delimited by “/”
key-value delimited by “:”
lenient about whitespace.

The SimpleCoder Decoder will need a much more complicated parser and will handle more types.

So with a post about string munching and one about date parsing in the rear-view let’s make a Decoder!

Finished playground code

remember how to inspect strings in general
decide about Dates specifically
write a really really simple decoder to walk through the process <=
inspect other peoples decoders
write the string -> tree step for SimpleCoder
finish the full decoder (with round trip tests)

Using the Decoder

First updating the struct to actually use Date in the struct and add an Int for good measure.

struct HousePlant:Codable {
    let commonName:String
    let whereAcquired:String?
    let dateAcquired:Date
    //NOTE THE VAR. `let` with a default wil not be decoded.
    var currentPropCount:Int = 0 
    let dateOfDeath:Date?
}

Thanks to some handy code completions in XCode we can peak at the synthesized Codable conformance:

    enum CodingKeys: CodingKey {
        case commonName
        case whereAcquired
        case dateAcquired
        case currentPropCount
        case dateOfDeath
    }
    
    init(from decoder: any Decoder) throws {
        let container = try decoder.container(keyedBy: CodingKeys.self)
        self.commonName = try container.decode(String.self, forKey: .commonName)
        self.whereAcquired = try container.decodeIfPresent(String.self, forKey: .whereAcquired)
        self.dateAcquired = try container.decode(Date.self, forKey: .dateAcquired)
        self.currentPropCount = try container.decodeIfPresent(Int.self, forKey: .currentPropCount)
        self.dateOfDeath = try container.decodeIfPresent(Date.self, forKey: .dateOfDeath)
    }
    
    func encode(to encoder: any Encoder) throws {
        var container = encoder.container(keyedBy: CodingKeys.self)
        try container.encode(self.commonName, forKey: .commonName)
        try container.encodeIfPresent(self.whereAcquired, forKey: .whereAcquired)
        try container.encode(self.dateAcquired, forKey: .dateAcquired)
        try container.encode(self.currentPropCount, forKey: .currentPropCount)
        try container.encodeIfPresent(self.dateOfDeath, forKey: .dateOfDeath)
    }

From the init(from decoder: any Decoder) one can tell that the Decoder needs to be able to produce values when given a CodingKey and a Type.

Using that init would look like:

//Have raw input
let serializedHousePlant = "commonName:spider plant/whereAcquired:Trader Joe's/dateAcquired: 2024-03-12T16:12:07Z / currentPropCount:8 "
//have a decoder that has a chance of doing the lexical analysis
let decoder = HousePlantDecoder(serializedHousePlant)
//Codable finishes 
let myHousePlant = try HousePlant(from: decoder)

Our decoder holds a potential HousePlant in its belly or BUST.

The Data Type

Using the same reference-box as SimpleCoder

final class SimpleCoderData<Value> {
    var storage:Value
    
    init(_ value: Value) {
        self.storage = value
    }
}

Declare and conform

Notice the fatalError on the SingleValueContainer and UnkeyedContainer? Well, a single serialized HousePlant will never need them.

struct HousePlantDecoder:Decoder {
    var data:SimpleCoderData<Dictionary<String,String>>
    
    var codingPath: [any CodingKey] = []
    var userInfo: [CodingUserInfoKey : Any] = [:]
    
    func container<Key>(keyedBy type: Key.Type) throws -> KeyedDecodingContainer<Key> where Key : CodingKey {
        KeyedDecodingContainer(HousePlantDecoderKDC(decoder: self))
    }
    
    func unkeyedContainer() throws -> any UnkeyedDecodingContainer {
        fatalError()
    }
    
    func singleValueContainer() throws -> any SingleValueDecodingContainer {
        fatalError()
    }
    
}

Parsing initializer

Can a Dictionary<String,String> even happen? If not, I don’t even want to bother making the Decoder.

The second parser from the string scanning post nicely balances speed and complexity.

extension HousePlantDecoder {
    static func parse(_ inputString:String) throws -> Dictionary<String,String> {
        let fullKeyValueList = inputString.split(separator: "/")
        var result:Dictionary<String,String> = [:]
        return try fullKeyValueList.reduce(into: result){ result, item in
            if let firstColonIndex = item.firstIndex(of: ":") {
                let key = item
                    .prefix(upTo: firstColonIndex)
                    .trimmingCharacters(in: .whitespacesAndNewlines)
                let value = item
                    .suffix(from: item.index(firstColonIndex, offsetBy: 1))
                    .trimmingCharacters(in: .whitespacesAndNewlines)
                result[String(key)] = String(value)
            } else {
                throw HousePlantError.notAKeyValuePair
            }
        }
    }

    init(_ inputString:String) throws {
        self.data = SimpleCoderData(try HousePlantDecoder.parse(inputString))
    }
}

Custom Functions for Containers

The containers will need to retrieve values from the storage, and throw errors if the values aren’t there.

Also, the date decoder from the previous post should go in there since our date isn’t in timeIntervalSinceReferenceDate format (the default).

extension HousePlantDecoder {
    func retrieveValue(forKey codingKey:CodingKey) throws -> String {
        guard let value = data.storage[codingKey.stringValue] else {
            throw DecodingError.keyNotFound(codingKey, DecodingError.Context(codingPath: codingPath, debugDescription: "Didn't find key in decoder data."))
        }
        return value
    }

    func _decodeDate(from value:String) throws -> Date {
        let format = if value.contains(":") {
                        Date.ISO8601FormatStyle.iso8601
                    } else {
                        Date.ISO8601FormatStyle.iso8601
                            .year().month().day()
                    }
        guard let date = try? format.parse(value) else {
            throw DecodingError.dataCorrupted(
                .init(codingPath: [], debugDescription: "String not in expected \(format.parseStrategy) format.")
            )
        }
        return date
    }
}

KeyedDecodingContainerProtocol

Almost the bare minimum to get it working.

struct HousePlantDecoderKDC<Key:CodingKey>:KeyedDecodingContainerProtocol {
    let decoder:HousePlantDecoder
    var codingPath: [any CodingKey] { decoder.codingPath }
    
    var allKeys: [Key] {
        decoder.data.storage.keys.compactMap({ Key(stringValue: $0) })
    }
    
    func contains(_ key: Key) -> Bool {
        decoder.data.storage.keys.contains(key.stringValue)
    }
    
    func decodeNil(forKey key: Key) throws -> Bool {
        //return true if the value should be .none
        //false if .some, and an attempt should be made to decode the
        //wrapped value
        if decoder.data.storage.keys.contains(key.stringValue) {
            let value = decoder.data.storage[key.stringValue]
            return value == "NULL" || value == "nil"
        } else {
            return true
        }
    }
    
    func decode(_ type: Bool.Type, forKey key: Key) throws -> Bool { fatalError() }
    
    func decode(_ type: String.Type, forKey key: Key) throws -> String {
        try decoder.retrieveValue(forKey: key)
    }
    
    func decode(_ type: Double.Type, forKey key: Key) throws -> Double { fatalError() }
    func decode(_ type: Float.Type, forKey key: Key) throws -> Float { fatalError() }

    func decode(_ type: Int.Type, forKey key: Key) throws -> Int {
        let value = try decoder.retrieveValue(forKey: key)
        if let int = Int(value) { return int }
        throw DecodingError.typeMismatch(type, DecodingError.Context(codingPath: codingPath, debugDescription: "Could not make \(type) from \(value)"))
    }

    func decode(_ type: Int8.Type, forKey key: Key) throws -> Int8 { fatalError() }
    func decode(_ type: Int16.Type, forKey key: Key) throws -> Int16 { fatalError() }
    func decode(_ type: Int32.Type, forKey key: Key) throws -> Int32 { fatalError() }
    func decode(_ type: Int64.Type, forKey key: Key) throws -> Int64 { fatalError() }
    func decode(_ type: UInt.Type, forKey key: Key) throws -> UInt { fatalError() }
    func decode(_ type: UInt8.Type, forKey key: Key) throws -> UInt8 { fatalError() }
    func decode(_ type: UInt16.Type, forKey key: Key) throws -> UInt16 { fatalError() }
    func decode(_ type: UInt32.Type, forKey key: Key) throws -> UInt32 { fatalError() }
    func decode(_ type: UInt64.Type, forKey key: Key) throws -> UInt64 { fatalError() }
    
    func decode<T>(_ type: T.Type, forKey key: Key) throws -> T where T : Decodable {
        switch type {
        case is Bool.Type: return try decode(Bool.self, forKey: key) as! T
        case is String.Type: return try decode(String.self, forKey: key) as! T
        case is Double.Type: return try decode(Double.self, forKey: key) as! T
        case is Float.Type: return try decode(Float.self, forKey: key) as! T
        case is Int.Type: return try decode(Int.self, forKey: key) as! T
        case is Int8.Type: return try decode(Int8.self, forKey: key) as! T
        case is Int16.Type: return try decode(Int16.self, forKey: key) as! T
        case is Int32.Type: return try decode(Int32.self, forKey: key) as! T
        case is Int64.Type: return try decode(Int64.self, forKey: key) as! T
        case is UInt.Type: return try decode(UInt.self, forKey: key) as! T
        case is UInt8.Type: return try decode(UInt8.self, forKey: key) as! T
        case is UInt16.Type: return try decode(UInt16.self, forKey: key) as! T
        case is UInt32.Type: return try decode(UInt32.self, forKey: key) as! T
        case is UInt64.Type: return try decode(UInt64.self, forKey: key) as! T
        case is Date.Type: 
            let value = try decoder.retrieveValue(forKey: key)
            print(value)
            return try decoder._decodeDate(from: value) as! T
        default:
            //probably should fatalError this too, but, YOLO.  
            let value = try decoder.retrieveValue(forKey: key)
            let tmpDecoder = try HousePlantDecoder(value)
            return try type.init(from:tmpDecoder)
        }
    }
    
    func decodeIfPresent<T>(_ type: T.Type, forKey key: Key) throws -> T? where T: Decodable {
      guard contains(key) else { return nil }
      return try decode(type, forKey: key)
    }
    
    func nestedContainer<NestedKey>(keyedBy type: NestedKey.Type, forKey key: Key) throws -> KeyedDecodingContainer<NestedKey> where NestedKey : CodingKey { fatalError() }
    func nestedUnkeyedContainer(forKey key: Key) throws -> any UnkeyedDecodingContainer { fatalError() }
    func superDecoder(forKey key: Key) throws -> any Decoder { fatalError() }
    func superDecoder() throws -> any Decoder {
        decoder
    }
}

Alteratively, the folks who wrote the Vapor PlaintextDecoder got a little fancy and decoded anything that’s a LosslessStringConvertible in one swoop. (All the FixedWidthIntegers!) I like it. That’s what I’ve done in the code linked at the top and bottom of this post. (I wanted to show the code completion code first)

// in func decode<T>(_ type: T.Type, forKey key: Key)
        case is LosslessStringConvertible.Type :
            let value = try decoder.retrieveValue(forKey: key)
            if let convertible = T.self as? LosslessStringConvertible.Type {
                return try decoder._decodeLossless(value, ofType:convertible) as! T
            } else {
                throw DecodingError.dataCorruptedError(
                    forKey: key,
                    in: self,
                    debugDescription: "Thought \(value) could be converted to \(type.self) as a LosslessStringConvertible. Something went wrong.")
            }

//add to decoder
    func _decodeLossless<L: LosslessStringConvertible>(_ value:String, ofType: L.Type) throws -> L {
        guard let result = L.init(value) else { 
            throw DecodingError.typeMismatch(
                L.self, 
                DecodingError.Context(codingPath: codingPath, 
                                      debugDescription: "Attempt to decode \(value) as \(L.self) failed.")
                ) 
            }
        return result
    }

Summary

Put that all together and one gets a HousePlant struct from a String. the most basic of Decoders, just to prove to myself I get the gist before I go hunting for Decoders on github to break down.

Playground Code

Date Parsing. Nose wrinkle.

Fri, 29 Mar 2024 00:00:00 +0000

remember how to inspect strings in general
decide about Dates specifically <=
write a really really simple decoder to walk through the process
inspect other peoples decoders
write the string -> tree step for SimpleCoder
finish the full decoder (with round trip tests)

The test HousePlant from the last post just totally punted on parsing dates, but I know I want to use them. So now what?

First step is to be clear on what a Date type is and what it isn’t. As we saw when writing the encoder, the default Codable representation for Date is timeIntervalSinceReferenceDate because that is what a Date’s core storage is (core | FoundationEssentials ), a TimeInterval. And what’s a TimeInterval? A type alias for a Double representing seconds.

Any other representation of a Date provided by a formatter is a convenience. Calendar and other libraries better serve working with days, hours, months (little d dates) as first-order concepts. One cannot have a Date with nil for hours. That makes no sense. Date has no Optional “hours” parameter. Date is a unidimensional time coordinate; it must be specific. One can represent spotty calendar style information with DateComponents, not Date.

The various date formatters one can use in Swift do an excellent job of creating these time coordinates we call a Date from Strings and vice versa. However, it could be easy to end up very frustrated with them if in fact one is trying to do withDate things that would be much better served by a different API. (For even more on this topic see conversations around SE-0329) (also this NSSpain X Dave DeLong talk)

As of 2021 there are new versions of theses formatters in Foundation. The WWDC21 video “What’s new in Foundation” (14:29) shows both the old style and the new and explains the motivation.

The WWDC22 RegEx videos both show working with the new formatter in the context of a RegEx.

swift-foundation has public implementations.

ParseStrategy protocol definition (FoundationEssentials)
ParseStrategy Extension on Date (FoundationInternationalization)
ICUDateFormatter Internal class doing the work. (FoundationInternationalization)
Date.FormatString definition (FoundationInternationalization)
extension Date.FormatString (FoundationInternationalization)
DateFieldSymbol (FoundationInternationalization)
DateFormatStyle
VerbatimFormatStyle (FoundationInternationalization)
Date.ISO8601FormatStyle (FoundationInternationalization)
Date parsing as mentioned in RegEx Pitch (5.7) (swift-evolution)
Regex String Processing Algorithms (swift-evolution)

FormatStyle

FormatStyle seems to be designed for UI representation, making sure that the user’s settings stay in charge of how a Date gets displayed. As a result a FormatStyle works differently depending on the device’s settings and localization. Output will behave a bit unpredictably to the developer. For example, 24 hour time display gets decided by the user’s system settings, not the developer. On the other hand, it also provides some pretty generous parsers as demonstrated below.

My Playground and Package Manager for some reason had differing opinions about my locale settings? (TODO?) I could have gotten more control by adding an explicit Locale (e.g. .locale(Locale(identifier: "en_US"))), but I decided to roll with it to see the different behaviors.

//Computer settings PDT. 
//In Playground. Fails in package until date string changed to 03/28/2024
let basicFormatStyle = Date.FormatStyle()
let basicDate = basicFormatStyle.parse("28/03/2024 03:12 PM")
let basicStringFromDate = basicFormatStyle.format(basicDate)
print("date:", basicDate, "string:", basicStringFromDate)
//date: 2024-03-28 22:12:00 +0000 string: 28/03/2024, 3:12 PM

If time doesn’t matter, one can ignore it by telling the format style which fields you do care about. Order doesn’t matter. Order will be determined by the Locale. Notice in the print out there still is a time, but its midnight in the device’s time zone relative to UTC, not the time from the String.

//Computer settings PDT. 
//In Playground. Fails in package until date string changed to month leading
let dateOnlyStyle:Date.FormatStyle = .dateTime.year().day().month()
let dateOnlyDate = dateOnlyStyle.parse("28/03/2024 03:12 PM")
let dateOnlyStringFromDate = dateOnlyStyle.format(dateOnlyDate)
print("date:", dateOnlyDate, "string:", dateOnlyStringFromDate)
//date: 2024-03-28 07:00:00 +0000 string: 28 Mar 2024

A FormatStyle ends up being super chill about the text it will take in. It took all the following text inputs (Playground examples. Switched to month leading in Package Manger. There does not seem to be a way to make a super chill year-leading formatter? Where does that?)

28/03/2024
28 Mar 2024
28-03-2024
28-Mar-2024
28   .-/  03   .-/ 2024

But not

28 fhujsflh 03 jifoalgf 2024

So it does seem to be limited to “the usual suspects” in terms of date delimiters.

I’ve been showing the showing the shortcuts. The full initializer looks like

Date.FormatStyle(date: Date.FormatStyle.DateStyle?, 
                 time: Date.FormatStyle.TimeStyle?, 
                 locale: Locale, 
                 calendar: Calendar, 
                 timeZone: TimeZone, 
                 capitalizationContext: FormatStyleCapitalizationContext)

For example (why this)

static let customFormat = Date.FormatStyle(date: .complete, 
                                   time: .complete, 
                                   locale: Locale(identifier: "zh_Hans_CN"), 
                                   calendar: Calendar(identifier: .chinese), 
                                   timeZone: TimeZone(identifier: "UTC")!, 
                                   capitalizationContext: .beginningOfSentence)

//Text(Date.now, format:Self.customFormat) 
//=> Second Month 19, 2024(jia-chen) at 21:37:17
// when it was Thursday Mar 28 14:37:17 and system time was set to 24hr time.

ISO8601FormatStyle

If less chill is desired, one can go for the built in ISO8601 format.

let dateString = "20240312 03:12:10"
let myDateFormat:Date.ISO8601FormatStyle = .iso8601.year().month().day().dateSeparator(.omitted)
let myDate = myDateFormat.parse(dateString2)
let stringFromDate = myDateFormat.format(myDate)
print("date:", myDate, "string:", stringFromDate)
//date: 2024-03-12 00:00:00 +0000 string: 20240312

With time omitted from the format, again the parser will just ignore it. Overall the ISO8601FormatStyle assumes the date data will be generated by machines and be read by machines. It has very limited configuration. It’s not interested in being a good sport. That’s not its job.

VerbatimFormatStyle and DateFormatString

If one wants less chill like with ISO8601FormatStyle, but in a custom style, VerbatimFormatStyle combined with a DateFormatString provides a similar functionality.

//for some reason Playground balks when not dynamic var
var isoFormatString:Date.FormatString {
    "\(year: .defaultDigits)-\(month: .twoDigits)-\(day: .twoDigits)"
}

let verbatimFormat:Date.VerbatimFormatStyle = .init(format: isoFormatString, timeZone: TimeZone.gmt, calendar: Calendar.current)
//need to drop down into the parseStrategy directly 
let verbatimDate = verbatimFormat.parseStrategy.parse("2024-03-12")
let stringFromVDate = verbatimFormat.format(myDate)
//date: 2024-03-12 00:00:00 +0000 string: 2024-03-12

This is exactly how to get 24 hour time, independent of the system settings.

var withTime:Date.FormatString { "\(year: .defaultDigits)-\(month: .twoDigits)-\(day: .twoDigits) \(hour: .twoDigits(clock: .twentyFourHour, hourCycle: .zeroBased)):\(minute: .twoDigits):\(second: .defaultDigits) \(timeZone: .identifier(.short))" }

let verbatimFormatWithTime:Date.VerbatimFormatStyle = .init(format: withTime, timeZone: TimeZone.gmt, calendar: Calendar.current)
print(verbatimFormatWithTime.format(Date.now))

The VerbatimFormatStyle does not appear to work with Date.FormatString literals (FB13700896, semi-embarrassing forum post (plenty of those)), but pure ParseStrategy approaches do.

If one wants flexibility and a custom style… well that’s where I’d abandon Date for DateComponent and pull out RegExBuilder. Maybe even make some custom components. OOOooo or maybe a custom scanner… no this is 2024 its gotta be ML… But as I officially have too many projects going and Decoders do not require the ability to take human generated input, not today.

ParseStrategy

Date initializers can now take a ParseStrategy. You can roll your own with a Date.FormatString, or extract one from an existing Date.FormatStyle (myFormatStyle.parseStrategy.format). Like VerbatimFormatStyle they don’t provide flexibility.

//THIS CODE DOES NOT WORK IN PLAYGROUNDS
func testDateParsing() throws {
    let dashString:Date.FormatString = "'y-M-d'"
    let strategy = Date.ParseStrategy(format:  dashString, timeZone: .gmt)
    let date = try Date("2024-03-12", strategy: strategy)
    print("newDate:", date)
    //newDate 2024-03-12 00:00:00 +0000
    let formatString = strategy.format //returns the format string.
    print(formatString)
    //''y-M-d''
    let date2 = try strategy.parse("2024-03-12")
    XCTAssertEqual(date, date2)
}

In this example I used a Date.FormatString literal, but THEY ARE NOT IDENTICAL to the old DateFormatter literals.

They’re going to take some getting used to.

Definition y (current year)
Definition Y (week of year?)
works: "'y'-'MM'-'dd' 'HH':'mm':'ss'", fails: "y'-'MM'-'dd' 'HH':'mm':'ss"
When the date string is let dateString = "20240312031210" the string has to be "'yyyyMMddHHmmss'" and the full initializer has to have the year as \(year: .extended(minimumLength: 4)) as .defaultdigits does not work.

If one wanted to extract a format string to pass it around, it would need a little massaging to get back into proper Date.FormatString condition.

var formatString:Date.FormatString {
    Date.FormatString(stringLiteral: "'\(someFormat.parseStrategy.format)'")
}

let strategy = Date.ParseStrategy(format:  formatString, timeZone: .gmt)
let date = try Date(dateString, strategy: strategy)

Example for the Decoder

My first crack at decoding a date actually went straight for initializing a ParseStrategy with a Date.FormatString. Which strategy depended on whether the time was included (based on presence of colon).

func _decodeDate(from value:String) throws -> Date {
    //gmt to make sure time fields are 0
    let strategy = if value.contains(":") {
                        //strings can work too, not identical to DateFormatter but similar
                        Date.ParseStrategy(format: "'yyyy-MM-dd' 'HH:mm:ss'",
                                            timeZone: .gmt)
                    } else {
                        Date.ParseStrategy(format: "\(year: .defaultDigits)-\(month: .twoDigits)-\(day: .twoDigits)", 
                                           timeZone: .gmt)
                    }

    guard let date = try? Date(value, strategy: strategy) else {
        throw DecodingError.dataCorrupted(
            .init(codingPath: [], debugDescription: "String not in expected \(strategy.format) format.")
        )
    }
    return date
}

On second pass I changed the code to use the ISO8601FormatStyle because that’s the industry standard. Additionally, I decided to require a time zone (distance from UTC as ±HH:mm or Z if UTC) if a time is also included. An Encoder should be able to provide that no problem.

func _decodeDate(from value:String) throws -> Date {
    //gmt ("Z")is the default. 
    //
    let format = if value.contains(":") {
                    Date.ISO8601FormatStyle.iso8601
                        //.dateTimeSeparator(.space)
                } else {
                    Date.ISO8601FormatStyle.iso8601
                        .year().month().day()
                }
    guard let date = try? format.parse(value) else {
        throw DecodingError.dataCorrupted(
            .init(codingPath: [], debugDescription: "String not in expected \(format.parseStrategy) format.")
        )
    }
    return date
}

I could try to be a lot more gracious with what strings I’ll accept, but I’m choosing not to be. When writing a Decoder, it’s important to know if the data will be coming from:

user input,
only your own Encoder,
or someone else’s serialization process

The question becomes: is a badly formatted date something to be gracious about, or an alarm that the data can’t be trusted. I wanted to think about how to accept more than one format style, but not go full general purpose calendar event storage.

Summary

Good. Enough. Next post, a super basic Decoder.

Wait, how do I scan text again?

Wed, 27 Mar 2024 00:00:00 +0000

The encoder spun our data out into a thread using serialization. A decoder crochets that thread back into an object.

If the data coming into the decoder is highly structured and narrow in scope it can actually be kind of relaxing to parse. In the Swift Talk decoder example (video (paywall) | github) the parsing ends up being pretty straight forward because their encoder only takes in a Route, and the decoder only produce a Route.

The general purpose mayhem of taking in any Codable that SimpleCoder performs is the whole point of an Encoder/Decoders pair (they are about the format, not the type) but it’s a lot to tackle in one bite. I’m not even using brackets to help calculate object relationships. Oof.

As a result making the decoder will be a much slower job.

remember how to inspect strings in general
decide about Dates specifically
write a really really simple decoder to walk through the process
inspect other peoples decoders
write the string -> tree step for SimpleCoder
finish the full decoder (with round trip tests)

In this post

First on the list, I’m going to catalog all the ways I know how to scan strings for information.

String.builtinThing (0.000195)
Moving the index (0.000113)
Regex (\: 0.000399, builder: 0.000176)
Scanner (v1 0.0000439, v2 0.0000512)
Scanner + Dynamic Sequence (0.0000261)
Custom char scanner (0.0000246)

The number in parenthesis is the average time to turn one example string into a Dictionary according to my XCTest set up. I’m focused on their relative order of magnitude.

    func testStaticParserJ() throws {
        measure {
            do {
                let dictJ = try HousePlantTest.parse_charScanner4(serializedHousePlant)
            } catch {
                fatalError()
            }
        }
    }

I’m going to delay talking about making a Parser Combinator or other more substantial parser class because that would be complete overkill for a “MyFirstDecoder” Decoder, even for me.

Using built in String functions

Let’s imagine there is a HousePlant struct and an encoded example. As long as we can get that string into a Dictionary<String, String>, we can make a HousePlant out of it. (I’m going to punt on Date parsing for now)

//Playground style
struct HousePlant:Codable {
    let commonName:String
    let whereAcquired:String?
    let dateAcquired:String
    let dateOfDeath:String?

    static func parse(_ inputString:String) throws -> Dictionary<String,String> {
        let fullKeyValueList = inputString.split(separator: "/")
        var result:Dictionary<String,String> = [:]
        return fullKeyValueList.reduce(into: result){ result, item in
            let kvSplit = item.split(separator: ":")
            let key = String(kvSplit[0]).trimmingCharacters(in: .whitespacesAndNewlines)
            let value = String(kvSplit[1]).trimmingCharacters(in: .whitespacesAndNewlines)
            result[key] = value
        }
    }
}

extension HousePlant {
    init?(_ inputString:String) {
        guard let dictionary = try? HousePlant.parse(inputString) else {
            return nil
        }
        print("made dictionary", dictionary)
        if let cN = dictionary["commonName"] {
            print("commonName")
            self.commonName = cN
        } else {
            return nil
        }
        if let dA = dictionary["dateAcquired"] {
            print("dateAcquired")
            self.dateAcquired = dA
        } else {
            return nil
        }
        self.whereAcquired = dictionary["whereAcquired"]
        self.dateOfDeath = dictionary["dateOfDeath"]
    }
}
let serializedHousePlant = "commonName:spider plant/whereAcquired:Trader Joe's/dateAcquired: 2024-03-12"
let housePlant = HousePlant(serializedHousePlant)
print(housePlant ?? "no plant")

String and StringProtocol have many built in functions to do basic scanning. In this case a split and a trim were good enough.

Move the index

The current parser works, but it could get into trouble. Imagine our starting text was instead

"commonName: spider plant /whereAcquired :Trader Joe's/dateAcquired: 2024-03-12 16:12:07"

The value contains a delimiter.

One way to accommodate strings with the delimiter character in them is to just stop at the first example of the delimiter. The quick and dirty way to do that would be to update the the current code to .split(separator: ":", maxSplits: 1)

Alternatively we can switch to moving through the string with a String.Index. An advantage being there’s no risk of throwing an index out of bounds error anymore.

    static func parse_stringIndex(_ inputString:String) throws -> Dictionary<String,String> {
        let fullKeyValueList = inputString.split(separator: "/")
        var result:Dictionary<String,String> = [:]
        return try fullKeyValueList.reduce(into: result){ result, item in
            if let firstColonIndex = item.firstIndex(of: ":") {
                let key = item
                    .prefix(upTo: firstColonIndex)
                    .trimmingCharacters(in: .whitespacesAndNewlines)
                let value = item
                    .suffix(from: item.index(firstColonIndex, offsetBy: 1))
                    .trimmingCharacters(in: .whitespacesAndNewlines)
                result[String(key)] = String(value)
            } else {
                throw HousePlantError.notAKeyValuePair
            }
        }
    }

RegEx example

I love regular expressions, although some do not. Understanding RegEx as a DSL for writing lexical analysis state machines improved how I use them. I’ve documented using them in Swift before to detect the header of a USD file (RegExBuilder version, back tracking to a literal)

SIDEBAR: To see the pros detect a header look at the Package Manger ToolsVersionParser. It’s a 685 line file.

The non-regex version works fine for this, so I don’t think I’d actually bother in a real project.

I typically start making a regex by opening https://regex101.com. The scanner will need a to capture a repeated group. The below examples were created with the /gm flags and the original test string.

Here’s the evolution process:

([^\/:]+) : works. simple. Tada. (one or more “not / or :” greedy.)(every item, could interleave.)
(?:(?:\A|\/)([^\/]+)(?:$|\/)) : Explicit “between /”, consumes the /. Doesn’t grab middle groups.
(?<=\/|^)(.*?)(?=\/|$) : look ahead and behind version switch to .*? (not greedy anything until delimiter)
(?<=\/|^)(?:(.*?):(.*?))(?=\/|$) : splits
(?:^|\A|\G)(.*?)(?:\/|$) :remove lookbehind, no split
(?:^|\A|\G)(?:(.+?):(.+?))(?:\/|$) : return the split
(?:^|\A|\G)(?:\s*(.+?)\s*:\s*(.+?)\s*)(?:\/|$) : trim white space
(?:^|\A|\G)(?:\s*(?<key>.+?)\s*:\s*(?<value>.+?)\s*)(?:\/|$) : add names

The final regex actually works for the new test string with the extra colon in it since the non-greedy indicator (the ?in (.+?)) will stop at the first : match.

static func parse_regex(_ inputString:String) throws -> Dictionary<String,String>{
    let pattern = /(?:^|\A|\G)(?:\s*(?<key>.+?)\s*:\s*(?<value>.+?)\s*)(?:\/|$)/
    var dictionary:Dictionary<String, String> = [:]
    let matches = inputString.matches(of: pattern)
    for match in matches {
        dictionary[String(match.output.key)] = String(match.output.value)
    }
    return dictionary
}

NOTE: to get this to run in a Swift Package (swift-tools-version: 5.10) with the /regex/ notation add the following to the Package.swift file.

let swiftSettings: [SwiftSetting] = [
    .enableUpcomingFeature("BareSlashRegexLiterals"),
]

for target in package.targets {
    target.swiftSettings = target.swiftSettings ?? []
    target.swiftSettings?.append(contentsOf: swiftSettings)
}

The same regex done with the RegEx Builder would look something like (generator to get started):

static func parse_regexLong(_ inputString:String) -> Dictionary<String,String>{
    let key = Reference(Substring.self)
    let value = Reference(Substring.self)
    let pattern = Regex {
        ChoiceOf {
            Anchor.startOfLine //^
            Anchor.startOfSubject //A
            Anchor.firstMatchingPositionInSubject //G
        }
        Regex {
            ZeroOrMore { CharacterClass.whitespace }
            Capture(as: key) {
                OneOrMore(.reluctant) { CharacterClass.any }
            }
            ZeroOrMore { CharacterClass.whitespace }
            ":"
            ZeroOrMore { CharacterClass.whitespace }
            Capture(as: value) {
                OneOrMore(.reluctant) { CharacterClass.any }
            }
            ZeroOrMore { CharacterClass.whitespace }
        }
        ChoiceOf {
            "/"
            Anchor.endOfLine
        }
    }
    var dictionary:Dictionary<String, String> = [:]
    let matches = inputString.matches(of: pattern)
    for match in matches {
        dictionary[String(match[key])] = String(match[value])
    }
    return dictionary
}

One can use bare slash syntax in the builder!


let pattern = Regex {
    /(?:^|\A|\G)/
    /(?:\s*(.+?)\s*:\s*(.+?)\s*)/
    /(?:\/|$)/
}

Do watch the WWDC22 Meet RegEx talk. It’s hilarious and informative.

Scanner example

A number of examples use the built in Scanner class. Scanner works like an Encoder in that you give the Scanner the string to analyze and it holds it in its own memory.

static func parse_scanner(_ inputString:String) throws -> Dictionary<String,String> {
    let scanner = Scanner(string: inputString)
    var dictionary:Dictionary<String,String> = [:]
    while !scanner.isAtEnd {
        var key = scanner.scanUpToString(":")
        key = key?.trimmingCharacters(in: .whitespacesAndNewlines)
        let _  = scanner.scanCharacter()
        var value = scanner.scanUpToString("/")
        value = value?.trimmingCharacters(in: .whitespacesAndNewlines)
        let _  = scanner.scanCharacter()
        if let key, let value {
            dictionary[key] = value
        } else {
            throw HousePlantError.notAKeyValuePair
        }
    }
    return dictionary
}

Notice I’m still trimming the whitespace. The default charactersToBeSkipped settings of the scanner will make the Scanner ignore any whitespace outside of a scan. For this string that means it will skip any “leading” white space for our keys and values (at the start of the string, immediately after the “:” or immediately after “/”). Once the scanner considers itself mid-scan it will no longer ignore the characters (discussion). That leaves all the trailing whitespace to lop off (and no .trimSuffix to do it).

Another Scanner example, this time adding the delimiters to the skipped characters:

static func parse_scanner2(_ inputString:String) throws -> Dictionary<String,String> {
    var dictionary:Dictionary<String,String> = [:]
    
    let itemDelimiter = CharacterSet(charactersIn: "/")
    let keyValueDelimiter = CharacterSet(charactersIn: ":")
    let kvDelimAndWhite = CharacterSet()
        .union(keyValueDelimiter)
        .union(.whitespacesAndNewlines)
    let allDelimAndWhite = CharacterSet()
        .union(itemDelimiter)
        .union(keyValueDelimiter)
        .union(.whitespacesAndNewlines)
    
    let scanner = Scanner(string: inputString)
    scanner.charactersToBeSkipped = allDelimAndWhite

    while !scanner.isAtEnd {
        var key = scanner.scanCharacters(from: kvDelimAndWhite.inverted)
        var value = scanner.scanCharacters(from: itemDelimiter.inverted)
        value = value?.trimmingCharacters(in: .whitespacesAndNewlines)
        
        if let key, let value {
            dictionary[key] = value
        } else {
            throw HousePlantError.notAKeyValuePair
        }
    }
    return dictionary
}

In this example a key should never have whitespace so the Scanner can run through whitespace characters as part of the delimiter search. No more trimming for the key.

Scanner + Dynamic Sequence

I once saw an interesting demo of combining a Scanner with a Dynamic Sequence (SE-0094 Review Post), which works especially well if one had multiple items to dig through. We aren’t parsing multiple HousePlants yet, so I’ll put a different example in for reference.

    func numbersInString(_ input:String) {
        print("numbersInString")
        let result = sequence(state: Scanner(string: input)) { scanner in
            let string = scanner.scanCharacters(from: .decimalDigits.inverted )
            print(string ?? "none found")
            return scanner.scanInt()
        }.map({ $0 })
        print(result)
    }
    
    func numbersInSubString(_ input:String) {
        print("numbersInSubString")
        let result = sequence(state: Scanner(string: input)) { scanner in
            scanner.scanUpToString("\n").map { subString in
                print(subString)
                return sequence(state: Scanner(string: subString)) { subScanner in
                    let string = subScanner.scanCharacters(from: .decimalDigits.inverted )
                    print(string ?? "none found")
                    return subScanner.scanInt()
                }.map({ $0 })
            }
        }.map({ $0 })
        print(result)
    }

    func testParse() {
        numbersInString("hda23hfw78hdjila2889\nhda7991hfw12hdjila9\nhufie281sufvns0938dhqqj8837")
        //[23, 78, 2889, 7991, 12, 9, 281, 938, 8837]

        numbersInSubString("hda23hfw78hdjila2889\nhda7991hfw12hdjila9\nhufie281sufvns0938dhqqj8837")
        //[[23, 78, 2889], [7991, 12, 9], [281, 938, 8837]]
    }

This style provides a chance for the Scanner to just jump to the end of the Substring when getting a value like with the String.split("/") examples. Apparently it still saves some time even though the key chars probably get scanned twice.

static func parse_subScanner(_ inputString:String) throws -> Dictionary<String,String> {
    var dictionary:Dictionary<String,String> = [:]
    
    let itemDelimiter = CharacterSet(charactersIn: "/")
    let keyValueDelimiter = CharacterSet(charactersIn: ":")
    let kvDelimAndWhite = CharacterSet()
        .union(keyValueDelimiter)
        .union(.whitespacesAndNewlines)
    let allDelimAndWhite = CharacterSet()
        .union(itemDelimiter)
        .union(keyValueDelimiter)
        .union(.whitespacesAndNewlines)
    
    let topScanner = Scanner(string: inputString)
    topScanner.charactersToBeSkipped = allDelimAndWhite
    
    sequence(state: topScanner) { topScanner in
        topScanner.scanCharacters(from: itemDelimiter.inverted).map { subString in
            let subScanner = Scanner(string: subString)
            subScanner.charactersToBeSkipped = allDelimAndWhite
            var key = subScanner.scanCharacters(from: kvDelimAndWhite.inverted)
            subScanner.charactersToBeSkipped = nil //turn off skipping!! 
            let _ = subScanner.scanCharacters(from: kvDelimAndWhite)
            let startValue:String.Index = subScanner.currentIndex
            let value:String? = subScanner.string.substring(from: startValue)
            if let key, let value {
                dictionary[key] = value
            }
        }
    }.map({ $0 })
    return dictionary
}

Hand coded scanner

Last example, walking through a String, char by char, appending to the dictionary as we go.

static func parse_charScanner(_ inputString:String) -> Dictionary<String,String>{
        var result: Dictionary<String,String> = [:]
        var isKey = true
        
        var currentKey = "".unicodeScalars
        var currentValue = "".unicodeScalars
        
        @inline(__always) func add() {
            let key = String(currentKey).trimmingCharacters(in: .whitespacesAndNewlines)
            let value = String(currentValue).trimmingCharacters(in: .whitespacesAndNewlines)
            result[key] = value
        }
        
        @inline(__always) func flush() {
            currentKey.removeAll()
            currentValue.removeAll()
            isKey = true
        }
        
        for c in inputString.unicodeScalars {
            if isKey {
                switch c {
                case ":":
                    isKey = false
                default:
                    currentKey.append(c)
                }
            } else {
                switch c {
                case "/":
                    add()
                    flush()
                default:
                    currentValue.append(c)
                }
            }
        }
        add()
        return result
    }

I did try a few different variations, including using UnsafeBytes and everything seemed slower or on par with this one.

Summary

I’ve got a handful of ways to get a Dictionary out of my “/:” formatted string. They do handle malformed Strings differently, but overall

The Scanner and the custom scanner are fastest.
The raw string regex is the most succinct, but the slowest.
The split-split with the match of one gets it done with the least rigamarole.

None of these let me handle items that won’t be a String yet. Next post… turning a String into a Date.

And what can I make a custom Encoder do?

Mon, 11 Mar 2024 00:00:00 +0000

Relevant repo: https://github.com/carlynorama/CoderExplorer/

After last post’s extensive review of options, I decided to write an Encoder along lines of the StackOverflow example in that

every item would be a String
the full codingPath would be in the key.

The implementation will be different.

The first test needed to pass?

struct TestStruct:Codable {
    let number:Int
    let text:String
    let sub:TestSubStruct
}

struct TestSubStruct:Codable {
    let numeral:Int
    let string:String
}

func testExample() throws {
    let encoder = SimpleCoder()
    
    let sub = TestSubStruct(numeral: 34, string: "world")
    let testItem = TestStruct(number: 12, text: "hello", sub: sub)
    let expected = "number:12/sub.numeral:34/sub.string:world/text:hello"
    XCTAssertEqual(expected, try encoder.encode(testItem))
    
}

“:” The delimiter between key and value
“/” The delimiter between items
“.” The delimiter between keys in nested values

Getting Started with a Keyed Container

Some initial decisions

Since not a lot of fancy is going to be happening, all of the smarts should live in the :Encoder so I don’t have to go chasing around the whole code base to find it.
I’ll avoid doing the nested-push-pop custom codingPath data type thing for now, I’ll just make new encoders. This should cause zero problems for this Encoder because every single key has its full path. We can just keep passing in the same data storage reference with no collisions or confusions.

Super basic public face of the Encoder system:

struct SimpleCoder {
    public func encode<E:Encodable>(_ value: E) throws -> String {
        let encoder = _SimpleEncoder()
        try value.encode(to: encoder)
        return encoder.value
    }
}

A Dumb-As-Rocks reference type for the data, a la the objc.io example

final class SimpleCoderData<Value> {
    var storage:Value
    
    init(_ value: Value) {
        self.storage = value
    }
}

The type that will eventually conform to Encoder. It will know how to turn the SimpleCoderData type into a String.

struct _SimpleEncoder {
    var data:SimpleCoderData<[String: String]>
    var codingPath: [CodingKey] = []
    
    //note: having lots of complicated work in a var
    //isn't the best. I'm following the API of an :Encoder
    //that didn't need to do any work to get its .value.  
    var value:String {
        var lines = data.storage.map { key, value in
            if key.isEmpty || key.contains("keyless"){
                return "\(value)"
            } else {
                return "\(key):\(value)"
            }
        }
        //To get a consistent order out since this isn't a 
        //sorted dictionary. Better for testing. 
        lines.sort()
        return lines.joined(separator: "/")
    }
}

extension _SimpleEncoder {
    init() {
        self.data = SimpleCoderData([:])
    }
}

SIDEBAR: discussion on Codable and value order re: OrderedDictionary

The _SimpleEncoder will also own adding values to the data var. EVERYTHING will need to have a key in this example so we only need one function that takes a non-optional CodingKey.

extension _SimpleEncoder {
    
    func encodeKey(key:CodingKey) -> String {
        (codingPath + [key]).map { $0.stringValue }.joined(separator: ".")
    }
    
    //to be called from containers
    func encode(_ value: String, forKey key:CodingKey) {
        data.storage[encodeKey(key: key)] = value
    }
}

Also in _SimpleEncoder, the converters for the basic types that the Encoding Containers will use. I’m going to follow the model of them all being called the same thing, but taking different value types in as parameters.

extension _SimpleEncoder {
    
    //called from the containers
    
    @inline(__always)
    func convert(_ value: some BinaryFloatingPoint) -> String {
        Double(value).description
    }
    
    @inline(__always)
    func convert(_ value: some FixedWidthInteger) throws -> String {
        guard let validatedValue = Int(exactly: value) else {
            throw EncodingError.invalidValue(value, EncodingError.Context(codingPath: codingPath, debugDescription: "Integer out of range."))
        }
        return validatedValue.description
    }
}

Now for the actual conformance - time to make up some names for the encoding containers.

extension _SimpleEncoder:Encoder {
    
    var userInfo: [CodingUserInfoKey : Any] { [:] }
    
    func container<Key>(keyedBy type: Key.Type) -> KeyedEncodingContainer<Key> where Key : CodingKey {
        KeyedEncodingContainer(SimpleEncoderKEC<Key>(encoder: self))
    }
    
    func unkeyedContainer() -> UnkeyedEncodingContainer {
        fatalError()
        //SimpleCoderUEC(encoder: self)
    }
    
    func singleValueContainer() -> SingleValueEncodingContainer {
        fatalError()
       //SimpleCoderSVEC(encoder: self)
    }
    
}

The KeyedEncodingContainer

I followed the pattern from the URIEncoder for my encoding containers, with the wrinkle that everything has to get a key. That’s not hard to envision with the KeyedContainer. We’ll get the first test case to pass by just implementing this one.

The init

struct SimpleEncoderKEC<Key: CodingKey> {
    let encoder: _SimpleEncoder
}

The custom functions

extension  SimpleEncoderKEC {
    private func _insertValue(_ converted:String, atKey key: Key) throws {
        try encoder.encode(converted, forKey: key)
    }

    private func _insertBinaryFloatingPoint(_ value: some BinaryFloatingPoint, atKey key: Key) throws {
        try _insertValue(encoder.convert(value), atKey: key)
    }

    private func _insertFixedWidthInteger(_ value: some FixedWidthInteger, atKey key: Key) throws {
        try _insertValue(encoder.convert(value), atKey: key)
    }
}

The conformance

Full disclosure, I did not write tests for all of the func encode() functions that I yanked from the URIEncoder.

extension SimpleEncoderKEC:KeyedEncodingContainerProtocol {
    var codingPath: [CodingKey] {
        encoder.codingPath
    }
    
    mutating func encodeNil(forKey key: Key) throws {
        //TODO: Decide about nils
        fatalError()
    }
    mutating func encode(_ value: Bool, forKey key: Key) throws { try _insertValue("\(value)", atKey: key) }
    mutating func encode(_ value: String, forKey key: Key) throws { try _insertValue(value, atKey: key) }
    mutating func encode(_ value: Double, forKey key: Key) throws { try _insertBinaryFloatingPoint(value, atKey: key) }
    mutating func encode(_ value: Float, forKey key: Key) throws { try _insertBinaryFloatingPoint(value, atKey: key) }
    mutating func encode(_ value: Int, forKey key: Key) throws { try _insertFixedWidthInteger(value, atKey: key) }
    mutating func encode(_ value: Int8, forKey key: Key) throws { try _insertFixedWidthInteger(value, atKey: key) }
    mutating func encode(_ value: Int16, forKey key: Key) throws { try _insertFixedWidthInteger(value, atKey: key) }
    mutating func encode(_ value: Int32, forKey key: Key) throws { try _insertFixedWidthInteger(value, atKey: key) }
    mutating func encode(_ value: Int64, forKey key: Key) throws { try _insertFixedWidthInteger(value, atKey: key) }
    mutating func encode(_ value: UInt, forKey key: Key) throws { try _insertFixedWidthInteger(value, atKey: key) }
    mutating func encode(_ value: UInt8, forKey key: Key) throws { try _insertFixedWidthInteger(value, atKey: key) }
    mutating func encode(_ value: UInt16, forKey key: Key) throws { try _insertFixedWidthInteger(value, atKey: key) }
    mutating func encode(_ value: UInt32, forKey key: Key) throws { try _insertFixedWidthInteger(value, atKey: key) }
    mutating func encode(_ value: UInt64, forKey key: Key) throws { try _insertFixedWidthInteger(value, atKey: key) }
    mutating func encode<T>(_ value: T, forKey key: Key) throws where T: Encodable {
        //notice the reference to the existing data! 
        //not using push and pop style like URIEncoder.
        var tmpEncoder = _SimpleEncoder(data: encoder.data)
        tmpEncoder.codingPath.append(key)
        try value.encode(to: tmpEncoder)
    }

    mutating func nestedContainer<NestedKey>(keyedBy keyType: NestedKey.Type, forKey key: Key)
        -> KeyedEncodingContainer<NestedKey> where NestedKey: CodingKey
    { fatalError() }

    mutating func nestedUnkeyedContainer(forKey key: Key) -> any UnkeyedEncodingContainer { fatalError() }
    mutating func superEncoder() -> any Encoder { fatalError() }
    mutating func superEncoder(forKey key: Key) -> any Encoder { fatalError() }
}

I’m super generous with the fatal errors at this point because I’ll go back in and fix them as they fire.

And this passes the first test!

UnkeyedContainer

The test

Let’s try this test that has Arrays in it.

struct MoreItems:Codable {
    let myDouble:Double
    let myFloat:Float
    let myArray:[InsideStruct]
}

struct InsideStruct:Codable {
    let insideDouble:Double
    let insideFloat:Float
    //let insideArray:[TestSubStruct]
}

func testArrayExample() throws {
    let encoder = SimpleCoder()
    
    let subItem1 = InsideStruct(insideDouble: 0.234, insideFloat: 2144.421)
    let subItem2 = InsideStruct(insideDouble: 5.926, insideFloat: 0.00132)
    let subItem3 = InsideStruct(insideDouble: 312421.4124214, insideFloat: 421421.223)
    
    let testItem = MoreItems(myDouble: 8921.41421, myFloat: 1182.12, myArray: [subItem1, subItem2, subItem3])
    let expected =  "myArray.0.insideDouble:0.234/"  +
                    "myArray.0.insideFloat:2144.4208984375/" +
                    "myArray.1.insideDouble:5.926/" +
                    "myArray.1.insideFloat:0.0013200000394135714/" +
                    "myArray.2.insideDouble:312421.4124214/" +
                    "myArray.2.insideFloat:421421.21875/" +
                    "myDouble:8921.41421/" +
                    "myFloat:1182.1199951171875"
    XCTAssertEqual(expected, try encoder.encode(testItem))
}

The init

Start off with the basic needs

struct SimpleCoderUEC {
    /// The associated encoder.
    let encoder: _SimpleEncoder
    private(set) var count: Int = 0
}

The custom functions

extension SimpleCoderUEC {

    private mutating func _appendValue(_ converted:String) throws {
        try encoder.encode(converted, forKey: nextIndexedKey())
    }
    private mutating func _appendBinaryFloatingPoint(_ value: some BinaryFloatingPoint) throws {
        try _appendValue(encoder.convert(value))
    }
    private mutating func _appendFixedWidthInteger(_ value: some FixedWidthInteger) throws {
        try _appendValue(encoder.convert(value))
    }
}

Key creation

Add what’s needed for index based keys.

extension SimpleCoderUEC {
    
    //Require a key
    private mutating func nextIndexedKey() -> CodingKey {
        let nextCodingKey = IndexedCodingKey(intValue: count)!
        count += 1
        return nextCodingKey
    }
    
    private struct IndexedCodingKey: CodingKey {
        let intValue: Int?
        let stringValue: String

        init?(intValue: Int) {
            self.intValue = intValue
            self.stringValue = intValue.description
        }

        init?(stringValue: String) {
            return nil
        }
    }
}

The conformance

Heavy on the fatal errors again.

extension SimpleCoderUEC: UnkeyedEncodingContainer {

    var codingPath: [any CodingKey] { encoder.codingPath }

    func nestedUnkeyedContainer() -> any UnkeyedEncodingContainer { fatalError() }

    mutating func nestedContainer<NestedKey>(keyedBy keyType: NestedKey.Type) -> KeyedEncodingContainer<NestedKey>
    where NestedKey: CodingKey {  fatalError()   }

    mutating func superEncoder() -> any Encoder { fatalError() }

    mutating func encodeNil() throws { fatalError() }
     
    mutating func encode(_ value: Bool) throws { try _appendValue("\(value)") }
    mutating func encode(_ value: String) throws { try _appendValue(value) }
    mutating func encode(_ value: Double) throws { try _appendBinaryFloatingPoint(value) }
    mutating func encode(_ value: Float) throws { try _appendBinaryFloatingPoint(value) }
    mutating func encode(_ value: Int) throws { try _appendFixedWidthInteger(value) }
    mutating func encode(_ value: Int8) throws { try _appendFixedWidthInteger(value) }
    mutating func encode(_ value: Int16) throws { try _appendFixedWidthInteger(value) }
    mutating func encode(_ value: Int32) throws { try _appendFixedWidthInteger(value) }
    mutating func encode(_ value: Int64) throws { try _appendFixedWidthInteger(value) }
    mutating func encode(_ value: UInt) throws { try _appendFixedWidthInteger(value) }
    mutating func encode(_ value: UInt8) throws { try _appendFixedWidthInteger(value) }
    mutating func encode(_ value: UInt16) throws { try _appendFixedWidthInteger(value) }
    mutating func encode(_ value: UInt32) throws { try _appendFixedWidthInteger(value) }
    mutating func encode(_ value: UInt64) throws { try _appendFixedWidthInteger(value) }
    mutating func encode<T>(_ value: T) throws where T: Encodable {
        //points to same data reference! 
        var tmpEncoder = _SimpleEncoder(data:encoder.data)
        tmpEncoder.codingPath = encoder.codingPath
        tmpEncoder.codingPath.append(nextIndexedKey())
        try value.encode(to: tmpEncoder)
    }
}

And that passes the test.

SingleValues

It may seem little funny to do the “simplest” case, single values, last. In this case we need to do some thinking about how to force them to have keys when they wouldn’t normally. I’m being a bit lazy and giving everything the phrase “keyless(UUID())” as a key. Using a UUID instead of the value itself means not worrying about duplicates or if the value contains the key delimiter.

The test

func testSingleValues() async throws {
    let encoder = SimpleCoder()
    
    let toEncodeInt = Int.random(in: Int.min...Int.max)
    let toEncodeText = "hello" //TODO: explore strings more
    let toEncodeBool = Bool.random()
    let toEncodeDouble = Double.random(in: Double.leastNonzeroMagnitude...Double.greatestFiniteMagnitude)
    let toEncodeFloat = Float.random(in: Float.leastNonzeroMagnitude...Float.greatestFiniteMagnitude)
    let toEncodeInt32 = Int32.random(in: Int32.min...Int32.max)
    
    let encodedInt = try encoder.encode(toEncodeInt)
    let encodedText = try encoder.encode(toEncodeText)
    let encodedBool = try encoder.encode(toEncodeBool)
    
    let encodedDouble = try encoder.encode(toEncodeDouble)
    let encodedFloat = try encoder.encode(toEncodeFloat)
    let encodedInt32 = try encoder.encode(toEncodeInt32)
    
    XCTAssertEqual(toEncodeInt.description, encodedInt)
    XCTAssertEqual(toEncodeText.description, encodedText)
    XCTAssertEqual(toEncodeBool.description, encodedBool)
    XCTAssertEqual(toEncodeDouble.description, encodedDouble)
    XCTAssertEqual(Double(toEncodeFloat).description, encodedFloat)
    XCTAssertEqual(toEncodeInt32.description, encodedInt32)
}

The init

struct SimpleCoderSVEC {
    let encoder: _SimpleEncoder
}

The custom functions

extension SimpleCoderSVEC {
    private func _setValue(_ converted:String) throws { 
        try encoder.encode(converted, forKey: SVECCodingKey(converted)) 
    }

    private func _setBinaryFloatingPoint(_ value: some BinaryFloatingPoint) throws {
        try _setValue(encoder.convert(value))
    }

    private func _setFixedWidthInteger(_ value: some FixedWidthInteger) throws {
        try _setValue(encoder.convert(value))
    }
}

Key creation

extension SimpleCoderSVEC {
    
    private struct SVECCodingKey: CodingKey {
        let intValue: Int?
        let stringValue: String

        init?(intValue: Int) {
            return nil
        }

        init?(stringValue: String) {
            self.stringValue = stringValue
            self.intValue = nil
        }
        
        init(_ forValue:String) {
            self.stringValue = "keyless\(UUID())"
            self.intValue = nil
        }
    }
}

The conformance

extension SimpleCoderSVEC: SingleValueEncodingContainer {

    var codingPath: [any CodingKey] { encoder.codingPath }

    func encodeNil() throws {
        fatalError()
    }
    
    
    func encode(_ value: Bool) throws { try _setValue("\(value)") }
    func encode(_ value: String) throws { try _setValue(value) }
    func encode(_ value: Double) throws { try _setBinaryFloatingPoint(value) }
    func encode(_ value: Float) throws { try _setBinaryFloatingPoint(value) }
    func encode(_ value: Int) throws { try _setFixedWidthInteger(value) }
    func encode(_ value: Int8) throws { try _setFixedWidthInteger(value) }
    func encode(_ value: Int16) throws { try _setFixedWidthInteger(value) }
    func encode(_ value: Int32) throws { try _setFixedWidthInteger(value) }
    func encode(_ value: Int64) throws { try _setFixedWidthInteger(value) }
    func encode(_ value: UInt) throws { try _setFixedWidthInteger(value) }
    func encode(_ value: UInt8) throws { try _setFixedWidthInteger(value) }
    func encode(_ value: UInt16) throws { try _setFixedWidthInteger(value) }
    func encode(_ value: UInt32) throws { try _setFixedWidthInteger(value) }
    func encode(_ value: UInt64) throws { try _setFixedWidthInteger(value) }
    func encode<T>(_ value: T) throws where T: Encodable {
        try value.encode(to: encoder)
    }
}

And again the test passes!

Fixing the Dates

Before going on to tackle more complicated data structures I want to sort out a few value types that don’t get encoded the way I want by default. Date, URL and Data. Starting with Date, this is the test to pass.

func testDate() throws {
    let encoder = SimpleCoder()
    let date = Date()

    let dateString = date.ISO8601Format(.iso8601)        
    let encodedDate = try encoder.encode(date)       
    XCTAssertEqual(dateString, encodedDate)
}

It doesn’t out of the gate. It fails with the error:

XCTAssertEqual failed: ("2024-03-07T14:55:56Z") is not equal to ("731516156.037366")

The one below does though, which shows us that the default Encoding for Date’s Codable implementation (core | FoundationEssential) is the timeIntervalSinceReferenceDate.

func testDate() throws {
    let encoder = SimpleCoder()
    let date = Date()

    let dateString = "\(date.timeIntervalSinceReferenceDate)"
    let encodedDate = try encoder.encode(date)
    XCTAssertEqual(dateString, encodedDate)
}

How to fix it? First off, the encoder has to catch that a Date has come into play. We’ll catch it in the func encode<T>(_ value: T) functions in all three encoding containers by adding a case statement to each.

mutating func encode<T>(_ value: T, 
                        forKey key: Key
                       ) throws where T: Encodable {
    switch value {
    case let value as Date: fatalError()
    default:
        //points to same data reference! 
        var tmpEncoder = _SimpleEncoder(data:encoder.data)
        tmpEncoder.codingPath.append(key)
        try value.encode(to: tmpEncoder)
    }
}

If we run the test again the Date still doesn’t get snagged by the error. It will be easier to hit inside the Keyed and Unkeyed containers first.

I got the Keyed and Unkeyed encoding containers working by adding the relevant functions, starting with a convert function like there is for Float and Int in the :Encoder.

//----  In :Encoder
func convert(_ value:Date) -> String {
    return value.ISO8601Format()
}

Then adding the wrapper functions to the encoding containers and updating the case statements

//----  In :KeyedEncodingContainerProtocol
private func _insertDate(_ value: Date, atKey key:Key) throws {
    try _insertValue(encoder.convert(value), atKey: key)
}
mutating func encode<T>(_ value: T, forKey key: Key) throws where T: Encodable {
    switch value {
    //...
    case let value as Date: try _insertDate(value, atKey: key)
    //...
    }
}

//----  In :UnkeyedEncodingContainer
private mutating func _appendDate(_ value:Date) throws {
    try _appendValue(encoder.convert(value))
}
mutating func encode<T>(_ value: T, forKey key: Key) throws where T: Encodable {
    switch value {
    //...
    case let value as Date: try _appendDate(value)
    //...
    }
}

Add Keyed and Unkeyed containers to the test:

func testDate() throws {
    let encoder = SimpleCoder()
    let date = Date()
    

    //--------- SingleValue (STILL FAILS)
    let dateString = date.ISO8601Format(.iso8601)
    let encodedDate = try encoder.encode(date)
    XCTAssertEqual(dateString, encodedDate)
    
    //--------- Keyed (PASSES)
    struct MiniWithDate:Encodable {
        let date:Date = Date()
    }
    
    let miniToTest = MiniWithDate()
    let structExpected = "date:\(miniToTest.date.ISO8601Format())"
    let encodedStruct = try encoder.encode(miniToTest)
    XCTAssertEqual(structExpected, encodedStruct)
    
    //--------- Unkeyed (PASSES)
    let dateArray = [Date(), Date(), Date(), Date()]
    let arrayExpected = dateArray.enumerated().map ({ index, value in
        "\(index):\(value.ISO8601Format())"
    }).joined(separator: "/")
    let encodedArray = try encoder.encode(dateArray)
    XCTAssertEqual(arrayExpected, encodedArray)
}

The Keyed and Unkeyed containers work and the SingleValue encoding container doesn’t hit the error! What gives?

The Date keeps prioritizing its defaults (core | FoundationEssential). I mentioned date handling in the last post, that the URIEncoder & the JSONEncoders and even the XML encoder all have to catch a stand alone Date value in their encodeValue/wrapGeneric/box functions. It was good to see the need for that in action.

Instead of a routing function inside the _SimpleCoder:Encoder, I’ll put an array of types that I’ve made special handlers for up in the public SimpleCoder. If my public function gets passed a value whose type is in that array, that’s when it will go to the special handler function inside the :Encoder. Since SimpleCoder is a learning exercise I want to contrast right up top why one might see the basic value.encode(to: encoder) pattern (no overrides) vs. something more like encoder.encode(for: value) -> some Output (overrides inside).

//----  Updated
struct SimpleCoder {
    let flaggedTypes:[Encodable.Type] = [Date.self]
    public func encode<E:Encodable>(_ value: E) throws -> String {
        let encoder = _SimpleEncoder()
        if flaggedTypes.contains(where: { $0 == E.self }) {
            try encoder.specialEncoder(for: value)
        } else {
            try value.encode(to: encoder)
        }
        return encoder.value
    }
}

//----  Added
extension _SimpleEncoder {
    func specialEncoder(for value: some Encodable) throws {
        var container = singleValueContainer()
        try container.encode(value)
    }
}

//----  In the :SingleValueEncodingContainer
private func _setDate(_ value:Date) throws {
    try _setValue(encoder.convert(value))
}
mutating func encode<T>(_ value: T, forKey key: Key) throws where T: Encodable {
    switch value {
    //...
    case let value as Date: try _setDate(value)
    //...
    }
}

The Date tests now all pass.

Fixing URLs

I’m also going to need URLs for the Lines data and URLs have an excessively complete default format for my needs. What do I mean? They come out looking like:

//singleValue
 "relative:http://www.example.com"
 //part of Struct
 "url.relative:http://www.example.com"
 //part of Array
 "0.relative:http://www.example.com/1.relative:http://www.example.com?testQuery=42/2.relative:http://www.example.com/3.relative:http://www.example.com"

When a URL gets created from any of the following possible init paths, it actually creates a key:value pair of the String “relative” and the URL’s relativeString. See a discussion

URL(string:"http://www.example.com")
FileManager.default.homeDirectoryForCurrentUser
var components = URLComponents(); components.scheme = "http"; components.host = "www.example.com"; components.queryItems = [URLQueryItem(name: "testQuery", value: "42")]
URL(dataRepresentation: "http://www.example.com".data(using: .utf8)!, relativeTo: nil, isAbsolute: true)

Why is that? Well, lets look at the Codable implementation for URL:

extension URL : Codable {
    private enum CodingKeys : Int, CodingKey {
        case base
        case relative
    }

    public init(from decoder: Decoder) throws {
        let container = try decoder.container(keyedBy: CodingKeys.self)
        let relative = try container.decode(String.self, forKey: .relative)
        let base = try container.decodeIfPresent(URL.self, forKey: .base)

        guard let url = URL(string: relative, relativeTo: base) else {
            throw DecodingError.dataCorrupted(
                                   DecodingError.Context(codingPath: decoder.codingPath,
                                                         debugDescription: "Invalid URL string.")
                                )
        }

        self = url
    }

    public func encode(to encoder: Encoder) throws {
        var container = encoder.container(keyedBy: CodingKeys.self)
        try container.encode(self.relativeString, forKey: .relative)
        if let base = self.baseURL {
            try container.encode(base, forKey: .base)
        }
    }
}

A complete URL actually has two pieces of information. Its path and whether it’s relative to a different URL. If one declares a URL like:

let relativeURL = URL(fileURLWithPath: "hello.jpg", relativeTo: URL(string:"http://www.example.com"))

It does in fact produce the complete set when encoded:

"relative:hello.jpg/base.relative:http://www.example.com"

For saving URLs to the Lines data I just want the URL’s absoluteString, so I’ll need another convert function.

struct SimpleCoder {
    let flaggedTypes:[Encodable.Type] = [Date.self, URL.self]
    public func encode<E:Encodable>(_ value: E) throws -> String {
        let encoder = _SimpleEncoder()
        if flaggedTypes.contains(where: { $0 == E.self}) {
            try encoder.specialEncoder(for: value)
        } else {
            try value.encode(to: encoder)
        }
        return encoder.value
    }
}

//----  In :Encoder
func convert(_ value:URL) -> String {
    return value.absoluteString()
}

//---- Also in all 3 encoding containers... Keyed example.  
private func _insertURL(_ value: URL, atKey key:Key) throws {
    try _insertValue(encoder.convert(value), atKey: key)
}  
mutating func encode<T>(_ value: T, forKey key: Key) throws where T: Encodable {
    switch value {
    //...
    case let value as URL: try _insertValue(value, forKey:Key)
    //...
    }
}

And again the tests work!

func testURL() throws {
    let encoder = SimpleCoder()

    //------------------ mostly HTTP Schema
    let http_url = URL(string: "http://www.example.com")!
    var components = URLComponents()
    components.scheme = "http"
    components.host = "www.example.com"
    components.queryItems = [URLQueryItem(name: "testQuery", value: "42")]
    let file_url = FileManager.default.homeDirectoryForCurrentUser
    
    let absoluteURL = URL(dataRepresentation: "http://www.example.com".data(using: .utf8)!, 
                            relativeTo: nil, isAbsolute: true)!
    
    let relativeURL = URL(fileURLWithPath: "hello.jpg", 
                            relativeTo: URL(string:"http://www.example.com"))
    
    //does not even touch my encoder
    let httpUrlString = http_url.relativeString //.absoluteString
    let httpEncodedURL = try encoder.encode(http_url)
    XCTAssertEqual(httpUrlString, httpEncodedURL)
    
    //--------- Keyed
    struct MiniWithHttpURL:Encodable {
        let url:URL = URL(string: "http://www.example.com")!
    }
    
    let httpMiniToTest = MiniWithHttpURL()
    let httpStructExpected = "url:\(httpMiniToTest.url.absoluteString)"
    let httpEncodedStruct = try encoder.encode(httpMiniToTest)
    XCTAssertEqual(httpStructExpected, httpEncodedStruct)
    
    //--------- Unkeyed
    let urlArray = [http_url, components.url!, absoluteURL, relativeURL, file_url]
    let httpArrayExpected = urlArray.enumerated().map ({ index, value in
        "\(index):\(value.absoluteString)"
    }).joined(separator: "/")
    let encodedURLArray = try encoder.encode(urlArray)
    XCTAssertEqual(httpArrayExpected, encodedURLArray)
}

Fixing Data

Failing initial test:

//FAILS
//XCTAssertEqual failed: ("Cg8U") is not equal to ("0:10/1:15/2:20")
func testData() throws {
    let encoder = SimpleCoder()
    let data = Data([10, 15, 20])
    let dataString = data.base64EncodedString()
    let encodedData = try encoder.encode(data)
    XCTAssertEqual(dataString, encodedData)
    
}

New convert:

@inline(__always)
func convert(_ value:Data) throws -> String {
    return value.base64EncodedString()
}

Do the same song and dance in the three encoding containers done for URL and Date
add Data.self to flaggedTypes

And again passing complete test.

func testData() throws {
    let encoder = SimpleCoder()
    
    //------------------ SingleValue
    let data = Data([10, 15, 20])
    let dataString = data.base64EncodedString()
    let encodedData = try encoder.encode(data)
    XCTAssertEqual(dataString, encodedData)
    
    //--------- Keyed
    struct MiniWithData:Encodable {
        let myData:Data = Data([10, 15, 20, 127, 0])
    }
    
    let miniToTest = MiniWithData()
    let structExpected = "myData:Cg8UfwA="
    let encodedStruct = try encoder.encode(miniToTest)
    XCTAssertEqual(structExpected, encodedStruct)
    
    //--------- Unkeyed
    let dataArray = [data, data, data, data]
    let structArray = [miniToTest, miniToTest]
    let arrayExpected = dataArray.enumerated().map ({ index, value in
        "\(index):\(value.base64EncodedString())"
    }).joined(separator: "/")
    let encodedArray = try encoder.encode(dataArray)
    XCTAssertEqual(arrayExpected, encodedArray)
    
    let structArrayExpected = "0.myData:Cg8UfwA=/1.myData:Cg8UfwA="
    let encodedStructArray = try encoder.encode(structArray)
    XCTAssertEqual(structArrayExpected, encodedStructArray)     
}

Not hard to pull off, but one can see that having to go in to every encoding container to add every little new custom override would get annoying so when I do the LineCoder I’ll handle it a little differently.

What to do about Optionals?

There are three ways to handle Optionals in an Encoder

add a specific marker for null values (put something in encodeNil(forKey key: Key) func)
silently ignore (put nothing in the encodeNil(forKey key: Key) func)
add encodeIfPresent() versions of the functions
- see DateComponents’ codable conformance for an encoder using them.
- more info since I don’t go this route: https://forums.swift.org/search?q=encodeIfPresent

And one can choose differently per container. Maybe a place holder makes sense for SingleValue situations, but Keyed null values can just be ignored. That’s the route I picked for SimpleCoder. It causes some interesting trouble for enums with optional associated values in arrays, but that’s for fixing another day.

//In KeyedEncodingContainer
    mutating func encodeNil(forKey key: Key) throws {
        //do nothing. 
        //try _insertValue("NULL", atKey: key)
    }
//In SingleValueEncodingContainer
    func encodeNil() throws { try _setValue("NULL") }

//In UnkeyedEncodingContainer
//That's right. NOTHING in the test calls this. 
//Not even the nested Optional arrays. 
   mutating func encodeNil() throws {
        fatalError()
        //try _appendValue("NULL")
    }

func testBasicNil() throws {
        let encoder = SimpleCoder()
        
        //------------------ SingleValue
        let optionalValue:Int? = nil
        let optionalString = "NULL"
        let encodedOptional = try encoder.encode(optionalValue)
        XCTAssertEqual(optionalString, encodedOptional)
        
        //--------- Keyed
        struct MiniStruct:Encodable {
            let noneInt:Int? = nil
            let someInt:Int? = 12
        }
        
        let miniToTest = MiniStruct()
        let structExpected = "someInt:12"
        let encodedStruct = try encoder.encode(miniToTest)
        XCTAssertEqual(structExpected, encodedStruct)
        
        
        func renderArray(_ array:[Int?], prepend:String = "") -> String {
            array.enumerated().map ({ index, value in
                let valueString = value != nil ? "\(value!)" : "NULL"
                return "\(prepend)\(index):\(valueString)"
            }).joined(separator: "/")
        }
        
        //--------- Unkeyed
        
        let array = [10, nil, 20, nil, 0]
        let arrayExpected = renderArray(array)
        let encodedArray = try encoder.encode(array)
        XCTAssertEqual(arrayExpected, encodedArray)
        
        let subArray:[[Int?]?] = [array, nil, array]
        let subArrayExpected = subArray.enumerated().map({
            if let notNil = $1 {
                return renderArray(notNil, prepend: "\($0).")
            } else {
                return "\($0):NULL"
            }
            }).joined(separator: "/")
        let encodedSubArray = try encoder.encode(subArray)
        XCTAssertEqual(subArrayExpected, encodedSubArray)
        
        struct MiniArrayStruct:Encodable {
            let myIntArray:[Int?] = [10, nil, 20, nil, 0]
        }
        let miniArratStructToTest = MiniArrayStruct()
        let structArray = [miniArratStructToTest, miniArratStructToTest]
        let structArrayExpected = structArray.enumerated().map ({ index, value in
            renderArray(value.myIntArray, prepend: "\(index).myIntArray.")
        }).joined(separator: "/")
        let encodedStructArray = try encoder.encode(structArray)
        XCTAssertEqual(structArrayExpected, encodedStructArray)   
    }

Adding enums

RawRepresentable

When getting started with enum’s it’s very useful to dive into the actual original swift-evolution proposal for what was called “swift-archival-serialization” and jump to the place where it shows what the synthesized conformance for enums would be.

    public func encode(to encoder: Encoder) throws {
        // Encode as a single value; no keys.
        try encoder.singleValueContainer().encode(self.rawValue)
    }

An enum always gets pushed through to a singleValueContainer.

And if we test this with a RawRepresentable conforming enum with a RawValue of String we see it works perfectly:

    func testEnum() throws {
        let encoder = SimpleCoder()

        enum Greeting:String,Codable {
            case hello, howdy, hola, hi, hiya
        }
        
        //------------------ SingleValue
        let enumValue:Greeting = .hiya
        let expectedEnumString = "hiya"
        let encodedEnum = try encoder.encode(enumValue)
        XCTAssertEqual(expectedEnumString, encodedEnum)
    }

but add an enum to a struct or array…

        struct MiniWithEnum:Encodable {
            let otherValue:Double = 42
            let myGreeting:Greeting = .hello
            let otherStringValue = "world"
        }
        
        let miniToTest = MiniWithEnum()
        let structExpected = "myGreeting:\(miniToTest.myGreeting)/otherStringValue:world/otherValue:42.0"
        let encodedStruct = try encoder.encode(miniToTest)
        XCTAssertEqual(structExpected, encodedStruct)

        //--------- Unkeyed
        let enumArray:[Greeting] = [.hello, .howdy, .hiya, .hola]
        let arrayExpected = enumArray.enumerated().map ({ index, value in
            "\(index):\(value)"
        }).joined(separator: "/")
        let encodedArray = try encoder.encode(enumArray)
        XCTAssertEqual(arrayExpected, encodedArray)

And we get failures like:

XCTAssertEqual failed: ("myGreeting:hello/otherStringValue:world/otherValue:42.0") is not equal to ("hello/otherStringValue:world/otherValue:42.0")

What happened? Well, even though I’m in a keyed container, my default behavior for anything my encoder doesn’t recognize is to let it do its own thing. For an enum that means it’s going through a SingleValueEncodingContainer (See this discussion). In the data store it does have a key, but it’s myGreeting.keyless(some UUID) which gets stripped out in our serialization step because it contains the text “keyless”.

I have some choices:

figure out a way to intercept all enums
write custom codable conformance for any enums to use with this encoder
Fix the Dictionary -> String step

I’m going to update the Dictionary -> String creation code because it will be easier than rerouting all enums or guaranteeing custom Codable implementations. Also my data store really relies on those unique keys so I can’t change the key-encoding function either.

    var value:String {
        var lines = data.storage.map { key, value in
            var mKey = key
            
            if mKey.contains("keyless") {
                mKey = cleanKey(mKey)
            }
            
            if mKey.isEmpty {
              return "\(value)"
            } else {
              return "\(mKey):\(value)"
            }
        }
        lines.sort()
        return lines.joined(separator: "/")
    }
    
    func cleanKey(_ key:String, keyDelimiter:String.Element = ".") -> String {
        //potential alternative approach:
        //just trim from the end until the last delimiter. 
        //TBD if there is a case where that would make a difference since the SVEC 
        //create a nested container. 
        let split = key.split(whereSeparator: { $0 == keyDelimiter }).filter({ !$0.contains("keyless")})
        return split.joined(separator: String(keyDelimiter))
    }

And now it all works.

Unspecified Enum

What happens when the enum isn’t RawRepresentable with a RawValue of String or Int?

    func testPlainEnum() throws {
        let encoder = SimpleCoder()

        enum FruitChoice:Codable {
            case strawberry, pineapple, dragon, kiwi, kumquat
        }
        
        //------------------ SingleValue
        let enumFruitValue:FruitChoice = .strawberry
        let expectedEnumFruitString = "strawberry"
        let encodedFruitEnum = try encoder.encode(enumFruitValue)
        XCTAssertEqual(expectedEnumFruitString, encodedFruitEnum) 
    }

Whelp. It catches one of the remaining fatalErrors() in the SimpleEncoderKEC at

    mutating func nestedContainer<NestedKey>(keyedBy keyType: NestedKey.Type, forKey key: Key)
        -> KeyedEncodingContainer<NestedKey> where NestedKey: CodingKey
    {
        fatalError()
    }

replacing that with

    mutating func nestedContainer<NestedKey>(keyedBy keyType: NestedKey.Type, forKey key: Key)
        -> KeyedEncodingContainer<NestedKey> where NestedKey: CodingKey
    {
        //fatalError()
        print("nestedKey:\(key.stringValue)")
        return encoder.container(keyedBy: NestedKey.self)
        
    }

One gets the TestFailure, AND one gets to see the reason for it; the enum values are coming through as the keys this time.

Test Case 
'-[CoderExplorerTests.CodeableStringExample testPlainEnum]' started.
nestedKey:strawberry
CoderExplorer/Tests/CoderExplorerTests/SimpleCoderTests.swift:251: 
error: -[CoderExplorerTests.CodableStringExample testPlainEnum] :
XCTAssertEqual failed: ("strawberry") is not equal to ("")

If I sent this enum to the JSONEncoder, it could handle it. The value would be an empty object.

enum FruitChoice:Encodable {
    case strawberry, pineapple, dragon, kiwi, kumquat
}
let JSONencoder = JSONEncoder()
let encodedData = try JSONencoder.encode(FruitChoice.strawberry)
let string = String(bytes: encodedData, encoding: .utf8) ?? ""
//string == {"strawberry":{}}

It would actually take some significant doing to catch an empty object in this particular encoder, so I’m going to show “fixing” the enum instead. These enums will no longer call the nestedContainer<NestedKey>(keyedBy:forKey:) function because the the custom encode(to:) function (as opposed to the synthesized one) knows the enum is flat and has no associated values.

Option 1: Stripping the key entirely

The right thing to do would be to make it RawRepresentable as a String as seen above, but if for some reason you can’t or want to have more control over the returned string:

enum FruitChoice:Encodable {
    case strawberry, pineapple, dragon, kiwi, kumquat
    
    public func encode(to encoder: Encoder) throws {
        var container = encoder.singleValueContainer()
        try container.encode("\(self)")
    }
}

Option 2: Forcing a key:value pair

Maybe instead the enum name would be useful to have in the data for some reason. The enum can return a dictionary.

enum FruitChoice:Encodable {
    case strawberry, pineapple, dragon, kiwi, kumquat
    
    public func encode(to encoder: Encoder) throws {
        //really a dictionary, but JSON terminology is leaking in
        let asObject = ["FruitChoice":"\(self)"]
        var container = encoder.singleValueContainer()
        let container = 
        try container.encode(asObject)
    }
}

Option 3: updating the CodingKeys (the “right way”)

Strictly more correct than the above method?

enum CodingKeys:CodingKey {
    case fruitChoice
}

public func encode(to encoder: Encoder) throws {
    var container = encoder.container(keyedBy: Self.CodingKeys.self)
    try container.encode("\(self)", forKey: .fruitChoice)
}

Associated Values

Newish to Codable is “Codable Synthesis for Enums with Associated Values”

Proposal
- on GitHub
- on Forum

Adding a mix of associated values to FruitChoice, what one will get is position based synthesized keys for the sub-object Swift thinks the associated value is part of.

enum FruitChoice:Encodable {
    case strawberry(String, Int), 
    case pineapple(Int), 
    case dragon(Double), 
    case kiwi(UInt8), 
    case kumquat(Date)
}
let JSONencoder = JSONEncoder()
let encodedData = try JSONencoder.encode(FruitChoice.strawberry("Everbearing", 5))
let string = String(bytes: encodedData, encoding: .utf8) ?? ""
//"{"strawberry":{"_0":"Everbearing","_1":5}}

One can name them inline:

enum FruitChoice:Encodable {
    case strawberry(name:String, cropCount:Int), 
    case pineapple(yearsToFruit:Int), 
    case dragon(metersTall:Double), 
    case kiwi(seedsIn1mmSlice:UInt8), 
    case kumquat(whenAquired:Date)
}

Or override the whole synthesized Codable conformance:

enum FruitChoice:Encodable {
    case strawberry(String, Int), pineapple(Double), kumquat(Date)
    
    enum CodingKeys:CodingKey {
        case strawberry
        case pineapple
        case kumquat
    }
    
     enum StrawberryCodingKeys: CodingKey {
        case name
        case cropCount
      }
    
    enum PineappleCodingKeys: CodingKey {
       case yearsToFruit
     }
    
    enum KumquatCodingKeys: CodingKey {
        case dateAquired
    }

    public func encode(to encoder: Encoder) throws {
      var container = encoder.container(keyedBy: CodingKeys.self)
      switch self {
          
      case .strawberry(let string, let int): 
          var nestedContainer = container.nestedContainer(
                                                keyedBy: StrawberryCodingKeys.self, 
                                                forKey: .strawberry)
          try nestedContainer.encode(string, forKey: .name)
          try nestedContainer.encode(int, forKey: .cropCount)
      case .pineapple(let double): 
          var nestedContainer = container.nestedContainer(
                                                keyedBy: PineappleCodingKeys.self, 
                                                forKey: .pineapple)
          try nestedContainer.encode(double, forKey: .yearsToFruit)
      case .kumquat(let date):
          var nestedContainer = container.nestedContainer(
                                                keyedBy: KumquatCodingKeys.self, 
                                                forKey: .kumquat)
          try nestedContainer.encode(date, forKey: .dateAquired)
      }
    }
}

I include that mostly to show how the nestedContainer call comes into play. Now that I’ll be needing that function again I have to fix it to work correctly.

    mutating func nestedContainer<NestedKey>(keyedBy keyType: NestedKey.Type, forKey key: Key)
        -> KeyedEncodingContainer<NestedKey> where NestedKey: CodingKey
    {
        //have to reversibly push the key onto the stack. 
        var tmpEncoder = _SimpleEncoder(data:encoder.data)
        tmpEncoder.codingPath = encoder.codingPath
        tmpEncoder.codingPath.append(key)
        return tmpEncoder.container(keyedBy: NestedKey.self)
    }

I’ll add that to all the nestedContainer(keyType:forKey:) calls. I’m creating a new _SimpleCoder in at least 5 places and counting now so I’ll make a function to do it:

    // in _SimpleCoder
    typealias DataStore = SimpleCoderData<[String: String]>
    func getEncoder(forKey key:CodingKey?, withData passedInData:DataStore) -> Self {
        var tmp = _SimpleEncoder(data:passedInData)
        tmp.codingPath = self.codingPath
        if let key {
            tmp.codingPath.append(key)
        }
        return tmp
    }

In _SimpleCoder the data storage never gets wiped clean so I could just use self.data in the function without passing it through, but I’d like to make it clear at the call location that the data matters for my own future reference.

For more about nested containers see also

The tests pass!

    func testAssociateValueEnum() throws {
        enum FruitChoice:Encodable {
            case strawberry(name:String, cropCount:Int), 
            case pineapple(yearsToFruit:Double), 
            case kumquat(dateAcquired:Date)
        }
        
        //------------------ SingleValue
        let enumFruitValue:FruitChoice = .strawberry(name:"Everbearing", cropCount:6)
        let expectedEnumFruitString = "strawberry.cropCount:6/strawberry.name:Everbearing"
        let encodedFruitEnum = try encoder.encode(enumFruitValue)
        XCTAssertEqual(expectedEnumFruitString, encodedFruitEnum)
            
        //--------- Keyed
        struct MiniWithEnum:Encodable {
            let otherValue:Double = 42
            let myFruit:FruitChoice = .pineapple(yearsToFruit:3.12)
            let otherStringValue = "world"
        }
        
        let miniToTest = MiniWithEnum()
        let structExpected = "myFruit.pineapple.yearsToFruit:3.12/otherStringValue:world/otherValue:42.0"
        let encodedStruct = try encoder.encode(miniToTest)
        XCTAssertEqual(structExpected, encodedStruct)
        
        //--------- Unkeyed
        let enumArray:[FruitChoice] = [.strawberry(name:"Everbearing", cropCount:6),
                                       .pineapple(yearsToFruit:1.23343), 
                                       .kumquat(dateAcquired:try Date("2024-03-10T21:36:24Z", 
                                                                      strategy: .iso8601))]

//        let arrayExpected = enumArray.enumerated().map ({ index, value in
//            "\(index).fruitChoice:\(value)"
//        }).joined(separator: "/")
        let arrayExpected = "0.strawberry.cropCount:6/"+
        "0.strawberry.name:Everbearing/"+
        "1.pineapple.yearsToFruit:1.23343/"+
        "2.kumquat.dateAquired:2024-03-10T21:36:24Z"
        let encodedArray = try encoder.encode(enumArray)
        XCTAssertEqual(arrayExpected, encodedArray)
        
    }

Adding a Dictionary

Using dictionaries that have an Int or String as a key and some Encodable as their value works like a charm. I’m not going to spend any time on them except to mention, yes, I wrote a test.

One can get other types to work as codable dictionary keys by conforming them to CodingKeyRepresentable

Many of the fixed-width int types conform automatically, but not all of them do or should

A passing test

func testSomeEncodableKeyDictionary() throws {
   
   enum FruitKey:String, Codable, CodingKeyRepresentable {
       case strawberry, banana, kiwi
   }

   let dictionaryA:Dictionary<FruitKey,UInt32> = [
       .strawberry:1,
       .banana:2,
       .kiwi: 3
   ]
   
   struct FlowerCodingKey: CodingKey {
       let stringValue: String
       var intValue: Int?
       
       init?(stringValue: String) {  self.stringValue = stringValue  }
       init?(intValue: Int) {
           self.stringValue = String(intValue)
           self.intValue = intValue
       }
       init(_ flower:Flower) {
           self.stringValue = flower.name
       }
   }
   
   //TODO: Codable shouldn't be necessary but my Encoder requested it?
   struct Flower:Hashable, Codable, CodingKeyRepresentable {
       let name:String
       init(_ name:String) {
           self.name = name
       }
       
       var codingKey: any CodingKey {
           FlowerCodingKey.init(self)
       }
       
       init?<T>(codingKey: T) where T : CodingKey {
           self.name = codingKey.stringValue
       }
   }

   let dictionaryB:Dictionary<Flower,Int> = [
       Flower("anemone"):7,
       Flower("freesia"):6,
       Flower("periwinkle"):5
   ]

   //------------------ SingleValue
   let expectedA = "banana:2/kiwi:3/strawberry:1"
   let expectedB = "anemone:7/freesia:6/periwinkle:5"
   let encodedA = try encoder.encode(dictionaryA)
   let encodedB = try encoder.encode(dictionaryB)
   XCTAssertEqual(expectedA, encodedA)
   XCTAssertEqual(expectedB, encodedB)

   //--------- Keyed
   struct MiniWithDict:Encodable {
       let first:Dictionary<FruitKey,UInt32>
       let second:Dictionary<Flower,Int>
   }

   let miniToTest = MiniWithDict(first: dictionaryA, second: dictionaryB)
   let structExpected = "first.banana:2/first.kiwi:3/first.strawberry:1/second.anemone:7/second.freesia:6/second.periwinkle:5"
   let encodedStruct = try encoder.encode(miniToTest)
   XCTAssertEqual(structExpected, encodedStruct)

   //--------- Unkeyed
   let dictArray = [dictionaryB, dictionaryB]
   let arrayExpected = "0.anemone:7/0.freesia:6/0.periwinkle:5/1.anemone:7/1.freesia:6/1.periwinkle:5"
   let encodedArray = try encoder.encode(dictArray)
   XCTAssertEqual(arrayExpected, encodedArray)
}

If one DOESN’T make the key CodingKeyRepresentable, one ends up with a interlaced output that the decoder will have to know to stitch back together.

func testUnofficialKeyDictionary() throws {
   struct MyKey:Hashable, Codable {
       var id:UInt8
   }
   
   let dictionaryA:Dictionary<MyKey, Int> = [
       MyKey(id: 127):1,
       MyKey(id: 126):2,
       MyKey(id: 125):3
   ]
   
   //NOTE: This test will only intermittently pass
   //because dictionary order is arbitrary. 
   //will be easier to test when can round trip with
   //decoder.
   let expectedA = "0.id:125/" +
                   "1:3/" +
                   "2.id:126/" +
                   "3:2/" +
                   "4.id:127/" +
                   "5:1"
   let encodedA = try encoder.encode(dictionaryA)
   //.sorted(by: { $0.key < $1.key }) not Encodable
   XCTAssertEqual(expectedA, encodedA)
}

What’s the deal with superEncoder()?

.superEncoder() gets called when a subclass wants to use it’s parent class’ encoder.

I did a couple of tests using my own class/subclass and one with AttributedString which uses a lot of .base64EncodedString() data. They’re not exactly super rigorous (why I’ve only added links), but it got me nominal coverage for:

from Mine: KeyedEncodingContainer.superEncoder()
from AttributedString: KeyedEncodingContainer.superEncoder(forKey key: Key)
from AttributedString: UnkeyedEncodingContainer.superEncoder()
Note about super and keys: https://forums.swift.org/t/codable-with-references/13885/20

The remaining fatalError()s

Nesting and unkeyed containers didn’t happen in either direction in my tests. Even where I expected them to. I could write my own custom objects that call them just for testing, but I’d rather wait until I understand who/what uses them in the wild for what.

KeyedEncodingContainerProtocol.nestedUnkeyedContainer(forKey key: Key)
UnkeyedEncodingContainer.nestedUnkeyedContainer()
UnkeyedEncodingContainer.nestedContainer(keyedBy keyType: NestedKey.Type)
UnkeyedEncodingContainer.encodeNil

More research:

https://forums.swift.org/search?q=UnkeyedEncodingContainer
https://github.com/search?q=encoder.unkeyedContainer%28%29&type=code
confirmation they aren’t common: https://forums.swift.org/t/codable-with-references/13885/17

A note about userInfo

Most encoder implementations I’ve seen essentially ignore the protocol defined userInfo dictionary, but the idea behind it is to pass down Encoder settings:

I’ve found at least one thing that uses it that way though:

https://forums.swift.org/t/codable-items-with-identity/60726/3

More research:

https://github.com/search?q=encoder.userInfo+language%3ASwift&type=code&l=Swift

I’m curious if it gets used with CodableWithConfiguration implementations?

CodableWithConfiguration

I have not implemented any custom handlers for things that are CodableWithConfiguration

https://www.andyibanez.com/posts/the-mysterious-codablewithconfiguration-protocol/

Summary

Encoders. Not so fiddly if one can be ruthless in vetting the input. Interminable to write if trying to be general purpose.

Next post will be the Decoder for this SimpleCoder which may cause it to be entirely rewritten!