Can I make my ESP32C6 HTTP client use HTTPTypes?

This article is part of a series.

So when we left off I had gotten the ESP32C6 to touch the internet using a very thin wrapper over C.

The work for the next two posts can be seen on the e_ImproveHTTPClient branch.

Reminder, in the terminal window, set up the environment if not using the VSCode extension.

. $HOME/esp/esp-idf/export.sh
cd $PROJECT_FOLDER
idf.py set-target esp32c6
# export SOME_PORT=`ls /dev/cu.usbmodem*`
# echo $SOME_PORT
idf.py build

I thought it would be an interesting exercise to see what it would take to bring the code from the HTTPTypes library into an embedded project.

I copied all the files from the core folder into a folder in my project and added them by hand to the CMake sources by hand:

# List of Swift source files to build.
target_sources(${COMPONENT_LIB}
    PRIVATE
    Main.swift
    Bool_Int.swift
    GPIOPin.swift
    DigitalIndicator.swift
    MomentaryInput.swift
    Delay.swift     
    ErrorHandler.swift
    WiFi.swift
    HTTP.swift
    HTTPRevisedTypes/HTTPField.swift
    HTTPRevisedTypes/HTTPFieldName.swift
    HTTPRevisedTypes/HTTPFields.swift
    HTTPRevisedTypes/HTTPParsedFields.swift
    HTTPRevisedTypes/HTTPRequest.swift
    HTTPRevisedTypes/HTTPResponse.swift
    HTTPRevisedTypes/ISOLatin1String.swift
    HTTPRevisedTypes/NIOLock.swift
)

I then got to work fixing all the compiler errors. A lot of these are covered in the Swift Embedded Examples Docs as caveats already.

Starting Easy

I’ve already been incredibly impressed how the esp-idf and Swift Embedded work together giving really nice String and print access with absolutely no work on my part, which is crazy. I’m not sure if people not coming from hardware development get how nice that is.

So it’s not a surprise to me that I needed to strip out the following string mashing related features:

I read that someone had gotten Codable working maybe? but its pretty slow. I ripped that all out without trying to get them to work.

No KeyPaths

Also an easy fix was changing the map functions that use a KeyPath to actual closures. In HTTPFields.swift

    public subscript(values name: HTTPField.Name) -> [String] {
        get {
            self.fields(for: name).map(\.value)
        }
        set {
            self.setFields(newValue.lazy.map { HTTPField(name: name, value: $0) }, for: name)
        }
    }

becomes

    public subscript(values name: HTTPField.Name) -> [String] {
        get {
            self.fields(for: name).map({$0.value})
        }
        set {
            self.setFields(newValue.lazy.map { HTTPField(name: name, value: $0) }, for: name)
        }
    }

Clarifying Error Types

HTTPParsedFields has a lot of throwing functions. They all need to be given an explicit Error type because any Error, as an Existential is not allowed in embedded.

From:

    private func validateFields() throws {
        guard self.fields[values: .contentLength].allElementsSame else {
            throw ParsingError.multipleContentLength
        }
        guard self.fields[values: .contentDisposition].allElementsSame else {
            throw ParsingError.multipleContentDisposition
        }
        guard self.fields[values: .location].allElementsSame else {
            throw ParsingError.multipleLocation
        }
    }

//...

extension HTTPRequest {
    public init(parsed fields: [HTTPField]) throws {
        var parsedFields = HTTPParsedFields()
        for field in fields {
            try parsedFields.add(field: field)
        }
        self = try parsedFields.request
    }
}

To:

    private func validateFields() throws(ParsingError) {
        guard self.fields[values: .contentLength].allElementsSame else {
            throw ParsingError.multipleContentLength
        }
        guard self.fields[values: .contentDisposition].allElementsSame else {
            throw ParsingError.multipleContentDisposition
        }
        guard self.fields[values: .location].allElementsSame else {
            throw ParsingError.multipleLocation
        }
    }

//...

extension HTTPRequest {
    public init(parsed fields: [HTTPField]) throws(HTTPParsedFields.ParsingError) {
        var parsedFields = HTTPParsedFields()
        for field in fields {
            try parsedFields.add(field: field)
        }
        self = try parsedFields.request
    }
}

The Doozey: HTTPFields underlying storage.

So the biggest fanciest thing in HTTPTypes is the different locks but on the field storage to make them concurrency safe.

There is a base class called _Storage that depending on the compiler information will pick one of two child classes: _StorageWithMutex or _StorageWithNIOLock

Swift Embedded with the esp-idf version I’m using has no Mutex and NIOLock hit the “unable to identify C library” error.

I could not add a child _StorageForEmbedded because the parent a template for the two children to implement their competing lock strategies.

    func withLock<Result>(_ body: () throws -> Result) rethrows -> Result {
            fatalError()
    }

So options include:

So in the mean time I just ripped out all the locking. In my demo that should be fine.

So Much STRIIIIIIINNNNGGGG.

So HTTPTypes is essentially a String handing library: comparing strings, parsing strings, inspecting strings. So even when getting all that done there’s one last library that needs to be explicitly added to prevent errors about missing _swift_stdlib types in during a build.

/Users/$USER/.espressif/tools/riscv32-esp-elf/esp-14.2.0_20241119/riscv32-esp-elf/bin/../lib/gcc/riscv32-esp-elf/14.2.0/../../../../riscv32-esp-elf/bin/ld: main.elf: hidden symbol `_swift_stdlib_getNormData' isn't defined

Looking at the Strings page in the Embedded Swift docs, right there in the list is getNormData.

HTTPRevisedTypes wanted

All of those can be found by inspecting the suggested library with strings piped to less

cd `swiftly use --print-location`
# update to be the target arch you're looking for
cd usr/lib/swift/embedded/riscv32-none-none-eabi
strings libswiftUnicodeDataTables.a | less
# inside less type
# /$SEARCH_ITEM
# to find the text, just q to exit.

This is some serious special sauce and it can be added to the project via CMake with

set(COMPILER_TARGET "riscv32-none-none-eabi")

find_program(SWIFTLY "swiftly")
IF(SWIFTLY)
  execute_process(COMMAND swiftly use --print-location OUTPUT_VARIABLE toolchain_path OUTPUT_STRIP_TRAILING_WHITESPACE)
  cmake_path(SET additional_lib_path NORMALIZE "${toolchain_path}/usr/lib/swift/embedded/${COMPILER_TARGET}")
ELSE()
  get_filename_component(compiler_bin_dir ${CMAKE_Swift_COMPILER} DIRECTORY)
  cmake_path(SET additional_lib_path NORMALIZE "${compiler_bin_dir}/../lib/swift/embedded/${COMPILER_TARGET}")
ENDIF()

target_link_directories(${COMPONENT_LIB} PUBLIC "${additional_lib_path}")
target_link_libraries(${COMPONENT_LIB}
    -Wl,--whole-archive
    swiftUnicodeDataTables
    -Wl,--no-whole-archive
    )

## debug message
message(" ----------::: ${additional_lib_path} :::---------- ")

Minimal Test

So… it compiles now, but can it see the types?

Adding a very minimal test

HTTP.swift

final class HTTPClient {

    func test() {
        let request = HTTPRequest(method: .get, scheme: "https", authority: "www.example.com", path: "/")
        print(request.scheme ?? "no scheme")
    }

    //....
}

Main.swift

  //Waiting for wifi to connect...
  delay(2000);
  let client = HTTPClient()
  client.getAndPrint(from: "example.com", route: "/")
  client.test()

And that at least worked?

Can HTTPRevisedTypes be a Library?

I have aspirations to be able to work with both the espressif sdk and the pico-sdk better than I can now. Specifically, I want to be able to compartmentalize code into libraries via CMake since both of those SDKs use it. I can when writing my own simple examples, but not when meshing with C based SDK’s.

As it is I tried a few different ways to change the HTTPRevisedTypes files into a included library, all of which failed for various reasons related to my general CMake weaknesses, not understanding some of what the esp-idf is doing under the hood to piece projects together, and Swift embedded’s relying on wholemodule mode, or maybe even the version of CMake that esp-idf 5.5 uses? I’m a few steps too low on the ladder to reach it yet.

I bought a book, but here is some additional reading to do.

It seems like there is a path forward treating HTTPRevisedTypes as a component, but I’m not confident how that will interact with the Swift requirements and that keeps it chained to the esp-idf. In the future I might give it a shot since it can be done as pure CMake?

In the mean time I want to focus on the client API, which could eventually use the request type under the hood.

This article is part of a series.