And what can I make a custom Encoder do?

This article is part of a series.

Relevant repo: https://github.com/carlynorama/CoderExplorer/

After last post’s extensive review of options, I decided to write an Encoder along lines of the StackOverflow example in that

The implementation will be different.

The first test needed to pass?

struct TestStruct:Codable {
    let number:Int
    let text:String
    let sub:TestSubStruct
}

struct TestSubStruct:Codable {
    let numeral:Int
    let string:String
}

func testExample() throws {
    let encoder = SimpleCoder()
    
    let sub = TestSubStruct(numeral: 34, string: "world")
    let testItem = TestStruct(number: 12, text: "hello", sub: sub)
    let expected = "number:12/sub.numeral:34/sub.string:world/text:hello"
    XCTAssertEqual(expected, try encoder.encode(testItem))
    
}

Getting Started with a Keyed Container

Some initial decisions

Super basic public face of the Encoder system:

struct SimpleCoder {
    public func encode<E:Encodable>(_ value: E) throws -> String {
        let encoder = _SimpleEncoder()
        try value.encode(to: encoder)
        return encoder.value
    }
}

A Dumb-As-Rocks reference type for the data, a la the objc.io example

final class SimpleCoderData<Value> {
    var storage:Value
    
    init(_ value: Value) {
        self.storage = value
    }
}

The type that will eventually conform to Encoder. It will know how to turn the SimpleCoderData type into a String.

struct _SimpleEncoder {
    var data:SimpleCoderData<[String: String]>
    var codingPath: [CodingKey] = []
    
    //note: having lots of complicated work in a var
    //isn't the best. I'm following the API of an :Encoder
    //that didn't need to do any work to get its .value.  
    var value:String {
        var lines = data.storage.map { key, value in
            if key.isEmpty || key.contains("keyless"){
                return "\(value)"
            } else {
                return "\(key):\(value)"
            }
        }
        //To get a consistent order out since this isn't a 
        //sorted dictionary. Better for testing. 
        lines.sort()
        return lines.joined(separator: "/")
    }
}

extension _SimpleEncoder {
    init() {
        self.data = SimpleCoderData([:])
    }
}

The _SimpleEncoder will also own adding values to the data var. EVERYTHING will need to have a key in this example so we only need one function that takes a non-optional CodingKey.

extension _SimpleEncoder {
    
    func encodeKey(key:CodingKey) -> String {
        (codingPath + [key]).map { $0.stringValue }.joined(separator: ".")
    }
    
    //to be called from containers
    func encode(_ value: String, forKey key:CodingKey) {
        data.storage[encodeKey(key: key)] = value
    }
}

Also in _SimpleEncoder, the converters for the basic types that the Encoding Containers will use. I’m going to follow the model of them all being called the same thing, but taking different value types in as parameters.

extension _SimpleEncoder {
    
    //called from the containers
    
    @inline(__always)
    func convert(_ value: some BinaryFloatingPoint) -> String {
        Double(value).description
    }
    
    @inline(__always)
    func convert(_ value: some FixedWidthInteger) throws -> String {
        guard let validatedValue = Int(exactly: value) else {
            throw EncodingError.invalidValue(value, EncodingError.Context(codingPath: codingPath, debugDescription: "Integer out of range."))
        }
        return validatedValue.description
    }
}

Now for the actual conformance - time to make up some names for the encoding containers.

extension _SimpleEncoder:Encoder {
    
    var userInfo: [CodingUserInfoKey : Any] { [:] }
    
    func container<Key>(keyedBy type: Key.Type) -> KeyedEncodingContainer<Key> where Key : CodingKey {
        KeyedEncodingContainer(SimpleEncoderKEC<Key>(encoder: self))
    }
    
    func unkeyedContainer() -> UnkeyedEncodingContainer {
        fatalError()
        //SimpleCoderUEC(encoder: self)
    }
    
    func singleValueContainer() -> SingleValueEncodingContainer {
        fatalError()
       //SimpleCoderSVEC(encoder: self)
    }
    
}

The KeyedEncodingContainer

I followed the pattern from the URIEncoder for my encoding containers, with the wrinkle that everything has to get a key. That’s not hard to envision with the KeyedContainer. We’ll get the first test case to pass by just implementing this one.

The init

struct SimpleEncoderKEC<Key: CodingKey> {
    let encoder: _SimpleEncoder
}

The custom functions

extension  SimpleEncoderKEC {
    private func _insertValue(_ converted:String, atKey key: Key) throws {
        try encoder.encode(converted, forKey: key)
    }

    private func _insertBinaryFloatingPoint(_ value: some BinaryFloatingPoint, atKey key: Key) throws {
        try _insertValue(encoder.convert(value), atKey: key)
    }

    private func _insertFixedWidthInteger(_ value: some FixedWidthInteger, atKey key: Key) throws {
        try _insertValue(encoder.convert(value), atKey: key)
    }
}

The conformance

Full disclosure, I did not write tests for all of the func encode() functions that I yanked from the URIEncoder.

extension SimpleEncoderKEC:KeyedEncodingContainerProtocol {
    var codingPath: [CodingKey] {
        encoder.codingPath
    }
    
    mutating func encodeNil(forKey key: Key) throws {
        //TODO: Decide about nils
        fatalError()
    }
    mutating func encode(_ value: Bool, forKey key: Key) throws { try _insertValue("\(value)", atKey: key) }
    mutating func encode(_ value: String, forKey key: Key) throws { try _insertValue(value, atKey: key) }
    mutating func encode(_ value: Double, forKey key: Key) throws { try _insertBinaryFloatingPoint(value, atKey: key) }
    mutating func encode(_ value: Float, forKey key: Key) throws { try _insertBinaryFloatingPoint(value, atKey: key) }
    mutating func encode(_ value: Int, forKey key: Key) throws { try _insertFixedWidthInteger(value, atKey: key) }
    mutating func encode(_ value: Int8, forKey key: Key) throws { try _insertFixedWidthInteger(value, atKey: key) }
    mutating func encode(_ value: Int16, forKey key: Key) throws { try _insertFixedWidthInteger(value, atKey: key) }
    mutating func encode(_ value: Int32, forKey key: Key) throws { try _insertFixedWidthInteger(value, atKey: key) }
    mutating func encode(_ value: Int64, forKey key: Key) throws { try _insertFixedWidthInteger(value, atKey: key) }
    mutating func encode(_ value: UInt, forKey key: Key) throws { try _insertFixedWidthInteger(value, atKey: key) }
    mutating func encode(_ value: UInt8, forKey key: Key) throws { try _insertFixedWidthInteger(value, atKey: key) }
    mutating func encode(_ value: UInt16, forKey key: Key) throws { try _insertFixedWidthInteger(value, atKey: key) }
    mutating func encode(_ value: UInt32, forKey key: Key) throws { try _insertFixedWidthInteger(value, atKey: key) }
    mutating func encode(_ value: UInt64, forKey key: Key) throws { try _insertFixedWidthInteger(value, atKey: key) }
    mutating func encode<T>(_ value: T, forKey key: Key) throws where T: Encodable {
        //notice the reference to the existing data! 
        //not using push and pop style like URIEncoder.
        var tmpEncoder = _SimpleEncoder(data: encoder.data)
        tmpEncoder.codingPath.append(key)
        try value.encode(to: tmpEncoder)
    }

    mutating func nestedContainer<NestedKey>(keyedBy keyType: NestedKey.Type, forKey key: Key)
        -> KeyedEncodingContainer<NestedKey> where NestedKey: CodingKey
    { fatalError() }

    mutating func nestedUnkeyedContainer(forKey key: Key) -> any UnkeyedEncodingContainer { fatalError() }
    mutating func superEncoder() -> any Encoder { fatalError() }
    mutating func superEncoder(forKey key: Key) -> any Encoder { fatalError() }
}

I’m super generous with the fatal errors at this point because I’ll go back in and fix them as they fire.

And this passes the first test!

UnkeyedContainer

The test

Let’s try this test that has Arrays in it.

struct MoreItems:Codable {
    let myDouble:Double
    let myFloat:Float
    let myArray:[InsideStruct]
}

struct InsideStruct:Codable {
    let insideDouble:Double
    let insideFloat:Float
    //let insideArray:[TestSubStruct]
}

func testArrayExample() throws {
    let encoder = SimpleCoder()
    
    let subItem1 = InsideStruct(insideDouble: 0.234, insideFloat: 2144.421)
    let subItem2 = InsideStruct(insideDouble: 5.926, insideFloat: 0.00132)
    let subItem3 = InsideStruct(insideDouble: 312421.4124214, insideFloat: 421421.223)
    
    let testItem = MoreItems(myDouble: 8921.41421, myFloat: 1182.12, myArray: [subItem1, subItem2, subItem3])
    let expected =  "myArray.0.insideDouble:0.234/"  +
                    "myArray.0.insideFloat:2144.4208984375/" +
                    "myArray.1.insideDouble:5.926/" +
                    "myArray.1.insideFloat:0.0013200000394135714/" +
                    "myArray.2.insideDouble:312421.4124214/" +
                    "myArray.2.insideFloat:421421.21875/" +
                    "myDouble:8921.41421/" +
                    "myFloat:1182.1199951171875"
    XCTAssertEqual(expected, try encoder.encode(testItem))
}

The init

Start off with the basic needs

struct SimpleCoderUEC {
    /// The associated encoder.
    let encoder: _SimpleEncoder
    private(set) var count: Int = 0
}

The custom functions

extension SimpleCoderUEC {

    private mutating func _appendValue(_ converted:String) throws {
        try encoder.encode(converted, forKey: nextIndexedKey())
    }
    private mutating func _appendBinaryFloatingPoint(_ value: some BinaryFloatingPoint) throws {
        try _appendValue(encoder.convert(value))
    }
    private mutating func _appendFixedWidthInteger(_ value: some FixedWidthInteger) throws {
        try _appendValue(encoder.convert(value))
    }
}

Key creation

Add what’s needed for index based keys.

extension SimpleCoderUEC {
    
    //Require a key
    private mutating func nextIndexedKey() -> CodingKey {
        let nextCodingKey = IndexedCodingKey(intValue: count)!
        count += 1
        return nextCodingKey
    }
    
    private struct IndexedCodingKey: CodingKey {
        let intValue: Int?
        let stringValue: String

        init?(intValue: Int) {
            self.intValue = intValue
            self.stringValue = intValue.description
        }

        init?(stringValue: String) {
            return nil
        }
    }
}

The conformance

Heavy on the fatal errors again.

extension SimpleCoderUEC: UnkeyedEncodingContainer {

    var codingPath: [any CodingKey] { encoder.codingPath }

    func nestedUnkeyedContainer() -> any UnkeyedEncodingContainer { fatalError() }

    mutating func nestedContainer<NestedKey>(keyedBy keyType: NestedKey.Type) -> KeyedEncodingContainer<NestedKey>
    where NestedKey: CodingKey {  fatalError()   }

    mutating func superEncoder() -> any Encoder { fatalError() }

    mutating func encodeNil() throws { fatalError() }
     
    mutating func encode(_ value: Bool) throws { try _appendValue("\(value)") }
    mutating func encode(_ value: String) throws { try _appendValue(value) }
    mutating func encode(_ value: Double) throws { try _appendBinaryFloatingPoint(value) }
    mutating func encode(_ value: Float) throws { try _appendBinaryFloatingPoint(value) }
    mutating func encode(_ value: Int) throws { try _appendFixedWidthInteger(value) }
    mutating func encode(_ value: Int8) throws { try _appendFixedWidthInteger(value) }
    mutating func encode(_ value: Int16) throws { try _appendFixedWidthInteger(value) }
    mutating func encode(_ value: Int32) throws { try _appendFixedWidthInteger(value) }
    mutating func encode(_ value: Int64) throws { try _appendFixedWidthInteger(value) }
    mutating func encode(_ value: UInt) throws { try _appendFixedWidthInteger(value) }
    mutating func encode(_ value: UInt8) throws { try _appendFixedWidthInteger(value) }
    mutating func encode(_ value: UInt16) throws { try _appendFixedWidthInteger(value) }
    mutating func encode(_ value: UInt32) throws { try _appendFixedWidthInteger(value) }
    mutating func encode(_ value: UInt64) throws { try _appendFixedWidthInteger(value) }
    mutating func encode<T>(_ value: T) throws where T: Encodable {
        //points to same data reference! 
        var tmpEncoder = _SimpleEncoder(data:encoder.data)
        tmpEncoder.codingPath = encoder.codingPath
        tmpEncoder.codingPath.append(nextIndexedKey())
        try value.encode(to: tmpEncoder)
    }
}

And that passes the test.

SingleValues

It may seem little funny to do the “simplest” case, single values, last. In this case we need to do some thinking about how to force them to have keys when they wouldn’t normally. I’m being a bit lazy and giving everything the phrase “keyless(UUID())” as a key. Using a UUID instead of the value itself means not worrying about duplicates or if the value contains the key delimiter.

The test

func testSingleValues() async throws {
    let encoder = SimpleCoder()
    
    let toEncodeInt = Int.random(in: Int.min...Int.max)
    let toEncodeText = "hello" //TODO: explore strings more
    let toEncodeBool = Bool.random()
    let toEncodeDouble = Double.random(in: Double.leastNonzeroMagnitude...Double.greatestFiniteMagnitude)
    let toEncodeFloat = Float.random(in: Float.leastNonzeroMagnitude...Float.greatestFiniteMagnitude)
    let toEncodeInt32 = Int32.random(in: Int32.min...Int32.max)
    
    let encodedInt = try encoder.encode(toEncodeInt)
    let encodedText = try encoder.encode(toEncodeText)
    let encodedBool = try encoder.encode(toEncodeBool)
    
    let encodedDouble = try encoder.encode(toEncodeDouble)
    let encodedFloat = try encoder.encode(toEncodeFloat)
    let encodedInt32 = try encoder.encode(toEncodeInt32)
    
    XCTAssertEqual(toEncodeInt.description, encodedInt)
    XCTAssertEqual(toEncodeText.description, encodedText)
    XCTAssertEqual(toEncodeBool.description, encodedBool)
    XCTAssertEqual(toEncodeDouble.description, encodedDouble)
    XCTAssertEqual(Double(toEncodeFloat).description, encodedFloat)
    XCTAssertEqual(toEncodeInt32.description, encodedInt32)
}

The init

struct SimpleCoderSVEC {
    let encoder: _SimpleEncoder
}

The custom functions

extension SimpleCoderSVEC {
    private func _setValue(_ converted:String) throws { 
        try encoder.encode(converted, forKey: SVECCodingKey(converted)) 
    }

    private func _setBinaryFloatingPoint(_ value: some BinaryFloatingPoint) throws {
        try _setValue(encoder.convert(value))
    }

    private func _setFixedWidthInteger(_ value: some FixedWidthInteger) throws {
        try _setValue(encoder.convert(value))
    }
}

Key creation

extension SimpleCoderSVEC {
    
    private struct SVECCodingKey: CodingKey {
        let intValue: Int?
        let stringValue: String

        init?(intValue: Int) {
            return nil
        }

        init?(stringValue: String) {
            self.stringValue = stringValue
            self.intValue = nil
        }
        
        init(_ forValue:String) {
            self.stringValue = "keyless\(UUID())"
            self.intValue = nil
        }
    }
}

The conformance

extension SimpleCoderSVEC: SingleValueEncodingContainer {

    var codingPath: [any CodingKey] { encoder.codingPath }

    func encodeNil() throws {
        fatalError()
    }
    
    
    func encode(_ value: Bool) throws { try _setValue("\(value)") }
    func encode(_ value: String) throws { try _setValue(value) }
    func encode(_ value: Double) throws { try _setBinaryFloatingPoint(value) }
    func encode(_ value: Float) throws { try _setBinaryFloatingPoint(value) }
    func encode(_ value: Int) throws { try _setFixedWidthInteger(value) }
    func encode(_ value: Int8) throws { try _setFixedWidthInteger(value) }
    func encode(_ value: Int16) throws { try _setFixedWidthInteger(value) }
    func encode(_ value: Int32) throws { try _setFixedWidthInteger(value) }
    func encode(_ value: Int64) throws { try _setFixedWidthInteger(value) }
    func encode(_ value: UInt) throws { try _setFixedWidthInteger(value) }
    func encode(_ value: UInt8) throws { try _setFixedWidthInteger(value) }
    func encode(_ value: UInt16) throws { try _setFixedWidthInteger(value) }
    func encode(_ value: UInt32) throws { try _setFixedWidthInteger(value) }
    func encode(_ value: UInt64) throws { try _setFixedWidthInteger(value) }
    func encode<T>(_ value: T) throws where T: Encodable {
        try value.encode(to: encoder)
    }
}

And again the test passes!

Fixing the Dates

Before going on to tackle more complicated data structures I want to sort out a few value types that don’t get encoded the way I want by default. Date, URL and Data. Starting with Date, this is the test to pass.

func testDate() throws {
    let encoder = SimpleCoder()
    let date = Date()

    let dateString = date.ISO8601Format(.iso8601)        
    let encodedDate = try encoder.encode(date)       
    XCTAssertEqual(dateString, encodedDate)
}

It doesn’t out of the gate. It fails with the error:

XCTAssertEqual failed: ("2024-03-07T14:55:56Z") is not equal to ("731516156.037366")

The one below does though, which shows us that the default Encoding for Date’s Codable implementation (core | FoundationEssential) is the timeIntervalSinceReferenceDate.

func testDate() throws {
    let encoder = SimpleCoder()
    let date = Date()

    let dateString = "\(date.timeIntervalSinceReferenceDate)"
    let encodedDate = try encoder.encode(date)
    XCTAssertEqual(dateString, encodedDate)
}

How to fix it? First off, the encoder has to catch that a Date has come into play. We’ll catch it in the func encode<T>(_ value: T) functions in all three encoding containers by adding a case statement to each.

mutating func encode<T>(_ value: T, 
                        forKey key: Key
                       ) throws where T: Encodable {
    switch value {
    case let value as Date: fatalError()
    default:
        //points to same data reference! 
        var tmpEncoder = _SimpleEncoder(data:encoder.data)
        tmpEncoder.codingPath.append(key)
        try value.encode(to: tmpEncoder)
    }
}

If we run the test again the Date still doesn’t get snagged by the error. It will be easier to hit inside the Keyed and Unkeyed containers first.

I got the Keyed and Unkeyed encoding containers working by adding the relevant functions, starting with a convert function like there is for Float and Int in the :Encoder.

//----  In :Encoder
func convert(_ value:Date) -> String {
    return value.ISO8601Format()
}

Then adding the wrapper functions to the encoding containers and updating the case statements

//----  In :KeyedEncodingContainerProtocol
private func _insertDate(_ value: Date, atKey key:Key) throws {
    try _insertValue(encoder.convert(value), atKey: key)
}
mutating func encode<T>(_ value: T, forKey key: Key) throws where T: Encodable {
    switch value {
    //...
    case let value as Date: try _insertDate(value, atKey: key)
    //...
    }
}

//----  In :UnkeyedEncodingContainer
private mutating func _appendDate(_ value:Date) throws {
    try _appendValue(encoder.convert(value))
}
mutating func encode<T>(_ value: T, forKey key: Key) throws where T: Encodable {
    switch value {
    //...
    case let value as Date: try _appendDate(value)
    //...
    }
}

Add Keyed and Unkeyed containers to the test:

func testDate() throws {
    let encoder = SimpleCoder()
    let date = Date()
    

    //--------- SingleValue (STILL FAILS)
    let dateString = date.ISO8601Format(.iso8601)
    let encodedDate = try encoder.encode(date)
    XCTAssertEqual(dateString, encodedDate)
    
    //--------- Keyed (PASSES)
    struct MiniWithDate:Encodable {
        let date:Date = Date()
    }
    
    let miniToTest = MiniWithDate()
    let structExpected = "date:\(miniToTest.date.ISO8601Format())"
    let encodedStruct = try encoder.encode(miniToTest)
    XCTAssertEqual(structExpected, encodedStruct)
    
    //--------- Unkeyed (PASSES)
    let dateArray = [Date(), Date(), Date(), Date()]
    let arrayExpected = dateArray.enumerated().map ({ index, value in
        "\(index):\(value.ISO8601Format())"
    }).joined(separator: "/")
    let encodedArray = try encoder.encode(dateArray)
    XCTAssertEqual(arrayExpected, encodedArray)
}

The Keyed and Unkeyed containers work and the SingleValue encoding container doesn’t hit the error! What gives?

The Date keeps prioritizing its defaults (core | FoundationEssential). I mentioned date handling in the last post, that the URIEncoder & the JSONEncoders and even the XML encoder all have to catch a stand alone Date value in their encodeValue/wrapGeneric/box functions. It was good to see the need for that in action.

Instead of a routing function inside the _SimpleCoder:Encoder, I’ll put an array of types that I’ve made special handlers for up in the public SimpleCoder. If my public function gets passed a value whose type is in that array, that’s when it will go to the special handler function inside the :Encoder. Since SimpleCoder is a learning exercise I want to contrast right up top why one might see the basic value.encode(to: encoder) pattern (no overrides) vs. something more like encoder.encode(for: value) -> some Output (overrides inside).

//----  Updated
struct SimpleCoder {
    let flaggedTypes:[Encodable.Type] = [Date.self]
    public func encode<E:Encodable>(_ value: E) throws -> String {
        let encoder = _SimpleEncoder()
        if flaggedTypes.contains(where: { $0 == E.self }) {
            try encoder.specialEncoder(for: value)
        } else {
            try value.encode(to: encoder)
        }
        return encoder.value
    }
}

//----  Added
extension _SimpleEncoder {
    func specialEncoder(for value: some Encodable) throws {
        var container = singleValueContainer()
        try container.encode(value)
    }
}

//----  In the :SingleValueEncodingContainer
private func _setDate(_ value:Date) throws {
    try _setValue(encoder.convert(value))
}
mutating func encode<T>(_ value: T, forKey key: Key) throws where T: Encodable {
    switch value {
    //...
    case let value as Date: try _setDate(value)
    //...
    }
}

The Date tests now all pass.

Fixing URLs

I’m also going to need URLs for the Lines data and URLs have an excessively complete default format for my needs. What do I mean? They come out looking like:

//singleValue
 "relative:http://www.example.com"
 //part of Struct
 "url.relative:http://www.example.com"
 //part of Array
 "0.relative:http://www.example.com/1.relative:http://www.example.com?testQuery=42/2.relative:http://www.example.com/3.relative:http://www.example.com"

When a URL gets created from any of the following possible init paths, it actually creates a key:value pair of the String “relative” and the URL’s relativeString. See a discussion

Why is that? Well, lets look at the Codable implementation for URL:

extension URL : Codable {
    private enum CodingKeys : Int, CodingKey {
        case base
        case relative
    }

    public init(from decoder: Decoder) throws {
        let container = try decoder.container(keyedBy: CodingKeys.self)
        let relative = try container.decode(String.self, forKey: .relative)
        let base = try container.decodeIfPresent(URL.self, forKey: .base)

        guard let url = URL(string: relative, relativeTo: base) else {
            throw DecodingError.dataCorrupted(
                                   DecodingError.Context(codingPath: decoder.codingPath,
                                                         debugDescription: "Invalid URL string.")
                                )
        }

        self = url
    }

    public func encode(to encoder: Encoder) throws {
        var container = encoder.container(keyedBy: CodingKeys.self)
        try container.encode(self.relativeString, forKey: .relative)
        if let base = self.baseURL {
            try container.encode(base, forKey: .base)
        }
    }
}

A complete URL actually has two pieces of information. Its path and whether it’s relative to a different URL. If one declares a URL like:

It does in fact produce the complete set when encoded:

"relative:hello.jpg/base.relative:http://www.example.com"

For saving URLs to the Lines data I just want the URL’s absoluteString, so I’ll need another convert function.

struct SimpleCoder {
    let flaggedTypes:[Encodable.Type] = [Date.self, URL.self]
    public func encode<E:Encodable>(_ value: E) throws -> String {
        let encoder = _SimpleEncoder()
        if flaggedTypes.contains(where: { $0 == E.self}) {
            try encoder.specialEncoder(for: value)
        } else {
            try value.encode(to: encoder)
        }
        return encoder.value
    }
}

//----  In :Encoder
func convert(_ value:URL) -> String {
    return value.absoluteString()
}

//---- Also in all 3 encoding containers... Keyed example.  
private func _insertURL(_ value: URL, atKey key:Key) throws {
    try _insertValue(encoder.convert(value), atKey: key)
}  
mutating func encode<T>(_ value: T, forKey key: Key) throws where T: Encodable {
    switch value {
    //...
    case let value as URL: try _insertValue(value, forKey:Key)
    //...
    }
}

And again the tests work!

func testURL() throws {
    let encoder = SimpleCoder()

    //------------------ mostly HTTP Schema
    let http_url = URL(string: "http://www.example.com")!
    var components = URLComponents()
    components.scheme = "http"
    components.host = "www.example.com"
    components.queryItems = [URLQueryItem(name: "testQuery", value: "42")]
    let file_url = FileManager.default.homeDirectoryForCurrentUser
    
    let absoluteURL = URL(dataRepresentation: "http://www.example.com".data(using: .utf8)!, 
                            relativeTo: nil, isAbsolute: true)!
    
    let relativeURL = URL(fileURLWithPath: "hello.jpg", 
                            relativeTo: URL(string:"http://www.example.com"))
    
    //does not even touch my encoder
    let httpUrlString = http_url.relativeString //.absoluteString
    let httpEncodedURL = try encoder.encode(http_url)
    XCTAssertEqual(httpUrlString, httpEncodedURL)
    
    //--------- Keyed
    struct MiniWithHttpURL:Encodable {
        let url:URL = URL(string: "http://www.example.com")!
    }
    
    let httpMiniToTest = MiniWithHttpURL()
    let httpStructExpected = "url:\(httpMiniToTest.url.absoluteString)"
    let httpEncodedStruct = try encoder.encode(httpMiniToTest)
    XCTAssertEqual(httpStructExpected, httpEncodedStruct)
    
    //--------- Unkeyed
    let urlArray = [http_url, components.url!, absoluteURL, relativeURL, file_url]
    let httpArrayExpected = urlArray.enumerated().map ({ index, value in
        "\(index):\(value.absoluteString)"
    }).joined(separator: "/")
    let encodedURLArray = try encoder.encode(urlArray)
    XCTAssertEqual(httpArrayExpected, encodedURLArray)
}

Fixing Data

Failing initial test:

//FAILS
//XCTAssertEqual failed: ("Cg8U") is not equal to ("0:10/1:15/2:20")
func testData() throws {
    let encoder = SimpleCoder()
    let data = Data([10, 15, 20])
    let dataString = data.base64EncodedString()
    let encodedData = try encoder.encode(data)
    XCTAssertEqual(dataString, encodedData)
    
}

New convert:

@inline(__always)
func convert(_ value:Data) throws -> String {
    return value.base64EncodedString()
}

And again passing complete test.

func testData() throws {
    let encoder = SimpleCoder()
    
    //------------------ SingleValue
    let data = Data([10, 15, 20])
    let dataString = data.base64EncodedString()
    let encodedData = try encoder.encode(data)
    XCTAssertEqual(dataString, encodedData)
    
    //--------- Keyed
    struct MiniWithData:Encodable {
        let myData:Data = Data([10, 15, 20, 127, 0])
    }
    
    let miniToTest = MiniWithData()
    let structExpected = "myData:Cg8UfwA="
    let encodedStruct = try encoder.encode(miniToTest)
    XCTAssertEqual(structExpected, encodedStruct)
    
    //--------- Unkeyed
    let dataArray = [data, data, data, data]
    let structArray = [miniToTest, miniToTest]
    let arrayExpected = dataArray.enumerated().map ({ index, value in
        "\(index):\(value.base64EncodedString())"
    }).joined(separator: "/")
    let encodedArray = try encoder.encode(dataArray)
    XCTAssertEqual(arrayExpected, encodedArray)
    
    let structArrayExpected = "0.myData:Cg8UfwA=/1.myData:Cg8UfwA="
    let encodedStructArray = try encoder.encode(structArray)
    XCTAssertEqual(structArrayExpected, encodedStructArray)     
}

Not hard to pull off, but one can see that having to go in to every encoding container to add every little new custom override would get annoying so when I do the LineCoder I’ll handle it a little differently.

What to do about Optionals?

There are three ways to handle Optionals in an Encoder

And one can choose differently per container. Maybe a place holder makes sense for SingleValue situations, but Keyed null values can just be ignored. That’s the route I picked for SimpleCoder. It causes some interesting trouble for enums with optional associated values in arrays, but that’s for fixing another day.

//In KeyedEncodingContainer
    mutating func encodeNil(forKey key: Key) throws {
        //do nothing. 
        //try _insertValue("NULL", atKey: key)
    }
//In SingleValueEncodingContainer
    func encodeNil() throws { try _setValue("NULL") }

//In UnkeyedEncodingContainer
//That's right. NOTHING in the test calls this. 
//Not even the nested Optional arrays. 
   mutating func encodeNil() throws {
        fatalError()
        //try _appendValue("NULL")
    }

func testBasicNil() throws {
        let encoder = SimpleCoder()
        
        //------------------ SingleValue
        let optionalValue:Int? = nil
        let optionalString = "NULL"
        let encodedOptional = try encoder.encode(optionalValue)
        XCTAssertEqual(optionalString, encodedOptional)
        
        //--------- Keyed
        struct MiniStruct:Encodable {
            let noneInt:Int? = nil
            let someInt:Int? = 12
        }
        
        let miniToTest = MiniStruct()
        let structExpected = "someInt:12"
        let encodedStruct = try encoder.encode(miniToTest)
        XCTAssertEqual(structExpected, encodedStruct)
        
        
        func renderArray(_ array:[Int?], prepend:String = "") -> String {
            array.enumerated().map ({ index, value in
                let valueString = value != nil ? "\(value!)" : "NULL"
                return "\(prepend)\(index):\(valueString)"
            }).joined(separator: "/")
        }
        
        //--------- Unkeyed
        
        let array = [10, nil, 20, nil, 0]
        let arrayExpected = renderArray(array)
        let encodedArray = try encoder.encode(array)
        XCTAssertEqual(arrayExpected, encodedArray)
        
        let subArray:[[Int?]?] = [array, nil, array]
        let subArrayExpected = subArray.enumerated().map({
            if let notNil = $1 {
                return renderArray(notNil, prepend: "\($0).")
            } else {
                return "\($0):NULL"
            }
            }).joined(separator: "/")
        let encodedSubArray = try encoder.encode(subArray)
        XCTAssertEqual(subArrayExpected, encodedSubArray)
        
        struct MiniArrayStruct:Encodable {
            let myIntArray:[Int?] = [10, nil, 20, nil, 0]
        }
        let miniArratStructToTest = MiniArrayStruct()
        let structArray = [miniArratStructToTest, miniArratStructToTest]
        let structArrayExpected = structArray.enumerated().map ({ index, value in
            renderArray(value.myIntArray, prepend: "\(index).myIntArray.")
        }).joined(separator: "/")
        let encodedStructArray = try encoder.encode(structArray)
        XCTAssertEqual(structArrayExpected, encodedStructArray)   
    }

Adding enums

RawRepresentable

When getting started with enum’s it’s very useful to dive into the actual original swift-evolution proposal for what was called “swift-archival-serialization” and jump to the place where it shows what the synthesized conformance for enums would be.

    public func encode(to encoder: Encoder) throws {
        // Encode as a single value; no keys.
        try encoder.singleValueContainer().encode(self.rawValue)
    }

An enum always gets pushed through to a singleValueContainer.

And if we test this with a RawRepresentable conforming enum with a RawValue of String we see it works perfectly:

    func testEnum() throws {
        let encoder = SimpleCoder()

        enum Greeting:String,Codable {
            case hello, howdy, hola, hi, hiya
        }
        
        //------------------ SingleValue
        let enumValue:Greeting = .hiya
        let expectedEnumString = "hiya"
        let encodedEnum = try encoder.encode(enumValue)
        XCTAssertEqual(expectedEnumString, encodedEnum)
    }

but add an enum to a struct or array…

        struct MiniWithEnum:Encodable {
            let otherValue:Double = 42
            let myGreeting:Greeting = .hello
            let otherStringValue = "world"
        }
        
        let miniToTest = MiniWithEnum()
        let structExpected = "myGreeting:\(miniToTest.myGreeting)/otherStringValue:world/otherValue:42.0"
        let encodedStruct = try encoder.encode(miniToTest)
        XCTAssertEqual(structExpected, encodedStruct)

        //--------- Unkeyed
        let enumArray:[Greeting] = [.hello, .howdy, .hiya, .hola]
        let arrayExpected = enumArray.enumerated().map ({ index, value in
            "\(index):\(value)"
        }).joined(separator: "/")
        let encodedArray = try encoder.encode(enumArray)
        XCTAssertEqual(arrayExpected, encodedArray)

And we get failures like:

XCTAssertEqual failed: ("myGreeting:hello/otherStringValue:world/otherValue:42.0") is not equal to ("hello/otherStringValue:world/otherValue:42.0")

What happened? Well, even though I’m in a keyed container, my default behavior for anything my encoder doesn’t recognize is to let it do its own thing. For an enum that means it’s going through a SingleValueEncodingContainer (See this discussion). In the data store it does have a key, but it’s myGreeting.keyless(some UUID) which gets stripped out in our serialization step because it contains the text “keyless”.

I have some choices:

I’m going to update the Dictionary -> String creation code because it will be easier than rerouting all enums or guaranteeing custom Codable implementations. Also my data store really relies on those unique keys so I can’t change the key-encoding function either.

    var value:String {
        var lines = data.storage.map { key, value in
            var mKey = key
            
            if mKey.contains("keyless") {
                mKey = cleanKey(mKey)
            }
            
            if mKey.isEmpty {
              return "\(value)"
            } else {
              return "\(mKey):\(value)"
            }
        }
        lines.sort()
        return lines.joined(separator: "/")
    }
    
    func cleanKey(_ key:String, keyDelimiter:String.Element = ".") -> String {
        //potential alternative approach:
        //just trim from the end until the last delimiter. 
        //TBD if there is a case where that would make a difference since the SVEC 
        //create a nested container. 
        let split = key.split(whereSeparator: { $0 == keyDelimiter }).filter({ !$0.contains("keyless")})
        return split.joined(separator: String(keyDelimiter))
    }

And now it all works.

Unspecified Enum

What happens when the enum isn’t RawRepresentable with a RawValue of String or Int?

    func testPlainEnum() throws {
        let encoder = SimpleCoder()

        enum FruitChoice:Codable {
            case strawberry, pineapple, dragon, kiwi, kumquat
        }
        
        //------------------ SingleValue
        let enumFruitValue:FruitChoice = .strawberry
        let expectedEnumFruitString = "strawberry"
        let encodedFruitEnum = try encoder.encode(enumFruitValue)
        XCTAssertEqual(expectedEnumFruitString, encodedFruitEnum) 
    }

Whelp. It catches one of the remaining fatalErrors() in the SimpleEncoderKEC at

    mutating func nestedContainer<NestedKey>(keyedBy keyType: NestedKey.Type, forKey key: Key)
        -> KeyedEncodingContainer<NestedKey> where NestedKey: CodingKey
    {
        fatalError()
    }

replacing that with

    mutating func nestedContainer<NestedKey>(keyedBy keyType: NestedKey.Type, forKey key: Key)
        -> KeyedEncodingContainer<NestedKey> where NestedKey: CodingKey
    {
        //fatalError()
        print("nestedKey:\(key.stringValue)")
        return encoder.container(keyedBy: NestedKey.self)
        
    }

One gets the TestFailure, AND one gets to see the reason for it; the enum values are coming through as the keys this time.

Test Case 
'-[CoderExplorerTests.CodeableStringExample testPlainEnum]' started.
nestedKey:strawberry
CoderExplorer/Tests/CoderExplorerTests/SimpleCoderTests.swift:251: 
error: -[CoderExplorerTests.CodableStringExample testPlainEnum] :
XCTAssertEqual failed: ("strawberry") is not equal to ("")

If I sent this enum to the JSONEncoder, it could handle it. The value would be an empty object.

enum FruitChoice:Encodable {
    case strawberry, pineapple, dragon, kiwi, kumquat
}
let JSONencoder = JSONEncoder()
let encodedData = try JSONencoder.encode(FruitChoice.strawberry)
let string = String(bytes: encodedData, encoding: .utf8) ?? ""
//string == {"strawberry":{}}

It would actually take some significant doing to catch an empty object in this particular encoder, so I’m going to show “fixing” the enum instead. These enums will no longer call the nestedContainer<NestedKey>(keyedBy:forKey:) function because the the custom encode(to:) function (as opposed to the synthesized one) knows the enum is flat and has no associated values.

Option 1: Stripping the key entirely

The right thing to do would be to make it RawRepresentable as a String as seen above, but if for some reason you can’t or want to have more control over the returned string:

enum FruitChoice:Encodable {
    case strawberry, pineapple, dragon, kiwi, kumquat
    
    public func encode(to encoder: Encoder) throws {
        var container = encoder.singleValueContainer()
        try container.encode("\(self)")
    }
}

Option 2: Forcing a key:value pair

Maybe instead the enum name would be useful to have in the data for some reason. The enum can return a dictionary.

enum FruitChoice:Encodable {
    case strawberry, pineapple, dragon, kiwi, kumquat
    
    public func encode(to encoder: Encoder) throws {
        //really a dictionary, but JSON terminology is leaking in
        let asObject = ["FruitChoice":"\(self)"]
        var container = encoder.singleValueContainer()
        let container = 
        try container.encode(asObject)
    }
}

Option 3: updating the CodingKeys (the “right way”)

Strictly more correct than the above method?

enum CodingKeys:CodingKey {
    case fruitChoice
}

public func encode(to encoder: Encoder) throws {
    var container = encoder.container(keyedBy: Self.CodingKeys.self)
    try container.encode("\(self)", forKey: .fruitChoice)
}

Associated Values

Newish to Codable is “Codable Synthesis for Enums with Associated Values”

Adding a mix of associated values to FruitChoice, what one will get is position based synthesized keys for the sub-object Swift thinks the associated value is part of.

enum FruitChoice:Encodable {
    case strawberry(String, Int), 
    case pineapple(Int), 
    case dragon(Double), 
    case kiwi(UInt8), 
    case kumquat(Date)
}
let JSONencoder = JSONEncoder()
let encodedData = try JSONencoder.encode(FruitChoice.strawberry("Everbearing", 5))
let string = String(bytes: encodedData, encoding: .utf8) ?? ""
//"{"strawberry":{"_0":"Everbearing","_1":5}}

One can name them inline:

enum FruitChoice:Encodable {
    case strawberry(name:String, cropCount:Int), 
    case pineapple(yearsToFruit:Int), 
    case dragon(metersTall:Double), 
    case kiwi(seedsIn1mmSlice:UInt8), 
    case kumquat(whenAquired:Date)
}

Or override the whole synthesized Codable conformance:

enum FruitChoice:Encodable {
    case strawberry(String, Int), pineapple(Double), kumquat(Date)
    
    enum CodingKeys:CodingKey {
        case strawberry
        case pineapple
        case kumquat
    }
    
     enum StrawberryCodingKeys: CodingKey {
        case name
        case cropCount
      }
    
    enum PineappleCodingKeys: CodingKey {
       case yearsToFruit
     }
    
    enum KumquatCodingKeys: CodingKey {
        case dateAquired
    }

    public func encode(to encoder: Encoder) throws {
      var container = encoder.container(keyedBy: CodingKeys.self)
      switch self {
          
      case .strawberry(let string, let int): 
          var nestedContainer = container.nestedContainer(
                                                keyedBy: StrawberryCodingKeys.self, 
                                                forKey: .strawberry)
          try nestedContainer.encode(string, forKey: .name)
          try nestedContainer.encode(int, forKey: .cropCount)
      case .pineapple(let double): 
          var nestedContainer = container.nestedContainer(
                                                keyedBy: PineappleCodingKeys.self, 
                                                forKey: .pineapple)
          try nestedContainer.encode(double, forKey: .yearsToFruit)
      case .kumquat(let date):
          var nestedContainer = container.nestedContainer(
                                                keyedBy: KumquatCodingKeys.self, 
                                                forKey: .kumquat)
          try nestedContainer.encode(date, forKey: .dateAquired)
      }
    }
}

I include that mostly to show how the nestedContainer call comes into play. Now that I’ll be needing that function again I have to fix it to work correctly.

    mutating func nestedContainer<NestedKey>(keyedBy keyType: NestedKey.Type, forKey key: Key)
        -> KeyedEncodingContainer<NestedKey> where NestedKey: CodingKey
    {
        //have to reversibly push the key onto the stack. 
        var tmpEncoder = _SimpleEncoder(data:encoder.data)
        tmpEncoder.codingPath = encoder.codingPath
        tmpEncoder.codingPath.append(key)
        return tmpEncoder.container(keyedBy: NestedKey.self)
    }

I’ll add that to all the nestedContainer(keyType:forKey:) calls. I’m creating a new _SimpleCoder in at least 5 places and counting now so I’ll make a function to do it:

    // in _SimpleCoder
    typealias DataStore = SimpleCoderData<[String: String]>
    func getEncoder(forKey key:CodingKey?, withData passedInData:DataStore) -> Self {
        var tmp = _SimpleEncoder(data:passedInData)
        tmp.codingPath = self.codingPath
        if let key {
            tmp.codingPath.append(key)
        }
        return tmp
    }

In _SimpleCoder the data storage never gets wiped clean so I could just use self.data in the function without passing it through, but I’d like to make it clear at the call location that the data matters for my own future reference.

For more about nested containers see also

The tests pass!

    func testAssociateValueEnum() throws {
        enum FruitChoice:Encodable {
            case strawberry(name:String, cropCount:Int), 
            case pineapple(yearsToFruit:Double), 
            case kumquat(dateAcquired:Date)
        }
        
        //------------------ SingleValue
        let enumFruitValue:FruitChoice = .strawberry(name:"Everbearing", cropCount:6)
        let expectedEnumFruitString = "strawberry.cropCount:6/strawberry.name:Everbearing"
        let encodedFruitEnum = try encoder.encode(enumFruitValue)
        XCTAssertEqual(expectedEnumFruitString, encodedFruitEnum)
            
        //--------- Keyed
        struct MiniWithEnum:Encodable {
            let otherValue:Double = 42
            let myFruit:FruitChoice = .pineapple(yearsToFruit:3.12)
            let otherStringValue = "world"
        }
        
        let miniToTest = MiniWithEnum()
        let structExpected = "myFruit.pineapple.yearsToFruit:3.12/otherStringValue:world/otherValue:42.0"
        let encodedStruct = try encoder.encode(miniToTest)
        XCTAssertEqual(structExpected, encodedStruct)
        
        //--------- Unkeyed
        let enumArray:[FruitChoice] = [.strawberry(name:"Everbearing", cropCount:6),
                                       .pineapple(yearsToFruit:1.23343), 
                                       .kumquat(dateAcquired:try Date("2024-03-10T21:36:24Z", 
                                                                      strategy: .iso8601))]

//        let arrayExpected = enumArray.enumerated().map ({ index, value in
//            "\(index).fruitChoice:\(value)"
//        }).joined(separator: "/")
        let arrayExpected = "0.strawberry.cropCount:6/"+
        "0.strawberry.name:Everbearing/"+
        "1.pineapple.yearsToFruit:1.23343/"+
        "2.kumquat.dateAquired:2024-03-10T21:36:24Z"
        let encodedArray = try encoder.encode(enumArray)
        XCTAssertEqual(arrayExpected, encodedArray)
        
    }

Adding a Dictionary

Using dictionaries that have an Int or String as a key and some Encodable as their value works like a charm. I’m not going to spend any time on them except to mention, yes, I wrote a test.

One can get other types to work as codable dictionary keys by conforming them to CodingKeyRepresentable

Many of the fixed-width int types conform automatically, but not all of them do or should

A passing test

func testSomeEncodableKeyDictionary() throws {
   
   enum FruitKey:String, Codable, CodingKeyRepresentable {
       case strawberry, banana, kiwi
   }

   let dictionaryA:Dictionary<FruitKey,UInt32> = [
       .strawberry:1,
       .banana:2,
       .kiwi: 3
   ]
   
   struct FlowerCodingKey: CodingKey {
       let stringValue: String
       var intValue: Int?
       
       init?(stringValue: String) {  self.stringValue = stringValue  }
       init?(intValue: Int) {
           self.stringValue = String(intValue)
           self.intValue = intValue
       }
       init(_ flower:Flower) {
           self.stringValue = flower.name
       }
   }
   
   //TODO: Codable shouldn't be necessary but my Encoder requested it?
   struct Flower:Hashable, Codable, CodingKeyRepresentable {
       let name:String
       init(_ name:String) {
           self.name = name
       }
       
       var codingKey: any CodingKey {
           FlowerCodingKey.init(self)
       }
       
       init?<T>(codingKey: T) where T : CodingKey {
           self.name = codingKey.stringValue
       }
   }

   let dictionaryB:Dictionary<Flower,Int> = [
       Flower("anemone"):7,
       Flower("freesia"):6,
       Flower("periwinkle"):5
   ]

   //------------------ SingleValue
   let expectedA = "banana:2/kiwi:3/strawberry:1"
   let expectedB = "anemone:7/freesia:6/periwinkle:5"
   let encodedA = try encoder.encode(dictionaryA)
   let encodedB = try encoder.encode(dictionaryB)
   XCTAssertEqual(expectedA, encodedA)
   XCTAssertEqual(expectedB, encodedB)

   //--------- Keyed
   struct MiniWithDict:Encodable {
       let first:Dictionary<FruitKey,UInt32>
       let second:Dictionary<Flower,Int>
   }

   let miniToTest = MiniWithDict(first: dictionaryA, second: dictionaryB)
   let structExpected = "first.banana:2/first.kiwi:3/first.strawberry:1/second.anemone:7/second.freesia:6/second.periwinkle:5"
   let encodedStruct = try encoder.encode(miniToTest)
   XCTAssertEqual(structExpected, encodedStruct)

   //--------- Unkeyed
   let dictArray = [dictionaryB, dictionaryB]
   let arrayExpected = "0.anemone:7/0.freesia:6/0.periwinkle:5/1.anemone:7/1.freesia:6/1.periwinkle:5"
   let encodedArray = try encoder.encode(dictArray)
   XCTAssertEqual(arrayExpected, encodedArray)
}

If one DOESN’T make the key CodingKeyRepresentable, one ends up with a interlaced output that the decoder will have to know to stitch back together.

func testUnofficialKeyDictionary() throws {
   struct MyKey:Hashable, Codable {
       var id:UInt8
   }
   
   let dictionaryA:Dictionary<MyKey, Int> = [
       MyKey(id: 127):1,
       MyKey(id: 126):2,
       MyKey(id: 125):3
   ]
   
   //NOTE: This test will only intermittently pass
   //because dictionary order is arbitrary. 
   //will be easier to test when can round trip with
   //decoder.
   let expectedA = "0.id:125/" +
                   "1:3/" +
                   "2.id:126/" +
                   "3:2/" +
                   "4.id:127/" +
                   "5:1"
   let encodedA = try encoder.encode(dictionaryA)
   //.sorted(by: { $0.key < $1.key }) not Encodable
   XCTAssertEqual(expectedA, encodedA)
}

What’s the deal with superEncoder()?

.superEncoder() gets called when a subclass wants to use it’s parent class’ encoder.

I did a couple of tests using my own class/subclass and one with AttributedString which uses a lot of .base64EncodedString() data. They’re not exactly super rigorous (why I’ve only added links), but it got me nominal coverage for:

The remaining fatalError()s

Nesting and unkeyed containers didn’t happen in either direction in my tests. Even where I expected them to. I could write my own custom objects that call them just for testing, but I’d rather wait until I understand who/what uses them in the wild for what.

More research:

A note about userInfo

Most encoder implementations I’ve seen essentially ignore the protocol defined userInfo dictionary, but the idea behind it is to pass down Encoder settings:

I’ve found at least one thing that uses it that way though:

More research:

I’m curious if it gets used with CodableWithConfiguration implementations?

CodableWithConfiguration

I have not implemented any custom handlers for things that are CodableWithConfiguration

Summary

Encoders. Not so fiddly if one can be ruthless in vetting the input. Interminable to write if trying to be general purpose.

Next post will be the Decoder for this SimpleCoder which may cause it to be entirely rewritten!

This article is part of a series.