Can I Swiftify that ESP32C6 HTTPClient more?

This article is part of a series.

I’m going to straight up lead with my future TODO list, because I’m not satisfied with where the code gets left off.

But back to what I was able to do. My current HTTP request gets managed by a whisper thin Swift wrapper around C.

I’d like to make it a little less whisper thin.

The results so far here:

Setting Up the Protocol

What’s my minimum viable client for the demo? Describe it as a protocol:

protocol HTTPClient {
  init(host: String, port:Int?)
  func fetch(_ path:String)
}

and a the client moves to an implementation.


struct  ServerInfo {
    //scheme is always http
    //let useHTTPS: Bool = true
    let host:String //TODO: will have to split to domain for dns lookup?
    let port:Int
    //let basePath = String //future
}

final class MyClient: HTTPClient {

    var defaultServer: ServerInfo

    init(host: String, port: Int? = nil) {
    if port == nil {
      self.defaultServer = ServerInfo(host: host, port: 80)
    } else {
      self.defaultServer = ServerInfo(host: host, port: port!)
    }
  }

}

In Main.swift using the following instantiation would lock it down so the subsequent code can only use the features of the class that comply with the protocol.

let exampleClient: some HTTPClient = MyClient(host: "example.com")

I don’t love this. As noted in my TODO’s I’d prefer to have a concrete HTTPClient type that links to a HTTPClientService protocol instead, more like the way the momentary button code is two steps away from the underlying hardware architecture.

I am also on the fence about having a client be attached to a specific server.

Resolve the DNS

To resolve the underlying DNS the standard trick is to use the c function getaddrinfo, even SwiftNIO uses it.

It writes the result to a C struct which has another C struct embedded in it passed as a reference:

struct addrinfo {
    int              ai_flags;     // AI_PASSIVE, AI_CANONNAME, etc.
    int              ai_family;    // AF_INET, AF_INET6, AF_UNSPEC
    int              ai_socktype;  // SOCK_STREAM, SOCK_DGRAM
    int              ai_protocol;  // use 0 for "any"
    size_t           ai_addrlen;   // size of ai_addr in bytes
    struct sockaddr *ai_addr;      // struct sockaddr_in or _in6
    char            *ai_canonname; // full canonical hostname

    struct addrinfo *ai_next;      // linked list, next node
};

struct sockaddr {
    unsigned short    sa_family;    // address family, AF_xxx
    char              sa_data[14];  // 14 bytes of protocol address
};

For the time being I chose to hold that result container in a class level variable, making sure to have a deinit for it, just in case. I tried holding the pointee instead of the pointer, which appeared to work, but fell apart when creating the socket.

I also hard coded a retry of 6 times, which a better library would let the developer config.


enum HTTPClientError: Error {
  case hostUnresolved

  var describe: String {
    return switch self {
    case .hostUnresolved: "hostUnresolved"
    }
  }
}

final class MyClient: HTTPClient {
    internal typealias AddressInfo = addrinfo

    var currentInfo: UnsafeMutablePointer<addrinfo>?

    var defaultServer: ServerInfo

    init(host: String, port: Int? = nil) {
        if port == nil {
        self.defaultServer = ServerInfo(host: host, port: 80)
        } else {
        self.defaultServer = ServerInfo(host: host, port: port!)
        }
    }

    //TODO: what if i just nil'd it? 
    deinit {
        if let currentInfo {
            freeaddrinfo(currentInfo)
        }
    }

    func test() {
      print("resolving...")
      do {
        try getAddressInfo(for: server.host, using: server.port)
      } catch let myError {
        print("Error info: \(myError.describe)")
      }
    }

    private func getAddressInfo(for host: String, using port: Int = 80) throws(HTTPClientError) {
        currentInfo = nil
        var retry = 6
        let local_host = host.utf8CString
        while retry > 0 {

        var hints: AddressInfo = addrinfo()
        hints.ai_socktype = SOCK_STREAM  //vs SOCK_DGRAM
        hints.ai_protocol = AF_INET  //IP version 4, vs 6 or unix
        let error = local_host.withContiguousStorageIfAvailable { host_name in
            return getaddrinfo(host_name.baseAddress, "\(port)", &hints, &currentInfo)
        }
        if error == 0 {
            print("got it.")
            retry = 0
            return
        } else {
            if error == nil {
            print("why wasn't contiguous storage available?")
            } else {
            print("getAddressInfo error: \(error!)")
            }
            retry = retry - 1
        }
        }
        currentInfo = nil
        throw HTTPClientError.hostUnresolved
    }
}

NOTE: I am having a problem that requires me to unplug and replug my board while the monitor is still running to get the host to resolve some of the time. I think this might be a esp-idf DNS cache problem? It does not always happen.

Connect the socket

I again created a top level variable to hold an “open socket”. It’s just a CInt that the underlying network code can use to locate which underlying hardware to use. Once a socket has been opened, it’s important to close it. Once a socket has been opened, this top level var should be returned to nil only after close() has been called. This will let my code know if it has done proper clean up.

The swift nio socket type:

final class MyClient: HTTPClient {
  internal typealias AddressInfo = addrinfo
  internal typealias SocketAddress = sockaddr
  internal typealias SocketHandle = CInt  //none of the functions use socklen_t?

  //...

  var currentInfo: UnsafeMutablePointer<AddressInfo>?
  var openSocket: SocketHandle?

  deinit {
    if let openSocket {
      close(openSocket)
    }
    if let currentInfo {
      freeaddrinfo(currentInfo)
    }
  }
  //...
}

    func test() {
      print("resolving...")
      do {
        try getAddressInfo(for: server.host, using: server.port)
        openSocket = try connectSocket()
        freeaddrinfo(currentInfo)  //addrinfo has a freshness value.
        assert(currentInfo == nil) //check to make sure that actually nil'd it too. 
      } catch let myError {
        print("Error info: \(myError.describe)")
      }
    }

If the connection is successful, return the socket handle.

private func connectSocket() throws(HTTPClientError) -> SocketHandle {
    if let currentInfo {
      print("I have AddressInfo")
      let socket: SocketHandle = socket(
        currentInfo.pointee.ai_family, currentInfo.pointee.ai_socktype, 0)
      if socket < 0 {
        throw HTTPClientError.couldNotMakeSocket
      }
      print("socketHandle: \(socket)")
      var retry = 6
      while retry > 0 {
        let connectValue = connect(
          socket, currentInfo.pointee.ai_addr, currentInfo.pointee.ai_addrlen)
        if connectValue == 0 {
          retry = 0
          return socket
        } else {
          print("connect returned \(connectValue)")
        }
        delay(500)
        retry = retry - 1
        print("retry connect \(retry)")
      }
      close(socket)
      throw HTTPClientError.connectionFailed
    }
    throw HTTPClientError.addressUnresolved
  }

Write to the Socket

In a future version of this client I’d like to formulate a HTTPRequest that gets handled to a ClientService that does the serialization, but for now it’s hand coded. Why are those “extra steps” a good idea? Well, because when dealing with protocols it’s REALLY REALLY easy to forget things like required terminating newlines, carriage returns, 0 and other final characters.

It’s helpful to be able to see the request on the receiving side, but since this isn’t a client running on my machine, there are a couple extra steps to troubleshooting with netcat. One has to turn off firewalls and use the ifconfig command (look for en0: ). It’s likely to be an address like 192.168.42.*** and then my client init would look like:

let exampleClient: some HTTPClient = MyClient(host: "192.168.42.***", port:8080)

(For an IP not on the same private network try something like ifconfig.me, but there’s a million of these sites.)

    func test() {
      print("resolving...")
      do {
        try getAddressInfo(for: server.host, using: server.port)
        openSocket = try connectSocket()
        freeaddrinfo(currentInfo)  //addrinfo has a freshness value.
        assert(currentInfo == nil) //check to make sure that actually nil'd it too. 
        try writeRequest(with: openSocket!, to: defaultServer.host, at: "/")
      } catch let myError {
        print("Error info: \(myError.describe)")
        //TODO: close socket if a socket related error? 
      }
    }

Notice my request type is a ContiguousArray<CChar> and I’m still using that withContiguousStorageIfAvailable closure like in the first pass at the client. C Strings are very different than Swift Strings and trying to pass a non-contiguous storage type (Swift String) to a contiguous storage type (C String) will end up with all sorts of bad data reads, etc. Very. Unsafe.

  private func writeRequest(with socket: SocketHandle, to host: String, at path: String)
    throws(HTTPClientError)
  {
    //DON'T FORGET THE CLOSING \r\n
    let request: ContiguousArray<CChar> =
      "GET \(path) HTTP/1.0\r\nHost: \(host)\r\nUser-Agent: \(userAgent)\r\n\r\n".utf8CString
    print("request length: \(request.count)")
    print("\(request)")

    let result = request.withContiguousStorageIfAvailable { request_buffer in
      let writeResponse = write(socket, request_buffer.baseAddress, request_buffer.count)
      print("writeResponse: \(writeResponse)")
      return writeResponse
    }

    if result != nil && result! < 0 {
      close(socket)
      self.openSocket = nil
      throw HTTPClientError.sendFailed
    }

    if result == nil {
      print("what happened?")
      //close socket? 
      throw HTTPClientError.unsendableRequest
    }
  }

Listen

This time I’ll show the function first because the return type is important, its a [UInt8] that DOES NOT null terminate the buffer. If this code was C the buffer would be a char* with null termination. char* becomes [CChar] which comes into Swift as a [Int8]. At the moment, far more String initializers take a [UInt8] than can accommodate the other two, including the initializer to the ISOLatin1String type in HTTPTypes.

If I was going to keep response buffer stored in a [CChar] I would have null terminated it to be consistent with how C stores strings. In fact the line to do so is commented out in the code below.

  private func listenForResponse() throws(HTTPClientError) -> [UInt8] {
    var message: [UInt8] = []
    var buffer: [UInt8] = Array(repeating: 0, count: 512)

    guard let openSocket else {
      throw HTTPClientError.noSocketOpen
    }
    let _ = buffer.withContiguousMutableStorageIfAvailable { buffer in
      var r: CInt
      repeat {
        r = read(openSocket, buffer.baseAddress, buffer.count - 1)
        print("bytes read: \(r)")
        //this buffer could be holding data from a previous read with
        //a longer r. Only copy r bytes. 
        message.append(contentsOf: buffer.prefix(Int(r)))
      } while r > 0
    }

    //done so close the socket. 
    close(openSocket)
    self.openSocket = nil

    //DO NOT NULL TERMINATE IF NOT USING CChar
    //message.append(0)

    return message
  }

Decisions about the byte type is null termination change what in String initializers are available. At the call site I’ve chosen to use String(validating: response, as: UTF8.self), which will return nil if the bytes aren’t all UTF8 codable characters.

Since we have a UInt8 array, another choice would be String(decoding: response, as: UTF8.self) which will replace invalid data with "\u{FFFD}" (official replacement character of the specials), but always return something.

There are MANY String initializers, and MANY of them in Swift 6 are deprecated in favor of the above 2, so those will be the ones I’ll lean on the most in future code.

Neither of the remaining two accept null terminated strings, and it seems like all the CString acceptors have been deprecated, so clipping off the 0 and passing it along seems like the best choice. But only to validating because decoding won’t work with a [Char]. But there is maybe one remaining static function? Anyway, this all seems a bit in flux.

//the one remaining static function. 
let message7 = String.decodeCString(response as! [UTF8.CodeUnit], as: UTF8.self, repairingInvalidCodeUnits: true)

The final fetch function:

  public func fetch(_ path: String) {
    fetch(path, from: defaultServer)
  }

    public func fetch(_ path: String, from server: ServerInfo) {
    do {
      print("resolving...")
      try getAddressInfo(for: server.host, using: server.port)
      print("connecting...")
      openSocket = try connectSocket()
      freeaddrinfo(currentInfo) 
      assert(currentInfo == nil)
      print("currentInfo is nil: \(currentInfo == nil)")
      print("writing...")
      try writeRequest(with: openSocket!, to: defaultServer.host, at: path)

      print("listening...")
      let response = try getResponse()
      //print("response count: \(response.count)")
      //for the future. 
      //let message = ISOLatin1String(response) 

      if let message = String(validating: response, as: UTF8.self) {
        print("message was valid UTF8")
        print(message)
      } else {
        print("message could not be decoded as UTF8")
      }
      
      print("done")

    } catch let myError {
      print("Error info: \(myError.describe)")
      if let openSocket {
        close(openSocket)
      }
      if let currentInfo {
        freeaddrinfo(currentInfo)
      }
    }
  }

Running this code on the Xiao should spit out the full example.com home page!

Summary

So still a lot left to do. Even on top of all I want in that TODO list at the top, a bare minimum is to do enough parsing to tell what the status code is.

That said, before I do that, I’ll probably take a look at the momentary button code so I can do this HTTP call “onPress” instead of on a timer.

This article is part of a series.