Protobuf and SwiftData with full exposure of the Proto object graph for queries

[Skoti]

To use SwiftProtobuf types in SwiftData models, they must conform to Codable.
Suppose you have:

@Model
final class Foo {
    var bar: Bar?
    
    init(bar: Bar? = nil) {
        self.bar = bar
    }
}

where Bar is a protobuf message type.

We cannot simply do:

extension Bar: Codable {}

public extension Message {
    init(from decoder: Swift.Decoder) throws {
        let container = try decoder.singleValueContainer()
        let data = try container.decode(Data.self)
        self = try Self(serializedBytes: data)
    }

    func encode(to encoder: Swift.Encoder) throws {
        let data = try serializedData()
        var container = encoder.singleValueContainer()
        try container.encode(data)
    }
}

because it will crash at runtime as SwiftData expects things to use a keyed container, and not a single-value one.

Ofc one can implement a custom init(from:) and encode(to:) per each message, but this is cumbersome.
It would be possible to write the implementation in one place (like above), if there were access to field names that SwiftProtobuf already generates, yet all related APIs are internal only (like visitors or _NampingMap).

So there are two possibilities:

• make Protobuf types Codable (the default synthesized behavior already relies on keyed containers) - this has nothing to do with how Protobuf encodes things (so might be confusing for some people)

OR (probably preferable)

• expose a way (like a custom Decoder and Visitor) to traverse each field and decode/encode into a keyed container, so folks wishing to opt into Codable can easily do this with just an extension Bar: Codable

Something like:

public extension Message {
    init(from decoder: Swift.Decoder) throws {
        init()
        try decodeMessage(decoder: CodableDecoder(decoder))
    }

    func encode(to encoder: Swift.Encoder) throws {
        try CodableEncoder(encoder).encode(self)
    }
}

Where CodableDecoder and CodableEncoder are provided by SwiftProtobuf.
They could even check for a custom userInfo key in the provided decoder/encoder to decide whether to use a single container (from 1st snippet) or a keyed container - so their use case is more flexible.

Would any route be of interest to implement?

[thomasvl]

fyi - Codable has come up before, see #657 for some of the past discussions.

You mention fully exposing the fields to Codable - if I'm follow that, I think it would likely be a very bad idea, as then you wouldn't actually be using the Protobuf Binary (or JSON) speced formats, you'd be using what every encoder the developer creates and scheme migration would have to be entirely handle via those approaches. i.e. - you would lose all of the protobuf spec guarentees.

If you really wanted Codable support for your App's use case, I see two potential options:

1. Make a simple struct that only has a Data field and put a helper to "get" the value that is generic to a protobuf message class. Within that method, you can parse the bytes into the request class and return it. For a "set" api, you can take any message and serialize it to update the actual Data that you then store. So the helper struct would be what goes into SwiftData and the extra api support going to/from protobuf types.
2. The other would be to do something like extension on Message, but also declare a custom set of coding keys, but only have one key for the binary payload, and then also provide init/encode that serialize the message and then store it under that one key.

As far as the library generally supporting Codable, I guess the second options above could be considered, but a fair amount of thought would have to be put into if we really want to support as an official api, and there's also the potential the library adding something like that could clash with some downstream code that did something similar themselves.

[Skoti]

fyi - Codable has come up before, see #657 for some of the past discussions.

I've seen it, but that was a different request. The OP thought of encapsulating the Protobuf binary encoding through a custom encoder and decoder to be used with any Codable conforming types, which is not possible. It might be though, with the CodableWithConfiguration, but that's a different story.

and scheme migration would have to be entirely handle via those approaches. i.e. - you would lose all of the protobuf spec guarentees.

Yes, I'm aware this separates from Protobuf binary compatibility and all the perks that come with it. Migrations would be required more often than with pure Protobuf.

---

Make a simple struct that only has a Data field and put a helper to "get" the value that is generic to a protobuf message class. Within that method, you can parse the bytes into the request class and return it. For a "set" api, you can take any message and serialize it to update the actual Data that you then store. So the helper struct would be what goes into SwiftData and the extra api support going to/from protobuf types.

That's not usable because you cannot create predicates based on a given Protobuf message, as you only have the data field:

let predicate = #Predicate<Foo> {
    $0.bar.data // can't check anything
}

The other would be to do something like extension on Message, but also declare a custom set of coding keys, but only have one key for the binary payload, and then also provide init/encode that serialize the message and then store it under that one key.

This works in theory, but then again Predicate won't return matching items:

struct Bar: Codable {
    enum CodingKeys: CodingKey {
        case data
    }

    var i: Int = .zero

    init(i: Int) {
        self.i = i
    }

    init(from decoder: any Decoder) throws {
        let c = try decoder.container(keyedBy: CodingKeys.self)
        if let data = try c.decodeIfPresent(Data.self, forKey: .data) {
            self.i = data.withUnsafeBytes { $0.load(as: Int.self) }
        }
    }

    func encode(to encoder: any Encoder) throws {
        var c = encoder.container(keyedBy: CodingKeys.self)
        if i != .zero {
            try c.encode(Swift.withUnsafeBytes(of: i) { Data($0) }, forKey: .data)
        }
    }
}

@Model
final class Foo {
    var bar: Bar

    init(bar: Bar) {
        self.bar = bar
    }
}

@ModelActor
actor MA {
    func create() throws {
        modelContext.insert(Foo(bar: .init(i: 9)))
        modelContext.insert(Foo(bar: .init(i: 4)))
        try modelContext.save()
    }

    func get(_ block: ([Foo]) -> Void) throws {
        print(try modelContext.fetchCount(FetchDescriptor<Foo>())) // this prints 2
        let predicate = #Predicate<Foo> {
            $0.bar.i > 0 // this should allow the predicate to return the two items but it has 0 matches :/
        }
        let result = try modelContext.fetch(.init(predicate: predicate))
        block(result)
    }
}

@Test func example() async throws {
    let mc = try ModelContainer(for: Foo.self, configurations: .init(isStoredInMemoryOnly: true))
    let ma = MA(modelContainer: mc)

    try await ma.create()
    try await ma.get() {
        print($0.count) // this prints 0 :(
        $0.forEach {
            print($0.bar.i)
       }
    }
}

I guess this has something to do with how SwiftData represents models underneath, and since the CodingKeys are not matching with the fields used in Predicate, it does not return any results 🤷‍♂ .

So the only solution is to support all field names in CodingKeys for each message, as I've described earlier.

---

As far as the library generally supporting Codable, I guess the second options above could be considered,

Yeah, the second option is nice as it's purely opt-in.

but a fair amount of thought would have to be put into if we really want to support as an official api,

If this won't be officially supported, then it would be great if at least the functionality (existing in the Protobuf library today) needed to achieve that could be exposed. For a start, I could make use of _NameMap, and its names(for number:) function for encoding, and number(forJSONName:) for decoding.

[thomasvl]

Ok, so that's a pretty big change in scope since you actually want to be able to do queries based on the proto object graph.

As far as exposing things in the name map, see the other issues where this as home up, it isn't something we've wanted to do at this time, but understand the desire and the protobuf libraries for other langauges do provide some means for this that we'd likely use for inspiration if/when something was added.

[thomasvl]

I should add, since it sounds like each user of this would have to re-implement proto scheme upgrading themselves, it does leave me wondering what protobuf is even bringing to the mix here as one of it's main advantages would be defeated.

[Skoti]

Protobuf was not brought to solve anything related to storage. Its main advantage is the common definition of models, the convenience of code generation, and data exchange between mobiles, peripherals, and backend.

It's just that additionally, for some of the models, an offline-mode support is desired (with full query support).
Hence, it's convenient to use proto structures in SwiftData models, instead of duplicating their definition.