YapDatabaseExtensions 2

Recreating YapDatabaseExtensions with protocol extensions in Swift 2.0

In YapDatabaseExtensions, I introduced protocols to support working with Swift value types in YapDatabase. Generic convenience APIs were provided to preserve type fidelity in Swift. Initial reaction has been fairly positive within the teams that I've introduced it to, and it's seen widespread adoption in all of my own projects. According to CocoaPods, a total of 12 apps use the framework! So it's safe to say that we're not talking about AFNetworking, or Reachability here. However, there is always room for improvement, and since the announcement of Swift 2.0, I've been looking at ways to improve things. This post will mostly cover the steps that I've gone through to utilize Swift 2.0 features to rectify weaknesses.

Archiving vs Coding

The protocols to support "archiving" value types into binary were called Saveable and Archiver, which is a little regrettable. Firstly of course, because Saveable isn't a word, but more importantly it doesn't really convey the purpose of the protocol. Naming stuff is hard.

Secondly, an archiving process means writing binary to storage. Coding is the process of converting native domain types (classes etc) into a binary representation, usually ready for archiving. In Cocoa, classes implement NSCoding and can then be used with NSKeyedArchiver and friends. For this framework, YapDatabase is effectively the archive storage. Therefore, to align things with Cocoa, I renamed Saveable to ValueCoding and Archiver to CodingType and it all makes a lot more sense.

/**
A generic protocol for a class which can 
encode/decode a value type.
*/
public protocol CodingType {

    typealias ValueType

    /// The value type which is being encoded/decoded
    var value: ValueType { get }

    /// Required initializer receiving the wrapped value type.
    init(_: ValueType)
}

/**
A generic protocol for value types which require
coding.
*/
public protocol ValueCoding {
    typealias Coder: CodingType
}

One great outcome of this, is that ValueCoding is now trivial to adopt. Just take your existing NSCoding capable class, give it a new name, adopt CodingType, and then define it as the Coder of your value type which then adopts ValueCoding. Something like this:

extension Product: ValueCoding {
    public typealias Coder = ProductCoder
}

The API which then supports encoding and decoding is available to all types which implement ValueCoding thanks to protocol extensions in Swift 2.0. We can provide a static method, decode and a property encoded to all types which implement ValueCoding correctly. I say "correctly" because the ValueType of a ValueCoding type's Coder must be equal to the type itself. This equality constraint is expressed using a generic where clause in an extension on ValueCoding.

/**
Static methods for decoding `AnyObject` to Self, and returning encoded object
of Self.
*/
extension ValueCoding where Coder: NSCoding, Coder.ValueType == Self {

    /**
    Decodes the value from a single object, if possible.
    For example

        let foo = Foo.decode(decoder.decodeObjectForKey("foo"))
    
    - parameter object: an optional `AnyObject` which if not nil should
    be of `Foo.Coder` type to work.
    - returns: an optional `Self`
    */
    public static func decode(object: AnyObject?) -> Self? {
        return Coder.decode(object)
    }

    /**
    Encodes the value type into its Coder.
    
    Typically this would be used inside of 
    `encodeWithCoder:` when the value is composed inside
    another `ValueCoding` or `NSCoding` type. For example:
    
        encoder.encodeObject(foo.encoded, forKey: "foo")
    */
    public var encoded: Coder {
        return Coder(self)
    }
}

The decode function on CoderType is implemented internally in a similar way, with support for working with sequences also available by default.

extension SequenceType where
    Generator.Element: ValueCoding,
    Generator.Element.Coder: NSCoding,
    Generator.Element.Coder.ValueType == Generator.Element {

    /**
    Encodes the sequence of value types into a sequence of objects.
    Typically this would be used inside of
    `encodeWithCoder:` when a sequence of values is
    composed inside another `ValueCoding` or 
    `NSCoding` type. For example:

        encoder.encodeObject(foos.encoded, forKey: "foos")
    */
    public var encoded: [Generator.Element.Coder] {
        return map { $0.encoded }
    }
}

Lastly, because value coding has a lot of potential usage outside of saving structs in YapDatabase, I have moved it to its own framework/pod called ValueCoding.

Metadata

YapDatabase has a nice feature where it will store objects at the same index (collection & key) alongside the primary object in another table. This is useful for, well, metadata. Previously in the extensions, the support for this was a bit of a disaster (sorry). Essentially it forced metadata to be stored additionally inside the primary object. There were also two additional protocols extending Persistable, to support different coding patterns for the MetadataType.

Now, both of these mistakes have been fixed by making the metadata property part of Persistable and doing away with a specialized protocol for metadata completely. The updated Persistable protocol is

public protocol Persistable: Identifiable {

    /// The nested type of the metadata.
    typealias MetadataType

    /// The YapDatabase collection name the type is stored in.
    static var collection: String { get }

    /// A metadata which is set when reading, and get when writing.
    var metadata: MetadataType? { get set }
}

The metadata property is an optional, which makes it possible for different values of the same type to opt into having metadata, which makes this very flexible. The property is required to be a getter and setter, this is because YapDatabaseExtensions will get the metadata to write it to the database, and when reading the primary object, it will attempt to read the metadata, and set the property if available, before returning the value. Therefore when an item is created for the first time, if appropriate, the metadata property is set before writing it to the database.

However, what about types which have no need for metadata? For these, we can implement this requirement of Persistable in an extension.

extension Persistable {

    /**
    Default metadata property. Implement this to re-define your
    own MetadataType.

        var metadata: MyMetadata? = .None
    */
    public var metadata: Void? {
        get { return .None }
        set { }
    }
}

Meaning that by default, all Persistable types have Void metadata. As the property is defined as part of the protocol, any implementation in a concrete type adopting the protocol can override this implementation. This makes the framework much easier to understand, as there is now only one protocol to adopt.

Generic return types

The write functions in YapDatabaseExtensions all receive an argument which is the generic type. This means that at the call-site (where the function is used) type inference will work, and there is no requirement to specify any types. However, for read functions, the return value is the generic, which means that for type inference to work, the type of the result needs to be specified. I find this make can make code fairly ugly.

// Writing functions will figure the type out
let written = connection.write(employee) 

// Reading requires type specification
let employee: Employee? = connection.readAtIndex(index)

It's also extremely error prone to write these functions, as the generic where clause needs to be specified for every function. What would be great would be to define the readAtIndex etc methods as part of a protocol, which can then be implemented in an extension.

Of course, the protocols needs a generic type, which is the item being written to or read from the database. With Swift 2.0, this all starts to look possible as we can create a generic protocol, constrain it with where clauses to support ValueCoding and them implement the whole API as methods in one go. Great! It would look like the code snippet below, which shows the where clause for a value type.

extension Readable where
    ItemType: ValueCoding,
    ItemType: Persistable,
    ItemType.Coder: NSCoding,
    ItemType.Coder.ValueType == ItemType {

    func readAtIndex(index: YapDB.Index) -> ItemType? {
        // etc - we'll come back to the implementation detail
    }
}

To make use of such a protocol, they need to be adopted by a concrete type. However, it's not possible for YapDatabase types to do that. YapDatabaseReadTransaction and YapDatabaseConnection are not generic types, they're Objective-C classes, and it is not possible to extend them with a generic parameter. So, we're sort of stuck. Although...

What if we define our own generic protocols for a database, connection, write transaction and read transaction? We're ultimately not going to be able to implement the functions on the read or write transaction protocols in extensions, because we cannot make them generic and have YapDatabaseReadTransaction adopt it. But we can move the generic functions to a protocol. The primary benefit in doing this, is that we can then unit test the functions totally removed from YapDatabase.

Looking at it from the outside in, the database needs to be able to create a new connection, and this connection needs to be generic.

public protocol DatabaseType {
    typealias Connection: ConnectionType

    func makeNewConnection() -> Connection
}

We can easily make YapDatabase conform to this. Note that we must define a new method and cannot hijack the existing newConnection() method as the Swift compiler would complain about the conflict.

extension YapDatabase: DatabaseType {
    public func makeNewConnection() -> YapDatabaseConnection {
        return newConnection()
    }
}

Similarly, ConnectionType can define the base read and write methods which we will require. Again, this protocol will require generic types for the read and write transactions.

public protocol ConnectionType {
    typealias ReadTransaction: ReadTransactionType
    typealias WriteTransaction: WriteTransactionType

    func read<Item>(block: ReadTransaction -> Item) -> Item

    func write<Item>(block: WriteTransaction -> Item) -> Item
}

The transaction types are pretty simple, here is the gist:

public protocol ReadTransactionType {
    func objectAtIndex(index: YapDB.Index) -> AnyObject?
}

public protocol WriteTransactionType: ReadTransactionType {
    func writeAtIndex(index: YapDB.Index, object: AnyObject, metadata: AnyObject?)
}

Now we can adopt these protocols on the relevant YapDatabase types. This is possible as the transaction protocols are not generic. The read and write APIs, which are generic functions, can be implemented in extensions on ReadTransactionType and WriteTransactionType. This is the best the we can do in terms generic functions. Trust me.

This allows us to write super fast unit tests.

func test__transaction__read_at_index_with_data() {
    configureForReadingSingle()
    let person: Person? = readTransaction.readAtIndex(index)
    XCTAssertNotNil(person)
    XCTAssertEqual(person!.identifier, item.identifier)
}

But what if...

No, seriously, you can't make generic functions work with non-generic types.

Hang on though, what if we flipped everything around?

Oh, that could be interesting... Like something similar to the decode and encoded extensions? Let's explore that possibility.

The read/write functionality could be provided via methods on a custom type. Say we have a type Employee, then we want this:

// Let me read an employee!
let employee = Employee.readFrom(database, atIndex: index)

This strategy looks promising because the API operating on the type which we previously were not able to make generic, is the framework consumer's own type.

Curried API

Taking this idea further, we can write a "curried" API in an extension on Persistable. Here the function returns a function that accepts a transaction. Like the example below for value coding types with object metadata:

extension Persistable where
    Self: ValueCoding,
    Self.Coder: NSCoding,
    Self.Coder.ValueType == Self,
    Self.MetadataType: NSCoding {

    /**
    Returns a closure which, given a read transaction will return
    the item at the given index.
    
    - parameter index: a YapDB.Index
    - returns: a (ReadTransaction) -> Self? closure.
    */
    public static func readAtIndex<
        ReadTransaction where
        ReadTransaction: ReadTransactionType>(index: YapDB.Index) -> ReadTransaction -> Self? {
            return { $0.readAtIndex(index) }
    }

    /**
    Returns a closure which, given a write transaction will write
    and return the item.
    
    - warning: Be aware that this will capure `self`.
    - returns: a (WriteTransaction) -> Self closure
    */
    public func write<WriteTransaction: WriteTransactionType>() -> WriteTransaction -> Self {
        return { $0.write(self) }
    }    

Which could be used like this:

// Read an item by key from the database in a new transaction 
// using a connection.
let item = connection.read(Item.readByKey(key))

// Write an item to the database in a new transaction 
// using a connection. 
let result = connection.write(item.write())

Persistable API

Taking this one final step further, we can define a full Persistable API. An intermediate type, which is generic over both ItemType and DatabaseType can implement a generic Readable as originally discussed. An extension on Persistable can provide an API to return a reader. For writing, this is a lot simpler, and doesn't require an intermediate helper. So, from the call site, it would look like this:

// Read an item by key from the database in a new transaction 
// using a connection.
let item = Item.read(connection).byKey(key)

// Write an sequence of items to the database 
// synchronously using a connection
items.write(connection) 

First, lets look at reading. We need to define the Readable protocol, and a struct which will act as the reader, Read.

/// Generic protocol for Reader types.
public protocol Readable {
    typealias ItemType
    typealias Database: DatabaseType

    var transaction: Database.Connection.ReadTransaction? { get }
    var connection: Database.Connection { get }
}

/// A generic structure used to read from a database type.
public struct Read<Item, D: DatabaseType>: Readable {
    public typealias ItemType = Item
    public typealias Database = D

    // etc, full implementation not shown

    internal init(_ transaction: D.Connection.ReadTransaction) {
        // etc, full implementation not shown
    }

    internal init(_ connection: D.Connection) {
        // etc, full implementation not shown
    }
}

The idea here is that on extensions to Readable we can implement readAtIndex etc using generic where clauses. And in fact, the code inside the extension will be identical for the 6 different type patterns. It's only the where constraints which are unique.

To create the Read value, we add an extension to Persistable, similar to the curried API above.

extension Persistable {
    public static func read<D: DatabaseType>(transaction: D.Connection.ReadTransaction) -> Read<Self, D> {
        return Read(transaction)
    }
}

And on Readable we can implement our API. Shown below is part of the value coding type with value coding metadata extension.

extension Readable where
    ItemType: ValueCoding,
    ItemType: Persistable,
    ItemType.Coder: NSCoding,
    ItemType.Coder.ValueType == ItemType,
    ItemType.MetadataType: ValueCoding,
    ItemType.MetadataType.Coder: NSCoding,
    ItemType.MetadataType.Coder.ValueType == ItemType.MetadataType {

    /**
    Reads the items at the indexes.

    - parameter indexes: a SequenceType of YapDB.Index values
    - returns: an array of `ItemType`
    */
    public func atIndexes<
        Indexes where
        Indexes: SequenceType,
        Indexes.Generator.Element == YapDB.Index>(indexes: Indexes) -> [ItemType] {
            return sync(atIndexesInTransaction(indexes))
    }

    // etc
}

For writing, our API is defined on Persistable with appropriate where clauses. The implementation of these are straightforward, and internally just use a combination of the functional and curried APIs. The example below is from the value types with value type metadata.

extension Persistable where
    Self: ValueCoding,
    Self.Coder: NSCoding,
    Self.Coder.ValueType == Self,
    Self.MetadataType: ValueCoding,
    Self.MetadataType.Coder: NSCoding,
    Self.MetadataType.Coder.ValueType == Self.MetadataType {

    /**
    Write the item synchronously using a connection.

    - parameter connection: a YapDatabaseConnection
    - returns: the receiver.
    */
    public func write<Connection: ConnectionType>(connection: Connection) -> Self {
        return connection.write(write())
    }
}

extension SequenceType where
    Generator.Element: Persistable,
    Generator.Element: ValueCoding,
    Generator.Element.Coder: NSCoding,
    Generator.Element.Coder.ValueType == Generator.Element,
    Generator.Element.MetadataType: ValueCoding,
    Generator.Element.MetadataType.Coder: NSCoding,
    Generator.Element.MetadataType.Coder.ValueType == Generator.Element.MetadataType {

    /**
    Write the items asynchronously using a connection.

    - parameter connection: a ConnectionType e.g. YapDatabaseConnection
    - parameter queue: a dispatch_queue_t for the completion block, defaults to the main queue.
    - parameter completion: an optional completion block which receives the receiver and returns void, defaults to .None
    */
    public func asyncWrite<Connection: ConnectionType>(connection: Connection, queue: dispatch_queue_t = dispatch_get_main_queue(), completion: ([Generator.Element] -> Void)? = .None) {
        return connection.asyncWrite(self, queue: queue, completion: completion)
    }
}

Hopefully this has shown that generics and protocol extensions in Swift 2.0 are very powerful language features. Using them we have been able to provide a broad API to perform read and write tasks with strict type safety for a 3rd party Objective-C framework. Additionally, we've been able to completely unit test these functions in isolation.

The general approach has been to first define the minimal interface required to support the API in a protocol. We adopted these protocols via extensions on the target classes, in this case YapDatabaseReadTransaction etc. When returning other framework objects we used a typealias, and defined new methods to avoid namespace clashes.

For generic parameters which are unknown, i.e. arguments or responses, avoid defining these methods in the protocol. Instead implement them in extensions with generic type classifiers.

For an object-orientated API, it is necessary to define a protocol to be extended. This can either be generic over Self as in the case of Persistable, or it can define generic nested types, which can be constrained, such as Readable. Combining these approaches allows framework consumers flexibility over their own architecture. If needed, we can return a generic type value which implements the generic protocol from an extension on a non-generic type.

Checkout the project for the full source code: danthorpe/YapDatabaseTransactions

Posted on Oct 14
Written by Daniel Thorpe