This is my research notebook. I'm an OSX / iOS indie developer. After 8 years of Objective-C I really enjoy Swift nowadays. Trying to publish all my research on Development, Swift & other technologies here.

Wed, 10 Jan 2018 #

Useful Optional Extensions

Optionals are a staple of Swift. I guess everybody will agree that they are a huge boon insofar as they force us to properly handle edge cases. The Optional language feature alone removes a whole category of bugs from the development process.

However, the API surface of Swift's optional is rather limited. The Swift documentation lists just a couple of methods / properties on Optional - if we ignore customMirror and debugDescription:

var unsafelyUnwrapped: Wrapped { get } 
func map<U>(_ transform: (Wrapped) throws -> U) rethrows -> U? 
func flatMap<U>(_ transform: (Wrapped) throws -> U?) rethrows -> U?

The reason why optionals are still very useful even though they have such a small amount of methods is that the Swift syntax makes up for it via features such as optional chaining, pattern matching, if let or guard let. In some situations, though, this manifests itself in unnecessary line noise. Sometimes, a very succinct method will let you express a concept in one short line of code instead of multiple lines of combined if let statements.

I've sifted through Swift Projects on Github as well as the optional implementations of other languages such as Rust, Scala, or C# in order to find a couple of useful additions to Optional. Below are 14 useful Optional extensions. I'll describe them by category and then give a couple of examples per category. Finally, I'll write a more involved example that uses several extensions at once.

1 Emptiness

extension Optional {
    /// Returns true if the optional is empty
    var isNone: Bool {
	return self == .none

    /// Returns true if the optional is not empty
    var isSome: Bool {
	return self != .none

Those are the most basic additions to the optional type. The implementation could also use a switch pattern match instead, but the nil comparison is much shorter. What I like about these additions is that they move the concept of an empty optional being nil away from your code. This might just as well be an implementation detail. Using optional.isSome feels much cleaner and less noisy than if optional == nil:

// Compare
guard leftButton != nil, rightButton != nil else { fatalError("Missing Interface Builder connections") }

// With
guard leftButton.isSome, rightButton.isSome else { fatalError("Missing Interface Builder connections") }

2 Or

extension Optional {
    /// Return the value of the Optional or the `default` parameter
    /// - param: The value to return if the optional is empty
    func or(_ default: Wrapped) -> Wrapped {
	return self ?? `default`

    /// Returns the unwrapped value of the optional *or*
    /// the result of an expression `else`
    /// I.e. optional.or(else: print("Arrr"))
    func or(else: @autoclosure () -> Wrapped) -> Wrapped {
	return self ?? `else`()

    /// Returns the unwrapped value of the optional *or*
    /// the result of calling the closure `else`
    /// I.e. optional.or(else: { 
    /// ... do a lot of stuff
    /// })
    func or(else: () -> Wrapped) -> Wrapped {
	return self ?? `else`()

    /// Returns the unwrapped contents of the optional if it is not empty
    /// If it is empty, throws exception `throw`
    func or(throw exception: Error) throws -> Wrapped {
	guard let unwrapped = self else { throw exception }
	return unwrapped

extension Optional where Wrapped == Error {
    /// Only perform `else` if the optional has a non-empty error value
    func or(_ else: (Error) -> Void) {
	guard let error = self else { return }

Another abstraction on the isNone / isSome concept is being able to specify instructions to be performed when the invariant doesn't hold. This saves us from having to write out if or guard branches and instead codifies the logic into a simple-to-understand method.

This concept is so useful, that it is defined in three distinct functions.

2.1 Default Value

The first one returns the wrapped value of the optional or a default value:

let optional: Int? = nil
print(optional.or(10)) // Prints 10

2.2 Default Closure

The second one is very similar to the first one, however it allows to return a default value from a closure instead.

let optional: Int? = nil
optional.or(else: secretValue * 32)

Since this uses the @autoclosure parameter we could actually use just the second or implementation. Then, using a just a default value would automatically be converted into a closure returning the value. However, I prefer having two separate implementations as that allows users to also write closures with more complex logic.

let cachedUserCount: Int? = nil
return cachedUserCount.or(else: {
   let db = database()
   guard db.failures.isEmpty else { return 0 }
   return db.amountOfUsers

A really nice use case for or is code where you only want to set a value on an optional if it is empty:

if databaseController == nil {
  databaseController = DatabaseController(config: config)

This can be replaced with the much nicer:

databaseController = databaseController.or(DatabaseController(config: config)

2.3 Throw an error

This is a very useful addition as it allows to merge the chasm between Optionals and Error Handling in Swift. Depending on the code that you're using, a method or function may express invalid behaviour by returning an empty optional (imagine accessing a non-existing key in a Dictionary) or by throwing an Error. Combining these two oftentimes leads to a lot of unnecessary line noise:

func buildCar() throws -> Car {
  let tires = try machine1.createTires()
  let windows = try machine2.createWindows()
  guard let motor = externalMachine.deliverMotor() else {
    throw MachineError.motor
  let trunk = try machine3.createTrunk()
  if let car = manufacturer.buildCar(tires, windows,  motor, trunk) {
    return car
  } else {
    throw MachineError.manufacturer

In this example, we're building a car by combining internal and external code. The external code (external_machine and manufacturer) choose to use optionals instead of error handling. This makes the code unnecessary complicated. Our or(throw:) function makes this much more readable:

func build_car() throws -> Car {
  let tires = try machine1.createTires()
  let windows = try machine2.createWindows()
  let motor = try externalMachine.deliverMotor().or(throw: MachineError.motor)
  let trunk = try machine3.createTrunk()
  return try manufacturer.buildCar(tires, windows,  motor, trunk).or(throw: MachineError.manufacturer)

2.4 Handling Errors

The code from the Throw an error section above becomes even more useful when you include the following free function that was proposed by Stijn Willems on Github. Thanks for the suggestion!

func should(_ do: () throws -> Void) -> Error? {
    do {
	try `do`()
	return nil
    } catch let error {
	return error

This free function (alternatively, you could make it a class method on optional) will perform a do {} catch {} block and return an error if and only if the closure `do` resulted in an error. Take, the following Swift code as an example:

do {
  try throwingFunction()
} catch let error {

This is one of the basic tennets of error handling in Swift, and it introduces quite a lot of line noise. With the free function above, you can reduce it to this simple on-liner:

should { try throwingFunction) }.or(print($0))

I feel that there're many situations where such a one-liner for error handling would be very beneficient.

2.5 Map

As we saw above, map and flatMap are the only methods that Swift offers on Optionals. However, even those can be improved a bit to be more versatile in many situations. There're two additional variations on map that allow defining a default value similar to how the or variants above are implemented:

extension Optional {
    /// Maps the output *or* returns the default value if the optional is nil
    /// - parameter fn: The function to map over the value
    /// - parameter or: The value to use if the optional is empty
    func map<T>(_ fn: (Wrapped) throws -> T, default: T) rethrows -> T {
	return try map(fn) ?? `default`

    /// Maps the output *or* returns the result of calling `else`
    /// - parameter fn: The function to map over the value
    /// - parameter else: The function to call if the optional is empty
    func map<T>(_ fn: (Wrapped) throws -> T, else: () throws -> T) rethrows -> T {
	return try map(fn) ?? `else`()

The first one will allow you to map the contents of an optional to a new type T. If the optional is empty, you can define a default value that should be used instead:

let optional1: String? = "appventure"
let optional2: String? = nil

// Without
print({ $0.count }) ?? 0)
print({ $0.count }) ?? 0)

// With 
print({ $0.count }, default: 0)) // prints 10
print({ $0.count }, default: 0)) // prints 0

The changes are minimal, but we're moving away from having to use the ?? operator and can instead express the operation more clearly with the default keyword.

The second variant is very similar. The main difference is that it accepts (again) a closure returning value T instead of value T. Here's a brief example:

let optional: String? = nil
print({ $0.count }, else: { "default".count })

3 Combining Optionals

This category contains four functions that allow you to define relations between multiple optionals.

extension Optional {
    /// Tries to unwrap `self` and if that succeeds continues to unwrap the parameter `optional`
    /// and returns the result of that.
    func and<B>(_ optional: B?) -> B? {
	guard self != nil else { return nil }
	return optional

    /// Executes a closure with the unwrapped result of an optional.
    /// This allows chaining optionals together.
    func and<T>(then: (Wrapped) throws -> T?) rethrows -> T? {
	guard let unwrapped = self else { return nil }
	return try then(unwrapped)

    /// Zips the content of this optional with the content of another
    /// optional `other` only if both optionals are not empty
    func zip2<A>(with other: Optional<A>) -> (Wrapped, A)? {
	guard let first = self, let second = other else { return nil }
	return (first, second)

    /// Zips the content of this optional with the content of another
    /// optional `other` only if both optionals are not empty
    func zip3<A, B>(with other: Optional<A>, another: Optional<B>) -> (Wrapped, A, B)? {
	guard let first = self,
	      let second = other,
	      let third = another else { return nil }
	return (first, second, third)

These four functions all share that they take an additional optional as a parameter and return another optional value. However, they're all quite different in what they achieve.

3.1 Dependencies

and<B>(_ optional) is useful if the unpacking of an optional is only required as a invariant for unpacking another optional:

// Compare
if user != nil, let account = userAccount() ...

// With
if let account = user.and(userAccount()) ...

In the example above, we're not interested in the unwrapped contents of the user optional. We just need to make sure that there is a valid user before we call the userAccount function. While this relationship is kinda codified in the user != nil line, I personally feel that the and makes it more clear.

3.2 Chaining

and<T>(then:) is another very useful function. It allows to chain optionals together so that the output of unpacking optional A becomes the input of producing optional B. Lets start with a simple example:

protocol UserDatabase {
  func current() -> User?
  func spouse(of user: User) -> User?
  func father(of user: User) -> User?
  func childrenCount(of user: User) -> Int

let database: UserDatabase = ...

// Imagine we want to know the children of the following relationship:
// Man -> Spouse -> Father -> Father -> Spouse -> children

// Without
let childrenCount: Int
if let user = database.current(), 
   let father1 = database.father(user),
   let father2 = database.father(father1),
   let spouse = database.spouse(father2),
   let children = database.childrenCount(father2) {
  childrenCount = children
} else {
  childrenCount = 0

// With
let children = database.current().and(then: { database.spouse($0) })
     .and(then: { database.father($0) })
     .and(then: { database.spouse($0) })
     .and(then: { database.childrenCount($0) })

There're a lot of improvements when using the version with and(then). First of all, you don't have to come up with superfluous temporary variable names (user, father1, father2, spouse, children). Second, we clearly have less code. Also, using the or(0) instead of a complicated let childrenCount is so much easier to read.

Finally, the original Swift example can easily lead to logic errors. You may not have noticed, but there's a bug in the example. When writing lines like that, copy paste errors can easily be introduced. Do you see the error?

Yeah, the children property should be created by calling database.childrenCount(spouse) but I wrote database.childrenCount(father2) instead. It is difficult to spot errors like that. The and(then:) example makes it much easier because it always relies on the same variable name $0.

3.3 Zipping

This is another variation on an existing Swift concept. The zip method on optional will allow us to combine multiple optionals and unwrap them together or not at all. I've just provided implementations for zip2 and zip3 but nothing prevents you from going up to zip22 (Well, maybe sanity and compiler speed).

// Lets start again with a normal Swift example
func buildProduct() -> Product? {
  if let var1 = machine1.makeSomething(),
    let var2 = machine2.makeAnotherThing(),
    let var3 = machine3.createThing() {
    return finalMachine.produce(var1, var2, var3)
  } else {
    return nil

// The alternative using our extensions
func buildProduct() -> Product? {
  return machine1.makeSomething()
     .zip3(machine2.makeAnotherThing(), machine3.createThing())
     .map { finalMachine.produce($0.1, $0.2, $0.3) }

Less code, clearer code, more beautiful code. However, as a downside, this code is also more involved. The reader has to know and understand zip in order to easily grasp it.

3.4 On

extension Optional {
    /// Executes the closure `some` if and only if the optional has a value
    func on(some: () throws -> Void) rethrows {
	if self != nil { try some() }

    /// Executes the closure `none` if and only if the optional has no value
    func on(none: () throws -> Void) rethrows {
	if self == nil { try none() }

These two short methods will allow you to perform side effects if an optional is empty or not. In contrast to the already discussed methods, these ignore the contents of the optional. So on(some:) will only execute the closure some if the optional is not empty but the closure some will not get the unwrapped contents of the optional.

/// Logout if there is no user anymore
self.user.on(none: { AppCoordinator.shared.logout() })

/// self.user is not empty when we are connected to the network
self.user.on(some: { AppCoordinator.shared.unlock() })

3.5 Various

extension Optional {
    /// Returns the unwrapped value of the optional only if
    /// - The optional has a value
    /// - The value satisfies the predicate `predicate`
    func filter(_ predicate: (Wrapped) -> Bool) -> Wrapped? {
	guard let unwrapped = self,
	    predicate(unwrapped) else { return nil }
	return self

    /// Returns the wrapped value or crashes with `fatalError(message)`
    func expect(_ message: String) -> Wrapped {
	guard let value = self else { fatalError(message) }
	return value

3.5.1 Filter

This is a simple method which works like an additional guard to only unwrap the optional if it satisfies a predictate. Here's an example. Imagine we want to upgrade all our old users to a premium account for sticking with us for a long time:

// Only affect old users with id < 1000
// Normal Swift
if let aUser = user, < 1000 { aUser.upgradeToPremium() }

// Using `filter`
user.filter({ $ < 1000 })?.upgradeToPremium()

Here, user.filter feels like a much more natural implementation. Also, it only implements what already exists for Swift's collections.

3.5.2 Expect

This is one of my favorites. Also, I shamelessly stole it from Rust. I'm trying very hard to never force unwrap anything in my codebase. Similar for implicitly unwrapped optionals.

However, this is tricky when working with interface builder outlets. A common pattern that I observed can be seen in the following function:

func updateLabel() {
  guard let label = valueLabel else {
    fatalError("valueLabel not connected in IB")
  label.text = state.title

The alternative solution, obviously, would be to just to force unwrap the label, as that leads to a crash just like fatalError. Then, I'd have to insert ! though, also it wouldn't give me a nice succinct description of what actually is wrong. The better alternative here is to use expect as implemented above:

func updateLabel() {
  valueLabel.expect("valueLabel not connected in IB").text = state.title

4 Example

So now that we've seen a couple of (hopefully) useful Optional extensions, I'll set up an example to better see how some of these extensions can be combined to simplify optional handling. First, we need a bit of context. Forgive me for the rather unconventional and impossible example:

You're working in the 80s at a shareware distributor. A lot of student programmers are working for you and writing new shareware apps and games every month. You need to keep track of how many were sold. For that, you recieve an XML file from accounting and you need to parse it and insert it into the database (isn't it awesome how in this version of the 80s there's Swift to love but also XML to hate?). Your software system has an XML parser and a database (both written in 6502 ASM of course) that implement the following protocols:

protocol XMLImportNode {
    func firstChild(with tag: String) -> XMLImportNode?
    func children(with tag: String) -> [XMLImportNode]
    func attribute(with name: String) -> String?

typealias DatabaseUser = String
typealias DatabaseSoftware = String
protocol Database {
    func user(for id: String) throws -> DatabaseUser
    func software(for id: String) throws -> DatabaseSoftware
    func insertSoftware(user: DatabaseUser, name: String, id: String, type: String, amount: Int) throws
    func updateSoftware(software: DatabaseSoftware, amount: Int) throws

A typical file looks like this (behold the almighty XML):

 <user name="" id="158">
   <package type="game" name="Maniac Mansion" id="4332" amount="30" />
   <package type="game" name="Doom" id="1337" amount="50" />
   <package type="game" name="Warcraft 2" id="1000" amount="10" />

Our original Swift code to parse the XML looks like this:

enum ParseError: Error {
    case msg(String)

func parseGamesFromXML(from root: XMLImportNode, into database: Database) throws {
    guard let users = root.firstChild(with: "users")?.children(with: "user") else {
	throw ParseError.msg("No Users")
    for user in users {
	guard let software = user.firstChild(with: "software")?
		.children(with: "package"),
	    let userId = user.attribute(with: "id"),
	    let dbUser = try? database.user(for: userId)
	    else { throw ParseError.msg("Invalid User") }
	for package in software {
	    guard let type = package.attribute(with: "type"),
	    type == "game",
	    let name = package.attribute(with: "name"),
	    let softwareId = package.attribute(with: "id"),
	    let amountString = package.attribute(with: "amount")
	    else { throw ParseError.msg("Invalid Package") }
	    if let existing = try? softwareId) {
		try database.updateSoftware(software: existing, 
					      amount: Int(amountString) ?? 0)
	    } else {
		try database.insertSoftware(user: dbUser, name: name, 
					      id: softwareId, 
					    type: type, 
					  amount: Int(amountString) ?? 0)

Lets apply what we learned above:

func parseGamesFromXML(from root: XMLImportNode, into database: Database) throws {
    for user in try root.firstChild(with: "users")
		    .or(throw: ParseError.msg("No Users")).children(with: "user") {
	let dbUser = try user.attribute(with: "id")
		    .and(then: { try? database.user(for: $0) })
		    .or(throw: ParseError.msg("Invalid User"))
	for package in (user.firstChild(with: "software")?
		    .children(with: "package")).or([]) {
	    guard (package.attribute(with: "type")).filter({ $0 == "game" }).isSome
		else { continue }
	    try package.attribute(with: "name")
		.zip3(with: package.attribute(with: "id"), 
		   another: package.attribute(with: "amount"))
		.map({ (tuple) -> Void in
		    switch try? tuple.1) {
		    case let e?: try database.updateSoftware(software: e, 
							       amount: Int(tuple.2).or(0))
		    default: try database.insertSoftware(user: dbUser, name: tuple.0, 
							   id: tuple.1, type: "game", 
						       amount: Int(tuple.2).or(0))
		}, or: { throw ParseError.msg("Invalid Package") })

If we look at this, then there're two things that immediately come to mind:

  1. Less Code
  2. More Complicated Looking Code

I deliberately went into overdrive when utilizing the various Optional extensions. Some of them fit better while others seem to be a bit misplaced. However, the key is not to solely rely on these extensions (like I did above) when using optionals but instead to mix and match where it makes most sense. Compare the two implementations and consider which from the second example you'd rather implement with Swift's native features and which feel better when using the Optional extensions.

That's all for today, thanks for reading!

If you read this far, you should follow me (@terhechte)
on Twitter

    Sun, 10 Dec 2017 #

    Patterns for Working With Associated Types

    1 Useful Patterns for Working Your Way Around Associated Types

    This blog post is based on a talk I gave at AppBuilders 2016 explaining protocols with associated types offering tips for using them.

    Swift is a powerful language with a very powerful type system. Among the features that define said type system are associated types. They can be defined on a protocol to allow implementors of the protocol to specialize certain types in a generic way:

    protocol Example {
      associatedtype Value
      var value: Value { get }

    In the snippet above, any type that implements the Example protocol has to define the Value type. Protocols with associated types can be understood as unfinished types. Compared to regular protocols, which can be used within Swift like normal types, those protocols can only be used as a generic constraint. This means that once your type requires an associated type, using it suddenly becomes much more complicated.

    The example below shows an example of finishing a type. By explicitly telling the compiler that the Value type is Int it is now able to understand ImplementExample fully.

    struct ImplementExample: Example {
      typealias Value = Int

    Associated types are useful for a certain kind of problems where subclassing and composition does allow you to build the right kind of abstractions. However, this is a seperate topic. The topic of this article, on the other hand, is what to do when you end up with associated types trouble.

    2 Associated Types Trouble

    The classic example of associated types trouble certainly is the following Swift error message:

    protocol 'Bookmarkable' can only be used as a generic constraint because it has Self 
    or associated type requirements
    var bookmarks: [Bookmarkable]

    This happens once your type conforms to a protocol which conforms to Equatable:

    protocol Bookmarkable: Equatable {
    struct User {
        var bookmarks: [Bookmarkable]

    Here, the problem is that Equatable contains a method == which has two paramters of type Self. Protocol Methods with Self parameters automatically opt in to associated types.

    In this article, we will be investigating several patterns that allow you to work your way around the associated type requirement or that show how such a type can be handled.

    3 Working Around Associated Types

    4 Make Your Types Equatable

    The first solution for the archetypical problem is also a really simple one. Instead of enforcing Equatable on your custom protocol, you can simply require your full fledged, final, types to conform to the Equatable protocol instead of your custom protocol. Consider the previously defined Bookmarkable protocol:

    protocol Bookmarkable {
    struct Bookmark: Bookmarkable, Equatable {
      var identifier: Int
    func ==(lhs: Bookmark, rhs: Bookmark) -> Bool {
      return lhs.identifier == rhs.identifier
    var myBookmarks: [Bookmark] = []

    In the example above, the Equatable requirement actually stems from the Bookmark type conforming to the Equatable protocol, not the Bookmarkable protocol itself. The actual Equatable information, however, lies in the new identifier property, which has been added to the Bookmark struct. As you can easily see, this also requires you to make the myBookmarks array require only elements of type Bookmark. A serious disgression if you're used to using protocols like partially anonymous types. A better solution, if your design allows for it, goes one step further by enforcing the new property which we introduced in this example.

    4.1 Equatable Properties

    Here, the idea is that we take one of the types that already implement Equatable in a proper way (i.e. Int, String, …) and add a new property requirement to our Bookmarkable protocol. Then, we can use this property to add Equatable support without actually implementing Equatable:

    protocol Bookmarkable {
        var identifier: Int { get }
    struct Bookmark: Bookmarkable {
        var identifier: Int
    var myBookmarks: [Bookmarkable] = []

    The main change, compared to the code above, is that the var identifier moved to the Bookmarkable protocol and that we removed the func ==.

    While this works better, it still has a major deficit. Since Bookmarkable does not directly comply with Equatable, you will not gain the standard library's methods that specifically deal with Equatable types. So instead of being able to call Array.contains like this:

    let ourBookmark = Bookmark(identifier: 0)
    let result = myBookmarks.contains(ourBookmark)

    You will have to use the more verbose closure-based version:

    let ourBookmark = Bookmark(identifier: 0)
    let result = myBookmarks.contains { (bookmark) -> Bool in
        return bookmark.identifier == ourBookmark.identifier

    5 Associated Types and Self

    Another vector which can introduce associated types into your codebase is the usage of Self:

    protocol Example {
      /// Indirect Associated Type
      var builder: Self { get }
      /// Indirect Associated Type
      func makeSomething(with example: Self)
    var myExamples: [Example] = []

    As you can see in the example above, using Self as a method parameter or using Self as a property type automatically introduces an associated type (like we saw with Equatable, earlier).

    The most helpful note here is that once you use a method instead of a property in order to return something of type Self you will not opt in to an associated type:

    protocol Example {
      /// No Indirect Associated Type
      func builder() -> Self
    var myExamples: [Example] = []

    This example works fine. No indirect associated type is introduced.

    6 Method-Only Types

    If your associated type requirement doesn't come from Equatable conformance but instead from your own use, you can double-check if you actually need these associated types.

    Take this example of a validator type:

    protocol Validator {
        associatedtype I
        func validate(_ input: I) -> Bool

    As the associated type is only used in one method, you can alternatively just make it a generic method and thus save yourself from introducing unnecessary unfinished types:

    protocol Validator {
        func validate<I>(_ input: I) -> Bool

    7 Hiding Behind Protocols

    This is an especially useful and flexible pattern. It can be used in many situations where you want to use protocols with associated types like a normal, full fledged type, but still be able to opt in to the generic part if necessary. The idea here is that you define two protocols that share common methods. Only one of those protocols contains associated types, the other does not. Your types conform to both protocols. This means that you can use the normal protocol as a type for all situations. If you, then, need to use the parts of the type that only affect the associated type, you can do so by means of a runtime cast.

    Begin by defining an associated Protocol ExampleAssociatedProtocol that is shadowed by a normal Protocol ExampleProtocol.

    /// The `Normal` Protocol
    protocol ExampleProtocol {
      var anyValue: Any { get }
    /// The Protocol with an associated type
    protocol ExampleAssociatedProtocol: ExampleProtocol {
      associatedtype Value
      /// Retrieving the actual associated type
      var value: Value { get }
    /// Conform to the `ExampleProtocol`
    extension ExampleAssociatedProtocol {
      var anyValue: Any {
        return value

    Now, you can use the ExampleProtocol as a normal type throughout your app in all situations where a protocol with an associated type would otherwise fail:

    struct World {
      var examples: [ExampleProtocol]
      let example: ExampleProtocol
      func generate() -> ExampleProtocol { 
        return example

    However, if you need to access the property that is specific to the ExampleAssociatedProtocol (value) then you can do so through at runtime.

    /// Custom type implementing `ExampleAssociatedProtocol`
    struct IntExample: ExampleAssociatedProtocol {
      var value: Int
    /// Custom type implementing `ExampleAssociatedProtocol`
    struct StringExample: ExampleAssociatedProtocol {
      var value: String
    /// Shadowing via `ExampleProtocol`
    let myExamples: [ExampleProtocol] = 
        [StringExample(value: "A"), IntExample(value: 10)]
    /// Runtime Casting
    for aNormalExample in myExamples {
      if let anAssociatedExample = aNormalExample as? IntExample {
      if let anAssociatedExample = aNormalExample as? StringExample {

    This will print "A10" as both types (IntExample and StringExample) are being identified at runtime via a cast from ExampleProtocol.

    8 Type Erasure

    8.1 The Problem

    Quite often, when Swift's associated types are dicussed, type erasure is mentioned as another solution to the problem of handling the issues that associated types bring along.

    Type Erasure in the context of associated types solves one particular problem. We'll use computers as an example. Back in the golden age of desktop operating systems, you could buy a desktop computer with many non-X86 CPU architectures: PowerPC, Alpha, Sparc, 68000, and so on. One of the many differences were the endianness of the architecture. Lets model these computers in Swift:

    protocol CPU {
        var littleEndian: Bool { get }
    struct PowerPC: CPU {
        let littleEndian = false
    struct X86: CPU {
        let littleEndian = true

    Next up, we want to define a protocol for a computer. It could be a desktop computer or a phone or maybe a game console, so we use a protocol. In order to model the CPU, we're using an associated type, so that the actual type can define the CPU:

    protocol Computer {
        associatedtype ProcessorType: CPU
        var processor: ProcessorType { get }
        var processorCount: Int { get }

    Based on this, we can now define a couple of systems:

    struct PowerMacG5: Computer {
        let processor = PowerPC()
        let processorCount = 2
    struct Xbox360: Computer {
        let processor = PowerPC()
        let processorCount = 1
    struct MacPro: Computer {
        let processor = X86()
        let processorCount = 1

    Now that we have all this, we'd like to perform a computation on all PowerPC based computers. I.e. something like:

    let powerComputers = [PowerMacG5(), Xbox360()]

    However, what would be the type of this? We can't use the Computer protocol, as it contains associated types. However, the associated types for the PowerMacG5 and the Xbox360 are the same, so in terms of types, Swift ought to understand that those things are kinda similar. However, there's no way to (easily) express this in the type system; both PowerMacG5 and Xbox360 are not the correct types for the array:

    // None of those work
    let powerComputers: [PowerMacG5] = [PowerMacG5(), Xbox360]
    let powerComputers: [Xbox360] = [PowerMacG5(), Xbox360]
    let powerComputers: [Computer] = [PowerMacG5(), Xbox360]

    Type erasure is a solution for this. The idea is to box the actual type into a generic wrapper so that Swift can coalesce around wrapper + type. The solution we're aiming for would look like this in the end:

    let powerComputers: [AnyComputer<PowerPC>] = [AnyComputer(PowerMacG5()), AnyComputer(Xbox360())]

    Now we would have our shared type, in this case it is AnyComputer<CPU>. Where does this mystic AnyComputer come from? We have to build it ourselves. This is a multi-step process, and requires quite a bit of boilerplate. We will start simple and expand step by step. This solution requires multiple types.

    8.2 An Abstract Class

    In essense, what we're going to build, is a generic wrapper (or box) that hosts a type conforming to a protocol with an associated type. It does so by implementing the requirements of the protocol and forwarding all invocations to the boxed type.

    The first new type we need for that is a base class that acts as a abstract class:

    class AnyComputerBase<Processor: CPU>: Computer {
        var processor: Processor {
        var processorCount: Int {

    This class should never be initialized, as it only provides an abstract template of what subclasses should implement. While other languages (like Java) allow explicitly marking classes as abstract, Swift doesn't offer us a way to do so. One solution to this is adding a fileprivate init to this class. However as that requires subclasses to be in the same file as this superclass, we can also just make the whole class private with an even better result. Now, other parts of the code won't even know about the existence of AnyComputerBase or even initialize it:

    private class AnyComputerBase<Processor: CPU>: Computer {

    Why do we even need this, and what does it do? As you can see, it just implements the Computer protocol by implementing the requirements and doing nothing in there. The more important part is that it moves the associated type from the protocol into a generic type for the class: AnyComputerBase<Processor: CPU>.

    Swift automatically figures out that Processor is the typealias for Computer.ProcessorType. However, when in doubt you can also add an extra typealias:

    class AnyComputerBase<Processor: CPU>: Computer {
      typealias ProcessorType = Processor

    8.3 A Box Type

    The next step is the most difficult to understand part of type erasure, which means that after this, it'll be easy. We will introduce another private type. This will be the actual box that houses our original type (the XBox360 or the PowerMac G5). Let's start by having a look at the code:

    private class AnyComputerBox<ConcreteComputer: Computer>: 
        private let internalComputer: ConcreteComputer
        override var processor: ConcreteComputer.ProcessorType {
    	return internalComputer.processor
        override var processorCount: Int {
    	return internalComputer.processorCount
        init(_ computer: ConcreteComputer) {
    	internalComputer = computer

    The most important concept here can be found in the very first line:

    private class AnyComputerBox<ConcreteComputer: Computer>: 

    Here, we define a new type AnyComputerBox which is generic over any computer (ConcreteComputer). This new type, then, is a subclass of our earlier abstract class AnyComputerBase. Remember that AnyComputerBase made the original ProcessorType of the Computer protocol generic by adding it as a generic parameter CPU. Now, our new box has a different generic type (Computer) and provides only its associated type ProcessorType to the abstract superclass. In a simpler explanation, this is what happens (in a mock language):

    1. Computer<CPU>
    2. AnyComputerBase<Processor: CPU>: Computer<CPU> where Computer.CPU = Processor
    3. AnyComputerBox<ConcreteComputer: Computer>: AnyComputerBase<ConcreteComputer.ProcessorType>

    So the box (AnyComputerBox) subclasses the abstract class and forwards in the Processor type via its own generic Computer type which also has a ProcessorType.

    Why do we do this? It makes the box generic over any computer so that any computer can be boxed into it.

    The rest of the class is simple. There's an internal computer internalComputer which is the actual type conforming to the Computer protocol. We're also overriding the two classes that are required by the protocol and forwarding the implementations of the internalComputer. Finally we have an initializer with a new ConcreteComputer (i.e. the Computer protocol).

    8.4 Putting it all together

    In the next and final step, we're building the actual type that will be used as the proverbial type eraser. Just as before, lets have a look at the code first:

    final class AnyComputer<Processor: CPU>: Computer {
        private let box: AnyComputerBase<Processor>
        var processor: Processor {
    	return box.processor
        var processorCount: Int {
    	return box.processorCount
        init<Concrete: Computer>(_ computer: Concrete) 
    	where Concrete.ProcessorType == Processor {
          box = AnyComputerBox(computer)

    This AnyComputer conforms to the Computer protocol and is generic over the CPU type that the protocol requires. Once again, we implement the protocol requirements (processor, and processorCount) and forward to a boxed type. This time we're forwarding to private let box: AnyComputerBase<Processor>. This box is set in the initializer where most of the magic happens:

    init<Concrete: Computer>(_ computer: Concrete) 
        where Concrete.ProcessorType == Processor {
      box = AnyComputerBox(computer)

    The problem with protocols with associated types is that you can't use them as property types. Here, init requires any type conforming to the Computer protocol. This is done by having a method-generic type Concrete that requires Computer conformance. Even more, we also add a constraint that makes sure that the generic Processor type of the new AnyComputer class is the same type as the associated type of the Concrete Computer type.

    And now comes the kicker: Since we cannot set a property as being of type Computer we, instead, have a property that is of AnyComputerBase with a generic type for the Processor. As our AnyComputerBox type is a subclass of AnyComputerBase we can literally put any box (that is a subclass of AnyComputerBase into this property. In this case, we're creating a new box with the Concrete Computer.

    Then we return the implementations of the contents of the box (i.e. the actual Concrete Computer) in our Computer implementations:

    var processorCount: Int {
        return box.processorCount

    8.5 Using It

    With all this machinery in place, we can finally use this in order to have different types (which share an associated type) in one container:

    let powerComputers: [AnyComputer<PowerPC>] = 
        [AnyComputer(PowerMacG5()), AnyComputer(Xbox360())]

    9 Conclusion

    Associated types are a powerful concept however they come with a fair share of difficulties. Most notably, as soon as you introduce an associated type you can't use it like you'd use normal full types. This article provided several patterns that make it a bit easier to handle associated type problems in your codebase. Each of these patterns has downsides though. In general, if you intend to use associated types in a protocol, one of the best solutions is to try to only use the types that implement this protocol instead of the protocol itself. Because then you don't even need those patterns.

    If you read this far, you should follow me (@terhechte)
    on Twitter

      Sun, 8 Oct 2017 #

      Taming SourceKitService for Less Xcode Memory Consumption

      1 Update [10/15/2017]

      It seems that Xcode 9.1 beta 2 fixes this issue:

      In my preliminary testing, everything worked fine. This feels really good.

      2 Original Article

      There were recently two popular Swift posts on Hacker News1 , 2, and one issue I saw coming up multiple times was the memory consumption of the tooling nee Xcode. One particular problem is that for some codebases the Swift sourcecode process SourceKitService consumes a huge amount of memory. I've had it rise to 30GB and beyond - at which point my system usually stalls and I'm not able to continue working for a couple of minutes.

      Oftentimes memory issues like these can be solved by reviewing your sourcecode with the same tools you also use to reduce your compile times. See:

      However, for some, complex, codebases this may not be enough. I've employed an awful little hack in order to at least keep my machine from stalling. I wrote a small little bash script that check the memory consumption of the SourceKitService every n seconds and if it goes beyond x megabytes of memory (by default 5.000) I restart it. I feel that this may be useful to some others so I'm sharing it here for posterity. Note that this is an awful hack and future versions of SourceKitService will probably (hopefully!) not need this anymore. Meanwhile, this might be of help to others:

      # Amount of seconds to wait between measures
      # Limit memory consumption to this many megabytes before killing the process
      while true; do 
        fields=`ps aux -m | grep -v grep | grep -i $name | tr -s ' '`
        mem=`echo $fields | cut -d ' ' -s -f 6| awk '{$1=$1/1024; print $1;}' | cut -d '.' -f 1`
        pid=`echo $fields | cut -d ' ' -s -f 2`
        if [ -z "$mem" ]; then
            echo "$name not running"
            sleep 15
        if [ "$mem" -gt $x ]; then
            echo "Killing $name pid $pid with mem $mem"
            kill -9 $pid
            sleep 5
        sleep $n

      To use this just paste that code into a file (say and do:

      chmod +x ./

      If you want to kill it, just hit CTRL=C.



      Why many developers still prefer Objective-C


      Dictionary and Set Improvements in Swift 4.0

      If you read this far, you should follow me (@terhechte)
      on Twitter

        Sat, 30 Sep 2017 #

        Value Types for Simple Difference Detection

        Following sage advice I received from John Sundell half a year ago (I am a slow learner) I will try to write about smaller pieces instead of focusing on longform content. This should make it easier to update appventure more often. Thanks John!.

        Today, I'll write about value types: Value types are a very useful addition to Swift. Objective-C did offer C struct types and (obviously) classical value types such as numbers, but Swift goes way beyond that by allowing us to also define more complex structures such as Array, Set, or Dictionary as value types. One of the best properties of value types is that they can easily be compared given that they're values:

        let one = [1, 2, 3]
        let two = [1, 2, 3]
        print(one == two) // returns true

        This way of easily comparing arrays is something we can use to implement a simple difference detection algorithm without using more heavy-weight solutions. Imagine you have a simple app that downloads a list of entries from a server and displays them in a table view. Once a minute you download a new list and reload the table view. This is not a particularly nice solution as you're reloading the table view even when there're no changes. If your data is defined as a struct and you implement the Equatable protocol, then you can simply use the equality operator to see if the table view needs to be reloaded:

        struct Data: Equatable {
          let username: String
          let userid: Int
          static func ==(lhs: Data, rhs: Data) -> Bool {
            return lhs.username == rhs.username && lhs.userid == rhs.userid
        let oldData: [Data] = currentData()
        let newData: [Data] = retrieveNewData()
        guard oldData != newData else {
        updateData(with: newData)

        However, it may be that your data is modelled using struct types, but that they're very complex and change often. So you've never implemented Equatable. Then you have three different options:

        1. Wait for Swift 4.1 which will hopefully merge a PR which will auto-generate Equatable for struct types if all properties of a struct also conform to Equatable.
        2. Use Krzysztof Zabłocki's Sourcery which is a meta programming framework that allows you to auto generate things like Equatable conformance for your types (and much more).
        3. This I will explain in more detail as it is also a pattern that you can use if your data is modelled using class types.

        The idea here is to store just the absolutely necessary information in a seperate difference detection cache. Imagine the following data model:

        final class Story {
          let story_id: Int
          let views: Int
          let last_updated: TimeInterval
          let description: String

        In this example, a Story will never change its id and description. In order to create a simple cache, we can now just use the information we absolutely need to determine a change and store them in tuples. Tuples with up to 6 elements will automatically generate Equatable conformance:

        let a = (1, 2, 3)
        let b = (1, 2, 3)
        print(a == b)

        With this in mind, we can generate our tuples:

        let newInformation: [(Int, Int, TimeInterval)] = 
{ ($0.story_id, $0.views, $0.last_updated) })
        guard newInformation != oldInformation else { return }

        This is a simple solution that leverages value types to give us an easy solution for comparing sets of data. However, if you also need to determine insertions, deletions and moves then you can still do so with value types, but you need a proper diff algorithm such as Dwifft.

        Nevertheless, for many simpler use cases it is good to remember that we can easily build a comparison cache of tuples or structs to determine general changes to data.

        If you read this far, you should follow me (@terhechte)
        on Twitter

          Fri, 15 Jul 2016 #

          Data in Swift 3 parsing a Doom WAD File

          1 From NSData to Data in Swift 3

          Swift 3 encompasses many different small and big changes to the language. One of them is the introduction of value type wrappers for common Foundation reference types such as NSData (Data) or NSDate (Date). These new types differ not only in their memory behaviour and name, their methods also differ from their reference-based counterparts1. From small changes to new method names up to big changes like completely removed functionalities, these new value types require some getting used to. This post will try to highlight some of the bigger changes happened to Data the value-based wrapper for NSData.

          Even better, after going through the basics, we will write a small example application that will read and parse a Doom 2 WAD file.

          2 Basic Differences

          One of the most common usage scenarios for NSData is the loading and writing of data via these calls:

          func writeToURL(_ url: NSURL, atomically atomically: Bool) -> Bool
          func writeToURL(_ url: NSURL, options writeOptionsMask: NSDataWritingOptions) throws
          // ... (implementations for file: String instead of NSURL)
          init?(contentsOfURL url: NSURL)
          init(contentsOfURL url: NSURL, options readOptionsMask: NSDataReadingOptions) throws
          // ... (implementations for file: String instead of NSURL)

          For those basic usages, very little changed. The new Data type offers these methods:

          init(contentsOf: URL, options: ReadingOptions)
          func write(to: URL, options: WritingOptions)

          Note that Data simplifies the various ways of reading and writing data from the file system into two calls while NSData offers multiple different methods.

          Another difference can be observed when comparing the methods on NSData with those on Data. While NSData offers 30 methods & properties, Data offers 130. This huge difference is easily explained via Swift's formidable Protocol Extensions. Data obtains many of those methods from the following protocols:

          • CustomStringConvertible
          • Equatable
          • Hashable
          • MutableCollection
          • RandomAccessCollection
          • RangeReplaceableCollection
          • ReferenceConvertible

          This adds functionality to Data which did not exist in NSData. Here's a small sample:

          func distance(from: Int, to: Int)
          func dropFirst(Int)
          func dropLast(Int)
          func filter((UInt8) -> Bool)
          func flatMap<ElementOfResult>((UInt8) -> ElementOfResult?)
          func forEach((UInt8) -> Void)
          func index(Int, offsetBy: Int, limitedBy: Int)
          func map<T>((UInt8) -> T)
          func max()
          func min()
          func partition()
          func prefix(Int)
          func reversed()
          func sort()
          func sorted()
          func split(separator: UInt8, maxSplits: Int, omittingEmptySubsequences: Bool)
          func reduce<Result>(Result, (partialResult: Result, UInt8) -> Result)

          As you can see, many functional approaches, such as mapping or filtering can now be done on the byte contents of Data types. This, to me, is a huge improvement over NSData. An example of the benefits this brings is how easily you can now subscript and compare data:

          var data = Data(bytes: [0x00, 0x01, 0x02, 0x03])  
          print(data[2]) // 2
          data[2] = 0x09
          print (data == Data(bytes: [0x00, 0x01, 0x09, 0x03])) // true

          Data also offers several new initializers which specifically handle other common Swift data types:

          init(bytes: Array<UInt8>)
          init<SourceType>(buffer: UnsafeMutableBufferPointer<SourceType>)
          init(repeating: UInt8, count: Int)

          3 GetBytes

          Another difference which you will run into if you're using Data to interact with lower level code such as C libraries is the distinct lack of the NSData getBytes method:

          // NSData
          func getBytes(_ buffer: UnsafeMutablePointer<Void>, length length: Int)

          There're many different usage scenarious for getBytes. One of the most common is when you need to parse a file and read the bytes into data types / variables. A common example: Say you want to read a binary file which encodes a list of items. The file is encoded as follows:

          Datatype Size Function
          Char 4 Header (ABCD)
          UInt32 4 Start of Data
          UInt32 4 Amount of items

          The file contains a 4 byte string "ABCD" tagging it as the correct file type. The next 4 bytes define the start of the actual data (i.e. where the header ends and the items begin), the final 4 bytes in the header define the amount of items stored in this file.

          Parsing this data with NSData is pretty straight forward:

          let data = ...
          var length: UInt32 = 0
          var start: UInt32 = 0
          data.getBytes(&start, range: NSRange(location: 4, length: 4))
          data.getBytes(&length, range: NSRange(location: 8, length: 4))

          This will return the correct result3. If your data does not contain C strings, there's an even easier way of doing this, you can simply define a struct with the correct fields and read the bytes directly into the struct:

          Datatype Size Function
          UInt32 4 Start of Data
          UInt32 4 Amount of items
          let data = ...
          struct Header { 
            let start: UInt32
            let length: UInt32
          var header = Header(start: 0, length: 0)
          data.getBytes(&header, range: NSRange(location: 0, length: 8))

          4 Data alternatives to getBytes

          However, if you're using Data this functionality is not available anymore. Instead, Data offers a new method:

          // Access the bytes in the data.
          func withUnsafeBytes<ResultType, ContentType>((UnsafePointer<ContentType>) -> ResultType)

          This method allows direct access of the our data's bytes from within a closure. Let's see a simple example:

          let data = Data(bytes: [0x01, 0x02, 0x03])
          data.withUnsafeBytes { (pointer: UnsafePointer<UInt8>) -> Void in
          // Prints: 
          // : 0x00007f8dcb77cc50
          // : 1

          Ok, now that we have an unsafe UInt8 pointer into our data, how does this help us? First of fall, we obviously need a different data type, and we're sure (we have to be!) that the data is indeed of this particular data type. We know that this data contains a Int32 type, so how do we decode it correctly?

          As we already have a unsafe pointer (of type UInt8) it is easy to move this into an unsafe pointer of our target type. UnsafePointer has a pointee property which returns the type that the pointer is pointing to as the correct type:

          let data = Data(bytes: [0x00, 0x01, 0x00, 0x00])
          let result = data.withUnsafeBytes { (pointer: UnsafePointer<Int32>) -> Int32 in
                return pointer.pointee
          //: 256

          As you can see, we created a byte Data instance, and returned the data as Int32 by defining an UnsafePointer<Int32> in the closure. You can shorten this code if the compiler is able to infer the result type from the context:

          let result: Int32 = data.withUnsafeBytes { $0.pointee }

          5 Lifetime of the data

          One important consideration of using withUnsafeBytes (apart from the fact that the whole operation is unsafe) is that the lifetime of the pointer you're accessing is limited to the lifetime of your closure. As the documentation notes:

          Warning The byte pointer argument should not be stored and used outside of the lifetime of the call to the closure.

          6 Generic Solution

          Now that we have a way of accessing raw bytes and casting them to the correct type, we ought to create a generic solution that allows us to perform this operation easily without the syntactical overhead. Also, we still did not account for the fact that we need to perform the operation on a subsequence of our data and not the whole Data instance. A generic solution would look like this:

          extension Data {
              func scanValue<T>(start: Int, length: Int) -> T {
          	return self.subdata(in: start..<start+length).withUnsafeBytes { $0.pointee }
          let data = Data(bytes: [0x01, 0x02, 0x01, 0x02])
          let a: Int16 = data.scanValue(start: 0, length: 1)
          // : 1

          Compared to our earlier code, this has a couple of notable differences:

          • We're using subdata to only scan the bytes of a specific slice of our Data.
          • We're using generics to support different possible data types for extraction

          7 To Data

          The opposite case, taking an existing variable and getting a Data buffer to the content, is not relevant for the Doom example below, but easy enough to implement:

          var variable = 256
          let data = Data(buffer: UnsafeBufferPointer(start: &variable, count: 1))
          print(data) // : <00010000 00000000>

          8 Parsing the Doom WAD file

          I've played a lot of Doom in my youth. I loved the game. I also created a lot of Doom levels and modified the WAD file to incorporate new sprites, textures, and more. So when I thought about a nice (and simple) example of how to parse a binary file, I remembered the layout of the WAD file which is pretty straightforward and easy to implement. So I wrote a simple app that reads a WAD file and lists the names of all the floor textures stored in the WAD4.

          The source code for this application is available on Github.

          The Doom WAD file layout is described in these two documents:

          However, for our simple example, we only need to understand a subset of the format. First, each WAD file begins with a header:

          Datatype Size Function
          Char 4 IWAD or PWAD string
          Int32 4 The number of lumps in the WAD
          Int32 4 Pointer to the location of the directory

          The first 4 bytes are spend to identify the file format. IWAD are official Doom WAD files, PWAD are patches containing additional information patched at runtime into the main WAD file. Our application will only read IWAD files. The next 4 bytes define the number of lumps in the WAD. Lumps are the individual items that the Doom engine operates with: Textures, Sprite-Frames, Text blocks, Models, etc. Each texture is a distinct lump. The final 4 bytes define the location of the directory. We'll explain the directory below, once we start parsing it. First, lets parse the header.

          8.1 Parsing the Header

          Reading a WAD file is straight forward:

          let data = try Data(contentsOf: wadFileURL, options: .alwaysMapped)

          Once we have the data, we need to parse the header. We're making heavy use of the scanValue Data extension we defined earlier.

          public func validateWadFile() throws {
              // Several Wad File definitions
              let wadMaxSize = 12, wadLumpsStart = 4, wadDirectoryStart = 8, wadDefSize = 4
              // A WAD file always starts with a 12-byte header.
              guard data.count >= wadMaxSize else { throw WadReaderError.invalidWadFile(reason: "File is too small") }
              // It contains three values:
              // The ASCII characters "IWAD" or "PWAD". Defines whether the WAD is an IWAD or a PWAD.
              let validStart = "IWAD".data(using: String.Encoding.ascii)!
              guard data.subdata(in: 0..<wadDefSize) == validStart else
              { throw WadReaderError.invalidWadFile(reason: "Not an IWAD") }
              // An integer specifying the number of lumps in the WAD.
              let lumpsInteger: Int32 = data.scanValue(start: wadLumpsStart, length: wadDefSize)
              // An integer holding a pointer to the location of the directory.
              let directoryInteger: Int32 = data.scanValue(start: wadDirectoryStart, length: wadDefSize)
              guard lumpsInteger > 0 && directoryInteger > Int32(wadMaxSize)
          	else {
          	    throw WadReaderError.invalidWadFile(reason: "Empty Wad File")

          You can find additional types (such as the WadReaderError enum) in the source on GitHub. The next step is to parse the directory, so that we get the addresses and sizes of the individual lumps.

          8.2 Parsing the Directory

          The directory associates names of lumps with the data that belong to them. It consists of a number of entries, each with a length of 16 bytes. The length of the directory is determined by the number given in the WAD header.

          Each of the 16 bytes entries follows the same format:

          Datatype Size Function
          Int32 4 The start of the lumps data in the file
          Int32 4 The size of the lump in bytes
          Char 8 An ASCII string defining the lump's name

          The name char is a bit more complicated. The documentation says:

          An ASCII string defining the lump's name. Only the characters A-Z (uppercase), 0-9, and [ ] - _ should be used in lump names (an exception has to be made for some of the Arch-Vile sprites, which use "\"). When a string is less than 8 bytes long, it should be null-padded to the tight byte.

          Note the last sentence. In C, a String is terminated with the null character (\0). This signifies to the system that the memory for the string ends here. Doom saves space by having an optional null character. When the string is less than 8 bytes long, it will contain a null character, when it is of the max length (8 bytes) the 8th byte will be the final character, not the null character.

            0 1 2 3 4 5 6 7  
          Short I M P \0 \0 \0 \0 \0 #
          Long F L O O R 4 _ 5 #

          See above for an example. The Short name has a null character after the last letter in position 3, the long name does not have a null character, instead the last letter is the 5 from the name FLOOR4_5. The # signifies the beginning of the next item / piece of memory.

          Before we venture into supporting this, lets first take care of the easier part, reading the start and size.

          Before we start, we should define a data structure that can store the information from the directory:

          public struct Lump {
              public let filepos: Int32
              public let size: Int32
              public let name: String

          Afterwards, we take the slice of data that constitutes our directory from the complete data instance.

          // Define the default size of a directory entry
          let wadDirectoryEntrySize = 16
          // Extract the directory slice from the main Data
          let directory = data.subdata(in: Int(directoryLocation)..<(Int(directoryLocation) + Int(numberOfLumps) * wadDirectoryEntrySize))

          Next, we can iterate over the Data in 16byte steps. This works great with Swift's stride function:

          for currentIndex in stride(from: 0, to: directory.count, by: wadDirectoryEntrySize) {
              let currentDirectoryEntry = directory.subdata(in: currentIndex..<currentIndex+wadDirectoryEntrySize)
              // An integer holding a pointer to the start of the lump's data in the file.
              let lumpStart: Int32 = currentDirectoryEntry.scanValue(start: 0, length: 4)
              // An integer representing the size of the lump in bytes.
              let lumpSize: Int32 = currentDirectoryEntry.scanValue(start: 4, length: 4)

          This was the easier part the next part is a bit more difficult.

          8.3 Parsing C Strings

          Remember, for each lump's name, we need to stop reading bytes into our Swift string once we reach a null terminator or once we reach 8 bytes. The very first thing to do is create a data slice with the relevant data:

          let nameData = currentDirectoryEntry.subdata(in: 8..<16)

          Swift offers great support for C String interoperability. This means that to create a string we just need to hand the data to a String initializer:

          let lumpName = String(data: nameData, encoding: String.Encoding.ascii)

          This works, though the result is not correct. This method ignores the null terminator, so that all names, even the short ones, are converted to 8byte strings. As an example, the lump for the IMP character name becomes IMP00000. This happens because Doom fills the remaining 5 bytes with null characters and String(data:encoding:) does not interpret them but creates a string of the full 8 bytes of the nameData.

          If we want to support null characters, Swift offers something else, a cString initializer which is defined for reading valid cStrings with null terminators:

          // Produces a string containing the bytes in a given C array, 
          // interpreted according to a given encoding.
          init?(cString: UnsafePointer<CChar>, encoding enc: String.Encoding)

          Note that it doesn't require a data instance as its parameter but an unsafePointer to CChars instead. We already know how to do that, so lets write the code:

          let lumpName2 = nameData.withUnsafeBytes({ (pointer: UnsafePointer<UInt8>) -> String? in
              return String(cString: UnsafePointer<CChar>(pointer), encoding: String.Encoding.ascii)

          This, again, doesn't work. In all cases where Doom's names are less than 8 characters, this code works flawlessly, but once we reach a 8 byte name without a null terminator, it will continue reading (into the next 16byte segment) until it finds the next valid null terminator. This results in long strings with random memory at the end.

          Since this logic is custom to Doom, we also need to implement custom code. As Data supports Swift's collection & sequence operations, we can just solve this in terms of reduce:

          let lumpName3Bytes = try nameData.reduce([UInt8](), { (a: [UInt8], b: UInt8) throws -> [UInt8] in
              guard b > 0 else { return a }
              guard a.count <= 8 else { return a }
              return a + [b]
          guard let lumpName3 = String(bytes: lumpName3Bytes, encoding: String.Encoding.ascii)
              else {
          	throw WadReaderError.invalidLup(reason: "Could not decode lump name for bytes \(lumpName3Bytes)")

          This code just reduces over the UInt8 bytes of our data and checks whether we have an early null terminator. This code works, though it is not necessarily fast as the data has to be moved through several abstractions.

          It would be better if we could solve this similarly to how the Doom engine does it. Doom just moves the pointer of the char* and checks for each char whether it is a null terminator in order to break early. As Doom is written in low level C code, it can just iterate over the raw pointer addresses.

          How would we implement this logic in Swift? We can actually do something quite similar in Swift by, again, utilizing withUnsafeBytes. Lets see:

          let finalLumpName = nameData.withUnsafeBytes({ (pointer: UnsafePointer<CChar>) -> String? in
              var localPointer = pointer
              for _ in 0..<8 {
          	guard localPointer.pointee != CChar(0) else { break }
          	localPointer = localPointer.successor()
              let position = pointer.distance(to: localPointer)
              return String(data: nameData.subdata(in: 0..<position),
          		  encoding: String.Encoding.ascii)
          guard let lumpName4 = finalLumpName else {
              throw WadReaderError.invalidLup(reason: "Could not decode lump name for bytes \(lumpName3Bytes)")

          Similar to our earlier uses of withUnsafeBytes we're receiving a pointer to the raw memory. pointer is a let constant, but we need to modify the variable, which is why we create a local mutable version in the first line 5.

          Afterwards, we're performing the main work. We loop from 0 to 8 and for each loop iteration we test whether the char that the pointer is pointing to (the pointee) is equal to the null terminator (CChar(0)). If it is equal to the null terminator, this means that we found the null terminator early, and we break. If it is not equal to the null terminator, we overwrite localPointer with its successor, i.e. the next position in memory after the current pointer. That way, we're iterating byte by byte over the contents of our memory.

          Once we're done, we calculate the distance between our original pointer and our localPointer. If we just advanced three times before finding a null terminator, the distance between the two pointers would be 3. This distance, finally, allows us to create a new String instance with the subdata of actual C String.

          This allows us to create a new Lump struct with the required data:

          lumps.append(Lump(filepos: lumpStart, size: lumpSize, name: lumpName4))

          When you look into the source, you will see ominous references to F_START and F_END. Doom marks the beginning and end of special lump regions with empty lumps with magic names. F_START / F_END enclose all the floor texture lumps. We will ignore this additional step in this tutorial.

          A screenshot from the final application:

          Not really impressive, I know. One of the next installments on this blog might concentrate on how to display those textures.

          9 Bridging to NSData

          I find the new Data easier to work with than NSData. Nevertheless, if you need NSData or if you need to use the getBytes method, there's an easy way to convert Data to NSData. The Swift documentation writes:

          This type provides “copy-on-write” behavior, and is also bridged to the Objective-C NSData class. You can wrap an instance of a custom subclass of NSData in struct Data by converting it using myData as Data.

          // Create a new Data Struct
          let aDataStruct = Data()
          // Get the underlying reference type NSData
          let aDataReference = aDataStruct as NSData

          Whenever you feel that what you're trying to do seems to be really hard with the Data type, it is easy to go back to NSData to use the well known tried and trusted methods. However, in general you should strive to use the new Data type whenever possible (except if you need reference semantics):



          Some, such as Date aren't even wrappers but completely new implementations


          Doom1, Doom2, Hexen, Heretic, or Ultimate Doom. Though I've only tested it with Doom1 Shareware


          Note we did not make sure that this is indeed an ABCD file by testing for the first 4 bytes, but that would be easy to add


          I kinda wanted to also display the textures but lacked the time to implement that.


          Swift 3 dropped support for the useful var annotation in closure or function bodies

          If you read this far, you should follow me (@terhechte)
          on Twitter