Ben Dodson

Freelance iOS, Apple Watch, and Apple TV Developer

Moonlight tvOS for Apple TV 4

A few nights ago I was attempting to finish playing Day of the Tentacle on my PC via a Steam Link in my bedroom. For some reason, whilst it all loaded correctly, it was really jerky to play with a severe delay between controller input and response on the screen. As I’d been playing it successfully using GameStream on my Nvidia Shield, I assumed correctly that the issue was with the Steam Link.

Not content to have to either a) go downstairs and use the Nvidia Shield or b) go to my office and play in front of my PC, I instead turned to my Apple TV as I knew there was an open source project called Moonlight that basically lets you run the GameStream technology on non-Nvidia devices like iPhone or iPad. I assumed there would be a version for Apple TV but unfortunately it turns out that the AVFoundation frameworks have a __TVOS_PROHIBITED flag on some key components that prevent it from working. That’s when I found a fork by kevsmithpublic that enabled tvOS support with some code changes and requiring you to remove the __TVOS_PROHIBITED flags from the AVFoundation framework1. Unfortunately, I wasn’t able to get it to pair to my PC and after a lot of searching it turns out that the fork was fairly old and hadn’t been updated to the latest specifications for GeForce Experience 3.0 and above.

Determined not to be beaten, I forked the project and then set about updating the code to support the latest Moonlight connections. This ended up being similar to the point and click adventure I was trying to play as I needed to constantly fetch the Apple TV from my bedroom, connect it to my mac, build the project, unplug the Apple TV, reconnect it in my bedroom, and then test if it actually worked (all because I didn’t have a spare monitor in my office as I use an iMac). After several hours (and a few beers) I had the project working and can now stream my games at 1080p from my PC to my Apple TV 4 and play them using an MFI controller.

You can download the project from my GitHub account.

  1. This means it can’t ever be used in an App Store version but it’s pretty cool nonetheless. I had no idea you could alter framework headers to enable things like this. It does beg the question why this framework is prohibited as it all clearly works. ↩︎

The Divide #15 - Virtually Reality

After a short summer break we’re back with episode 15 of The Divide podcast which is all about virtual reality. John has a fancy new HTC Vive, I’m trying to convince myself I don’t need one, and Chris just wants to be on the bridge with Captain Janeway…

You can get The Divide from these fine outlets:

Don’t forget to leave a review on iTunes and follow us on Twitter via @PodcastDivide.

Media Library privacy flaw fixed in iOS 10

As I’ve mentioned many times before, iOS had a pretty terrible privacy flaw in that apps didn’t need any permissions in order to read through your media library. This was an issue as it meant you could be fingerprinted easily and tracked across various apps1. Thankfully, this has now been fixed in iOS 102.

In this article, I’ll explain how to update your apps to support this new privacy requirement. Before I do that, I’m going to show you what happens if you run an app built against the iOS 9.3 SDK (or earlier) on iOS 10:

[app name] would like to access Apple Music

You’ll be prompted3 to allow permissions as soon as any media library code is encountered be that on app startup or in a background process such as that used by Music Tracker. If you decline to give permission then the app will quit to the home screen and you will not be able to use it. This is a big change from previous permission switches whereby apps built against the old SDK would be exempt from new permissions (i.e. if your iOS 7 SDK compiled app ran on iOS 8, it wouldn’t crash because it wasn’t using the new location privacy options). Personally I like this change as it allows you to see clearly which apps were abusing these APIs such as Canary and Google Calendar.

Bottom line: If you are using MPMediaQuery or similar in your app, you’d better update it with the iOS 10 SDK as soon as possible as otherwise you are going to get a lot of crashes if your users don’t allow the permission (or a lot of awkward questions if you shouldn’t be using this API).

How do you update your app to request permission for these APIs? First of all, lets whip up a basic example with Swift 3 that will pull the title of the first track in your music library:

@IBAction func buttonPressed(_ sender: AnyObject) {
    let query = MPMediaQuery.songs()
    if let items = query.items, let item = items.first {
        NSLog("Title: \(item.title)")
    }
}

In keeping with other permissions based APIs such as photos, contacts, and calendars, iOS 10 requires that you add a new key to your Info.plist file to explain why you want to use this permission; for music library access, this key is NSAppleMusicUsageDescription. If you don’t add this key, your app will crash as soon as you try and access an MPMediaQuery with the following message:

This app has crashed because it attempted to access privacy-sensitive data without a usage description. The app’s Info.plist must contain an NSAppleMusicUsageDescription key with a string value explaining to the user how the app uses this data.

With the NSAppleMusicUsageDescription key in place, you will now be given a standard permission dialogue when you first try and access the users media library. If the user chooses “Don’t Allow”, then the media query will fail and any subsequent calls will request in the query.items property being nil. If they choose “OK”, then nothing happens (as execution of the code is not suspended and there is no callback). To fix this, we need to use the MPMediaLibrary.authorizationStatus() and requestAuthorization((MPMediaLibraryAuthorizationStatus) -> Void) APIs that were added in iOS 9.3 to do something like this:

@IBAction func buttonPressed(_ sender: AnyObject) {
    MPMediaLibrary.requestAuthorization { (status) in
        if status == .authorized {
            self.runMediaLibraryQuery()
        } else {
            self.displayMediaLibraryError()
        }
    }
}

func runMediaLibraryQuery() {
    let query = MPMediaQuery.songs()
    if let items = query.items, let item = items.first {
        NSLog("Title: \(item.title)")
    }
}

func displayMediaLibraryError() {
    var error: String
    switch MPMediaLibrary.authorizationStatus() {
    case .restricted:
        error = "Media library access restricted by corporate or parental settings"
    case .denied:
        error = "Media library access denied by user"
    default:
        error = "Unknown error"
    }
    
    let controller = UIAlertController(title: "Error", message: error, preferredStyle: .alert)
    controller.addAction(UIAlertAction(title: "OK", style: .default, handler: nil))
    present(controller, animated: true, completion: nil)
}

You first need to request authorization and then run your code only if the status is authorized. If not, then you should display an error specific to whether or not the request was denied or restricted (usually by corporate or parental controls).

I’ve put an example project on GitHub which uses the code above by way of demonstration.

I’m extremely grateful to everyone that has helped share my posts about this issue and to the engineers at Apple for fixing this privacy flaw. I’ll update this post should anything change between now and the expected public release of iOS 10 in September.

  1. Whilst I had no concrete proof of this at the time of writing those articles, it looks like I was right that this was a widespread problem↩︎

  2. I don’t want to take all the credit for this but I honestly don’t think this would have been fixed if I hadn’t raised the issue repeatedly over the last 7 months. ↩︎

  3. In earlier builds of iOS 10 you’d be asked “[App Name] Would Like to Access Apple Music” (as shown in screenshots above) but as of iOS 10 beta 4 this has been changed to the more appropriate “[App Name] Would Like to Access Apple Music and Your Media Library”. ↩︎

Talking Shop with Ben Dodson

I was recently interviewed by the lovely folks at Cushion1 as part of their Talking Shop series about my life as a freelancer. You can read the full interview on their website but I’ve also put their questions and my answers below:

How did you get your start?

I started off strictly as a web developer, moving from agency to agency, not really enjoying what I was doing. At the same time, Apple had just released the iPhone and, shortly afterwards, the App Store. I thought iOS development looked fun, so I quit my job and started teaching myself how to do it.

Where did your first clients come from?

I had been freelancing as a web developer on the side for a number of years, so I had a few clients through that. Coincidentally, when I decided to freelance full-time, a friend from Twitter reached out and asked if I would be interested in appearing on The Gadget Show. For the segment the show wanted a freelance developer and a small company to build two separate apps, then see which was more popular. At the time, I had never built an app, so naturally, I said yes. It forced me to learn the ropes pretty quickly! I cobbled this app together and it ended up doing really well, scoring a feature from the App Store. From there, clients just kept coming.

What type of projects do you work on now?

Now there’s also the iPad, Apple Watch, Apple TV, so I work on a variety of projects — everything from little apps to find the best steak restaurant nearby to bigger ones for clients like L’Oreal and Glenfiddich.

What does your day-to-day look like?

At this point, I try to exclusively work from home. In the beginning, clients would insist on coming to their office. Those days would always be spent in meetings, talking about nothing, and not getting much done. Nowadays, I push for a video call in lieu of all that. I have clients I’ve never even spoken to, that I work with via email or Slack. They’re the best.

What has been the greatest challenge of kind of being a freelancer?

For me, it’s been time management. It’s so easy to procrastinate. Then you hit a deadline and realize that you’ve done nothing, you have to tell the client, but you don’t want to tell the client, so things get worse quickly. The biggest challenge for me was getting to a point where I can switch my work mode on and off.

What’s next for you?

A few years ago, I had my own app, an iPhone game called WallaBee. For about three and a half years, I focused on that and was extremely adamant about not hiring anyone else. In retrospect, some help might have made sense, but I was worried about being able to pay someone reliably month to month. It was a lot of fun, but eventually, I burned out and sold it.

Ultimately, I’d like to do something like that again, but at the moment, I enjoy freelancing.

It’s a constant battle between doing things that you want to do, but not earning any money, and doing the things that are less enjoyable, but make money.

I’m sure that statement resonates with a lot of freelancers.

When I was young and single, it was easy to spend twenty hours a day working on my pet project for no pay. The money earned mattered less because at the end of the day, it was just me. But when you have a family with mouths to feed and bills to pay, you find yourself taking less of those risks and opting for projects that provide a bit more security.

What advice would you give to someone just starting out?

Keep it simple. I spent a long time researching and reading up on various how-to-get-things-done philosophies, buying every to-do list manager there was. Now I just have a basic list, make my own invoices, and track everything else in Cushion. If you spend a fortune on loads of tools and to-do lists and all that, you end up spending more time tweaking your setup for productivity than actually being productive.

  1. If you’re a freelancer, you should definitely check out Cushion. It’s an awesome online tool for keeping track of your clients and scheduling - I’ve been using it for nearly a year and it has helped me avoid overbooking myself. I’m super excited for their invoicing tools which they are launching soon. ↩︎

Creating passwordless user accounts within iOS apps

I’m currently in the process of writing an app which will store some information for each user on a server; the stored data should be available to the user on all of their devices. Usually there are two ways of dealing with this:

  1. Build a user access system such that the user has to register to use the service. They’ll need to log in to the app (usually with email address and password) on each of their devices.

  2. Use CloudKit to store all of the information in an Apple provided database.

The first option is overkill for this fairly simple app1 and the second is too restrictive as it means I need to use Apple’s specific data modelling system. However, there is a third way which uses a bit of each of them…

When a user is logged in to an Apple account on their device2, they are automatically signed into the iCloud system. With CloudKit, every app gets its own identifier based on both the CloudKit container and user account which doesn’t change. By retrieving this identifier, you can be 100% sure of the user that is using your app without knowing anything personal about them and without requiring them to manually log into your app.

To get the identifier, you first need to enable CloudKit within the iCloud section of the Capabilities panel in Xcode (which will require updating your provisioning profile). Then, it is a simple case of importing the CloudKit framework and using the following code:

let container = CKContainer.defaultContainer()
container.fetchUserRecordIDWithCompletionHandler { (recordID, error) in
    guard let recordID = recordID else {
        NSLog("Error: \(error")
        return
    }
    
    NSLog("Identifier: \(recordID.recordName)")    
}

The identifier returned will be 34 characters long and look something like this:

_e990774f93dd6625b11af6d40fceb310

Once you have that, you can then send it to your server to match it against whatever content you want to link to this particular user. Now whenever they use your app on any device with the same Apple account, they’ll have access to everything without the need for a manual account creation process and without having to hand over any personal information. The entire process is completely silent and is very secure as the identifier is generated from both the Apple account and your CloudKit container - it can’t be used by other apps nor can it be reverse-engineered to give you personal details of the user.

I find the code above a bit messy to be used multiple times throughout an application (not to mention the import CloudKit requirement) and so I’ve wrapped this up in a very basic Swift class that can be called as such:

AppleCloudIdentifier.fetch { (identifier, error) in
    guard let identifier = identifier else {
        NSLog("Error: \(error)")
        return
    }
    
    NSLog("Cloud identifier: \(identifier)")
}

I’ve named it AppleCloudIdentifier and the code is available on GitHub.

  1. It’ll also lead to a sharp drop-off in users as people don’t like having to create multiple accounts all over the place. You could implement something like Facebook Login but then you still have an issue in that you are requiring a fair amount of personal information upfront which you probably don’t need. ↩︎

  2. I’ll hazard a guess that 99.9% of iOS devices are logged into an Apple account. They need to be in order to download apps, send iMessages, create backups, etc, so very unlikely you wouldn’t be. ↩︎

The Divide #14 - WWDC 2016, 'iTunes with tabs'

The 14th episode of The Divide podcast is now available in which we take look at the new announcements about watchOS, tvOS, macOS, and iOS in a special episode almost as long as the WWDC keynote itself. We also have a brief chat about the latest news from E3 and Chris gets upset about the Stay Puft Marshmallow Man.

You can get The Divide from these fine outlets:

Don’t forget to leave a review on iTunes and follow us on Twitter via @PodcastDivide.

iOS 10 Wishlist

I was recently interviewed by Fast Company for an article they wrote about iOS features they’d like to see at WWDC this year. A couple of my suggestions were featured in the article but I thought I’d publish my full iOS 10 wishlist.

System Defaults

One of the things that I feel has been long missing from iOS is the ability to choose a default app for certain tasks such as email, web browser, phone, or notes. I would love to see a way for developers to make their app conform to some set of standards that would allow them to be selected as a default app. For example, Skype could conform to the “phone app” standard so when you tap a phone number in your mail app it’ll launch Skype and dial the number rather than using the Phone app. Similarly, email links could go to Polymail and web links to Chrome.

Siri API

I’ve long wanted a full API for Siri but so far there hasn’t even been a stepping stone. I imagine the first phase would go hand in hand with the System Defaults plan I outlined and that apps that adopt a certain standard would be able to show up in Siri. For example, if Spotify conformed to being a “music app” then saying “Hey Siri, Play C’est La Via by B*Witched” would start that track playing in Spotify rather than Music.

Control Center Buttons

When you swipe up from the bottom of the screen on iOS, you get Control Center which gives you quick access to certain features (like WiFi), controls (for music / video), and apps (Torch, Timer, etc). I’d like the ability to a) use 3D touch to get more options when using the features section (so I can choose a new WiFi network) and b) be able to choose the apps and features that show up in that area. I never need quick access to the calculator.

CarPlay

I’ve been using CarPlay for around a year now (via an after market Alpine iLX-700) and I absolutely love it. However, 3rd party apps are limited to a few selected audio apps like Spotify and Overcast which use a simple template that doesn’t work very well. I’d like to see support for any 3rd party app to use CarPlay and for it to use UIKit so that any interface can be displayed. I noticed in March that iOS 9.3 added a UIUserInterfaceIdiom for CarPlay so hopefully this is coming.

AirPlay

I have a number of AirPlay devices around the house and one of the things that frustrates me most is that I can’t use Siri to control them. I’d like to be able to say “Hey Siri, play my cooking playlist in the kitchen” and have it just work. I’ve been able to hack this functionality together via some Applescript and a fake HomeKit setup on my Mac called Homebridge but it’d be nice to have it built in natively. Also, iTunes on the Mac can AirPlay to multiple speakers at once whereas iOS can’t.

The Divide #13 - Consoles

The 13th episode of The Divide podcast is now available in which we discuss video game consoles from our first tentative steps in the 8-bit era to the almost HD consoles of the modern day. We also discuss the Flic button, Day of the Tentacle, the Nvidia GTX 1080, Stellaris, and our predictions for WWDC.

You can get The Divide from these fine outlets:

Don’t forget to leave a review on iTunes and follow us on Twitter via @PodcastDivide.

HomeKit, AirPlay, and controlling iTunes with Siri

A few days ago I wrote an article about getting my Flic button to work with HomeKit. Since then, I’ve had a few requests from people wondering how I was getting iTunes to work with HomeKit such that it could start a playlist on some AirPlay speakers. Today I’ve released a couple of HomeBridge plugins to NPM and I’ll detail how I’ve got my system working.

First of all, my entire setup is powered by the awesome Homebridge system which I’ve written about at great length. I found a plugin from Dan Budiac called homebridge-applescript which allowed you to run a line of applescript via a system of fake switches (so a different script could be called when the switch is turned on and off). This is great for simple one-liners (like “tell application iTunes to pause”) but I needed something a bit longer for dealing with AirPlay. I forked Dan’s project to create my own Homebridge plugin that would work with a path to an AppleScript file rather than a single line of AppleScript:

{
	"accessory": "ApplescriptFile",
	"name": "Kitchen Music",
	"on": "/Users/bendodson/Dropbox/Scripts/kitchenMusic.applescript",
	"off": "/Users/bendodson/Dropbox/Scripts/stopMusic.applescript"
}

This allowed me to then use my existing alarm clock script in order to make iTunes connect to my kitchen speakers and play a specific playlist1 when my “kitchen speakers” switch is turned on and disconnect from AirPlay and pause iTunes when it is turned off.

This worked well but it has a few problems as the music is coming from a Mac in a different room; there is no way to change the volume, skip a track, or do a basic play / pause toggle2 without going upstairs to do it manually on iTunes. After thinking about it for a little while, I was able to get the track skipping and play /pause done fairly easily. First of all, I use the following config:

{
    "accessory": "ApplescriptFile",
    "name": "Music Play Pause Control",
    "on": "/Users/bendodson/Dropbox/Scripts/playMusic.applescript",
    "off": "/Users/bendodson/Dropbox/Scripts/pauseMusic.applescript"
},
{
    "accessory": "ApplescriptFile",
    "name": "Music Track Control",
    "on": "/Users/bendodson/Dropbox/Scripts/nextTrack.applescript",
    "off": "/Users/bendodson/Dropbox/Scripts/previousTrack.applescript"
}

I won’t detail the AppleScript for each of these as they are fairly basic.

Next, I used the ‘Scenes’ feature of HomeKit to set up named scenes that related to each toggle. For example, “iTunes next track” is a scene that ensures the “Music Track Control” accessory is turned on whereas “iTunes previous track” ensures that accessory is off. The play / pause controls work in a similar way. Once activated, this meant I could say things like “Hey Siri, iTunes next track” or “Hey Siri, pause iTunes”.

Whilst scenes can be incredibly useful due to the ability to use any text you want, they can’t be used for something like volume where you want to use a specific amount3. Unfortunately HomeKit doesn’t have the concept of speakers so there isn’t a way to naturally create an accessory that deals with volume so I improvised and created a fake lightbulb that controlled an AppleScript via the brightness property; homebridge-applescript-file-lightbulb4.

{
    "accessory": "ApplescriptFileLightbulb",
    "name": "iTunes volume",
    "on": "",
    "off": "",
    "brightness": "/Users/bendodson/Dropbox/Scripts/volume.applescript"
}

The volume.applescript file looks like this:

on run argv
	set v to item 1 of argv
	activate application "iTunes"
	tell application "iTunes"
		set sound volume to v
	end tell
end run

With this in place, I can say “Hey Siri, set iTunes volume to 50%” and it will work. This is because lightbulbs have a brightness setting and Siri interprets “set [name] to [percentage]” as being “change the brightness of this device” just in this case we are hitting an AppleScript with our brightness value. This has been a fun little exercise in trying to hack HomeKit into what I want to use it for. I’m hopeful that iOS 10 will add native support for HomeKit speakers as then this could be done without reverting to AppleScript and I’d be able to stream from my iPhone via Siri but it’s pretty cool for the time being.

You can find both of my Homebridge plugins on NPM and GitHub.

  1. Actually my own setup has two playlists. I have a playlist for Monday through Saturday with my music but on Sunday my wife gets up first so I have a different playlist for her. The code for that is if weekday of (current date) is Sunday then ↩︎

  2. At first I’d get around this by just using “turn off kitchen music” followed by “turn on kitchen music” as the playlist was shuffled but hardly an ideal solution. ↩︎

  3. I could have done the same as the track control AppleScript and had a “iTunes volume up” and “iTunes volume down” scene to increase or decrease by 10% each time but that would be a pain if you wanted to change the volume by a large amount. ↩︎

  4. I shouldn’t be allowed to name things. ↩︎

Great British Bee Count 2016

Last year I worked on rebuilding the Great British Bee Count iPhone app for Friends of the Earth. Today, an updated version of the app has gone live to support this years count which runs from 19th May until 30th June 2016:

There are several updates including a redesigned bee picker, more details of the bees, and a list of plants suitable for attracting bees sponsored by Waitrose. The app is also now universal with support for both the iPhone and iPad1.

You can check out the Great British Bee Count on the App Store (it’s free) or learn more about the bee cause.

  1. This came about due to a restriction in the App Store. If you search for an iPhone-only app on the iPad, you have to manually select “iPhone-only” when searching which nobody in their right mind does (as iPhone apps run on iPad just fine in a scaled mode). To make it so that searching “Great British Bee Count” on the iPad App Store would show the app, it had to be a universal app but the client didn’t want to have a custom design for the iPad but continue to use the iPhone scaling. There isn’t an elegant way to do that (as turning on Universal mode will cause AutoLayout to treat the iPad like an iPad) so I solved this by writing a custom view controller that would take the iPhone 4s XIB files and then simply scale them upwards with an affine transform. I also added an extra border on the sides that matches the yellow background to avoid the black boxing that the previous version had. This works great on all iPads (including the iPad Pro) but the text on some items is quite blurry on the non-retina iPad 2 due to a bug within the Apple Frameworks; it’s intelligent enough to use the retina imagery so that images look smooth but not on text apparently. ↩︎

« Older Entries Newer Entries »