'Automatically' Picking Icons Using Machine Learning

When you schedule a block in the app Hour Blocks, an icon is automatically picked based on the title:

This has led to many people asking me what 'automatically' really means in this context. The answer is a little complicated, as the meaning has been constantly changing over time.

Brute Force Matching

Initially, the term 'automatically' was completely avoided. All I was doing, was just picking out certain keywords and seeing if they matched up to an icon. If you were having Dinner with Emma, then Dinner would trigger a match to the food icon. If there was no match, then the default Hour Blocks icon would be displayed instead.

While this approach worked, for the most part, it wasn't sustainable. What about a snack? What about brunch? What about a buffet? Setting up rules for all of these possibilities would end up being a tangled mess. Outside of the obvious issue of a tangled mess of rules, there were other edge cases too, such as scheduling Working Out, where the first 4 characters of Work would end up triggering the briefcase icon. Fortunately, it was around this time that I started experimenting with CoreML.

A Trained Data Set

Anonymous analytics had given me real-world insight into what users were scheduling in Hour Blocks, and how they were phrasing their blocks. From this, and my own experience with the app, I managed to compile a data set of over 500 titles with corresponding icons. This data set was then fed into CreateML, generating a CoreML model that simply took the title of a scheduled Hour Block to determine an icon match. Not only did this massively clean up the icon generation section of the codebase, but a lot more icons were being matched as a result of not being bound to a fixed set of matched names.

However, this was a double-edged sword, as a lot of false positives were being matched. Since this model was just trying to best match up a collection of characters to a specific string, a lot of titles that had no real matches were being matched for the sake of it. Despite the false positives, this solution worked well overall, as it covered a good amount of the blocks being scheduled. And if it didn't, it was extremely easy to update without having to touch the codebase. I just had to add any glaring exceptions into the dataset, regenerate the model, and ship it as part of an app update.

I was still searching for a better solution. This was such a core part of the Hour Blocks experience, so it was important to me to get it as close to perfect as possible. And that's when CoreML 3 came along.

Similar Words in a BlockDomain

At WWDC19, just weeks after the trained data set solution went live, Apple unveiled CoreML 3 which contained something called NLEmbedding. NLEmbedding allowed developers to be able to extract a bank of 'similar and related words' from a single word whilst also providing a score on the similarity of each word. When I heard about this, I began to wonder if I could build a foundation similar to the initial Brute Force Matching system, and use this 'similar words' API as a layer on top. This opened up a huge number of potential benefits:

1. A much higher number of blocks would be correctly matched, as there would be a huge bank of similar words easily accessible on-device, with a scoring system to determine how good a match is
2. There would be little room for any false positives, as it would essentially function like the brute force matching system at the core
3. There would be no need to collect the titles of the blocks people were adding as I was no longer building a dataset, making this solution far more privacy-centric
4. Since this API supported 8 languages, it would be much easier to localise Hour Blocks for different languages, as I'd just have to localise the keywords

The core of this system is the BlockDomain, which expresses a 'category' with a keyword, and a corresponding icon. When an Hour Block is created, each word's BlockDomain is determined by the closest match of a similar word of a particular domain. In the case of Violin Practice, the word Violin would be matched with a high score to the Music domain, whereas Practice would be matched with a fairly low score to the Competition domain. As a result, Violin Practice is matched to the Music domain, resulting in a nice little music note icon with the card.

While implementing this system, it also occurred to me that since a large portion of these blocks would now be matched to a BlockDomain, this could power a suggestions system. By simply logging on-device, how often and when a user schedules a certain a BlockDomain, the recommendation could be fed back to them in an incredibly useful manner. If I have Lunch with John every Monday and Tuesday, but Lunch with Rahul every Friday, the app would only log the domain of Lunch and just simply suggest Have Lunch on these days at those times. Having this generalised suggestions system allows for users to be able to express the same activity in different ways and different spellings, but still have the suggestions system pick it up.

The BlockDomain system launched with the huge Hour Blocks 3.0 release. Since that release, over 60% of blocks are being automatically matched with an icon, whereas before Hour Blocks 3.0, that figure was less than 30%. Since the suggestions system was launching for the first time in that release, I had no idea what kind of usage rate to expect from it. Ever since Hour Blocks 3.0, a quarter of all scheduled blocks come from the suggestions system, which I'm extremely pleased with.

What's Next?

The BlockDomain system felt like a culmination of perfecting the core Hour Blocks experience. As a result, it's managed to stick around to this day as we approach the launch of Hour Blocks 6.0- almost a whole year! Twice as long as the original 2 systems combined!

While this system has continued to be the foundation of icon generation in Hour Blocks, and now suggestions, I've still got my eyes wide open. New domains are still being added over time, the score thresholds are always being tweaked, and new ideas are still being entertained as future solutions.

If you haven't already, check out all of this technology in action by downloading Hour Blocks: Day Planner for iOS, iPadOS and macOS by clicking here.