Update, The Second: Welp, I was wrong. I assumed that Facebook PMs and engineers were smart. Of course they were going to get found out modifying content via In-App Browsers, just as this post warned they could. It's long past time for Google and Apple to act to curb this abuse via App Store policy, and regulators interested in gatekeeper shenanigans should take notice.
At first glance, the market for mobile browsers looks roughly functional. The 85% global-share OS (Android) has historically facilitated browser choice and diversity in browser engines. Engine diversity is essential, as it is the mechanism that causes competition to deliver better performance, capability, privacy, security, and user controls. More on that when we get to iOS.
Tech pundits and policymakers are generally older and wealthier than the median user and likely formed expectations of browsers on the desktop. They may, therefore, tend to think about mobile browser competition through the lends of desktop browsing. To recap:
Users can freely choose desktop browsers with differing UIs, search engines, privacy features, security properties, and underlying engines.
Browsers update quickly, either through integrated auto-update mechanisms or via fast OS updates (e.g., ChromeOS).
Browsers bundled with desktop OSes represent the minority of browser usage, indicating a healthy market for replacements.
Popular native apps usually open links in users' chosen browsers and don't undermine the default behaviour of link clicks.[1]
Each point highlights a different aspect of ecosystem health. Together, these properties show how functioning markets work: clear and meaningful user choice creates competitive pressure that improves products over time. Users select higher quality products in the dimensions they care about most, driving progress.
The mobile ecosystem appears to retain these properties, but the resemblance is only skin deep. Understanding how mobile OSes undermine browser choice requires an understanding of OS and browser technology. It's no wonder that few commenters are connecting the dots.[2]
How bad is the situation? It may surprise you to learn that until late last year only Safari could be default browser on iOS. It may further disorient to know that competitors are still prevented from using their own browser engines.
Meanwhile, the #2 and #3 sources of web traffic on Android — Google's search box and Facebook's native apps — do not respect browser choice. Users can have any browser with any engine they like, but it's unlikely to be used. The Play Store is little more than a Potemkin Village of browser choice; a vibrant facade to hide the rot.
Registering to handle link taps is only half the battle. For a browser to be the user's agent, it must also receive navigations. Google's Search App and Facebook's apps undermine these choices in slightly different ways.[3] This defangs the privacy and security choices made through browsers. Developers suffer higher costs when they cannot escape Google, Facebook, and Apple's walled gardens or effectively reach users through the web.
Web engineers frequently refer to browsers as "User Agents", a nod to their unique role in giving users the final say over how the web is experienced. A silent erosion of browser choice has transferred power away from users, depositing it with dominant platforms and apps. To understand how this sell-out happened under our noses, literally, let's look at how mobile and desktop differ.
The predominant desktop situation is straightforward:
Browsers handle links, and non-browsers defer loading http and https URLs to the system, which in turn invokes the user's default browser. This flow is what gives links utility. If the players involved (OSes, browsers, or referring apps) violate aspects of the contract, user choice in browsers has less effect.
"What, then, is a 'browser'?" you might ask? I've got a long blog post brewing on this, but jumping to the end, an operable definition is:
A browser is an application that can register with an OS to handle http and https navigations by default.
No matter how an OS facilitates browser choice, it's this ability to replace the default handler for links that defines browsers. How often links lead users to their browser defines the meaningfulness of this choice.
Modern browsers like Chrome and Samsung Internet support a long list of features that make web apps powerful and keep users safe. Both pass all eighteen feature tests in Thomas Steiner's excellent 🕵️ PWA Feature Detector
Mobile browsers started in a remarkably resource-constrained environment. First-generation iOS and Android smartphones were slow single-core, memory-impoverished affairs, leading mobile OSes to adopt heuristics for killing background apps to reclaim memory. This helped ensure the whole system remained responsive.
But background task killing created problems for link-heavy apps. Launching the browser placed linking apps in the background and browser UI didn't provide affordances for returning to referring applications. This reduced the probability users would return, hurting engagement.
Being put in the background also increased the likelihood of a linking app being killed.[5] It can take seconds to re-start the original app and restore UI state, an experience that gets worse on low-end devices that are most likely to evict apps in the first place.
To compensate, engagement-thirsty apps began including "In-App Browsers" ("IABs") to keep links from bouncing users to browsers. Contrary to any plain-language understanding of "a browser", these IABs cannot be installed as browsers, even where OSes enabled browser choice. Instead, they load content referred by their hosting native app in system-provided WebViews.
The benefits to apps that adopt WebView-based IABs are numerous:
WebViews are system components designed for use within other apps. They do not place embedders in the background where the system may kill them to reclaim resources. This reduces friction and commensurately increases "engagement" metrics.[6]
As they are now "the browser", they can provide UI that makes returning to the host application easier than continuing on the web.
Because they lean on the system-provided WebView component, they do not need to pay the expense of a heavier app download to support rendering HTML, running JavaScript, decoding images, or loading network resources.
Apps can customise UI to add deeper integrations, e.g., "pinning" images from a hosted page to Pinterest.
WebViews allow embedders to observe and modify network traffic (regardless of encryption).
WebViews can monitor user input, passwords, site content, and system auto-filled credentials.
In the unlikely scenario users are happy for browsers to forget their saved passwords, login state, privacy preferences, extensions, and accessibility settings, this could, in theory, be a win-win. In practice it is a hidden, ecosystem-wide tax.
WebViews are the source of much confusion in debates around apps and browser choice. Thankfully, the situation is only complicated rather than complex.
There are two dimensions in play:
Can the app register with an OS to handle http/https navigations by default?
If so, it's a browser regardless of the underlying engine.
If not, it's something else; a "content renderer" or an IAB.
Does the app include its own web engine?
If so, it's integrated — e.g., an "Integrated Browser".
If not, it's WebView-powered, e.g. a "WebView Browser" or "WebView IAB".
So, a browser can be WebView-based, and so can an IAB. But neither has to be.
A View that displays web pages.
...
In most cases, we recommend using a standard web browser, like Chrome, to deliver content to the user.
WebViews have a long history in mobile OSes, filling several roles:
Rendering HTML on behalf of the first-party application developer.
Displaying cooperating, second-party content like ads.
Providing the core of browsers, whose job is to display third-party content. The original Android Browser used early-Android's system WebView, for instance.
The use of WebViews in non-browser apps is appropriate for first and second-party content. Here, apps are either rendering their own web content or the content can be expected to know about the limits imposed by the WebView implementation. Instead of breaking content, WebViews rendering first and second party content can help apps deliver better experiences without additional privacy and security concerns.
All bets are off regarding WebViews and third-party content. Remember, WebViews are not browsers.
WebViews support core features for rendering web content, along with hooks that allow embedders to "light up" permission-based APIs (e.g., webcam access). Making a full browser out of a WebView requires a lot of additional UI and glue code.
Basic navigation and window management features to (e.g. window.open() and <a target="_blank"> which are critical to some site monetization features).
Friction reducing OS integrations such as:
Web Payments (streamlined e-commerce checkout)
Web OTP (for easier/faster sign-in)
Web Share
Hardware access APIs, notably:
Geolocation
Camera/mic (getUserMedia())
Web Bluetooth
WebUSB
Web Serial
WebHID
WebMIDI
Web NFC
Filesystem Access
Re-engagement features including:
PWA installation and home screen shortcuts for sites
Push Notifications
Few (if any) WebView browsers implement all of these features, even when underlying system WebViews provide the right hooks.
The situation is even more acute in WebView IABs, where features are often broken even when they appear to be available to developers. Debugging content in IAB franken-browsers is challenging, and web developers are often blind to the volume of traffic they generate, meaning they may not even understand how broken their experiences are.
How can that be? Web developers are accustomed to real browsers and industry standard tools, analytics, and feature dashboards do break out or highlight IABs. The biggest IAB promulgators (Facebook, Pinterest, Snap, etc.) are complicit, investing nothing in clarifying the situation.
Neither users nor developers understand Facebook, Pinterest, or Google Go as browsers. If they did, they would be livid at the poor quality and broken feature set. WebView IABs strip users of choice, and technical limits they impose prevent web developers from any recourse to real browsers.
No documentation is available for third-party web developers from any of the largest WebView IAB makers. This scandalous free-riding is shady, but not surprising. It is, however, all the more egregious for the subtlety and scale of breakage.
Thanks to IAB shenanigans, Facebook is the third largest Android "browser"-maker. If it employs a single developer relations engineer or doc writer to cover these issues, I'm unaware of it. Meanwhile, forums are full of melancholy posts recounting myriad ways these submarine renderers break features that work in other browsers.
Update (Oct '21): How feature-deprived are WebView IABs? Several months after this post was published, and with no apparent irony, Facebook dropped support for WebViews as a login mechanism to Facebook itself. That's right, Facebook's own app is now an unsupported browser for the purposes of logging in to Facebook.
"Facebook Mobile Browser" relies on the system WebView built from the same Chromium revision as the installed copy of Chrome. Despite the common code lineage and exceedingly low cost to Facebook to develop, it fails to support half of the most meaningful PWA features, cutting third-party web developers off at the knees.
WebView IAB makers have been given "the first 80%" of a browser. Development and distribution of critical components is also subsidised by OS vendors. Despite these considerable advantages, WebView IABs universally fail to keep up their end of the bargain.
First-party developers can collaborate with their IAB colleagues to build custom access to any feature they need.
Likewise, second-party developers expect less and their content will not appear to be broken — ads are generally not given broad feature access.
But third-party developers? They are helpless to understand why an otherwise browser-presenting environment is subtly, yet profoundly, broken.
There are still users browsing with a Chrome 37 engine (7 years ago), not because they don't update their browsers but because it's Facebook Mobile Browser on Android 5 using a webview. Facebook does NOT honor user browser choice leaving that user with an old engine. +
These same app publishers request (and heavily use) features within real browsers they do not enable for others, even when spotted the bulk of the work. Perhaps browser and platform vendors should consider denying these apps access to capabilities they undermine for others.
The consequences of WebView IABs on developers are noteworthy, but it's the impacts on users that inspire confusion and rage.
Consider again the desktop reference scenario:
Clicking links takes users to their browser, assuming they are not already in a browsesr. If a link from an email application points to example.com, previous login state and passwords are not forgotten. Saved addresses and payment information are readily available, speeding up checkout flows. Most importantly, accessibility settings and privacy preferences are consistently applied.
By contrast, WebView IABs fracture state, storing it in silos within each application. This creates a continuous partial amnesia, where privacy settings, accessibility options, passwords, logins, and app state are frequently lost.
The resulting confusion doesn't hurt apps that foist WebView IABs on unsuspecting users and developers. The costs are borne by publishers and users, harming the larger web ecosystem. IABs are, in this understanding, a negative externality.
Does anyone expect anything one does on a website loaded from a link within Facebook, Instagram, or Google Go can be monitored by those apps? That passwords can be collected? That all sites you visit can be tracked?[8]
To be clear, there's no record of these apps using this extraordinary access in overtly hostile ways, but even the unintended side-effects reduce user control over data and security.(August '22) Facebook has been caught red-handed absuing this power to track users within their IAB browser without explicit consent. Sanctions, App Store policies, and opt-out mechanisms are overdue.
The WebView IAB sleight of hand is to act as a browser when users least expect it, but never to cop to the privacy implications of silently undermining user choice.
To address this challenge, Apple introduced SFSafariViewController ("SFSVC")[9] and Google followed suit with Chrome Custom Tabs protocol ("CCT"). Both systems let native apps to skip the work of building WebView IABs and, instead, provide an OS-wide mechanism for invoking the user's default browser over top of a native app.
Like WebView IABs, CCT and SFSVC address background eviction and lost app state. However, because they invoke the user's actual browser, they also prevent user confusion. They also provide the complete set of features supported by the user's default browser, improving business outcomes for web publishers.
These solutions come at the cost of flexibility for app developers who lose access to snoop on page content, read network traffic, or inject custom behavior. Frustratingly, no OS or App Store mandate their use for IAB needs. More on this shortly.
CCT working as intended from the Twitter native app. Samsung Internet is set as the default browser and loads links within the app. Important developer-facing features work and privacy settings are respected.
Well, it is. At least in the default configuration. Despite the clunky inclusion of "Chrome" in the name, the CCT library and protocol are browser-agnostic. A well-behaved CCT-invoking-app (e.g., Twitter for Android) will open URLs in the CCT-provided IAB-alike UI via Firefox, Brave, Samsung Internet, Edge, or Chrome if they are the system default browser.
@slightlylate I recently was talking to my Dad about the Web and asked what browser he uses and he showed me what he does: He searches for the Web site in the Google search widget and then just uses the results page Chrome tab as his entire browser. His default browser is not set to Chrome.
Who would do this, you might ask? None other than Google's own Search App; you know the one that comes on every reputable Android device via the ubiquitous home screen search widget.
AGSA's homescreen widget; the text box that launched two billion phones. Links followed from search results always load in Chrome via CCT, regardless which browser users have set as default.
Known as the "Android Google Search App" ("AGSA", or "AGA"), this humble text input is the source of a truly shocking amount of web traffic; traffic that all goes to Chrome, no matter the user's choice of browser.
Early on, there were justifiable reasons to hard-code Chrome. Before support for CCT was widespread, some browsers exhibited showstopper bugs.
Fast-forward to 2021 and those bugs are long gone, but the hard-coding persists. Today, the primary effect is to distort the market for browsers and undermine user choice. This subverts privacy and makes it hard for alternative browsers to compete on a level playing field.
This is admittedly better than the wholesale neutering of important features by WebView IABs, but when users change browsers, continuious partial amnesia on the web gets worse. A Hobson's Choice of browser.
'Powered By Chrome': Google's Search App disregarding browser choice on a system with Samsung Internet set as the default browser.
Google can (and should) revert to CCT's default behavior which respects user choice. Since AGSA uses CCT to load web pages rather than a WebView, this would be a nearly trivial code change. CCT's core design is sound and has enormous potential if made mandatory in place of WebView IABs. The Android and Play teams could mandate better behavior in IABs to improve user privacy.
There's reason to worry that this is unlikely.
Instead of addressing frequent developer requests for features in the CCT library, the Chrome team has invested heavily in the "WebLayer" project. You can think of WebLayer like a WebView-with-batteries-included, repairing issues related to missing features but continuing to fracture state and user choice.
There is a weakly positive case for WebLayer. For folks making browsers, WebLayer dramatically reduces the amount of custom glue code needed to light up adavanced features. In the context of IABs, however, WebLayer looks set to entrench user-hostile patterns even further.
Subversion of choice is a dispiriting trend in search apps. Stealing traffic without any effort to honestly earn a spot as the user's preferred browser is, at best, uncouth and adopting WebLayer will not meaningfully improve the user experience or privacy of these amnesiac browsing experiences.
Google Go, the Google app for iOS, and Microsoft's Bing app for Android all capture outbound links in WebView IABs, subverting browser choice and rubbishing features for developers. If there's any mercy, it's that their low use limits the impact on the ecosystem.
Google Go's WebView IAB is just as broken as Facebook's. As the default Search app on Android Go devices, it creates new challenges for the web in emerging markets.
Google and Apple could prevent this bad behavior through App Store policies and technical changes. They have the chance to lead, to show they aren't user-hostile, and remove a permission structure for lousy behaviour that less scrupulous players exploit. More on that in a moment.
Imagine if automakers could only use one government-mandated engine model across all cars and trucks.
Different tires and upholstery only go so far. With the wrong engine, many jobs cannot be done, rendering whole classes of vehicles pointless. If the mandated engine were particularly polluting, choosing a different model would have little effect on emissions.
That's the situation iOS creates regarding browsers today. The only recourse it to buy a phone running a different OS.
iOS matters because wealthy users carry iPhones. It's really as simple as that. Even when Apple's products fail to gain a numerical majority of users in a market, the margin contribution of iOS users can dominate all other business considerations.
Apple has deigned to allow "browsers" in its App Store since 2012. Those apps could not be browsers in a meaningful sense because they could not replace Safari as the default handler for http/https links.
But Apple has taken care to ensure that choice is only skin deep. Browsers on Windows, Linux, ChromeOS, Android, and MacOS can be Integrated Browsers, including their own competing engines. iOS, meanwhile, restricts browsers to shells over the system-provided WebView.
Unlike WebView browsers on other OSes, Apple locks down these components in ways that prevent competition in additional areas, including restrictions on network stacks that block improved performance, new protocols, or increased privacy. These restrictions make some sense in the context of WebView IABs, but extending them to browsers only serves to deflect pressure from Apple to improve their browser.
Perhaps it would be reasonable for iOS to foreclose competition from integrated browsers if it also kept other native apps from accessing powerful features. Such policies would represent a different view of what computing should be. However, Apple is happy to provide a wide variety of scary features to unsafe native applications, so long as they comply with the coercive terms of its App Store.
Apple forestalls this threat by keeping the web on iOS from feature parity. Outlawing true browser choice leaves only Apple's own, farcially under-powered, Safari/WebKit browser/engine...and there's precious little that other WebView browsers can do to improve the situation at a deep level.[11]
Seeing a Web App I worked on used by *Apple* to justify that the Web is a viable platform on iOS is bullshit
The Web can be an ideal place to build apps but Apple is consistently dragging their heals on implementing the Web APIs that would allow them to compete with native apps twitter.com/stopsatgreen/status/1389593307219701760
In addition, by refusing to let any other Web browser engines run on iOS. They are preventing any other browser filling in the feature gap. Truly holding back Web Apps on iOS.
I have defended Apple's choice to restrict web browsers on their platform before and I still do but they can't have their cake and eat it to.
They should not hold back Web Apps with one hand and then turn around and say that Web Apps can compete with native apps.
Developer anger only hints at the underlying structural rot. 25+ years of real browser competition has driven waves of improvements in security, capability, and performance. Competition has been so effective that browsers now represent most computing time on OSes with meaningful browser choice.
Hollowing out choice while starving Safari and WebKit of resources managed to put the genie back in the bottle. Privacy, security, performance, and feature evolution all suffer when the competition is less vibrant — and that's how Apple likes it.
A vexing issue for commentators regarding Apple's behaviour in this area is that of "market definition". What observers should understand is that, in the market for browsers, the costs that a browser vendor can inflict on web developers extend far beyond the market penetration for their specific product.
A typical (but misleading) shorthand for this is "standards conformanc". While Apple's engine falls woefully short on support for basic standards, that isn't even the beginning of the negative impacts.[12] Because the web is an open, interoperable platform, web developers build sites to reach the vast majority of browsers from a single codebase.
When browsers with more than ~10% share fail to add a feature or exhibit nasty bugs, developers must spend more to work around these limitations. When important APIs go missing, entire classes of content may simply be viewed as unworkable.
The cost of these capability gaps is steep. When the web cannot deliver experiences that native apps can (a very long list), businesses must build entirely different apps using Apple's proprietary tools. These apps, not coincidentally, can only be distributed via Apple's high-tax App Store.
A lack of meaningful choice in browsers leads directly to higher costs for users and developers across the mobile ecosystem even for folks that don't use Apple's products. Apple's norm-eroding policies have created a permission struture for bad actors like Facebook. Apple's leadership in the race to the bottom has inspired a burgeoning field of fast-followers.
Browser choice is not unrelated to other objectionable App Store policies. Keeping the web from competing is part and parcel of an architecture of control that tilts commerce into coercive, centralising stores, even though safer, browser-based alternatives would otherwise be possible.
Here's a quick summary of the systems and variations we've seen thus far, as well as their impacts on user choice:
System
Respects Choice
Notes
Integrated Browsers
Yes
Maximizes impact of choice
WebView Browsers
Yes
Reduces diversity in engines; problematic when the only option (iOS).
WebView IABs
No
Undermines user choice, reduces engine diversity, and directly harms developers through lower monetisation and feature availability (e.g., Facebook, Google Go).
Chrome Custom Tabs (CCT)
Partial
WebView IABs replacement, preserves choice by default (e.g. Twitter). Problematic when configured to ignore user preferences (e.g. AGA).
WebLayer
No
Like WebView with better feature support. Beneficial when used in place of WebViews for browsers. Problematic when used as a replacement for WebView IABs.
SFSafariViewController
Partial
Similar to CCT in spirit, but fails to support multiple browsers.
Proposals to repair the situation must centre on the effectiveness of browser choice.
Some policymakers have suggested browser choice ballots, but these will not be effective if user choice is undermined no matter which browser they choose. Interventions that encourage brand-level choice cannot have a positive effect until the deeper positive impacts of choice are assured.
Thankfully, repairing the integrity of browser choice in the mobile ecosystem can be accomplished with relatively small interventions. We only need to ensure that integrated browsers are universally available and that when third-party content is displayed, user choice of browser is respected.
Repairing the IAB situation will likely require multiple steps, given the extreme delay in new Android OS revisions gaining a foothold in the market. Thankfully, many fixes don't need OS updates:
Google should update the CCT system to respect browser choice when loading third-party content and require updates to CCT-using apps to this new behaviour within six months.
Verification of first-party content for use with specific engines is possible thanks to the Digital Asset Links infrastructure that underpins Trusted Web Activities, the official mechanism for putting web apps in the Play Store.
AGSA and Google Go should respect user choice via CCT.
Android's WebView and WebLayer should be updated with code to detect a new HTTP header value sent with top-level documents that cause the URL to be opened in the user's default browser (or a CCT for that browser) instead.
These systems update out-of-band every six weeks on 90+% of devices, delivering quick relief.
Such an opt-out mechanism preserves WebViews for first-party and second-party use-cases (those sites will simply not set the new header) while giving third-parties a fighting chance at being rendered in the user's default browser.
Apps that are themselves browsers (can be registered as default http/https handlers) would be exempt, preserving the ability to build WebView browsers. "Browserness" can be cheaply verified via an app's manifest.
Google should provide access to all private APIs currently reserved to Chrome, including but not limited to the ability to install web applications to the system (a.k.a. "WebAPKs").
Future releases of Android should bolster these improvements by creating system-wide opt-out of WebView and WebLayer IABs.
Play policy enforcement of rules regarding CCT, WebView, and WebLayer respect for user and developer choice will also be necessary. Such enforcement is not challenging for Google, given its existing binary analysis infrastructure.
Together, these small changes can redress the worst anti-web, anti-user, anti-developer, and anti-choice behaviour of Google and Facebook regarding Android browsers, putting users back in control of their data and privacy along the way.
iOS begins from a more troubling baseline but with somewhat better IAB policies. What's undermining user choice there require deeper, OS-level fixes, including:
Integrated browser choice, including access to APIs that iOS restricts to Safari today, such as:
The ability to create sandboxed subprocesses for renderers.
Push Notifications APIs.
Adding web apps to the home screen, including PWA installation.
Support in Web.app for alternative engine runtimes to ensure that home screen shortcuts and PWAs run in the correct context.
SFSafariViewController support for browsers other than Safari.
All apps that load non-consenting third-party websites (outside of edge cases like authentication flows) in IABs should be required to update to SFSafariViewController,
Apple's WebViews should support Content-Security-Policy: frame-ancestors 'system-default'
The proposal for a header to allow sites to demand CCT/SFSVC instead of a WebView IAB may seem complex, but it is technically straightforward and can be implemented very quickly.
Websites would include a tag (or the equivalent HTTP header) in top-level pages like this:
OS vendors would update their system WebViews to respect this tag and invoke CCT if encountered in a top-level document. This is compatible with the existing ecosystem, as no first-party content (help pages) or second-party integration (ad network) would send these headers, existing apps would not need to be updated. Websites could incrementally add the hint and benefit from the new behavior.
Android's WebView component auto-updates with Chrome, ensuring huge reach for such a fix in a short time. iOS updates are fused to OS upgrades, but iOS users tend to upgrade quickly. The net effect is that we should expect such a policy to begin to have a large, positive effect in less than 6 months.
What about apps that try to subvert the default behavior? App store policies can be easily formulated to punish this sort of poor behavior. There's a great deal of evidence that these policies work, at least for the "head" of an app catalog, and would surely condition Facebook's behavior.
The mobile web is a pale shadow of its potential because the vehicle of progress that has delivered consistent gains for two decades has silently been eroded to benefit native app platforms and developers. These attacks on the commons have at their core a shared disrespect for the sanctity of user choice, substituting the agenda of app and OS developers for mediation by a user's champion.
This power inversion has been as corrosive as it has been silent, but it is not too late. OSes and app developers that wish to take responsibility can start today to repair their own rotten, choice-undermining behaviour and put users back in control of their browsing, their data, and their digital lives.
The ball's in your court, platforms.
Deepest thanks to Eric Lawrence and Kevin Marks for their thoughtful feedback on drafts of this post.
Windows 10, for example includes several features (taskbar search box, lock screen links) that disrespect a user's choice of default browser. This sort of shortcut-taking in the competition for user attention has a long and discouraging history, but until relatively recently was viewed as "out of bounds". Mobile has shifted the Overton Window.
A decade of degraded norms around browser choice by mobile OSes has made these sorts of unreasonable tie-ins less exceptional. The work-a-day confusion of following links on mobile helps to create a permission structure that enables ever-more bad behaviour. The Hobbesian logic of power-begets-success is fundamentally escalatory, forcing those without a priori privilege into a paranoid mode, undercutting attempts to differentiate products in a market on their merits.
Fixing mobile won't be sufficient to unwind desktop's increasingly negative dark patterns, of course. But that's no reason to delay. Centering user's choices on their most personal devices can do much to reset the expectations of PMs and managers across the industry as to which tactics are, in fact, above-board. ↩︎
It's less clear why Mozilla is MIA in at least making noise about the situation. Their organisation has a front-row seat to the downsides of undermined user choice. The inability to project the benefits of their engine into the lives of their mobile users materially harms their future business and differentiation prospects.
It seems unlikely (if plausible) that the Firefox OS experience has so thoroughly burned management that there is no scope for mobile risk-taking, even if constrained to jawboning or blog posts.
If any organisation can credibly, independently connect the dots, it should be the Mozilla Foundation. One hopes they do. ↩︎
The history, competitive pressures, and norms of Android app developers caused many smaller apps to capture clicks (and user data), failing to send navigations onward.
A shortlist of notable apps that undermine user choice via IABs would include:
Facebook Messenger
Instagram
Pinterest
Snapchat
Microsoft Bing Search
Some apps that previously (ab)used WebViews for IABs in the pre-CCT era switched over to that choice-respecting mechanism, notably Twitter. ↩︎
This definition of "a browser" may sit uncomfortably with folks accustomed to the impoverished set of choices Apple made possible on iOS until late last year. In particular, folks will undoubtedly note that "alternative browsers" were available in the App Store much earlier, including a Chrome-branded app since at least 2012.
Not all applications that can load web pages are browsers. Only apps that can become the user's agent in browsing the web are. Until nine months ago, iOS only supported Safari as a proper browser. "Alternative browsers" could only traverse link space when users began browsing within them. They were impotent to support users more broadly, unable to consistently assist users, modulate harmful aspects of content, or project user preferences into sites. Without the ability to catch all navigations sent to the OS, users who downloaded these programs suffered frequent computing amnesia. User preferences were only respected if users started browsing from within a specific app. Incidental navigations, however, were subject to Apple's monopoly on link handling and whatever choices Safari projected.
In this way, iOS undermined choice and competition. OSes that prevent users from freely picking their agent in navigating the web most of the time cannot, therefore, be said to support browser choice — no matter how many directed-browsing apps they allow to list in their stores. ↩︎
Problems related to background task killing can, of course, be avoided by building a web app instead of a native app one. When users remain in a browser across sites, there's no heavy process switch between pages. Developers tried this path for a while but quickly found themselves at an impossible feature disadvantage. Lack of Push Notifications alone proved business-defining, and Apple's App Store policies explicitly forbid web apps in their store.
To be discovered where users are looking for apps and access business-critical features, mobile platforms effectively forced all developers into app stores. A strong insinuation that things would not go well for them in app stores if they used web technologies (via private channels, naturally) reliably accompanied this Sophie's choice.
Platforms played these user-and-developer hostile games in mobile's early days to dig a moat of OS-exclusive apps. Exclusives create friction for users considering a switch to a different OS. Platform owners know the cost of re-developing apps for each OS means when independent software vendors invest heavily in their proprietary systems, it becomes less likely that those developers can deliver quality experiences on their competitor's system.
App developers only have so many hours in the day, and it costs enormous amounts, both initially and in an ongoing way, to re-build features for each additional platform. The web is a portable applications platform, and portability is a bug to proprietary platform owners. The combination of engine neglect, feature gap expansion, and app store policies against web participation — explicit and implied — proved a shockingly effective "fix".
The story of feature-gap coercion and "app store lottery" games illuminate the backdrop of a new normal that none of us should accept. ↩︎
Many have adroitly covered the perspective and ethical distortions within social media firms caused by the relentless pursuit of "north star" metrics. There's little new I can add.
I can, however, confirm some uncharitable takes of their detractors are directionally correct. One cannot engage with engineers and PMs from these organisations for a decade without learning something about their team's values.
The blinkered pursuit of growth via "make number go up"-OKRs creates blind spots that are managed as exogenous crises. The health of the ecosystems around them is unfailingly subordinate to questions of competitive positioning. The hermetically circular logic of "we're changing the world for the better"does create incentives to undermine user autonomy, safety, and choice.
The jury is no longer out. Change is possible, but it will not come from within. But "unintended consequences!" special pleading weighs heavily. To improve this situation, folks must understand it sufficient depth to mandate maximally effective, competition-and-choice-enhancing interventions that carry the lightest footprint.
In the long list of dangerous, anti-competitive, opacity-increasing ills of modern tech products, the hollowing out of browser choice may seem small-time. Issues of content recommendation radicalisation, "persuasive design" dark patterns, source-of-funds ads opacity, and buried data collection controls surely deserve more attention. However, it would be a missed opportunity not to put users back in control of this aspect of their digital lives whilst the opportunity presents itself. ↩︎
Social apps strip-mining ecosystems they didn't build for their benefit while deflecting responsibility for downside consequences?
Facebook engineers have noted that the FB IAB is important in fighting bad behaviour on their social network. We should take these claims at face value.
Having done so, several further questions present themselves:
Why, then, is this system not opt-in? Presumably Facebook can convince a representative subset of users to enable it while preserving browser choice for the vast majority.
Why is this not a game-over problem for Facebook's desktop website?
If it's necessary to keep users within a browser that Facebook owns end-to-end, why not simply allow Facebook's native apps to be browsers. It's a simple Android manifest change that would put them back into line with the norms and expectations of the broader web community and allow them to compete for user's browsing time on the up-and-up. Not doing so suggests they have something to hide and may be ashamed of this browser that, by their calculations, keeps users safer.
The need for more information to protect users may be real, but undermining choice for all is a remedy that, at least with the information that's public thus far, seems very tough to justify. ↩︎
iOS didn't support browser choice at the time of SFSafariViewController's introduction and appeared only to have acquiesced to minimal (and initially broken) browser choice under regulatory duress. It is hardly surprising, then, that Apple hasn't updated SFSafariViewController to work with other default browsers the way CCT does.
For reasons that seem to boil down to Great Power calculations and myopic leadership focus on desktop, none of the major browser vendors has publicly challenged these rules or the specious, easily-debunked arguments offered to support them.
To recap, Apple has at various points argued that the blatantly anti-competitive policies against integrated browsers are necessary because Apple cannot allow programs to run Just-In-Time (JIT) compilers for languages like JavaScript due to safety concerns. Apple's WebKit framework is the only program on the system allowed to contain such a JIT.
Commenters forwarding these claims, as a rule, do not understand browser architecture. Any modern browser can suffer attacks against the privileged "parent" process, JIT or not. These "sandbox escapes" are not less likely for the mandated use of WebKit; indeed, by failing to expose APIs for sandboxed process creation, Apple prevents others from bringing stronger protections to users. iOS's security track record, patch velocity, and update latency for its required-use engine is not best-in-class.
User security would be meaningfully improved were Apple to allow integrated browsers that demonstrated an Apple-esqe-or-better patch velocity. Such a policy is not hard to formulate, and the ability for apps running on top of the OS to update without slow, painful-for-users update processes would meaningfully improve patch rates versus today's OS-update-locked cadence for WebKit.
Some commenters claim that browsers might begin to provide features that some users deem (without evidence) unnecessary or unsafe if alternative engines were allowed. These claims are doubly misinformed.
Alternative WebView browsers can already add features through JavaScript monkey-patching. There's no substantive security or privacy benefit in forcing browser vendors to re-build them in a contorted (but still allowable) way on top of WebViews. Indeed, bringing an integrated engine to iOS would do much to prevent one-off security issues that have been a frequent occurrence in such WebView browser feature extensions. Securing a single codebase is more straightforward than having to analyse multiple platform-specific workarounds. Engine choice will improve security, in part, by focusing limited security reviewer time on fewer attack vectors. Of course, a functioning market for browsers will still allow users to pick from under-powered, less secure, slower-updating, feature-light browsers as they can today; Safari, for example.
Misdirection about JITs and per-feature security posture are technically wanting but serve ably distract from iOS's deeper restrictions. Capable integrated browsers need access to a suite of undocumented APIs and capabilities Apple currently reserves to Safari, including the inability to create processes, set tighter sandboxing boundaries, or efficiently decode alternative media formats. Opening these APIs to competing integrated browsers would pave the way to safer, faster, more capable computing for iPhone owners.
Others have argued on Apple's behalf that if engine competition were allowed, Chromium's (Open Source) Blink engine would become ubiquitous on iOS, depriving the ecosystem of diversity in engines. This argument is seemingly offered with a straight face to defend the very policies that have prevented effective engine diversity to date. Mozilla ported Gecko twice, but was never allowed to bring its benefits to iOS users. In addition to being self-defeating regarding engine choice, this fear also seems to ignore the best available comparison points. Safari is the default browser for MacOS and has maintained a healthy 40-50% share for many years, despite healthy competition from other integrated browsers (Chrome, Firefox, Opera, Edge, etc.). Such an outcome is at least as likely on iOS.
Sitting under all of these arguments are, I suspect, more salient concerns to Apple's executives to resist increasing RAM in the iPhone's Bill of Materials. In the coerced status quo, Apple can drive device margins by provisioning relatively little in the way of (expensive) RAM components while still supporting multitasking. A vital aspect of this penny-pinching is to maximise sharing of "code pages" between programs. If alternative browsers suddenly began bringing their engines, code page sharing would not be as effective, requiring more RAM in Apple's devices to provide good multitasking experiences. More RAM could help deliver increased safety and choice to users, but would negatively impact Apple's bottom line.
Undermining user choice in browsers has, in this way, returned significant benefits — to AAPL shareholders, anyway. ↩︎
Engine developers possess outsized ability within standards bodies to deny new features and designs the ability to become standards in the first place. The Catch-22 is easy to spot once you know to look for it, but casual observers are often unacquainted with the way feature development on the web works.
In a nutshell, its often the case features are shipped by browsers ahead of final, formal inclusion in web standards. Specifications are documents that describe the working of a system. Some specifications are ratified by Standards Development Organisations (SDOs) like the World Wide Web Consortium (W3C) or Internet Engineering Task Force (IETF) as "web standards". Thanks to wide implementation and unambiguous IP licensing, standards increase market confidence and adoption of designs. But no new feature's specification begins life as a standard.
Market testing of proposed standards ("running code" in IETF-speak) are essential for the progress of any platform, and pejorative claims that a feature in this state is "proprietary" is misleading. This bleeds into active deception when invoked by other vendors who neither propose alternatives to solve developer challenges nor participate in shaping proposals in open collaboration.
Withholding engagement, then claiming that someone else is proceeding unilaterally — when your input would remove the stain — is a rhetorical Möbius strip. ↩︎
Git Worktrees appear to solve a set of challenges I encounter when working on this blog:
Maintenance branches for 11ty and other dependencies come and go with some frequency.
Writing new posts on parallel branches isn't fluid when switching frequently.
If I incidentally mix some build upgrades into a content PR, it can be difficult to extract and re-apply if developed in a single checkout.
Worktrees hold the promise of parallel working branch directories without separate backing checkouts. Tutorials I've found seemed to elide some critical steps, or required deeper Git knowledge than I suspect is common (I certainly didn't have it!).
After squinting at man pages for more time than I'd care to admit and making many mistakes along the way, here is a short recipe for setting up worktrees for a blog repo that, in theory, already exists at github.com/example/workit:
## # Make a directory to hold a branches, including main ##
$ cd /projects/ $ mkdir workit $ cd workit $ pwd # /projects/workit
## # Next, make a "bare" checkout into `.bare/` ##
## # Tell Git that's where the goodies are via a `.git` # file that points to it ##
$ echo"gitdir: ./.bare"> .git
## # *Update* (2021-09-18): OPTIONAL # # If your repo is going to make use of Git LFS, at # this point you should stop and edit `.bare/config` # so that the `[remote "origin"]` section reads as: # # [remote "origin"] # url = git@github.com:example/workit.git # fetch = +refs/heads/*:refs/remotes/origin/* # # This ensures that new worktrees do not attempt to # re-upload every resource on first push. ##
## # Now we can use worktrees. # # Start by checking out main; will fetch repo history # and may therefore be slow. ##
$ git worktree add main # Preparing worktree (checking out 'main') # ... # Filtering content: 100% (1226/1226), 331.65 MiB | 1.17 MiB/s, done. # HEAD is now at e74bc877 do stuff, also things
## # From here on out, adding new branches will be fast ##
$ git worktree addtest # Preparing worktree (new branch 'test') # Checking out files: 100% (2216/2216), done. # HEAD is now at e74bc877 do stuff, also things
## # Our directory structure should now look like ##
$ ls-la # total 4 # drwxr-xr-x 1 slightlyoff eng 38 Jul 7 23:11 . # drwxr-xr-x 1 slightlyoff eng 964 Jul 7 23:04 .. # drwxr-xr-x 1 slightlyoff eng 144 Jul 7 23:05 .bare # -rw-r--r-- 1 slightlyoff eng 16 Jul 7 23:05 .git # drwxr-xr-x 1 slightlyoff eng 340 Jul 7 23:11 main # drwxr-xr-x 1 slightlyoff eng 340 Jul 7 23:05 test
## # We can work in `test` and `main` independently now ##
Thankfully, commands like git worktree list and git worktree remove are relatively WYSIWYG by comparison to the initial setup.
Perhaps everyone else understands .git file syntax and how it works with --bare checkouts, but I didn't. Hopefully some end-to-end exposition can help drive adoption of this incredibly useful feature.
Update (June 16th, 2021): Folks attempting to build mobile web games have informed me that the Fullscreen APIremains broken on iOS for non-video elements. This hobbles gaming and immersive media experiences in a way that is hard to overstate. Speaking of being hobbled, the original post gave Apple credit for eventually shipping a useable implementation of IndexedDB. It seems this was premature.
Apple's iOS browser (Safari) and engine (WebKit) are uniquely under-powered. Consistent delays in the delivery of important features ensure the web can never be a credible alternative to its proprietary tools and App Store.
This is a bold assertion, and proving it requires overwhelming evidence. This post mines publicly available data on the pace of compatibility fixes and feature additions to assess the claim.
Misdirections often derail the debate around browsers, the role of the web, and App Store policies on iOS. Classics of the genre include:
Apple's just focused on performance!
...that feature is in Tech Preview
Apple's trying, they just added <long-awaited feature>
These points can be simultaneously valid and immaterial to the web's fitness as a competent alternative to native app development on iOS.
We have to check reservoir levels and seasonal rainfall to know if we're in a drought. It might be raining features right this instant, but weather isn't climate. We should look at trends rather than individual releases to understand the gap Apple created and maintains between the web and native.
Before we get to measuring water levels, I want to make some things excruciatingly clear.
First, what follows is not a critique of individuals on the Safari team or the WebKit project; it is a plea for Apple to fund their work adequately[2] and allow competition. They are, pound for pound, some of the best engine developers and genuinely want good things for the web. Apple Corporate is at fault, not Open Source engineers or the line managers who support them.
Second, projects having different priorities at the leading edge is natural and healthy. So is speedy resolution and agreement. What's unhealthy is an engine trailing far behind for many years. Even worse are situations that cannot be addressed through browser choice. It's good for teams to be leading in different areas, assuming that the "compatible core" of features continues to expand at a steady pace. We should not expect uniformity in the short run — it would leave no room for leadership[3].
Lastly, while this post does measure the distance Safari lags, let nobody mistake that for the core concern: iOS App Store policies that prevent meaningful browser competition are at issue here.
Safari trails competing macOS browsers by roughly the same amount, but it's not a crisis because genuine browser choice enables meaningful alternatives.
macOS Safari is compelling enough to have maintained 40-50% share for many years amidst stiff competition. Safari has many good features, and in an open marketplace, choosing it is entirely reasonable.
As an engineer on a browser team, I've been privy to the blow-by-blow of various performance projects, benchmark fire drills, and the ways performance marketing impacts engineering priorities.
All modern browsers are fast, Chromium and Safari/WebKit included. No browser is always fastest. As reliably as the Sun rises in the East, new benchmarks launch projects to re-architect internals to pull ahead. This is as it should be.
Healthy competitions feature competitors trading the lead with regularity. Performance Measurement is easy to get wrong. Spurious reports of "10x worse" performance merit intense scepticism, as they tend instead to be mismeasurement. This makes sense given the intense focus of all browser teams on performance.
All browsers are deep into the optimisation journey, forcing complex tradeoffs. Improving things for one type of device or application can regress them for others. Significant gains today tend to come from (subtly) breaking contracts with developers in the hopes users won't notice. There isn't a massive gap in focus on performance engineering between engines.
Small gaps and a frequent hand-off of the lead imply differences in capability and correctness aren't the result of one team focusing on performance while others chase different goals[4].
Finally, the choice to fund feature and correctness work is not mutually exclusive to improving performance. Many delayed features on the list below would allow web apps to run faster on iOS. Internal re-architectures to improve correctness often yield performance benefits too.
Web developers are a hearty bunch; we don't give up at the first whiff of bugs or incompatibility between engines. Deep wells of knowledge and practice centre on the question: "how can we deliver a good experience to everyone despite differences in what their browsers support?"
Adaptation is a way of life for skilled front enders.
The cultural value of adaptation has enormous implications. First, web developers don't view a single browser as their development target. Education, tools, and training all support the premise that supporting more browsers is better (ceteris paribus), creating a substantial incentive to grease squeaky wheels. Therefore, bridging the gap between leading and trailing-edge browsers is an intense focus of the web development community. Huge amounts of time and effort are spent developing workarounds (preferably with low runtime cost) for lagging engines[5]. Where workarounds fail, cutting features and UI fidelity is understood to be the right thing to do.
Compatibility across engines is key to developer productivity. To the extent that an engine has more than 10% share (or thereabouts), developers tend to view features it lacks as "not ready". It's therefore possible to deny web developers access to features globally by failing to deliver them at the margin.
A single important, lagging engine can make the whole web less competitive this way.
To judge the impact of iOS along this dimension, we can try to answer a few questions:
How far behind both competing engines is Safari regarding correctness?
When Safari has implemented essential features, how often is it far ahead? Behind?
Tests that fail only in a given browser. Lower is better.
The yellow Safari line is a rough measure of how often other browsers are compatible, but Safari's implementation is wrong. Conversely, the much lower Chrome and Firefox lines indicate Blink and Gecko are considerably more likely to agree and be correct regarding core web standards[6].
Stable-channel Compat 2021 results over time. Higher is better. Tip-of-tree improvements are visible in WebKit. Sadly, these take quarters to reach devices because Apple ties WebKit features to the slow cadence of OS releases.
In almost every area, Apple's low-quality implementation of features WebKit already supports requires workarounds. Developers would not need to find and fix these issues in Firefox (Gecko) or Chrome/Edge/Brave/Samsung Internet (Blink). This adds to the expense of developing for iOS.
Engines add features at different rates, and the Confluence graphs illuminate both the absolute scale of differences and the pace at which releases add new features. The data is challenging to compare across those graphs, so I extracted it to produce a single chart:
Chrome
Firefox
Safari
Count of APIs available from JavaScript by Web Confluence.
Higher is better.
In line with Web Platform Tests data, Chromium and Firefox implement more features and deliver them to market more steadily. From this data, we see that iOS is the least complete and competitive implementation of the web platform, and the gap is growing. At the time of the last Confluence run, the gap had stretched to nearly 1000 APIs, doubling since 2016.
To understand if intuitions formed by the Web Confluence data are directionally correct, we need to look more deeply at the history of feature development and connect APIs to the types of applications they enable.
Browser release notes and caniuse tables since Blink forked from WebKit in 2013[7] capture the arrival of features in each engine over an even longer period than either WPT or the Confluence dataset. This record can inform a richer understanding of how individual features and sets of capabilities unlock new types of apps.
Browsers sometimes launch new features simultaneously (e.g., CSS Grid and ES6). More often, there is a lag between the first and the rest. To provide a sizeable "grace period", and account for short-run differences in engine priorities, we look primarily at features with a gap of three years or more[8].
What follows is an attempt at a full accounting of features launched in this era. A summary of each API and the impact of its absence accompanies every item.
It's healthy for engines to have different priorities, leading every browser to avoid certain features. Still, mistakes have been made, and Chrome has missed several APIs for 3+ years:
Image carousels and other touch-based UIs are smoother and easier to build using this feature. Differences within the Blink team about the correct order to deliver this vs. Animation Worklets led to regrettable delays.
Makes "fixed" elements in scroll-based UIs easier to build. The initial implementation was removed from Blink post-fork and re-implemented on new infrastructure several years later.
Next-generation video codecs, supported in many modern chips, but also a licensing minefield. The open, royalty-free codec AV1 has been delivered instead.
Some features in this list were launched in Safari but were not enabled for other browsers forced to use WebKit on iOS (e.g. Service Workers, getUserMedia). In these cases, only the delay to shipping in Safari is considered.
Audio Worklets are a fundamental enabler for rich media and games on the web. Combined with WebGL2/WebGPU and WASM threading (see below), Audio Worklets unlock more of a device's available computing power, resulting in consistently good sound without fear of glitching.
After years of standards discussion and the first delivered to other platforms in 2018, iOS 14.5 finally shipped Audio Worklets this week.
Had Apple shipped a usable version in either of the first two attempts, IndexedDB would not have made the three-year cut. The release of iOS 10 finally delivered a workable version, bringing the lag with Chrome and Firefox to four and five years, respectively.
Critical for gaming with a mouse. Still not available for iOS or iPadOS.
Update: some commenters seem to sneer the the idea of using a mouse for gaming on iOS, but it has been reported to browser teams as a key feature by the teams building game streaming PWAs. It appears a sizeable set of users use external mice and keyboards with their iPads, and the entire categories of games are functionally unusable on these platforms without Pointer Lock.
Royalty-free codecs and containers; free alternatives to H.264/H.265 with competitive compression and features. Lack of support forces developers to spend time and money transcoding and serving to multiple formats (in addition to multiple bitrates).
Supported only for use in WebRTC but not the usual mechanisms for media playback (<audio> and <video>). Either delayed 9 years or still not available, depending on use.
element.animate(), a subset of the full API, has enabled developers to more easily create high-performance visual effects with a lower risk of visual stuttering in Chrome and Firefox since 2014.
The impact of missing Web Performance APIs is largely a question of scale: the larger the site or service one attempts to provide on the web, the more important measurement becomes.
Delayed two to four years, depending on how one counts.
Not every feature blocked or delayed on iOS is transformative, and this list omits cases that were on the bubble (e.g., the 2.5 year lag for BigInt). Taken together, the delays Apple generates, even for low-controversy APIs, makes it challenging for businesses to treat the web as a serious development platform.
It's also possible that APIs delivered on every other platform, but not yet available on any iOS browser (because Apple), may unlock whole categories of experiences on the web.
While dozens of features are either currently, or predicted to be, delayed multiple years by Apple, a few high-impact capabilities deserve particular mention:
WebGPU will also unlock richer GPU compute for the web, accelerating machine learning and media applications. WebGPU is likely to ship in Chrome in late 2021. Despite years of delay in standards bodies at the behest of Apple engineers, the timeline for WebGPU on iOS is unclear. Keen observers anticipate a minimum of several years of additional delay.
Web Assembly ("WASM") is supported by all browsers today, but extensions for "threading" (the ability to use multiple processor cores together) are missing from iOS.
Threading support enables richer and smoother 3D experiences, games, AR/VR apps, creative tools, simulations, and scientific computing. The history of this feature is complicated, but TL;DR, they are now available to sites that opt in on every platform save iOS. Worse, there's no timeline and little hope of them becoming available soon.
Combined with delays for Audio Worklets, modern graphics APIs, and Offscreen Canvas, many compelling reasons to own a device have been impossible to deliver on the web.[11]
Now in development in WebKit after years of radio silence, WebXR APIs provide Augmented Reality and Virtual Reality input and scene information to web applications. Combined with (delayed) advanced graphics APIs and threading support, WebXR enables immersive, low-friction commerce and entertainment on the web.
Support for a growing list of these features has been available in leading browsers across other platforms for several years. There is no timeline from Apple for when web developers can deliver equivalent experiences to their iOS users (in any browser).
These omissions mean web developers cannot compete with their native app counterparts on iOS in categories like gaming, shopping, and creative tools.
Developers expect some lag between the introduction of native features and corresponding browser APIs. Apple's policy against browser engine choice adds years of delays beyond the (expected) delay of design iteration, specification authoring, and browser feature development.
These delays prevent developers from reaching wealthy users with great experiences on the web. This gap, created exclusively and uniquely by Apple policy, all but forces businesses off the web and into the App Store where Apple prevents developers from reaching users with web experiences.
One might imagine five-year delays for 3D, media, and games might be the worst impact of Apple's policies preventing browser engine progress. That would be mistaken.
The next tier of missing features contains relatively uncontroversial proposals from standards groups that Apple participates in or which have enough support from web developers to be "no-brainers". Each enables better quality web apps. None are expected on iOS any time soon:
Likely to ship in Chromium later this year, enables smooth animation based on scrolling and swiping, a key interaction pattern on modern mobile devices.
No word from Apple on if or when this will be available to web developers on iOS.
Reduces data use and improves page load performance.
Fewer of these features are foundational (e.g. SIMD). However, even those that can be emulated in other ways still impose costs on developers and iOS users to paper over the gaps in Apple's implementation of the web platform. This tax can, without great care, slow experiences for users on other platforms as well[12].
Beyond these relatively uncontroversial (MIA) features lies an ocean of foreclosed possibility. Were Apple willing to allow the sort of honest browser competition for iOS that macOS users enjoy, features like these would enable entirely new classes of web applications. Perhaps that's the problem.
Some crucial features (shipped on every other OS) that Apple is preventing any browser from delivering to iOS today, in no particular order:
It's difficult to overstate the challenges posed by a lack of push notifications on a modern mobile platform. Developers across categories report a lack of push notifications as a deal-killer, including:
Chat, messaging, and social apps (for obvious reasons)
Apple's maintenance of this feature gap between native and web (despite clear underlying support for the mechanism) and unwillingness to allow other iOS browsers to improve the situation[13], combined with policies that prevent the placement of web content in the App Store, puts a heavy thumb on the scale for discovering content built with Apple's proprietary APIs.
Enables web apps to play media while in the background. It also allows developers to plug into (and configure) system controls for back/forward/play/pause/etc. and provide track metadata (title, album, cover art).
Lack of this feature prevents entire classes of media applications (podcasting and music apps like Spotify) from being plausible.
In development now, but if it ships this fall (the earliest window), web media apps will have been delayed more than five years.
Dramatically improve page loading performance on sites that provide an offline experience using Service Workers.
Multiple top-10 web properties have reported to Apple that lack of this feature prevents them from deploying more resilient versions of their experiences (including building PWAs) for users on iOS.
Improves the smoothness of 3D and media applications by moving rendering work to a separate thread. For latency-sensitive use-cases like XR and games, this feature is necessary to consistently deliver a competitive experience.
An addition to the Web Components system that powers applications like YouTube and Apple Music. Declarative Shadow DOM can improve loading performance and help developers provide UI for users when scripts are disabled or fail to load.
Indispensable for improving the quality of sites and avoid breakage due to browser deprecations. Modern versions also let developers know when applications crash, helping them diagnose and repair broken sites.
Keeps the screen from going dark or a screen saver taking over. Important for apps that present boarding passes and QR codes for scanning, as well as and presentation apps (e.g. PowerPoint or Google Slides).
Helps developers avoid prompting users for permissions that might be duplicative with apps already on the system. Particularly important for avoiding duplicated push notifications.
Allows applications to upload and download bulk media efficiently with progress indicators and controls. Important for reliably syncing playlists of music or videos for offline or synchronising photos/media for sharing.
Allows installed web apps to receive sharing intents via system UI, enabling chat and social media apps to help users post content more easily.
The list of missing, foundational APIs for media, social, e-commerce, 3d apps, and games is astonishing. Essential apps in the most popular categories in the App Store are impossible to attempt on the web on iOS because of feature gaps Apple has created and perpetuates.
An area where browsers makers disagree fervently, but where Chromium-based browsers have forged ahead (Chrome, Edge, Samsung Internet, Opera, UC, etc.) is access to hardware devices. While not essential to most "traditional" web apps, these features are foundational for vibrant categories like education and creative music applications. iOS Safari supports none of them today, while Chromium browsers on other OSes enable these apps on the web:
Allows Bluetooth Low Energy devices to safely communicate with web apps, eliminating the need to download heavyweight applications to configure individual IoT devices.
Provides safe access to USB devices from the web, enabling new classes of applications in the browser from education to software development and debugging.
Enables safe connection to input devices not traditionally supported as keyboards, mice, or gamepads.
This API provides safe access to specialised features of niche hardware over a standard protocol they already support without proprietary software or unsafe native binary downloads.
A uniform API for accessing sensors standard in phones, including Gyroscopes, Proximity sensors, Device Orientation, Acceleration sensors, Gravity sensors, and Ambient Light detectors.
Each entry in this inexhaustive list can block entire classes of applications from credibly being possible on the web. The real-world impact is challenging to measure. Weighing up the deadweight losses seems a good angle for economists to investigate. Start-ups not attempted, services not built, and higher prices for businesses forced to develop native apps multiple times could, perhaps, be estimated.
The data agree: Apple's web engine consistently trails others in both compatibility and features, resulting in a large and persistent gap with Apple's native platform.
Apple wishes us to accept that:
It is reasonable to force iOS browsers to use its web engine, leaving iOS on the trailing edge.
The web is a viable alternative on iOS for developers unhappy with App Store policies.
One or the other might be reasonable. Together? Hmm.
Parties interested in the health of the digital ecosystem should look past Apple's claims and focus on the differential pace of progress.
Full disclosure: for the past twelve years I have worked on Chromium at Google, spanning both the pre-fork era where potential features for Chrome and Safari were discussed within the WebKit project, as well as the post-fork epoch. Over this time I have led multiple projects to add features to the web, some of which have been opposed by Safari engineers.
Today, I lead Project Fugu, a collaboration within Chromium that is directly responsible for the majority of the device APIs mentioned above. Microsoft, Intel, Google, Samsung, and others are contributing to this work, and it is being done in the open with the hope of standardisation, but my interest in its success is large. My front-row seat allows me to state unequivocally that independent software developers are clamouring for these APIs and are ignored when they request support for them from Apple. It is personally frustrating to be unable to deliver these improvements for developers who wish to reach iOS users — which is all developers. My interests and biases are plain.
Previously, I helped lead the effort to develop Service Workers, Push Notifications, and PWAs over the frequent and pointed objections of Apple's engineers and managers. Service Worker design was started as a collaboration between Google, Mozilla, Samsung, Facebook, Microsoft, and independent developers looking to make better, more reliable web applications. Apple only joined the group after other web engines had delivered working implementations. The delay in availability of Service Workers (as well as highly-requested follow-on features like Navigation Preload) for iOS users and developers interested in serving them well, likewise, carries an undeniable personal burden of memory.
iOS is unique in disallowing the web from participating in its only app store. macOS's built-in App Store has similar anti-web terms, but macOS allows multiple app stores (e.g. Steam and the Epic Store), along with real browser choice.
Android and Windows directly include support for web apps in their default stores, allow multiple stores, and facilitate true browser choice. ↩︎
Failing adequate staffing for the Safari and WebKit teams, we must insist that Apple change iOS policy to allow competitors to safely fill the gaps that Apple's own skinflint choices have created. ↩︎
Claims that I (or other Chromium contributors) would happily see engine homogeneity could not be more wrong. ↩︎
Some commenters appear to confuse unlike hardware for differences in software. For example, an area where Apple is absolutely killing it is CPU design. Resulting differences in Speedometer scores between flagship Android and iOS devices are demonstrations of Apple's domineering lead in mobile CPUs.
A-series chips have run circles around other ARM parts for more than half a decade, largely through gobsmacking amounts of L2/L3 cache per core. Apple's restrictions on iOS browser engine choice have made it difficult to demonstrate software parity. Safari doesn't run on Android, and Apple won't allow Chromium on iOS.
Thankfully, the advent of M1 Macs makes it possible to remove hardware differences from comparisons. For more than a decade, Apple has been making tradeoffs and unique decisions in cache hierarchy, branch prediction, instruction set, and GPU design. Competing browser makers are just now starting to explore these differences and adapt their engines to take full advantage of them.
As that is progressing, the results are coming back into line with the situation on Intel: Chromium is roughly as fast, and in many cases much faster, than WebKit.
The lesson for performance analysis is, as always, that one must always double-and-triple-check to ensure you actually measure what you hope to. ↩︎
Ten years ago, trailing-edge browsers were largely the detritus of installations that could not (or would not) upgrade. The relentless march of auto-updates has largely removed this hurdle. The residual set of salient browser differences in 2021 is the result of some combination of:
Market-specific differences in browser update rates; e.g., emerging markets show several months of additional lag between browser release dates and full replacement
Increasingly rare enterprise scenarios in where legacy browsers persist (e.g., IE11)
Differences in feature support between engines
As other effects fade away, the last one comes to the fore. Auto-updates don't do as much good as they could when the replacement for a previous version lacks features developers need. Despite outstanding OS update rates, iOS undermines the web at large by projecting the deficiencies of WebKit's leading-edge into every browser on every iOS device. ↩︎
Perhaps it goes without saying, but the propensity for Firefox/Gecko to implement features with higher quality than Safari/WebKit is a major black eye for Apple.
A scrappy Open Source project without ~$200 billion in the bank is doing what the world's most valuable computing company will not: investing in browser quality and delivering a more compatible engine across more OSes and platforms than Apple does.
This should be reason enough for Apple to allow Mozilla to ship Gecko on iOS. That they do not is all the more indefensible for the tax it places on web developers worldwide. ↩︎
Where I was aware they were not accurate — often related to releases in which features first appeared — or where they disagreed, original sources (browser release notes, contemporaneous blogs) have been consulted to build the most accurate picture of delays.
The presence of features in "developer previews", beta branches, or behind a flag that users must manually flip have not been taken into account. This is reasonable based on several concerns beyond the obvious: that developers cannot count on the feature when it is not fully launched, mooting any potential impact on the market:
Some features linger for many years behind these flags (e.g. WebGL2 in Safari).
Features not yet available on release branches may still change in their API shape, meaning that developers would be subject to expensive code churn and re-testing to support them in this state.
Browser vendors universally discourage users from enabling experimental flags manually
Competing engines led WebKit on dozens of features not included in this list because of the 3+ year lag cut-off.
The data shows that, as a proportion of features landed in a leading vs. trailing way, it doesn't much matter which timeframe one focuses on. The proportion of leading/lagging features in WebKit remains relatively steady. One reason to omit shorter time periods is to reduce the impact of Apple's lethargic feature release schedule.
Even when Apple's Tech Preview builds gain features at roughly the same time as Edge, Chrome, or Firefox's Beta builds, they may be delayed in reaching users (and therefore becoming available to developers) because of the uniquely slow way Apple introduces new features. Unlike leading engines that deliver improvements every six weeks, the pace of new features arriving in Safari is tied to Apple's twice-a-year iOS point release cadence. Prior to 2015, this lag was often as bad as a full year. Citing only features with a longer lag helps to remove the impact of such release cadence mismatch effects to the benefit of WebKit.
It is scrupulously generous to Cupertino's case that features with a gap shorter than three years were omitted. ↩︎
One effect of Apple's forced web engine monoculture is that, unlike other platforms, issues that affect WebKit impact every other browser on iOS too.
Not only do developers suffer an unwelcome uniformity of quality issues, users are impacted negatively when security issues in WebKit create OS-wide exposure to problems that can only be repaired at the rate OS updates are applied. ↩︎
The three-year delay in Apple implementing Pointer Events for iOS is in addition to delays due to Apple-generated licensing drama within the W3C regarding standardisation of various event models for touch screen input. ↩︎
During the drafting of this post, iOS 14.5 was released and with it, Safari 14.1.
The Web Assembly community was understandably excited and began to test the claim, but could not seem to make the feature work as hoped.
Soon after, Apple updated it's docs and provided details on what was, in fact, added. Infrastructure that will eventually be critical to a WASM Threading solution in WebKit was made available, but it's a bit like an engine on a test mount: without the rest of the car, it's beautiful engineering without the ability to take folks where they want to go.
WASM Threads for iOS had seen their shadow and six more months of waiting (minimum) are predicted. At least we'll have one over-taxed CPU core to keep us warm. ↩︎
It's perverse that users and developers everywhere pay a tax for Apple's under-funding of Safari/WebKit development, in effect subsidising the world's wealthiest firm. ↩︎
Safari uses a private API not available to other iOS browsers for installing web apps to the home screen.
Users who switch their browser on iOS today are, perversely, less able to make the web a more central part of their computing life, and the inability for other browsers to offer web app installation creates challenges for developers who must account for the gap and recommend users switch to Safari in order to install their web experience. ↩︎
A silly little PWA has been brewing over the past couple of weekends to make a desktop-compatible version of a mobile-only native app using Web Bluetooth.
I'm incredibly biased of course, but the Project Fugu 🐡 APIs are a lot of fun. There's so much neat stuff we can build in the browser now that HID, Serial, NFC, Bluetooth and all the rest are available. It has been a relaxing pandemic distraction to make time to put them through their paces, even if developing them is technically the day job too.
Needless to say, browsers that support Web Bluetooth are ultra-modern[1]. There's no point in supporting legacy browsers that don't have this capability, which means getting to use all the shiny new stuff; no polyfills or long toolchains. Fun!
In building UI with lit-html, trigger rendering without littering code with calls to render(...) can be a challenge. Lots of folks use data store libraries that provide a callback life-cycle, but they seem verbose and I only want things to be as complicated as they are complex.
Being reactive to data changes without a passel of callbacks only needs:
An object that can be an event source for listeners
Some way to be notified of data changes
That's it! Before modern runtimes, we needed verbose, explicit API surface. But it's 2021, and we can do more with less now thanks to Proxies and subclassable EventTarget.
Hewing to the "data down, events up" convention of Web Components, here's a little function my small app is using instead of a "state management" library:
/** * * proxyFor.js * * Utilities for wrapping an object or graph in a Proxy * that dispatches `ondatachange` events to an EventTarget. * */
// // Bookeeping // let objToProxyMap =newWeakMap();
// // proxyFor() supports two modes, "currentOnly" or "shallow" // proxying (which is the default) avoids creating new wrappers // for objects extracted from properties on the passed object. // This is faster but requires that developers understand the // limitations. // // The other mode, deep proxies, create new wrappers around // each object returned from any property. This mode recursively // wraps sub-objects in proxies that notify the passed // EventTarget. // // Note that in the case of an object that is itself an // EventTarget, one can pass it as both the first and second // argument to notify on it directly. This also provides // flexibility so that a notifying proxy can be wired up // to send change events to a *different* object. // exportletproxyFor=(thing, eventTarget=null, currentOnly=true, path=[])=>{
// Bail if not an object, or if already proxied if(shouldNotProxy(thing)){ return thing; } if(!eventTarget){ console.error( "Missing eventTarget. Could not proxy for", thing); return thing; }
let dataProperties = currentOnly ? newSet(Object.keys(thing)):null;
// Debounce to once per rAF let updateUI =(()=>{ let uiUpdateId; returnfunction(obj, tmpl, node, evt){ if(!node){return;} if(uiUpdateId){ cancelAnimationFrame(uiUpdateId); uiUpdateId =null; } uiUpdateId =requestAnimationFrame(()=>{ // Logs/renders once per frame console.log(evt.type, Date.now()); render(tmpl(obj), node); }); } })();
// Wire the template to be re-rendered from data let main = document.querySelector("main");
app.addEventListener("datachange",(evt)=>{ // Not debounced, called in quick succession by // setters in `doStuff` updateUI(app, mainTemplate, main, evt); });
TL;DR: A lot has changed since 2017 when we last estimated a global baseline resource budget of 130-170KiB per-page. Thanks to progress in networks and browsers (but not devices), a more generous global budget cap has emerged for sites constructed the "modern" way. We can now afford ~100KiB of HTML/CSS/fonts and ~300-350KiB of JS (gzipped). This rule-of-thumb limit should hold for at least a year or two. As always, the devil's in the footnotes, but the top-line is unchanged: when we construct the digital world to the limits of the best devices, we build a less usable one for 80+% of the world's users.
Bad individual experiences can colour expectations of the entire ecosystem. Your company's poor site performance can manifest as lower engagement, higher bounce rates, or a reduction in conversions. While this local story is important, it isn't the whole picture. If a large enough proportion of sites behave poorly, performance hysteresis may colour user views of all web experiences.
Unless a site is launched from the home screen as a PWA, sites are co-mingled. Pages are experienced as a series of taps, flowing effortlessly across sites; a river of links. A bad experience in the flow is a bad experience of the flow, with constituent parts blending together.[1]
If tapping links tends to feel bad...why keep tapping? It's not as though slow websites are the only way to access information. Plenty of native apps are happy to aggregate content and serve it up in a reliably fast package, given half a chance. The consistency of those walled gardens is a large part of what the mobile web is up against — and losing to.
Poor performance of sites that link to and from yours negatively impacts engagement on your site, even if it is consistently snappy. Live by the link, die by the link.
The harmful business impact of poor performance is constantly re-validated. Big decreases in performance predictably lead to (somewhat lower) decreases in user engagement and conversion. The scale of the effect can be deeply situational or hard to suss out without solid metrics, but it's there.
Variance contributes another layer of concern; high variability in responsiveness may create effects that perceptually dominate averages, or even medians. If 9 taps in 10 respond in 100ms, but every 10th takes a full second, what happens to user confidence and engagement? These deep-wetware effects and their cross-origin implications mean that your site's success is, partially, a function of the health of the commons.
The default global baseline is a ~$200 Android device on a 400Kbps link with a 400ms round-trip-time ("RTT"). This translates into a budget of ~130-170KB of critical-path resources, depending on composition — the more JS you include, the smaller the bundle must be.
A $200USD device at the time featured 4-8 (slow, in-order, low-cache) cores, ~2GiB of RAM, and pokey MLC NAND flash storage. The Moto G4, for example.
The 2017 baseline represented a conservative, but evidence-driven, interpretation of the best information I could get regarding network performance, along with trend lines regarding device shipment volumes and price points.[2] Getting accurate global information that isn't artificially reduced to averages remains an ongoing challenge. Performance work often focuses on high percentile users (the slowest), after all.
Since then, the metrics conversation has moved forward significantly, culminating in Core Web Vitals, reported via the Chrome User Experience Report to reflect the real-world experiences of users.
Median mobile JavaScript payloads have only grown since 2016, now hovering above 400KiB of script transferred, or nearly 2.5MiB of uncompressed JS.
A silver lining on this dark cloud is that mobile JavaScript payload growth paused in 2020. How soon will low-end and middle-tier phones be able to handle such chonky payloads? If the median site continued to send 3x the recommended amount of script, when would the web start to feel usable on most of the world's devices?
Good news has consistently come from the steady pace of browser progress. The single largest improvements visible in traces come from improvedparsing and off-thread compilation of JavaScript. This step-change, along with improvements to streaming compilation, has helped to ensure that users are less likely to notice the unreasonably-sized JS payloads that "modern" toolchains generate more often than not. Better use of more cores (moving compilation off-thread) has given sites that provide HTML and CSS content a fighting chance of remaining responsive, even when saddled with staggering JS burdens.
The residual main-thread compilation, allocation, and script-driven DOM/Layout tasks pose a challenge for delivering a good user experinece.[3] As we'll see below, CPUs are not improving fast enough to cope with frontend engineers' rosy resource assumptions. If there is unambiguously good news on the tooling front, multiple popular tools now include options to prevent sending first-party JS in the first place (Next.js, Gatsby), though the JS community remains in stubborn denial about the costs of client-side script. Hopefully, toolchain progress of this sort can provide a more accessible bridge as we transition costs to a reduced-script-emissions world.
The 2017 baseline post included a small model for thinking about how to think about how various factors of a page's construction influence the likelihood of hitting a 5-second first load goal.
The hard floor of that model (~1.6s) came from the contributions DNS, TCP/IP, and TLS connection setup over a then-reasonable 3G network baseline, leaving only 3400ms to work with, fighting Nagle and weak CPUs the whole way. Adding just one extra connection to a CDN for a critical path resource could sink the entire enterprise. Talk about a hard target.
Four years later, has anything changed? I'm happy to report that it has. Not as much as we'd like, of course, but the worldwide baseline has changed enormously. How? Why?
Key to this growth is the effect of Reliance Jio's entry into the carrier market. Before Jio's disruptive pricing and aggressive rollout, data services in India were among the most expensive in the world relative to income and heavily reliant on 3G outside wealthier "tier 1" cities. Worse, 3G service often performed like 2G in other markets, thanks to under-provisioning and throttling by incumbent carriers.
In 2016, Jio swept over the subcontinent like a monsoon dropping a torrent of 4G infrastructure and free data rather than rain.
5G looks set to continue a bumpy rollout for the next half-decade. Carriers make different frequency band choices in different geographies, and 5G performance is heavily sensitive to mast density, which will add confusion for years to come. Suffice to say, 5G isn't here yet, even if wealthy users in a few geographies come to think of it as "normal" far ahead of worldwide deployment[4].
Whatever progress runtimes and networks have made in the past half-decade, browsers are stubbornly situated in the devices carried by real-world users, and the single most important thing to understand about the landscape of devices your sites will run on is that they are not new phones.
This makes some intuitive sense: smartphones are not in their first year (and haven't been for more than a dozen years), and most users do not replace their devices every year. Most smartphone sales today are replacements (that is, to users who have previously owned a smartphone), and the longevity of devices continues to rise.
The true median device from 2016 sold at about ~$200 unlocked. This year's median device is even cheaper, but their performance is roughly equivalent. Expect continued performance stasis at the median for the next few years. This is part of the reason I suggested the Moto G4 last year and recommend it or the Moto G5 Plus this year.
Median devices continue to be different from averages, but we can squint a little as we're abstracting over multi-year cohorts. The worldwide ASP 18 months ago was ~$300USD, so the average performance in the deployed fleet can be represented by a $300 device from mid-2019. The Moto G7 very much looks the part.
Compared to devices wealthy developers carry, the performance is night and (blinding) day. However shocking a 6x difference in single-thread CPU performance might be, it's nothing compared to where we should be setting the global baseline: the P75+ device. Using our little mental model for device age and replacement, we can naively estimate what the 75th percentile (or higher) device could be in terms of device price + age, either by tracking devices at half the ASP at half the replacement age, or by looking at ASP-priced devices 3/4 of the way through the replacement cycle. Today, either method returns a similar answer.
This is inexact for dozens of reasons, not least of all markets with lower ASPs not yet achieving smartphone saturation. Using a global ASP as a benchmark can further mislead thanks to the distorting effect of ultra-high-end prices rising while shipment volumes stagnate. It's hard to know which way these effects cut when combined, so we're going to make a further guess: we'll take half the average price and note how wrong this likely is[6].
So what did $150USD fetch in 2019?
Say hello to the Moto E6! This 2GiB RAM, Android 9 stalwart features the all-too classic lines of a Quad-core A53 (1.4GHz, small mercies) CPU, tastefully presented in a charming 5.5" package.
You might recall the Moto G4 as a baseline recommendation from 2016 for forward-looking performance work. It's also the model we sent several dozen of to Pat Meenan — devices that power webpagetest.org/easy to this day. There was no way to know that the already-ageing in-order, near-zero-cache A53 + too-hot-to-frequency-scale 28nm process duo would continue to haunt us five year on. Today's P75 devices tell the story of the yawning Performance Inequality Gap: performance for those at the top end continues to accelerate away from the have-nots who are perpetually stuck with 2014's hardware.
Regardless, the overall story for hardware progress remains grim, particularly when we recall how long device replacement cycles are:
Tap for a larger version. Updated Geekbench 4 single-core scores for each mobile price-point.
Recall that single-core performance most directly translates into speed on the web. Android-ecosystem SoC performance is, in a word, disastrous.
How bad is it?
We can think about each category in terms of years behind contemporary iPhone releases:
2020's high-end Androids sport the single-core performance of an iPhone 8, a phone released in Q3'17
mid-priced Androids were slightly faster than 2014's iPhone 6
low-end Androids have finally caught up to the iPhone 5 from 2012
You're reading that right: single core Android performance at the low end is both shockingly bad and dispiritingly stagnant.
Tap for a larger version. Android ecosystem SoC's fare slightly better on multi-core performance, but the Performance Inequality Gap is growing there, too.
The multi-core performance shows the same basic story: iOS's high-end and the most expensive Androids are pulling away from the volume-shipment pack. The fastest Androids predictably remain 18-24 months behind, owing to cheapskate choices about cache sizing by Qualcomm, Samsung Semiconductor, MediaTek, and the rest. Don't pay a lot for an Android-shaped muffler.
Chip design choices and silicon economics are the defining feature of the still-growing Performance Inequality Gap.
Things continue to get better and better for the wealthy, leaving the rest behind. When we construct a digital world to the limits of the best devices, the worse an experience we build, on average, for those who cannot afford iPhones or $800 Samsung flagships.
It is perhaps predictable that, instead of presenting a bulwark against stratification, technology outcomes have tracked society's growing inequality. A yawning chasm of disparities is playing out in our phones at the same time it has come to shape our economic and political lives. It's hard to escape thinking they're connected.
Developers, particularly in Silicon Valley firms, are definitionally wealthy and enfranchised by world-historical standards. Like upper classes of yore, comfort ("DX") comes with courtiers happy to declare how important comfort must surely be. It's bunk, or at least most of it is.
As frontenders, our task is to make services that work well for all, not just the wealthy. If improvements in our tools or our comfort actually deliver improvements in that direction, so much the better. But we must never forget that measurable improvement for users is the yardstick.
Instead of measurement, we seem to suffer a proliferation of postulates about how each new increment of comfort must surely result in better user experiences. But the postulates are not tied to robust evidence. Instead, experience teaches that it's the process of taking care to attend to those least-well-off that changes outcomes. Trickle-down user experience from developer-experience is, in 2021, as fully falsified as the Laffer Curve. There's no durable substitute for compassion.
It's in that spirit that I find it important to build to a strawperson baseline device and network target. Such a benchmark serves as a stand-in until one gets situated data about a site or service's users, from whom we can derive informed conclusions about user needs and appropriate resource budgets.
A global baseline is doubly important for generic frameworks and libraries, as they will be included in projects targeted at global-baseline users, whether or not developers have that in mind. Tools that cannot fit comfortably within, say, 10% of our baseline budget should be labelled as desktop-only or replaced in our toolchains. Many popular tools over the past 5 years have been blithely punctured these limits relative to the 2017 baseline, and the results have been predictably poor.
Given all the data we've slogged through, we can assemble a sketch of the browsers, devices, and networks today's P50 and P75 users will access sites from. Keep in mind that this baseline might be too optimistic, depending on your target market. Trust but verify.
OpenSignal's global report on connection speeds (pdf) suggest that WebPageTest's default 4G configuration (9Mbps w/ 170ms RTT) is a reasonable stand-in for the P75 network link. WPT's LTE configuration (12Mbps w/ 70ms RTT) may actually be too conservative for a P50 estimate today; something closer to 25Mbps seems to approach the worldwide median today. Network progress has been astonishing, particularly channel capacity (bandwidth). Sadly, data on latency is harder to get, even from Google's perch, so progress there is somewhat more difficult to judge.
As for devices, the P75 situation remains dire and basically unchanged from the baseline I suggested more than five years ago. Slow A53 cores — a design first introduced nearly a decade ago — continue to dominate the landscape, differing only slightly in frequency scaling (through improved process nodes) and GPU pairing. The good news is that this will change rapidly in the next few years. The median new device is already benefiting; modern designs and fab processes are powering a ramp-up in performance for a $300USD device, particularly in multi-core workloads. But the hardware future is not evenly distributed, and web workloads aren't heavily parallel.
Astonishingly, I believe this means that for at least the next year we should consider the venerable Moto G4 to still be our baseline. If you can't buy one new, the 2020-vintage Moto E6 does a stirring rendition of the classic; it's basically the same phone, after all. The Moto E7 Plus is a preview of better days to come, and they can't arrive soon enough.
Putting it all together, where does that leave us? Back-of-the napkin, and accepting our previous targets of 5 seconds for first load and two seconds for second load, what can we afford?
Plugging the new P75 numbers in, the impact of network improvements are dramatic. Initial connection setup times are cut in half, dropping from 1600ms to 700ms, which frees up significant headroom to transmit more data in the same time window. The channel capacity increase to 9Mbps is enough to transmit more than 4MiB (megabytes) of content over a single connection in our 5 second window (1 Byte == 8 bits, so 9Mbps is just over ~1MBps). Of course, if most of this content is JavaScript, it must be compiled and run, shrinking our content window back down significantly. Using a single connection (thanks to the magic of HTTP/2), a site composed primarily of JavaScript loaded perfectly can, in 2021, weigh nearly 600KiB and still hit the first-load bar.
That's a very fine point to balance on, though. A single additional TCP/TLS connection setup in the critical path reduces the amount by 100KiB (as do subsequent critical-path network handshakes). Serialized requests for data, high TTFBs, and ever-present font serving issues make these upper limits far higher than what a site shooting for consistently good load times should be aiming for. In practice, you can't actually afford 600KiB of content if your application is build in the increasingly popular "single page app" style.
As networks have improved, client-side CPU time now dominates script download, adding further variability. Not all script parses and runs in the same amount of time for the same size. The contribution of connection setup also looms large.
If you want to play with the factors and see how they contribute, you can try this slightly updated version of 2017's calculator (launch a larger version in a new tab here):
Tap to try the interactive version.
This model of browser behaviour, network impacts, and device processing time is, of course, wrong for your site. It could be wrong in ways big and small. Perhaps your content is purely HTML, CSS, and images which would allow for a much higher budget. Or perhaps your JS isn't visually critical path, but still creates long delays because of delayed fetching. And nobody, realistically, can predict how much main-thread work a given amount of JS will cause. So it's an educated guess with some padding built in to account for worst-case content-construction (which is more common than you or I would like to believe).
Conservatively then, assuming at least 2 connections need to be set up (burning ~1400 of our 5000ms), and that script resources are in the critical path, the new global baseline leaves space for ~100KiB (gzipped) of HTML/CSS/fonts and 300-350KiB of JavaScript on the wire (compressed). Critical path images for LCP, if any, need to subtract their transfer size from one of these buckets, with trades against JS bundle size potentially increasing total download budget, but the details matter a great deal. For "modern" pages, half a megabyte is a decent hard budget.
Is that a lot? Well, 2-4x our previous budget, and as low-end CPUs finally begin to get faster over the next few years, we can expect it to loosen further for well-served content.
Sadly, most sites aren't perfectly served or structured, and most mobile web pages send more than this amount of JS. But there's light at the end of the tunnel: if we can just hold back the growth of JS payloads for another year or two — or reverse the trend slightly — we might achieve a usable web for the majority of the world's users by the middle of the decade.
Getting there involves no small amount of class traitorship; the frontend community will need to value the commons over personal comfort for a little while longer to ease our ecosystem back toward health. The past 6 years of consulting with partner teams has felt like a dark remake of Groundhog Day, with a constant parade of sites failing from the get-go thanks to Framework + Bundler + SPA architectures that are mismatched to the tasks and markets at hand. Time (and chips) can heal these wounds if we only let it. We only need to hold the line on script bloat for a few years for devices and networks to overtake the extreme, unconscionable excesses of the 2010's.
I mentioned near the start of this too-long piece that metrics better than Time-to-Interactive have been developed since 2017. The professionalism with which the new Core Web Vitals (CWV) have been developed, tested, and iterated on inspires real confidence. CWV's constituent metrics and limits work to capture important aspects of page experiences as users perceive them on real devices and networks.
As RUM (field) metrics rather than bench tests, they represent ground truth for a site, and are reported in aggregate by the Chrome User Experience Report (CrUX) pipeline. Getting user-population level data about the performance of sites without custom tooling is a 2016-era dream come true. Best of all, the metrics and limits can continue to evolve in the future as we gain confidence that we can accurately measure other meaningful components of a user's journey.
These field metrics, however valuable, don't yet give us enough information to guide global first-best-guess making when developing new sites. Continuing to set performance budgets is necessary for teams in development, not least of all because conformance can be automated in CI/CD flows. CrUX data collection and first-party RUM analytics of these metrics require live traffic, meaning results can be predicted but only verified once deployed.
Both budgets and RUM analytics will continue to be powerful tools in piercing the high-performance privilege bubble that surrounds today's frontend discourse. Grounding our choices against a baseline and measuring how it's going for real users are the only proven approaches I know that reliably help teams take the first step toward frontend's first and highest calling: a web that's truly for everyone.
One subtle effect of browser attempts to avoid showing users blank white screens is that optimisations meant to hide latency may lead users to misattribute which sites are slow.
Consider a user tapping a link away from your site to a slow, cross-origin site. If the server takes a long time to respond, the referring page will be displayed for as long as it takes for the destination server to return with minimally parseable HTML content. Similarly, if a user navigates to your site, but the previous site's onunload handlers take a long time to execute (an all too common issue), no matter how fast your site responds, the poor performance of the referring page will make your site appear to load slowly.
Even without hysteresis effects, the performance of referrers and destinations can hurt your brand. ↩︎
A constant challenge is the tech world's blindness to devices carried by most users. Tech press coverage is over-fitted to the high-end phones sent for review (to say nothing of the Apple marketing juggernaut), which doesn't track with shipment volumes.
Smartphone markers (who are often advertisers) want attention for the high-end segment because it's where the profits are, and the tech press rarely pushes back. Column inches covering the high-end remains heavily out of proportion to sales by price-point. In a world with better balance, most articles would be dedicated to announcements of mid-range devices from LG and Samsung.
There's infinitely more to say about these biases and how they manifest. It's constantly grating that Qualcomm, Samsung Semiconductor, MediaTek, and other Android SoC vendors get a pass (even at the high end) on their terrible price/performance.[7:1]
Were we to suffer an outbreak of tenacious journalism in our tech press, coverage of the devastatingly bad CPUs or the galling OS update rates amongst Android OEMs might be among the early signs. ↩︎
In the worst cases, off-thread compilation has no positive impact on client-side rendered SPAs. If an experience is blocked on a script to begin requesting data or generating markup from it, additional cores can't help. Some pages can be repaired with a sprinkling of <link rel=preload> directives.
More frequently, "app"-centric architectures promulgated by popular framework starter kits leave teams with immense piles of JS to whittle down. This is expensive, gruelling work in the Webpack mines; not all the "developer experience" folks signed up for.
The React community, in particular, has been poorly served by a lack of guidance, tooling, and support from Facebook. Given the scale of the disaster that has unfolded there, it's shocking that in 2021 (and after recent re-development of the documentation site) there remains no guidance about bundle sizes or performance budgets in React's performance documentation.
Facebook, meanwhile, continues to staff a performance team, sets metrics and benchmarks, and generally exhibits the sort of discipline about bundle sizes and site performance that allows teams to succeed with any stack. Enforced latency and performance budgets tend to have that effect.
As ever, take the money and introductions VCs offer, but interpret their musings about what "everyone knows" and "how things are" like you might Henry Kissinger's geopolitical advise: toxic, troubling — and above all — unfailingly self-serving. ↩︎
Device Average Selling Price (APSs) are both directionally helpful and deeply misleading.
We see this misfeature in the wild through press treatment of large headline increases in ASP and iOS share when Android device sales fall even slightly. Instead of digging into the shape of the distribution and relative time series inputs, the press regularly writes this up as the sudden success of some new Apple feature. That's always possible, but a slightly stylised version of these facts more easily pass Occam's test: wealthy users are less sensitive in their technology refresh timing decisions.
This makes intutive sense: in a recession, replacing consumer electronics is a relative luxury, leading the most price-sensitive buyers — overwhelmingly Android users — to push off replacing otherwise functional devices.
Voila! A slight dip in the overwhelming volume of Android purchases appears to bolster both ASP and iOS share since iOS devices are almost universally more expensive. ASP might be a fine diagnostic metric in limited cases, but it always pays to develop a sense for the texture of your data! ↩︎
While the model of median device representation presented here is hand-wavey, I've cross-referenced the results with Google's internal analysis and the model holds up OK. Native app and SDK developers with global reach can use RAM, CPU speeds, and screen DPI to bucket devices into high/medium/low tiers, and RAM has historically been a key signifier of device performance overall. Indeed, some large sites I've worked with have built device model databases to bucket traffic based on system RAM to decide which versions/features to serve.
Our ASP + age model could be confounded by many factors: ASPs deviating more than expected year-on-year, non-linearities in price/performance (e.g., due to a global chip shortage), or sudden changes in device replacement rates. Assuming these factors hold roughly steady, so will the model's utility. Always gut-check the results of a model! Caveat emptor.
In our relatively steady-state, 90+% of active smartphones were sold in the last 4 years. This makes some intuitive sense: batteries degrade, screens crack, and disks fill up. Replacement happens.
If our napkin-back modelling has any bias, it's slightly too generous about device prices (too high) and age (too young) given the outsized historical influence of wealthier markets achieving smartphone saturation ahead of emerging one. Given that a new baseline is being set in an environment where we know the bottom-end will start to rise in terms of performance over the next few years (instead of staying stagnant), this bias, while hard to quantify, doesn't seem deeply problematic. ↩︎
While process nodes are starting to move forward in a hopeful way, the overall design tradeoffs of major Android-ecosystem SoC vendors are not. Recently updated designs for budget parts finally get out-of-order dispatch and better parallelism.
Sadly, the feeble caches they're mated to ensure whatever gains are unlocked by better microarchitectures, plus higher frequencies, go largely to waste. It's not much good to clock a core faster or give it more ability to dispatch ops in if it's only spinning up to a high power state to stall on main memory, 100s of cycles away.
Apple consistently wins benchmarks one both perf and power by throwing die space at the ARBs and caches to feed their hungry, hungry cores (among other improvements to core design). Any vendor with an ARM Architectural License can play this game, but either Qualcomm et al. are too stingy to fork out $200K (or $800K?) for a full license to customise the core of the thing they market, or they don't want to spend more to fund the extra design time or increase mm^2 of die space (both of which might cut into margins).
Whatever the reason, these designs are (literally) hot garbage, dollar-for-dollar, compared to the price/performance of equivalent-generation Apple designs. It's a trickle-down digital divide, on a loop. The folks who suffer are end-users who can't access services and information they need quickly because developers in the privilege bubble remain insulated by wealth from all of these effects, oblivious to the ways that a market failure several layers down the stack keeps them ignorant of their malign impacts on users. ↩︎↩︎
