Acko.net

HTML is Dead, Long Live HTML

2025-08-06T00:00:00+02:00

Rethinking DOM from first principles

Browsers are in a very weird place. While WebAssembly has succeeded, even on the server, the client still feels largely the same as it did 10 years ago.

Enthusiasts will tell you that accessing native web APIs via WASM is a solved problem, with some minimal JS glue.

But the question not asked is why you would want to access the DOM. It's just the only option. So I'd like to explain why it really is time to send the DOM and its assorted APIs off to a farm somewhere, with some ideas on how.

I won't pretend to know everything about browsers. Nobody knows everything anymore, and that's the problem.

The 'Document' Model

Few know how bad the DOM really is. In Chrome, document.body now has 350+ keys, grouped roughly like this:

This doesn't include the CSS properties in document.body.style of which there are... 660.

The boundary between properties and methods is very vague. Many are just facades with an invisible setter behind them. Some getters may trigger a just-in-time re-layout. There's ancient legacy stuff, like all the onevent properties nobody uses anymore.

The DOM is not lean and continues to get fatter. Whether you notice this largely depends on whether you are making web pages or web applications.

Most devs now avoid working with the DOM directly, though occasionally some purist will praise pure DOM as being superior to the various JS component/templating frameworks. What little declarative facilities the DOM has, like innerHTML, do not resemble modern UI patterns at all. The DOM has too many ways to do the same thing, none of them nice.

connectedCallback() {
  const
    shadow = this.attachShadow({ mode: 'closed' }),
    template = document.getElementById('hello-world')
      .content.cloneNode(true),
    hwMsg = `Hello ${ this.name }`;

  Array.from(template.querySelectorAll('.hw-text'))
    .forEach(n => n.textContent = hwMsg);

  shadow.append(template);
}

Web Components deserve a mention, being the web-native equivalent of JS component libraries. But they came too late and are unpopular. The API seems clunky, with its Shadow DOM introducing new nesting and scoping layers. Proponents kinda read like apologetics.

The achilles heel is the DOM's SGML/XML heritage, making everything stringly typed. React-likes do not have this problem, their syntax only looks like XML. Devs have learned not to keep state in the document, because it's inadequate for it.

For HTML itself, there isn't much to critique because nothing has changed in 10-15 years. Only ARIA (accessibility) is notable, and only because this was what Semantic HTML was supposed to do and didn't.

Semantic HTML never quite reached its goal. Despite dating from around 2011, there is e.g. no or tag, when those were well-established idioms. Instead, an article inside an article is probably a comment. The guidelines are... weird.

There's this feeling that HTML always had paper-envy, and couldn't quite embrace or fully define its hypertext nature, and did not trust its users to follow clear rules.

Stewardship of HTML has since firmly passed to WHATWG, really the browser vendors, who have not been able to define anything more concrete as a vision, and have instead just added epicycles at the margins.

Along the way even CSS has grown expressions, because every templating language wants to become a programming language.

Editability of HTML remains a sad footnote. While technically supported via contentEditable, actually wrangling this feature into something usable for applications is a dark art. I'm sure the Google Docs and Notion people have horror stories.

Nobody really believes in the old gods of progressive enhancement and separating markup from style anymore, not if they make apps.

Most of the applications you see nowadays will kitbash HTML/CSS/SVG into a pretty enough shape. But this comes with immense overhead, and is looking more and more like the opposite of a decent UI toolkit.

The Slack input box

Off-screen clipboard hacks

Lists and tables must be virtualized by hand, taking over for layout, resizing, dragging, and so on. Making a chat window's scrollbar stick to the bottom is somebody's TODO, every single time. And the more you virtualize, the more you have to reinvent find-in-page, right-click menus, etc.

The web blurred the distinction between UI and fluid content, which was novel at the time. But it makes less and less sense, because the UI part is a decade obsolete, and the content has largely homogenized.

CSS is inside-out

CSS doesn't have a stellar reputation either, but few can put their finger on exactly why.

Where most people go wrong is to start with the wrong mental model, approaching it like a constraint solver. This is easy to show with e.g.:


  ...
  ...


  ...
  ...

The first might seem reasonable: divide the parent into two halves vertically. But what about the second?

Viewed as a set of constraints, it's contradictory, because the parent div is twice as tall as... itself. What will happen instead in both cases is the height is ignored. The parent height is unknown and CSS doesn't backtrack or iterate here. It just shrink-wraps the contents.

If you set e.g. height: 300px on the parent, then it works, but the latter case will still just spill out.

Outside-in and inside-out layout modes

Instead, your mental model of CSS should be applying two passes of constraints, first going outside-in, and then inside-out.

When you make an application frame, this is outside-in: the available space is divided, and the content inside does not affect sizing of panels.

When paragraphs stack on a page, this is inside-out: the text stretches out its containing parent. This is what HTML wants to do naturally.

By being structured this way, CSS layouts are computationally pretty simple. You can propagate the parent constraints down to the children, and then gather up the children's sizes in the other direction. This is attractive and allows webpages to scale well in terms of elements and text content.

CSS is always inside-out by default, reflecting its document-oriented nature. The outside-in is not obvious, because it's up to you to pass all the constraints down, starting with body { height: 100%; }. This is why they always say vertical alignment in CSS is hard.

Use flex grow and shrink for spill-free auto-layouts with completely reasonable gaps

The scenario above is better handled with a CSS3 flex box (display: flex), which provides explicit control over how space is divided.

Unfortunately flexing muddles the simple CSS model. To auto-flex, the layout algorithm must measure the "natural size" of every child. This means laying it out twice: first speculatively, as if floating in aether, and then again after growing or shrinking to fit:

This sounds reasonable but can come with hidden surprises, because it's recursive. Doing speculative layout of a parent often requires full layout of unsized children. e.g. to know how text will wrap. If you nest it right, it could in theory cause an exponential blow up, though I've never heard of it being an issue.

Instead you will only discover this when someone drops some large content in somewhere, and suddenly everything gets stretched out of whack. It's the opposite of the problem on the mug.

To avoid the recursive dependency, you need to isolate the children's contents from the outside, thus making speculative layout trivial. This can be done with contain: size, or by manually setting the flex-basis size.

CSS has gained a few constructs like contain or will-change, which work directly with the layout system, and drop the pretense of one big happy layout. It reveals some of the layer-oriented nature underneath, and is a substitute for e.g. using position: absolute wrappers to do the same.

What these do is strip off some of the semantics, and break the flow of DOM-wide constraints. These are overly broad by default and too document-oriented for the simpler cases.

This is really a metaphor for all DOM APIs.

The Good Parts?

That said, flex box is pretty decent if you understand these caveats. Building layouts out of nested rows and columns with gaps is intuitive, and adapts well to varying sizes. There is a "CSS: The Good Parts" here, which you can make ergonomic with sufficient love. CSS grids also work similarly, they're just very painfully... CSSy in their syntax.

But if you designed CSS layout from scratch, you wouldn't do it this way. You wouldn't have a subtractive API, with additional extra containment barrier hints. You would instead break the behavior down into its component facets, and use them à la carte. Outside-in and inside-out would both be legible as different kinds of containers and placement models.

The inline-block and inline-flex display models illustrate this: it's a block or flex on the inside, but an inline element on the outside. These are two (mostly) orthogonal aspects of a box in a box model.

Text and font styles are in fact the odd ones out, in hypertext. Properties like font size inherit from parent to child, so that formatting tags like can work. But most of those 660 CSS properties do not do that. Setting a border on an element does not apply the same border to all its children recursively, that would be silly.

It shows that CSS is at least two different things mashed together: a system for styling rich text based on inheritance... and a layout system for block and inline elements, nested recursively but without inheritance, only containment. They use the same syntax and APIs, but don't really cascade the same way. Combining this under one style-umbrella was a mistake.

Worth pointing out: early ideas of relative em scaling have largely become irrelevant. We now think of logical vs device pixels instead, which is a far more sane solution, and closer to what users actually expect.

SVG is natively integrated as well. Having SVGs in the DOM instead of just as tags is useful to dynamically generate shapes and adjust icon styles.

But while SVG is powerful, it's neither a subset nor superset of CSS. Even when it overlaps, there are subtle differences, like the affine transform. It has its own warts, like serializing all coordinates to strings.

CSS has also gained the ability to round corners, draw gradients, and apply arbitrary clipping masks: it clearly has SVG-envy, but falls very short. SVG can e.g. do polygonal hit-testing for mouse events, which CSS cannot, and SVG has its own set of graphical layer effects.

Whether you use HTML/CSS or SVG to render any particular element is based on specific annoying trade-offs, even if they're all scalable vectors on the back-end.

In either case, there are also some roadblocks. I'll just mention three:

text-ellipsis can only be used to truncate unwrapped text, not entire paragraphs. Detecting truncated text is even harder, as is just measuring text: the APIs are inadequate. Everyone just counts letters instead.

position: sticky lets elements stay in place while scrolling with zero jank. While tailor-made for this purpose, it's subtly broken. Having elements remain unconditionally sticky requires an absurd nesting hack, when it should be trivial.

The z-index property determines layering by absolute index. This inevitably leads to a z-index-war.css where everyone is putting in a new number +1 or -1 to make things layer correctly. There is no concept of relative Z positioning.

For each of these features, we got stuck with v1 of whatever they could get working, instead of providing the right primitives.

Getting this right isn't easy, it's the hard part of API design. You can only iterate on it, by building real stuff with it before finalizing it, and looking for the holes.

Oil on Canvas

So, DOM is bad, CSS is single-digit X% good, and SVG is ugly but necessary... and nobody is in a position to fix it?

Well no. The diagnosis is that the middle layers don't suit anyone particularly well anymore. Just an HTML6 that finally removes things could be a good start.

But most of what needs to happen is to liberate the functionality that is there already. This can be done in good or bad ways. Ideally you design your system so the "escape hatch" for custom use is the same API you built the user-space stuff with. That's what dogfooding is, and also how you get good kernels.

A recent proposal here is HTML in Canvas, to draw HTML content into a , with full control over the visual output. It's not very good.

While it might seem useful, the only reason the API has the shape that it does is because it's shoehorned into the DOM: elements must be descendants of to fully participate in layout and styling, and to make accessibility work. There are also "technical concerns" with using it off-screen.

One example is this spinny cube:

To make it interactive, you attach hit-testing rectangles and respond to paint events. This is a new kind of hit-testing API. But it only works in 2D... so it seems 3D-use is only cosmetic? I have many questions.

Again, if you designed it from scratch, you wouldn't do it this way! In particular, it's absurd that you'd have to take over all interaction responsibilities for an element and its descendants just to be able to customize how it looks i.e. renders. Especially in a browser that has projective CSS 3D transforms.

The use cases not covered by that, e.g. curved re-projection, will also need more complicated hit-testing than rectangles. Did they think this through? What happens when you put a dropdown in there?

To me it seems like they couldn't really figure out how to unify CSS and SVG filters, or how to add shaders to CSS. Passing it thru canvas is the only viable option left. "At least it's programmable." Is it really? Screenshotting DOM content is 1 good use-case, but not what this is sold as at all.

The whole reason to do "complex UIs on canvas" is to do all the things the DOM doesn't do, like virtualizing content, just-in-time layout and styling, visual effects, custom gestures and hit-testing, and so on. It's all nuts and bolts stuff. Having to pre-stage all the DOM content you want to draw sounds... very counterproductive.

From a reactivity point-of-view it's also a bad idea to route this stuff back through the same document tree, because it sets up potential cycles with observers. A canvas that's rendering DOM content isn't really a document element anymore, it's doing something else entirely.

Canvas-based spreadsheet that skips the DOM entirely

The actual achilles heel of canvas is that you don't have any real access to system fonts, text layout APIs, or UI utilities. It's quite absurd how basic it is. You have to implement everything from scratch, including Unicode word splitting, just to get wrapped text.

The proposal is "just use the DOM as a black box for content." But we already know that you can't do anything except more CSS/SVG kitbashing this way. text-ellipsis and friends will still be broken, and you will still need to implement UIs circa 1990 from scratch to fix it.

It's all-or-nothing when you actually want something right in the middle. That's why the lower level needs to be opened up.

Where To Go From Here

The goals of "HTML in Canvas" do strike a chord, with chunks of HTML used as free-floating fragments, a notion that has always existed under the hood. It's a composite value type you can handle. But it should not drag 20 years of useless baggage along, while not enabling anything truly novel.

The kitbashing of the web has also resulted in enormous stagnation, and a loss of general UI finesse. When UI behaviors have to be mined out of divs, it limits the kinds of solutions you can even consider. Fixing this within DOM/HTML seems unwise, because there's just too much mess inside. Instead, new surfaces should be opened up outside of it.

WebGPU-based box model

My schtick here has become to point awkwardly at Use.GPU's HTML-like renderer, which does a full X/Y flex model in a fraction of the complexity or code. I don't mean my stuff is super great, no, it's pretty bare-bones and kinda niche... and yet definitely nicer. Vertical centering is easy. Positioning makes sense.

There is no semantic HTML or CSS cascade, just first-class layout. You don't need 61 different accessors for border* either. You can just attach shaders to divs. Like, that's what people wanted right? Here's a blueprint, it's mostly just SDFs.

Font and markup concerns only appear at the leaves of the tree, where the text sits. It's striking how you can do like 90% of what the DOM does here, without the tangle of HTML/CSS/SVG, if you just reinvent that wheel. Done by 1 guy. And yes, I know about the second 90% too.

The classic data model here is of a view tree and a render tree. What should the view tree actually look like? And what can it be lowered into? What is it being lowered into right now, by a giant pile of legacy crud?

Alt-browser projects like Servo or Ladybird are in a position to make good proposals here. They have the freshest implementations, and are targeting the most essential features first. The big browser vendors could also do it, but well, taste matters. Good big systems grow from good small ones, not bad big ones. Maybe if Mozilla hadn't imploded... but alas.

Platform-native UI toolkits are still playing catch up with declarative and reactive UI, so that's that. Native Electron-alternatives like Tauri could be helpful, but they don't treat origin isolation as a design constraint, which makes security teams antsy.

There's a feasible carrot to dangle for them though, namely in the form of better process isolation. Because of CPU exploits like Spectre, multi-threading via SharedArrayBuffer and Web Workers is kinda dead on arrival anyway, and that affects all WASM. The details are boring but right now it's an impossible sell when websites have to have things like OAuth and Zendesk integrated into them.

Reinventing the DOM to ditch all legacy baggage could coincide with redesigning it for a more multi-threaded, multi-origin, and async web. The browser engines are already multi-process... what did they learn? A lot has happened since Netscape, with advances in structured concurrency, ownership semantics, FP effects... all could come in handy here.

* * *

Step 1 should just be a data model that doesn't have 350+ properties per node tho.

Don't be under the mistaken impression that this isn't entirely fixable.

I is for Intent

2024-02-05T00:00:00+01:00

Why your app turned into spaghetti

"I do not like your software sir,
your architecture's poor.

Your users can't do anything,
unless you code some more.

This isn't how it used to be,
we had this figured out.

But then you had to mess it up
by moving into clouds."

There's a certain kind of programmer. Let's call him Stanley.

Stanley has been around for a while, and has his fair share of war stories. The common thread is that poorly conceived and specced solutions lead to disaster, or at least, ongoing misery. As a result, he has adopted a firm belief: it should be impossible for his program to reach an invalid state.

Stanley loves strong and static typing. He's a big fan of pattern matching, and enums, and discriminated unions, which allow correctness to be verified at compile time. He also has strong opinions on errors, which must be caught, logged and prevented. He uses only ACID-compliant databases, wants foreign keys and triggers to be enforced, and wraps everything in atomic transactions.

He hates any source of uncertainty or ambiguity, like untyped JSON or plain-text markup. His APIs will accept data only in normalized and validated form. When you use a Stanley lib, and it doesn't work, the answer will probably be: "you're using it wrong."

Stanley is most likely a back-end or systems-level developer. Because nirvana in front-end development is reached when you understand that this view of software is not just wrong, but fundamentally incompatible with the real world.

I will prove it.

State Your Intent

Take a text editor. What happens if you press the up and down arrows?

The keyboard cursor (aka caret) moves up and down. Duh. Except it also moves left and right.

The editor state at the start has the caret on line 1 column 6. Pressing down will move it to line 2 column 6. But line 2 is too short, so the caret is forcibly moved left to column 1. Then, pressing down again will move it back to column 6.

It should be obvious that any editor that didn't remember which column you were actually on would be a nightmare to use. You know it in your bones. Yet this only works because the editor allows the caret to be placed on a position that "does not exist." What is the caret state in the middle? It is both column 1 and column 6.

To accommodate this, you need more than just a View that is a pure function of a State, as is now commonly taught. Rather, you need an Intent, which is the source of truth that you mutate... and which is then parsed and validated into a State. Only then can it be used by the View to render the caret in the right place.

To edit the intent, aka what a classic Controller does, is a bit tricky. When you press left/right, it should determine the new Intent.column based on the validated State.column +/- 1. But when you press up/down, it should keep the Intent.column you had before and instead change only Intent.line. New intent is a mixed function of both previous intent and previous state.

The general pattern is that you reuse Intent if it doesn't change, but that new computed Intent should be derived from State. Note that you should still enforce normal validation of Intent.column when editing too: you don't allow a user to go past the end of a line. Any new intent should be as valid as possible, but old intent should be preserved as is, even if non-sensical or inapplicable.

Functionally, for most of the code, it really does look and feel as if the state is just State, which is valid. It's just that when you make 1 state change, the app may decide to jump into a different State than one would think. When this happens, it means some old intent first became invalid, but then became valid again due to a subsequent intent/state change.

This is how applications actually work IRL. FYI.

Knives and Forks

I chose a text editor as an example because Stanley can't dismiss this as just frivolous UI polish for limp wristed homosexuals. It's essential that editors work like this.

The pattern is far more common than most devs realize:

A tree view remembers the expand/collapse state for rows that are hidden.

Inspector tabs remember the tab you were on, even if currently disabled or inapplicable.

Toggling a widget between type A/B/C should remember all the A, B and C options, even if mutually exclusive.

All of these involve storing and preserving something unknown, invalid or unused, and bringing it back into play later.

More so, if software matches your expected intent, it's a complete non-event. What looks like a "surprise hidden state transition" to a programmer is actually the exact opposite. It would be an unpleasant surprise if that extra state transition didn't occur. It would only annoy users: they already told the software what they wanted, but it keeps forgetting.

The ur-example is how nested popup menus should work: good implementations track the motion of the cursor so you can move diagonally from parent to child, without falsely losing focus:

This is an instance of the goalkeeper's curse: people rarely compliment or notice the goalkeeper if they do their job, only if they mess up. Successful applications of this principle are doomed to remain unnoticed and unstudied.

Validation is not something you do once, discarding the messy input and only preserving the normalized output. It's something you do continuously and non-destructively, preserving the mess as much as possible. It's UI etiquette: the unspoken rules that everyone expects but which are mostly undocumented folklore.

This poses a problem for most SaaS in the wild, both architectural and existential. Most APIs will only accept mutations that are valid. The goal is for the database to be a sequence of fully valid states:

The smallest possible operation in the system is a fully consistent transaction. This flattens any prior intent.

In practice, many software deviates from this ad-hoc. For example, spreadsheets let you create cyclic references, which is by definition invalid. The reason it must let you do this is because fixing one side of a cyclic reference also fixes the other side. A user wants and needs to do these operations in any order. So you must allow a state transition through an invalid state:

This requires an effective Intent/State split, whether formal or informal.

Because cyclic references can go several levels deep, identifying one cyclic reference may require you to spider out the entire dependency graph. This is functionally equivalent to identifying all cyclic references—dixit Dijkstra. Plus, you need to produce sensible, specific error messages. Many "clever" algorithmic tricks fail this test.

Now imagine a spreadsheet API that doesn't allow for any cyclic references ever. This still requires you to validate the entire resulting model, just to determine if 1 change is allowed. It still requires a general validate(Intent). In short, it means your POST and PUT request handlers need to potentially call all your business logic.

That seems overkill, so the usual solution is bespoke validators for every single op. If the business logic changes, there is a risk your API will now accept invalid intent. And the app was not built for that.

If you flip it around and assume intent will go out-of-bounds as a normal matter, then you never have this risk. You can write the validation in one place, and you reuse it for every change as a normal matter of data flow.

Note that this is not cowboy coding. Records and state should not get irreversibly corrupted, because you only ever use valid inputs in computations. If the system is multiplayer, distributed changes should still be well-ordered and/or convergent. But the data structures you're encoding should be, essentially, entirely liberal to your user's needs.

Consider git. Here, a "unit of intent" is just a diff applied to a known revision ID. When something's wrong with a merge, it doesn't crash, or panic, or refuse to work. It just enters a conflict state. This state is computed by merging two incompatible intents.

It's a dirty state that can't be turned into a clean commit without human intervention. This means git must continue to work, because you need to use git to clean it up. So git is fully aware when a conflict is being resolved.

As a general rule, the cases where you actually need to forbid a mutation which satisfies all the type and access constraints are small. A good example is trying to move a folder inside itself: the file system has to remain a sensibly connected tree. Enforcing the uniqueness of names is similar, but also comes with a caution: falsehoods programmers believe about names. Adding (Copy) to a duplicate name is usually better than refusing to accept it, and most names in real life aren't unique at all. Having user-facing names actually requires creating tools and affordances for search, renaming references, resolving duplicates, and so on.

Even among front-end developers, few people actually grok this mental model of a user. It's why most React(-like) apps in the wild are spaghetti, and why most blog posts about React gripes continue to miss the bigger picture. Doing React (and UI) well requires you to unlearn old habits and actually design your types and data flow so it uses potentially invalid input as its single source of truth. That way, a one-way data flow can enforce the necessary constraints on the fly.

The way Stanley likes to encode and mutate his data is how programmers think about their own program: it should be bug-free and not crash. The mistake is to think that this should also apply to any sort of creative process that program is meant to enable. It would be like making an IDE that only allows you to save a file if the code compiles and passes all the tests.

Trigger vs Memo

Coding around intent is a very hard thing to teach, because it can seem overwhelming. But what's overwhelming is not doing this. It leads to codebases where every new feature makes ongoing development harder, because no part of the codebase is ever finished. You will sprinkle copies of your business logic all over the place, in the form of request validation, optimistic local updaters, and guess-based cache invalidation.

If this is your baseline experience, your estimate of what is needed to pull this off will simply be wrong.

In the traditional MVC model, intent is only handled at the individual input widget or form level. e.g. While typing a number, the intermediate representation is a string. This may be empty, incomplete or not a number, but you temporarily allow that.

I've never seen people formally separate Intent from State in an entire front-end. Often their state is just an adhoc mix of both, where validation constraints are relaxed in the places where it was most needed. They might just duplicate certain fields to keep a validated and unvalidated variant side by side.

There is one common exception. In a React-like, when you do a useMemo with a derived computation of some state, this is actually a perfect fit. The eponymous useState actually describes Intent, not State, because the derived state is ephemeral. This is why so many devs get lost here.

const state = useMemo( () => validate(intent), [intent] );

Their usual instinct is that every action that has knock-on effects should be immediately and fully realized, as part of one transaction. Only, they discover some of those knock-on effects need to be re-evaluated if certain circumstances change. Often to do so, they need to undo and remember what it was before. This is then triggered anew via a bespoke effect, which requires a custom trigger and mutation. If they'd instead deferred the computation, it could have auto-updated itself, and they would've still had the original data to work with.

e.g. In a WYSIWYG scenario, you often want to preview an operation as part of mouse hovering or dragging. It should look like the final result. You don't need to implement custom previewing and rewinding code for this. You just need the ability to layer on some additional ephemeral intent on top of the intent that is currently committed. Rewinding just means resetting that extra intent back to empty.

You can make this easy to use by treating previews as a special kind of transaction: now you can make preview states with the same code you use to apply the final change. You can also auto-tag the created objects as being preview-only, which is very useful. That is: you can auto-translate editing intent into preview intent, by messing with the contents of a transaction. Sounds bad, is actually good.

The same applies to any other temporary state, for example, highlighting of elements. Instead of manually changing colors, and creating/deleting labels to pull this off, derive the resolved style just-in-time. This is vastly simpler than doing it all on 1 classic retained model. There, you run the risk of highlights incorrectly becoming sticky, or shapes reverting to the wrong style when un-highlighted. You can architect it so this is simply impossible.

The trigger vs memo problem also happens on the back-end, when you have derived collections. Each object of type A must have an associated type B, created on-demand for each A. What happens if you delete an A? Do you delete the B? Do you turn the B into a tombstone? What if the relationship is 1-to-N, do you need to garbage collect?

If you create invisible objects behind the scenes as a user edits, and you never tell them, expect to see a giant mess as a result. It's crazy how often I've heard engineers suggest a user should only be allowed to create something, but then never delete it, as a "solution" to this problem. Everyday undo/redo precludes it. Don't be ridiculous.

The problem is having an additional layer of bookkeeping you didn't need. The source of truth was collection A, but you created a permanent derived collection B. If you instead make B ephemeral, derived via a stateless computation, then the problem goes away. You can still associate data with B records, but you don't treat B as the authoritative source for itself. This is basically what a WeakMap is.

In database land this can be realized with a materialized view, which can be incremental and subscribed to. Taken to its extreme, this turns into event-based sourcing, which might seem like a panacea for this mindset. But in most cases, the latter is still a system by and for Stanley. The event-based nature of those systems exists to support housekeeping tasks like migration, backup and recovery. Users are not supposed to be aware that this is happening. They do not have any view into the event log, and cannot branch and merge it. The exceptions are extremely rare.

It's not a system for working with user intent, only for flattening it, because it's append-only. It has a lot of the necessary basic building blocks, but substitutes programmer intent for user intent.

What's most nefarious is that the resulting tech stacks are often quite big and intricate, involving job queues, multi-layered caches, distribution networks, and more. It's a bunch of stuff that Stanley can take joy and pride in, far away from users, with "hard" engineering challenges. Unlike all this *ugh* JavaScript, which is always broken and unreliable and uninteresting.

Except it's only needed because Stanley only solved half the problem, badly.

Patch or Bust

When factored in from the start, it's actually quite practical to split Intent from State, and it has lots of benefits. Especially if State is just a more constrained version of the same data structures as Intent. This doesn't need to be fully global either, but it needs to encompass a meaningful document or workspace to be useful.

It does create an additional problem: you now have two kinds of data in circulation. If reading or writing requires you to be aware of both Intent and State, you've made your code more complicated and harder to reason about.

More so, making a new Intent requires a copy of the old Intent, which you mutate or clone. But you want to avoid passing Intent around in general, because it's fishy data. It may have the right types, but the constraints and referential integrity aren't guaranteed. It's a magnet for the kind of bugs a type-checker won't catch.

I've published my common solution before: turn changes into first-class values, and make a generic update of type Update be the basic unit of change. As a first approximation, consider a shallow merge {...value, ...update}. This allows you to make an updateIntent(update) function where update only specifies the fields that are changing.

In other words, Update looks just like Update and can be derived 100% from State, without Intent. Only one place needs to have access to the old Intent, all other code can just call that. You can make an app intent-aware without complicating all the code.

If your state is cleaved along orthogonal lines, then this is all you need. i.e. If column and line are two separate fields, then you can selectively change only one of them. If they are stored as an XY tuple or vector, now you need to be able to describe a change that only affects either the X or Y component.

const value = { hello: 'text', foo: { bar: 2, baz: 4 }, }; const update = { hello: 'world', foo: { baz: 50 }, }; expect( patch(value, update) ).toEqual({ hello: 'world', foo: { bar: 2, baz: 50 }, });

So in practice I have a function patch(value, update) which implements a comprehensive superset of a deep recursive merge, with full immutability. It doesn't try to do anything fancy with arrays or strings, they're just treated as atomic values. But it allows for precise overriding of merging behavior at every level, as well as custom lambda-based updates. You can patch tuples by index, but this is risky for dynamic lists. So instead you can express e.g. "append item to list" without the entire list, as a lambda.

I've been using patch for years now, and the uses are myriad. To overlay a set of overrides onto a base template, patch(base, overrides) is all you need. It's the most effective way I know to erase a metric ton of {...splats} and ?? defaultValues and != null from entire swathes of code. This is a real problem.

You could also view this as a "poor man's OT", with the main distinction being that a patch update only describes the new state, not the old state. Such updates are not reversible on their own. But they are far simpler to make and apply.

It can still power a global undo/redo system, in combination with its complement diff(A, B): you can reverse an update by diffing in the opposite direction. This is an operation which is formalized and streamlined into revise(…), so that it retains the exact shape of the original update, and doesn't require B at all. The structure of the update is sufficient information: it too encodes some intent behind the change.

With patch you also have a natural way to work with changes and conflicts as values. The earlier WYSIWIG scenario is just patch(commited, ephemeral) with bells on.

The net result is that mutating my intent or state is as easy as doing a {...value, ...update} splat, but I'm not falsely incentivized to flatten my data structures.

Instead it frees you up to think about what the most practical schema actually is from the data's point of view. This is driven by how the user wishes to edit it, because that's what you will connect it to. It makes you think about what a user's workspace actually is, and lets you align boundaries in UX and process with boundaries in data structure.

Remember: most classic "data structures" are not about the structure of data at all. They serve as acceleration tools to speed up specific operations you need on that data. Having the reads and writes drive the data design was always part of the job. What's weird is that people don't apply that idea end-to-end, from database to UI and back.

SQL tables are shaped the way they are because it enables complex filters and joins. However, I find this pretty counterproductive: it produces derived query results that are difficult to keep up to date on a client. They also don't look like any of the data structures I actually want to use in my code.

A Bike Shed of Schemas

This points to a very under-appreciated problem: it is completely pointless to argue about schemas and data types without citing specific domain logic and code that will be used to produce, edit and consume it. Because that code determines which structures you are incentivized to use, and which structures will require bespoke extra work.

From afar, column and line are just XY coordinates. Just use a 2-vector. But once you factor in the domain logic and etiquette, you realize that the horizontal and vertical directions have vastly different rules applied to them, and splitting might be better. Which one do you pick?

This applies to all data. Whether you should put items in a List or a Map largely depends on whether the consuming code will loop over it, or need random access. If an API only provides one, consumers will just build the missing Map or List as a first step. This is O(n log n) either way, because of sorting.

The method you use to read or write your data shouldn't limit use of everyday structure. Not unless you have a very good reason. But this is exactly what happens.

A lot of bad choices in data design come down to picking the "wrong" data type simply because the most appropriate one is inconvenient in some cases. This then leads to Conway's law, where one team picks the types that are most convenient only for them. The other teams are stuck with it, and end up writing bidirectional conversion code around their part, which will never be removed. The software will now always have this shape, reflecting which concerns were considered essential. What color are your types?

{ order: [4, 11, 9, 5, 15, 43], values: { 4: {...}, 5: {...}, 9: {...}, 11: {...}, 15: {...}, 43: {...}, }, );

For List vs Map, you can have this particular cake and eat it too. Just provide a List for the order and a Map for the values. If you structure a list or tree this way, then you can do both iteration and ID-based traversal in the most natural and efficient way. Don't underestimate how convenient this can be.

This also has the benefit that "re-ordering items" and "editing items" are fully orthogonal operations. It decomposes the problem of "patching a list of objects" into "patching a list of IDs" and "patching N separate objects". It makes code for manipulating lists and trees universal. It lets you to decide on a case by case basis whether you need to garbage collect the map, or whether preserving unused records is actually desirable.

Limiting it to ordinary JSON or JS types, rather than going full-blown OT or CRDT, is a useful baseline. With sensible schema design, at ordinary editing rates, CRDTs are overkill compared to the ability to just replay edits, or notify conflicts. This only requires version numbers and retries.

Users need those things anyway: just because a CRDT converges when two people edit, doesn't mean the result is what either person wants. The only case where OTs/CRDTs are absolutely necessary is rich-text editing, and you need bespoke UI solutions for that anyway. For simple text fields, last-write-wins is perfectly fine, and also far superior to what 99% of RESTy APIs do.

A CRDT is just a mechanism that translates partially ordered intents into a single state. Like, it's cool that you can make CRDT counters and CRDT lists and whatnot... but each CRDT implements only one particular resolution strategy. If it doesn't produce the desired result, you've created invalid intent no user expected. With last-write-wins, you at least have something 1 user did intend. Whether this is actually destructive or corrective is mostly a matter of schema design and minimal surface area, not math.

The main thing that OTs and CRDTs do well is resolve edits on ordered sequences, like strings. If two users are typing text in the same doc, edits higher-up will shift edits down below, which means the indices change when rebased. But if you are editing structured data, you can avoid referring to indices entirely, and just use IDs instead. This sidesteps the issue, like splitting order from values.

For the order, there is a simple solution: a map with a fractional index, effectively a dead-simple list CRDT. It just comes with some overhead.

Using a CRDT for string editing might not even be enough. Consider Google Docs-style comments anchored to that text: their indices also need to shift on every edit. Now you need a bespoke domain-aware CRDT. Or you work around it by injecting magic markers into the text. Either way, it seems non-trivial to decouple a CRDT from the specific target domain of the data inside. The constraints get mixed in.

If you ask me, this is why the field of real-time web apps is still in somewhat of a rut. It's mainly viewed as a high-end technical problem: how do we synchronize data structures over a P2P network without any data conflicts? What they should be asking is: what is the minimal amount of structure we need to reliably synchronize, so that users can have a shared workspace where intent is preserved, and conflicts are clearly signposted. And how should we design our schemas, so that our code can manipulate the data in a straightforward and reliable way? Fixing non-trivial user conflicts is simply not your job.

Most SaaS out there doesn't need any of this technical complexity. Consider that a good multiplayer app requires user presence and broadcast anyway. The simplest solution is just a persistent process on a single server coordinating this, one per live workspace. It's what most MMOs do. In fast-paced video games, this even involves lag compensation. Reliable ordering is not the big problem.

The situations where this doesn't scale, or where you absolutely must be P2P, are a minority. If you run into them, you must be doing very well. The solution that I've sketched out here is explicitly designed so it can comfortably be done by small teams, or even just 1 person.

The (private) CAD app I showed glimpses of above is entirely built this way. It's patch all the way down and it's had undo/redo from day 1. It also has a developer mode where you can just edit the user-space part of the data model, and save/load it.

When the in-house designers come to me with new UX requests, they often ask: "Is it possible to do ____?" The answer is never a laborious sigh from a front-end dev with too much on their plate. It's "sure, and we can do more."

If you're not actively aware the design of schemas and code is tightly coupled, your codebase will explode, and the bulk of it will be glue. Much of it just serves to translate generalized intent into concrete state or commands. Arguments about schemas are usually just hidden debates about whose job it is to translate, split or join something. This isn't just an irrelevant matter of "wire formats" because changing the structure and format of data also changes how you address specific parts of it.

In an interactive UI, you also need a reverse path, to apply edits. What I hope you are starting to realize is that this is really just the forward path in reverse, on so many levels. The result of a basic query is just the ordered IDs of the records that it matched. A join returns a tuple of record IDs per row. If you pre-assemble the associated record data for me, you actually make my job as a front-end dev harder, because there are multiple forward paths for the exact same data, in subtly different forms. What I want is to query and mutate the same damn store you do, and be told when what changes. It's table-stakes now.

With well-architected data, this can be wired up mostly automatically, without any scaffolding. The implementations you encounter in the wild just obfuscate this, because they don't distinguish between the data store and the model it holds. The fact that the data store should not be corruptible, and should enforce permissions and quotas, is incorrectly extended to the entire model stored inside. But that model doesn't belong to Stanley, it belongs to the user. This is why desktop applications didn't have a "Data Export". It was just called Load and Save, and what you saved was the intent, in a file.

Having a universal query or update mechanism doesn't absolve you from thinking about this either, which is why I think the patch approach is so rare: it looks like cowboy coding if you don't have the right boundaries in place. Patch is mainly for user-space mutations, not kernel-space, a concept that applies to more than just OS kernels. User-space must be very forgiving.

If you avoid it, you end up with something like GraphQL, a good example of solving only half the problem badly. Its getter assembles data for consumption by laboriously repeating it in dozens of partial variations. And it turns the setter part into an unsavory mix of lasagna and spaghetti. No wonder, it was designed for a platform that owns and hoards all your data.

* * *

Viewed narrowly, Intent is just a useful concept to rethink how you enforce validation and constraints in a front-end app. Viewed broadly, it completely changes how you build back-ends and data flows to support that. It will also teach you how adding new aspects to your software can reduce complexity, not increase it, if done right.

A good metric is to judge implementation choices by how many other places of the code need to care about them. If a proposed change requires adjustments literally everywhere else, it's probably a bad idea, unless the net effect is to remove code rather than add.

I believe reconcilers like React or tree-sitter are a major guide stone here. What they do is apply structure-preserving transforms on data structures, and incrementally. They actually do the annoying part for you. I based Use.GPU on the same principles, and use it to drive CPU canvases too. The tree-based structure reflects that one function's state just might be the next function's intent, all the way down. This is a compelling argument that the data and the code should have roughly the same shape.

You will also conclude there is nothing more nefarious than a hard split between back-end and front-end. You know, coded by different people, where each side is only half-aware of the other's needs, but one sits squarely in front of the other. Well-intentioned guesses about what the other end needs will often be wrong. You will end up with data types and query models that cannot answer questions concisely and efficiently, and which must be babysat to not go stale.

In the last 20 years, little has changed here in the wild. On the back-end, it still looks mostly the same. Even when modern storage solutions are deployed, people end up putting SQL- and ORM-like layers on top, because that's what's familiar. The split between back-end and database has the exact same malaise.

None of this work actually helps make the app more reliable, it's the opposite: every new feature makes on-going development harder. Many "solutions" in this space are not solutions, they are copes. Maybe we're overdue for a NoSQL-revival, this time with a focus on practical schema design and mutation? SQL was designed to model administrative business processes, not live interaction. I happen to believe a front-end should sit next to the back-end, not in front of it, with only a thin proxy as a broker.

What I can tell you for sure is: it's so much better when intent is a first-class concept. You don't need nor want to treat user data as something to pussy-foot around, or handle like it's radioactive. You can manipulate and transport it without a care. You can build rich, comfy functionality on top. Once implemented, you may find yourself not touching your network code for a very long time. It's the opposite of overwhelming, it's lovely. You can focus on building the tools your users need.

This can pave the way for more advanced concepts like OT and CRDT, but will show you that neither of them is a substitute for getting your application fundamentals right.

In doing so, you reach a synthesis of Dijkstra and anti-Dijkstra: your program should be provably correct in its data flow, which means it can safely break in completely arbitrary ways.

Because the I in UI meant "intent" all along.

More:

On Variance and Extensibility

Climbing Mount Effect

APIs are About Policy

Get in Zoomer, We're Saving React

2022-09-23T00:00:00+02:00

Looking back, and forward

Lately, it seems popular to talk smack about React. Both the orange and red site recently spilled the tea about how mean Uncle React has been, and how much nicer some of these next-gen frameworks supposedly are.

I find this bizarre for two reasons:

Most next-gen React spin-offs strike me as universally regressive, not progressive.

The few exceptions don't seem to have any actual complex, battle-hardened apps to point to, to prove their worth.

Now, before you close this tab thinking "ugh, not another tech rant", let me first remind you that a post is not a rant simply because it makes you angry. Next, let me point out that I've been writing code for 32 years. You should listen to your elders, for they know shit and have seen shit. I've also spent a fair amount of time teaching people how to get really good at React, so I know the pitfalls.

You may also notice that not even venerated 3rd party developers are particularly excited about React 18 and its concurrent mode, let alone the unwashed masses. This should tell you the React team itself is suffering a bit of an existential crisis. The framework that started as just the V in MVC can't seem to figure out where it wants to go.

So this is not the praise of a React fanboy. I built my own clone of the core run-time, and it was exactly because its limitations were grating, despite the potential there. I added numerous extensions, and then used it to tackle one of the most challenging domains around: GPU rendering. If one person can pull that off, that means there's actually something real going on here. It ties into genuine productivity boons, and results in robust, quality software, which seems to come together as if by magic.

To put it differently: when Figma recently announced they were acquired for $20B by Adobe, we all intuitively understood just how much of an exceptional black swan event that was. We know that 99.99…% of software companies are simply incapable of pulling off something similar. But do we know why?

Where we came from

If you're fresh off the boat today, React can seem like a fixture. The now-ancient saying "Nobody ever got fired for choosing IBM" may as well be updated for React. Nevertheless, when it appeared on the scene, it was wild: you're going to put the HTML and CSS in the JavaScript? Are you mad?

Yes, it was mad, and like Galileo, the people behind React were completely right, for they integrated some of the best ideas out there. They were so right that Angular pretty much threw in the towel on its abysmal two-way binding system and redesigned it to adopt a similar one-way data flow. They were so right that React also dethroned the previous fixture in web land, jQuery, as the diff-based Virtual DOM obsoleted almost all of the trickery people were using to beat the old DOM into shape. The fact that you could use e.g. componentDidUpdate to integrate legacy code was just a conceit, a transition mechanism that spelled out its own obsolescence as soon as you got comfortable with it.

Many competing frameworks acted like this wasn't so, and stuck to the old practice of using templates. They missed the obvious lesson here: every templating language inevitably turns into a very poor programming language over time. It will grow to add conditionals, loops, scopes, macros, and other things that are much nicer in actual code. A templating language is mainly an inner platform effect. It targets a weird imagined archetype of someone who isn't allergic to code, but somehow isn't smart enough to work in a genuine programming language. In my experience, this archetype doesn't actually exist. Designers don't want to code at all, while coders want native expressiveness. It's just that simple.

Others looked at the Virtual DOM and only saw inefficiency. They wanted to add a compiler, so they could reduce the DOM manipulations to an absolute minimum, smugly pointing to benchmarks. This was often just premature optimization, because it failed to recognize the power of dynamic languages: that they can easily reconfigure their behavior at run-time, in response to data, in a turing-complete way. This is essential for composing grown up apps that enable user freedom. The use case that most of the React spin-offs seem to be targeting is not apps but web sites. They are paving cow paths that are well-worn with some minor conveniences, while never transcending them.

var RouterMixin = { contextTypes: { router: React.PropTypes.object.isRequired }, // The mixin provides a method so that // components don't have to use the // context API directly. push: function(path) { this.context.router.push(path) } }; var Link = React.createClass({ mixins: [RouterMixin], handleClick: function(e) { e.stopPropagation(); // This method is defined in RouterMixin. this.push(this.props.to); }, render: function() { return ( {this.props.children} ); } }); module.exports = Link;

React circa 2016

It's also easy to forget that React itself had many architectural revisions. When old farts like me got in on it, components still had mix-ins, because genuine classes were a distant dream in JS. When ES classes showed up, React adopted those, but it didn't fundamentally change the way you structured your code. It wasn't until React 16.8 (!) that we got hooks, which completely changed the way you approached it. This reduced the necessary boilerplate by an order of magnitude, and triggered a cambrian explosion of custom hook development. That is, at least until the buzz wore off, and only the good ideas remained standing.

Along the way, third party React libraries have followed a similar path. Solutions like Redux appeared, got popular, and then were ditched as people realized the boilerplate just wasn't worth it. It was a necessary lesson to learn.

This legacy of evolution is also where the bulk of React's perceived bloat sits today. As browsers evolved, as libraries got smarter, and as more people ditched OO, much of it is now indeed unnecessary for many use cases. But while you can tweak React with a leaner-and-meaner reimplementation, this doesn't fundamentally alter the value proposition, or invalidate the existing appeal of it.

The fact remains that before React showed up, nobody really had any idea how to make concepts like URL routers, or drag and drop, or UI design systems, truly sing, not on the web. We had a lot of individual pieces, but nothing solid to puzzle them together with. Nevertheless, there is actual undiscovered country beyond, and that's really what this post is about: looking back and looking forward.

If there's one solid criticism I've heard of React, it's this: that no two React codebases ever look alike. This is generally true, but it's somewhat similar to another old adage: that happy families all look alike, but every broken family is broken in its own particular way. The reason bad React codebases are bad is because the people who code it have no idea what they're supposed to be doing. Without a model of how to reason about their code in a structured way, they just keep adding on hack upon hack, until it's better to throw the entire thing away and start from scratch. This is no different from any other codebase made up as they go along, React or not.

Where React came from is easy to explain, but difficult to grok: it's the solution that Facebook arrived at, in order to make their army of junior developers build a reliable front-end, that could be used by millions. There is an enormous amount of hard-earned experience encoded in its architecture today. Often though, it can be hard to sort the wheat from the chaff. If you stubbornly stick to what feels familiar and easy, you may never understand this. And if you never build anything other than a SaaS-with-forms, you never will.

I won't rehash the specifics of e.g. useEffect here, but rather, drop in a trickier question: what if the problem people have with useEffect + DOM Events isn't the fault of hooks at all, but is actually the fault of the DOM?

I only mention it because when I grafted an immediate-mode style interaction model onto my React clone instead, I discovered that complex gesture controllers suddenly became 2-3x shorter. What's more, declaring data dependencies that "violate the rules of React" wasn't an anti-pattern at all: it was actually key to the entire thing. So when I hear that people are proudly trying to replace dependencies with magic signals, I just shake my head and look elsewhere.

Which makes me wonder… why is nobody else doing things like this? Immediate mode UI isn't new, not by a long shot. And it's hardly the only sticking point.

Mac OS X Leopard - 2007

Where we actually came from

Here's another thing you may not understand: just how good old desktop software truly was.

The gold standard here is Mac OS X, circa 2008. It was right before the iPhone, when Apple was still uniquely focused on making its desktop the slickest, most accessible platform around. It was a time when sites like Ars Technica still published real content, and John Siracusa would lovingly post multi-page breakdowns of every new release, obsessing over every detail for years on end. Just imagine: tech journalists actually knowing the ins-and-outs of how the sausage was made, as opposed to copy/pasting advertorials. It was awesome.

This was supported by a blooming 3rd party app ecosystem, before anyone had heard of an App Store. It resulted in some genuine marvels, which fit seamlessly into the design principles of the platform. For example, Adium, a universal instant messenger, which made other open-source offerings seem clunky and downright cringe. Or Growl, a universal notification system that paired seamlessly with it. It's difficult to imagine this not being standard in every OS now, but Mac enthusiasts had it years before anyone else.

The monopolistic Apple of today can't hold a candle to the extended Apple cinematic universe from before. I still often refer to the Apple Human Interface Guidelines from that era, rather than the more "updated" versions of today, which have slowly but surely thrown their own wisdom in the trash.

The first section of three, Application Design Fundamentals, has almost nothing to do with Macs specifically. You can just tell from the chapter titles:

The Design Process

Characteristics of Great Software

Human Interface Design

Prioritizing Design Decisions

Like another favorite, The Design of Everyday Things, it approaches software first and foremost as tools designed for people to use. The specific choices made in app design can be the difference between something that's a joy to use and something that's resented and constantly fought against.

So what exactly did we lose? It's quite simple: by moving software into the cloud and turning them into web-based SaaS offerings, many of the basic affordances that used to be standard have gotten watered down or removed entirely. Here are some examples:

Menus let you cross over empty space and other menu items, instead of strictly enforcing hover rectangles.

You can drag and drop the file icon from a document's titlebar e.g. to upload it, instead of having to go look for it again.

Holding keyboard modifiers like CTRL or ALT is reflected instantly in menus, and used to make power-user features discoverable-yet-unobtrusive.

And here are some more:

You can browse years of documents, emails, … and instantly draft new ones. Fully off-line, with zero lag.

You can sort any table by any column, and it will remember prior keys to produce a stable sort for identical values.

Undo/redo is standard and expected, even when moving entire directories around in the Finder.

Copy/pasting rich content is normal, and entirely equivalent to dragging and dropping it.

When you rename or move a file that you're editing, its window instantly reflects the new name and location.

You can also drag a file into an "open file" dialog, to select it there.

When downloading a file, the partial download has a progress bar on the icon. It can be double clicked to resume, or even copied to another machine.

It's always amusing to me to watch a power user switch to a Mac late in life, because much of their early complaints stem from not realizing there are far more obvious ways to do what they've trained themselves to do in a cumbersome way.

On almost every platform, PDFs are just awful to use. Whereas out-of-the-box on a Mac, you can annotate them to your heart's content, or drag pages from one PDF to another to recompose it. You can also sign them with a signature read from your webcam, for those of us who still know what pens are for. This is what happens when you tell companies like Adobe to utterly stuff it and just show them how it's supposed to be done, instead of waiting for their approval. The productivity benefits were enormous.

As an aside, if all of this seems quaint or positively boomeresque, here's a tip: forcing yourself to slow down and work with information directly, with your hands, manipulating objects physical or virtual, instead of offloading it all to a cloud… this is not an anti-pattern. Neither is genuine note taking on a piece of paper. You should try it sometime.

At the time, many supposed software experts scoffed at Apple, deriding their products as mere expensive toys differentiated purely by "marketing". But this is the same company that seamlessly transitioned its entire stack from PowerPC, to x86, to x64, and eventually ARM, with most users remaining blissfully unaware this ever took place.

This is what the pinnacle of our craft can actually look like.

Apple didn't just knock it out of the park when it came to the OS or the overall UI: they also shipped powerful first-party apps like iMovie and Keynote, which made competing offerings look positively shabby. Steve Jobs used them for his own keynotes, arguably the best in the business.

Similarly, what set the iPhone apart was not just its touch interface, but that they actually ported a mature media and document stack to mobile wholesale. At that time, the "mobile web" was a complete and utter joke, and it would take Android years to catch up, whether it was video or music, or basic stuff like calendar invites and contacts.

It has nothing to do with marketing. Indeed, while many companies have since emulated and perfected their own Apple-style pitch, almost no-one manages to get away from that tell-tale "enterprise" feel. They don't know or care how their users actually want to use their products: the people in charge don't have the first clue about the fundamentals of product design. They just like shiny things when they see them.

iMovie - 2010

The Reactive Enterprise

What does any of this have to do with React? Well it's very simple. Mac OS X was the first OS that could actually seriously claim to be reactive.

The standard which virtually everyone emulated back then was Windows. And in Windows, the norm—which mostly remains to this day—is that when you query information, that information is fetched once, and never updated. The user was just supposed to know that in order to see it update, they had to manually refresh it, either by bumping a selection back and forth, or by closing and reopening a dialog.

Windows 95

The same applied to preferences: in Windows land, the established pattern was to present a user with a set of buttons, the triad of Ok, Cancel and Apply. This is awful, and here's why. If you click Ok, you are committing to a choice you haven't yet had the chance to see the implications of. If you click Cancel, you are completely discarding everything you did, without ever trying it out. If you click Apply, it's the same as pressing Ok, just the window stays open. None of the 3 buttons let you interact confidently, or easily try changes one by one, reinforcing the idea that it's the user's fault for being "bad at computers" if it doesn't do what they expect, or they don't know how to back out.

The bold Mac solution was that toggling a preference should take effect immediately. Even if that choice affects the entire desktop, such as changing the UI theme. So if that's not what you wanted, you simply clicked again to undo it right away. Macs were reactive, while Windows was transactional. The main reason it worked this way was because most programmers had no clue how to effectively make their software respond to arbitrary changes, and Microsoft couldn't go a few years without coming up with yet another ill-conceived UI framework.

This divide has mostly remained, with the only notable change being that on mobile devices, both iOS and Android tend to embrace the reactive model. However, given that much of the software used is made partially or wholly out of web views, this is a promise that is often violated and rarely seen as an inviolable constraint. It's just a nice-to-have. Furthermore, while it has become easier to display reactive information, the crucial second half of the equation—interaction—remains mostly neglected, also by design.

Tacoma Narrows bridge collapse (1940)

Hyatt Regency walkway collapse (1981)

I'm going to be cheeky and say if there's anyone who should take the blame for this, it's back-end engineers and the technology choices they continue to make. The very notion of "back-end" is a fallacy: it implies that one can produce a useful, working system, without ever having to talk to end-users.

Just imagine how alien this concept would be to an engineer before the software revolution happened: it'd be like suggesting you build a bridge without ever having to think about where it sits or who drives over it, because that's just "installation" and "surfacing". In civil engineering, catastrophes are rare, and each is a cautionary tale, never to be repeated: the loss of life was often visceral and brutal. But in software, we embraced never learning such lessons.

A specific evil here is the legacy of SQL and the associated practices, which fragments and normalizes data into rigid tables. As a result, the effect of any change is difficult to predict, and virtually impossible to reliably undo or synchronize after the fact.

This is also the fault of "enterprise", in a very direct sense: SQL databases and transactions are mainly designed to model business processes. They evolved to model bureaucratic workflows in actual enterprises, with a clear hierarchy of command, a need to maintain an official set of records, with the ability for auditing and oversight.

However, such classic enterprises were of course still run by people, by individuals. The bulk of the work they did was done offline, producing documents, spreadsheets and other materials through direct interaction and iteration. The bureaucracy was a means to an end, it wasn't the sole activity. The idea of an organization or country run entirely on bureaucracy was the stuff people made satirical movies about.

And yet, many jobs now follow exactly this template. The activity is entirely coordinated and routed through specific SaaS apps, either off-the-shelf or bespoke, which strictly limit the available actions. They only contain watered down mockeries of classic desktop concepts such as files and folders, direct manipulation of data, and parallel off-line workstreams. They have little to no affordances for drafts, iteration or errors. They are mainly designed to appeal to management, not the riff-raff.

The promise of adopting such software is that everything will run more smoothly, and that oversight becomes effortless thanks to a multitude of metrics and paper trails. The reality is that you often replace tasks that ordinary, capable employees could do themselves, with a cumbersome and restrictive process. Information becomes harder to find, mistakes are more difficult to correct, and the normal activity of doing your job is replaced with endless form filling, box-ticking and notification chasing. There is a reason nobody likes JIRA, and this is it.

What's more, by adopting SaaS, companies put themselves at the mercy of someone else's development process. When dealing with an unanticipated scenario, you often simply can't work around it with the tools given, by design. It doesn't matter how smart or self-reliant the employees are: the software forces them to be stupid, and the only solution is to pay the vendor and wait 3 months or more.

For some reason, everyone has agreed that this is the way forward. It's insane.

Oracle Cloud with AI Bullshit

Circling Back

Despite all its embedded architectural wisdom, this is a flaw that React shares: it was never meant to enable user freedom. Indeed, the very concept of Facebook precludes it, arguably the world's biggest lock-in SaaS. The interactions that are allowed there are exactly like any other SaaS: GET and POST to a monolithic back-end, which enforces rigid processes.

As an app developer, if you want to add robust undo/redo, comfy mouse interactions and drag-n-drop, keyboard shortcuts, and all the other goodies that were standard on the desktop, there are no easy architectural shortcuts available today. And if you want to add real-time collaboration, practically a necessity for real apps, all of these concerns spill out, because they cannot be split up neatly into a wholly separate front-end and back-end.

A good example is when people mistakenly equate undo/redo with a discrete, immutable event log. This is fundamentally wrong, because what constitutes an action from user's point of view is entirely different from how a back-end engineer perceives it. For example undo/redo needs to group multiple operations to enable sensible, logical checkpoints… but it also needs to do so on the fly, for actions which are rapid and don't conflict.

If you don't believe me, go type some text in your text editor and see what happens when you press CTRL-Z. It won't erase character by character, but did you ever think about that? Plus, if multiple users collaborate, each needs their own undo/redo stack, which means you need the equivalent of git rebasing and merging. You'd be amazed how many people don't realize this.

If we want to move forward, surely, we should be able to replicate what was normal 20 years ago?

Real-time databases

There are a few promising things happening in the field, but they are so, so rare… like the slow-death-and-rebirth of Firebase into open-source alternatives and lookalikes. But even then, robust real-time collaboration remains a 5-star premium feature.

Similarly, big canvas-based apps like Figma, and scrappy upstarts like TLDraw have to painstakingly reinvent all the wheels, as practically all the relevant knowledge has been lost. And heaven forbid you actually want a decent, GPU-accelerated renderer: you will need to pay a dedicated team of experts to write code nobody else in-house can maintain, because the tooling is awful and also they are scared of math.

What bugs me the most is that the React dev team and friends seem extremely unaware of any of this. The things they are prioritizing simply don't matter in bringing the quality of the resulting software forward, except at the margins. It'll just load the same HTML a bit faster. If you stubbornly refuse to learn what memo(…) is for, it'll render slightly less worse. But the advice they give for event handling, for data fetching, and so on… for advanced use it's simply wrong.

A good example is that GraphQL query subscriptions in Apollo split up the initial GET from the subsequent SUBSCRIBE. This means there is always a chance one or more events were dropped in between the two. Nevertheless, this is how the library is designed, and this is what countless developers are doing today. Well okay then.

Another good example is implementing mouse gestures, because mouse events happen quicker than React can re-render. Making this work right the "proper way" is an exercise in frustration, and eventually you will conclude that everything you've been told about non-HTML-element useRef is a lie: just embrace mutating state here.

In fact, despite being told this will cause bugs, I've never had any issues with it in React 17. This leads me to suspect that what they were really doing was trying to prevent people from writing code that would break in React 18's concurrent mode. If so: dick move, guys. Here's what I propose: if you want to warn people about "subtle bugs", post a concrete proof, or GTFO.

* * *

If you want to build a truly modern, robust, desktop-class web app with React, you will find that you still need to pretty much make apple pie from scratch, by first re-inventing the entire universe. You can try starting with the pre-made stuff, but you will hit a wall, and/or eventually corrupt your users' data. It's simply been my experience, and I've done the React real-time collaboration rodeo with GPU sprinkles on top multiple times now.

Crucially, none of the React alternatives solve this, indeed, they mostly just make it worse by trying to "helpfully" mutate state right away. But here's the annoying truth: you cannot skip learning to reason about well-ordered orchestration. It will just bite you in the ass, guaranteed.

What's really frustrating about all this is how passive and helpless the current generation of web developers seem to be in all this. It's as if they've all been lulled into complacency by convenience. They seem afraid to carve out their own ambitious paths, and lack serious gusto for engineering. If there isn't a "friendly" bot spewing encouraging messages with plenty of 👏 emoji at every turn, they won't engage.

As someone who took a classical engineering education, which included not just a broad scientific and mathematical basis, but crucially also the necessary engineering ethos, this is just alien to me. Call me cynical all you want, but it matches my experience. Coming after the generation that birthed Git and BitTorrent, and which killed IE with Firefox and Konqueror/WebKit, it just seems ridiculous.

Fuck, most zoomers don't even know how to dance. I don't mean that they are bad at dancing, I mean they literally won't try, and just stand around awkwardly.

Just know: nobody else is going to do it for you. So what are you waiting for?

OSCMS Talk: Designer Eye for the Geek Guy/Gal

2007-02-14T00:00:00+01:00

Update: I've posted the presentation slides and a video is available as well.

I'll be attending the OSCMS conference in Sunnyvale CA at Yahoo next month. Aside from a repeat of my DrupalCon jQuery talk, (though with a bit more examples) I just submitted another proposal for a talk. It's something that I've wanted to do for a while now:

In meetings and lectures across the globe, people are made to endure hideous presentation slides featuring some of the wildest colors, clip art and typography. Many websites are so confusingly laid out, that you get dizzy from the overload of boxes, images or links. And every day, people receive resumés, invoices and ads ... *cue lightning and thunder* set in the Comic Sans font.

It's enough to make the average designer's hair turn blue, fall out, morph into a ninja and stab him/her in the eyes.

But, all hope is not lost! Contrary to popular belief, graphical design is not some arcane voodoo magic, but a straightforward discipline that values experience, reusability, elegance and good tools just like programming. Just like code, there are plenty of objective ways to measure the quality of a design. However, just like art is subjective, so may two programmers disagree on which implementation is the best. No designer is born with a genetic sense of proportion... it's just that while you were busy writing BASIC code on your C64, they were busy drawing superheroes.

I myself am an engineering geek who's never had any sort of formal design or art training, but has earned the title of "design nazi" on numerous occasions.

This session will teach geeks some basic principles about graphical design (especially on the web), from a geek perspective. This means we won't talk about "visually balanced design" but "here's a good approach to spacing". Soon, you'll be hearing the oooh's and aaah's when you don your designer hat.

You can vote on the session page if you're interested.

Farbtastic Color Picker

2006-07-14T00:00:00+02:00

jQuery Plug-in

Farbtastic is a jQuery plug-in that can add one or more color picker widgets into a page through JavaScript. Each widget is then linked to an existing element (e.g. a text field) and will update the element's value when a color is selected.

Download Farbtastic 1.2 - January 8, 2007 (License: GPL).

Demo

Farbtastic uses layered transparent PNGs to render a saturation/luminance gradient inside of a hue circle. No Flash, no pixel sized divs.

Click and drag over the selector to try it out.

Color:

Basic Usage

Include farbtastic.js and farbtastic.css in your HTML:

Add a placeholder div and a text field to your HTML, and give each an ID:

Add a ready() handler to the document which initializes the color picker and link it to the text field with the following syntax:

$(document).ready(function() { $('#colorpicker').farbtastic('#color'); });

See demo1.html and demo2.html for an example (jquery.js is not included!).

Styling

The color picker is a block-level element and is 195x195 pixels large. You can control the position by styling your placeholder (e.g. floating it).

Note that the black/white gradients inside wheel.png and mask.png were generated programmatically and cannot be recreated easily in an image editing program.

Advanced Usage

jQuery Method

$(...).farbtastic() $(...).farbtastic(callback)

This creates color pickers in the selected objects. callback is optional and can be a:

DOM Node, jQuery object or jQuery selector: the color picker will be linked to the selected element(s) by syncing the value (for form elements) and color (all elements).

Function: this function will be called whenever the user chooses a different color. It should have the following signature:

function callback(color) { ... }

With color the chosen color in hex representation (e.g. '#123456').

Object

$.farbtastic(placeholder) $.farbtastic(placeholder, callback)

Invoking $.farbtastic(placeholder) is the same as using $(placeholder).farbtastic() except that the Farbtastic object is returned instead of the jQuery object. This allows you to use the Farbtastic methods and properties below.

Note that there is only one Farbtastic object per placeholder. If you call $.farbtastic(placeholder) twice with the same placeholder, you will get the same object back each time.

The optional callback argument behaves exactly as for the jQuery method.

Methods

.linkTo(callback)

Allows you to set a new callback. Any existing callbacks are removed. See above for the meaning of callback.

.setColor(string)

Sets the picker color to the given color in hex representation (e.g. '#123456').

.setColor([h, s, l])

Sets the picker color to the given color in normalized HSL (0..1 scale).

Properties

.linked

The elements (jQuery object) or callback function this picker is linked to.

.color

Current color in hex representation (e.g. '#123456').

.hsl

Current color in normalized HSL (e.g. [0.3, 0.4, 0.5]).