Acko.net

Yak Shading

2015-09-27T00:00:00+02:00

Data-Driven Geometry

MathBox primitives need to take arbitrary data, transform it on the fly, and render it as styled geometry based on their attributes. Done as much as possible on the graphics hardware.

Three.js can render points, lines, triangles, but only with a few predetermined strategies. The alternative is to write your own vertex and fragment shader and do everything from scratch. Each new use case means a new ShaderMaterial with its own properties, so called uniforms. If the stock geometry doesn't suffice, you can make your own triangles by filling a raw BufferGeometry and assign custom per-vertex attributes. Essentially, to leverage GPU computation with Three.js—most engines, really—you have to ignore most of it.

Virtual Geometry

Shader computations are mainly rote transforms. For example, if you want to draw a line between two points, you'll have to make a long rectangle, made out of two triangles. But this simple idea gets complicated quickly once you add corner joins, depth scaling, 3D clipping, and so on. Doing this to an entire data set at once is what GPUs are made for, through vertex shaders which transform points.

Vertex shaders can only do 1-to-1 mappings. This isn't a problem by itself. You can use a gather approach to do N-to-1 mapping, where all the necessary data is pre-arranged into attribute arrays, with the data repeated and interleaved per vertex as necessary.

The proper tool for this is a geometry shader: a program that creates new geometry by N-to-M mapping of data, like making triangles out of points. WebGL doesn't support geometry shaders, won't any time soon, but you can emulate them with texture samplers. A texture image is just a big typed array, and you have random access unlike vertex attributes.

The original geometry acts only as a template, directing the shader's real data lookups. You lose some performance this way, but it's not too bad. Any procedural sampling pattern works, drawing 1 shape or 10,000. As textures can be rendered to, not just from, this also enables transform feedback, using the result of one pass to create new geometry in another.

All geometry rendered this way is 100% static as far as Three.js is concerned. New values are uploaded directly to GPU memory just before the rendering starts. The only gotcha is handling variable size input, because reallocation is costly. Pre-allocating a larger texture is easy, but clipping off the excess geometry in an O(1) fashion on the JS side is hard. In most cases there's the work around of dynamically generating degenerate triangles in a shader, which collapse down to invisible edges or points. This way, MathBox can accept variable sized arrays in multiple dimensions and will do its best to minimize disruption. If attribute instancing was more standard in WebGL, this wouldn't be such an issue, but as it stands, the workarounds are very necessary.

Vertex Party

If you squint very hard it looks a bit like React for live geometry. Except instead of a diffing algorithm, there's a few events, some texture uploads, a handful of draw calls and then an idle CPU. It's ideal for drawing thousands of things that look similar and follow the same rules. It can handle not just basic GL primitives like lines or triangles, but higher level shapes like 3D arrows or sprites.

My first prototype of this was my last christmas demo. It was messy and tedious to make, especially the shaders, but it performed excellently: the final scene renders ~200,000 triangles. Despite being a layer around Three.js … around WebGL … around OpenGL … around a driver … around a GPU … performance has far exceeded my expectations. Even complex scenes run great on my Android phone, easily 10x faster than MathBox 1, in some cases more like 1000x.

Of course compared to cutting edge DirectX or OpenCL (not a typo), this is still very limited. In today's GPUs, the charade of attributes, geometries, vertices and samples has mostly been stripped away. What remains is buffers and massive operations on them, exposed raw in new APIs like AMD's Mantle and iOS's Metal. My vertex trickery acts like a polyfill, virtualizing WebGL's capabilities to bring them closer to the present. It goes a bit beyond what geometry shaders can provide, but still lacks many useful things like atomic append queues or stream compaction.

For large geometries, the set up cost can be noticeable though. Shader compilation time also grows with transform complexity, doubly so on Windows where shaders are recompiled to HLSL / Direct3D. This makes drawing ops the heaviest MathBox primitives to spawn and reallocate. You could call this the MathBox version of the dreaded 'paint' of HTML. Once warmed up though, most other properties can be animated instantly, including the data being displayed: this is the opposite of how HTML works. Hence you can mostly spawn things ahead of time, revealing and hiding objects as needed, with minimal overhead and jank at runtime.

This all relies on carefully constructed shaders which have to be wired up in all their individual permutations. This needed to be solved programmatically, which is where we go last.

MathBox²

2015-09-27T00:00:00+02:00

PowerPoint Must Die

"I think a lot of mathematics is really about how you understand things in your head. It's people that did mathematics, we're not just general purpose machines, we're people. We see things, we feel things, we think of things. A lot of what I have done in my mathematical career has had to do with finding new ways to build models, to see things, do computations. Really get a feel for stuff.

It may seem unimportant, but when I started out people drew pictures of 3-manifolds one way and I started drawing them a different way. People drew pictures of surfaces one way and I started drawing them a different way. There's something significant about how the representation in your head profoundly changes how you think.

It's very hard to do a brain dump. Very hard to do that. But I'm still going to try to do something to give a feel for 3-manifolds. Words are one thing, we can talk about geometric structures. There are many precise mathematical words that could be used, but they don't automatically convey a feeling for it. I probably can't convey a feeling for it either, but I want to try."
– William Thurston, The Mystery of 3-Manifolds (Video)

How do you convince web developers—heck, people in general—to care about math? This was the challenge underlying Making Things With Maths, a talk I gave three years ago. I didn't know either, I just knew why I liked this stuff: demoscene, games, simulation, physics, VR, … It had little to do with what passed for mathematics in my own engineering education. There we were served only eyesore PowerPoints or handwritten overhead transparencies, with simplified graphs, abstract flowcharts and rote formulas, available on black and white photocopies.

Smart people who were supposed to teach us about technology seemed unable to teach us with technology. Fixing this felt like a huge challenge where I'd have to start from scratch. This is why the focus was entirely on showing rather than telling, and why MathBox 1 was born. It's how this stuff looks and feels in my head, and how I got my degree: by translating formulas into mental pictures, which I could replay and reason about on demand.

PowerPoint Syndrome

Initially I used MathBox like an embedded image or video: compact diagrams, each a point or two in a presentation. My style quickly shifted though. I kept on finding ways to transform from one visualization to another. Not for show, but to reveal the similarities and relationships underneath. MathBox encouraged me to animate things correctly, leveraging the actual models themselves, instead of doing a visual morph from A to B. Each animation became a continuous stream of valid examples, a quality both captivating and revealing.

For instance, How to Fold a Julia Fractal is filled with animations of complex exponentials, right from the get go. This way I avoid the scare that ($ e^{i\pi} $) is a meaningful expression; symbology and tau-tology never have a chance to obscure geometrical workings. Instead a web page that casually demonstrates conformal mapping and complex differential equations got 340,000 visits. Despite spotty web browser support and excluding all mobile phones for years.

The next talk, Making WebGL Dance, contained elaborate long takes worthy of an Alfonso Cuarón film, with only 3 separate shots for the bulk of a 30 minute talk. The lesson seemed obvious: the slides shouldn't have graphics in them, rather, the graphics should have slides in them. The diagnosis of PowerPoint syndrome is then the constant trashing of context from one slide to the next. A traditional blackboard doesn't have this problem: you build up diagrams slowly, by hand, across a large surface, erasing selectively and only when you run out of space.

It's not just about permanence and progression though, it's also about leveraging our natural understanding of shape, scale, color and motion. Think of how a toddler learns to interact with the world: poke, grab, chew, spit, smash. Which evolves into run, jump, fall, get back up again. Humans are naturals at taking multiple cases of "If I do this, that will happen" and turning it into a consistent, functional model of how things work. We learn language by bootstrapping random jibberish into situational meaning, converging on a shared protocol.

That said, I find the usual descriptions of how people experience language and thought foreign. Instead, when Temple Grandin speaks about visual thinking, I nod vigorously. Thought to me is analog concepts and sensory memories, remixed with visual and other simulations. It builds off the quantities and qualities present in spatial and temporal notions, which appear built-in to us.

Speech and writing is then a program designed to reconstruct particular thoughts in a compatible brain. There are a multitude of evolving languages, they can be used elegantly, bluntly, incomprehensibly, but the desired output remains the same. In my talks, armed with weapons-grade C2-continuous animations, it is easy to transcode my film reel into words, because the slides run themselves. The string of concepts already hangs in the air, I only add the missing grammar that links them up. This is a puzzle our brains are so good at solving, we usually do it without thinking.

Language is the ability of thoughts to compute their own source code.

(It's not proof, I just supply pudding.)

Manifold Dreams

I don't say all this to up my Rain Man cred, but to lay to rest the recurring question of where my work comes from. I translate the pictures in my head to HD, in order to learn from and refine the view. As I did with quaternions: I struggled to grok the hypersphere, it wouldn't fit together right. So I wrote the code to trace out geodesics in color and fly around in it, and suddenly the twisting made sense. Hence my entire tutorial was built to replicate the same discovery process I went through myself.

There was one big problem: scenes now consisted of diagrams of diagrams, which meant working around MathBox more than with it. Performance issues arose as complexity grew. Above all there was a total lack of composability in the components. None of this could be fixed without ripping out significant pieces, so doing it incrementally seemed futile. I started from scratch and set off to reinvent all the wheels.

$$ \text{MathBox}^2 = \int_1^2 \text{code}(v) dv $$

MathBox 2 was inevitably going to suffer second-system syndrome, parts would be overengineered. Rather than fight it, I embraced it and effectively wrote a strange vector GPU driver in CoffeeScript. (Such is life, this is a blueprint meant to be simplified and made obsolete over time, not expanded upon.) It's a freight train straight to the heart of a graphics card, combining low-level and high-level in a way that feels novel 🐴 when you use it, squeezing 🐴 through a very small opening.

What was tedious before, now falls out naturally. If I format the scene above as XML/JSX, it becomes:

In order to make these pieces behave, a bunch of additional attributes are applied, most of which are strings or values, some of which are functions/code, either JavaScript or GLSL:

id="1" scale={300}>
  id="2" proxy={true} position={[0, 0, 3]} />
  id="3" speed={1/4}>
    id="4" bend={1}>
      id="5" code="
uniform float cos1;
uniform float sin1;
uniform float cos2;
uniform float sin2;
uniform float cos3;
uniform float sin3;
uniform float cos4;
uniform float sin4;

vec4 getRotate4D(vec4 xyzw, inout vec4 stpq) {
  xyzw.xy = xyzw.xy * mat2(cos1, sin1, -sin1, cos1);
  xyzw.zw = xyzw.zw * mat2(cos2, sin2, -sin2, cos2);
  xyzw.xz = xyzw.xz * mat2(cos3, sin3, -sin3, cos3);
  xyzw.yw = xyzw.yw * mat2(cos4, sin4, -sin4, cos4);

  return xyzw;
}"
 cos1=>{(t) => Math.cos(t * .111)} sin1=>{(t) => Math.sin(t * .111)} cos2=>{(t) => Math.cos(t * .151 + 1)} sin2=>{(t) => Math.sin(t * .151 + 1)} cos3=>{(t) => Math.cos(t * .071 + Math.sin(t * .081))} sin3=>{(t) => Math.sin(t * .071 + Math.sin(t * .081))} cos4=>{(t) => Math.cos(t * .053 + Math.sin(t * .066) + 1)} sin4=>{(t) => Math.sin(t * .053 + Math.sin(t * .066) + 1)} />
      id="6">
        id="7" rangeX={[-π/2, π/2]} rangeY={[0, τ]} width={129} height={65} expr={(emit, θ, ϕ, i, j) => {
        q1.set(0, 0, Math.sin(θ), Math.cos(θ));
        q2.set(0, Math.sin(ϕ), 0, Math.cos(ϕ));
        q1.multiply(q2);
        emit(q1.x, q1.y, q1.z, q1.w);
      }} live={false} channels={4} />
        id="8" color="#3090FF" />
        id="9" rangeX={[-π/2, π/2]} rangeY={[0, τ]} width={129} height={65} expr={(emit, θ, ϕ, i, j) => {
        q1.set(0, Math.sin(θ), 0, Math.cos(θ));
        q2.set(Math.sin(ϕ), 0, 0, Math.cos(ϕ));
        q1.multiply(q2);
        emit(q1.x, q1.y, q1.z, q1.w);
      }} live={false} channels={4} />
        id="10" color="#20A000" />
        id="11" rangeX={[-π/2, π/2]} rangeY={[0, τ]} width={129} height={65} expr={(emit, θ, ϕ, i, j) => {
        q1.set(Math.sin(θ), 0, 0, Math.cos(θ));
        q2.set(0, 0, Math.sin(ϕ), Math.cos(ϕ));
        q1.multiply(q2);
        emit(q1.x, q1.y, q1.z, q1.w);
      }} live={false} channels={4} />
        id="12" color="#DF2000" />
      >
    >
  >
>

Phew. That's how you make a 4D diagram with Hopf fibration as far as the eye can see. Except it's not actually JSX, that's just me and my pretty-printer pretending.

Geometry Streaming

The key is the data itself. It's an array of points mostly, but how that data is laid out and interpreted determines how useful it can be.

Most basic primitives come in fixed size chunks. Particles are single points, lines have two points, triangles have three points. Polygons and polylines have N points. So it made sense to have a tuple of N points be the basic logical unit. You can think in logical pieces of geometry, rather than raw points or individual triangles, unlike GL.

Each primitive maps over data in a standard way. Feed an array of points to a line, you get a polyline. Feed a matrix of points to a surface and you get a grid mesh. Simple. But feed a voxel to a vector, and you get a 3D vector field. The general idea is that drawing 1 of something should be as easy as drawing 100×100×100.

This is particularly useful for custom data expressions, which stream in live or procedural data. They now receive an emit(x, y, z, w) function, for emitting a 4-vector like XYZW or RGBA. This is little more than an inlineable call to fill a floatArray[i++] = x, quite a lot faster than returning an array or object.

mathbox
  .interval({
    expr: function (emit, x, i, t) {
      y = Math.sin(x + t);
      emit(x,  y);
      emit(x, -y);
    },
    width:   64,
    items:    2,
    channels: 2,
  })
  .vector({
    color: 0x3090FF,
    width: 3,
    start: true,
  });

Emitting 64 2D vectors on an interval, 2 points each.

More importantly it lets you emit N points in one iteration, which makes the JS expressions themselves feel like geometry shaders. The result feeds into one or more styled drawing ops. The number of emit calls has to be constant, but you can always knock out or mask the excess geometry.

emit = switch channels
  when 1 then (x) ->
    array[i++] = x
    ++j
    return

  when 2 then (x, y) ->
    array[i++] = x
    array[i++] = y
    ++j
    return

  when 3 then (x, y, z) ->
    array[i++] = x
    array[i++] = y
    array[i++] = z
    ++j
    return

  when 4 then (x, y, z, w) ->
    array[i++] = x
    array[i++] = y
    array[i++] = z
    array[i++] = w
    ++j
    return

Both the expression and emitter will be inlined into the stream's iteration loop.

consume = switch channels
  when 1 then (emit) ->
    emit array[i++]
    ++j
    return

  when 2 then (emit) ->
    emit array[i++], array[i++]
    ++j
    return

  when 3 then (emit) ->
    emit array[i++], array[i++], array[i++]
    ++j
    return

  when 4 then (emit) ->
    emit array[i++], array[i++], array[i++], array[i++]
    ++j
    return

Closures of Hanoi

(4-in-1)²

GPUs can operate on 4×1 vectors and 4×4 matrices, so working with 4D values is natural. Values can also be referenced by 4D indices. With one dimension reserved for the tuples, that leaves us 3 dimensions XYZ. Hence MathBox arrays are 3+1D. This is for width, height, depth, while the tuple dimension is called items. It does what it says on the tin, creating 1D W, 2D W×H and 3D W×H×D arrays of tuples. Each tuple is made of N vectors of up to 4 channels each.

Thanks to cyclic buffers and partial updates, history also comes baked in. You can use a spare dimension as a free time axis, retaining samples on the go. You can .set('history', N) to record a short log of a whole array over time, indefinitely.

All of this is modular: a data source is something that can be sampled by a 4D pointer from GLSL. Underneath, arrays end up packed into a regular 2D float texture, with "items × width" horizontally and "height × depth" vertically. Each 'pixel' holds a 1/2/3/4D point.

Mapping a 4D 'pointer' to the real 2D UV coordinates is just arithmetic, and so are operators like transpose and repeat. You just swap the XY indices and tell everyone downstream that it's now this big instead. They can't tell the difference.

You can create giant procedural arrays this way, including across rectangular texture size limits, as none of them actually exist except as transient values deep inside a GPU core. Until you materialize them by rendering to a texture using the memo primitive. Add in operators like interpolation and convolution and it's a pretty neat real-time finishing kit for data.

Continued in Part 2.

MathBox²

2015-09-27T00:00:00+02:00

Part 2

Continued from Part 1.

I-Can't-Believe-It's-Not-React

Underneath sits a large codebase driving it, 200+ files in JS/CS alone, not including dependencies that aren't my own. Much of it is infrastructure necessary to pull off certain tricks consistently: you can draw 2.5D lines with grace, render arbitrary Unicode text in GL, sync HTML to GPU-computed geometry, and do all this with GLSL code composed on the fly, including your own. Nobody needs all of this.

These wildly different strategies are actually all abstracted into the DOM path or the GL path, with a quacks-like-React component as the glue in the DOM path.

...

// Define VDOM handler to clone real DOM elements
var clone = MathBox.DOM.createClass({
  render: function (el, props, children) {
    var element = children.cloneNode(true);
    return element;
  },
});

// Define VDOM handler to format 'latex' into an HTML span
var latex = MathBox.DOM.createClass({
  render: function (el) {
    this.props.innerHTML = katex.renderToString(this.children);
    return el('span', this.props);
  }
});

The interface with the HTML DOM.

Appearances deceive however,
as MathBox's own DOM is an entirely different beast.

Most of the code is for initialization only, building up a reactive machine by combining components. Once assembled it lets the GPU do most of the crunching, while relying on the JS VM to inline and optimize the chewy outside.

At the top level, MathBox is plain old JavaScript, used like this:

/* Easy Mode */

// Bootstrap MathBox and Three.js
var mathbox = mathBox();
if (mathbox.fallback) { throw Error("No WebGL support.") };

// Make MathBox primitives
var view =
  mathbox
  .set({
    scale: null,
  })
  .camera({
    proxy: true,
    position: [0, 0, 3]
  })
  .polar({
    range: [[-2, 2], [-1, 1], [-1, 1]],
    scale: [2, 1, 1],
    bend: .25
  });

view.interval({
  width: 48,
  expr: function (emit, x, i, t) {
    // Emit sine wave
    y = Math.sin(x + t / 4) * .5 + .75;
    emit(x, y);
  },
  channels: 2,
})
.line({
  color: 0x30C0FF,
  width: 16,
})
.resample({
  width: 8,
})
.point({
  color: 0x30C0FF,
  size: 60,
})
.html({
  width:  8,
  expr: function (emit, el, i, j, k, l, t) {
    // Emit random latex
    var color = ['#30D0FF','#30A0FF'][i%2];
    var a = Math.round(t + i) % symbols.length;
    var b = Math.round(Math.sqrt(t * t + Math.sin(t + i * i) + 5));
    emit(el(latex, {style: {color: color}},
      '\\sqrt{\\text{LaTeX} + '+(i + b)+' \\pi^{'+symbols[a]+'}}'));
  },
})
.dom({
  snap: false,
  offset: [0, 32],
  depth: 0,
  zoom: 1,
  outline: 2,
  size: 20,
});

// ...

The WebGL Canvas bootstrapper is a separate piece though, it's wrapping Threestrap, the little non-framework that could. It lets you spawn a fully functioning GL canvas in one exceedingly configurable call. It takes care of browser support, resizing with retina, CSS alignment, warm-up and more.

If you prefer instead, you can spawn a bare MathBox context in a Three.js scene of your choosing, but you'll have to babysit it:

/* Simple Mode */

// Vanilla Three.js
var renderer = new THREE.WebGLRenderer();
var scene = new THREE.Scene();
var camera = new THREE.PerspectiveCamera(60, WIDTH / HEIGHT, .01, 1000);

// Insert into document
document.body.appendChild(renderer.domElement);

// MathBox context
var context = new MathBox.Context(renderer, scene, camera).init();
var mathbox = context.api;

// Set size
renderer.setSize(WIDTH, HEIGHT);
context.resize({ viewWidth: WIDTH, viewHeight: HEIGHT });

// Place camera and set background
camera.position.set(0, 0, 3);
renderer.setClearColor(new THREE.Color(0xFFFFFF), 1.0);

// MathBox elements
view = mathbox
.set({
  focus: 3,
})
.cartesian({
  range: [[-2, 2], [-1, 1], [-1, 1]],
  scale: [2, 1, 1],
});
// ...

// Render frames
var frame = function () {
  requestAnimationFrame(frame);
  context.frame();
  renderer.render(scene, camera);
};
requestAnimationFrame(frame);

Despite looking like a monolith, it really isn't, it was merely a matter of convenience and sanity to not decouple it more until its shape had stabilized. Minimal builds are, for now, left as an exercise to the reader. I've split up the thinking and design into several articles, mirroring the architecture. However, you don't need to know all this to use MathBox 2, they are for people who want to know the how and why... the document model, the geometry core and the shader assembly.

In putting it all together, the devil's in the details of course. Depending on your imagination, it's either much more or much less powerful than you want. There's still far too much to cover: slideshows, keyframe tracks, fov-calibrated units, z-indexes, atlas retexting, … Most of this is unsurprising in that it all works. You can define a keyframe interpolation between two value or emitter expressions, and watch the smoothly lerped data go. Animation tracks are tied to triggers like clocks and slides, which lets them fit naturally in presentations.


  
  index={1}>
    
    steps={2}>
      
      
        
        
        
        
        
        script={[
            {props: {color: 'red'}},
            {props: {color: 'blue'}},
          ]} >
      
    
    
    
      
      
        
        
        
        script={[
                       {props: {expr: (emit, x) => emit(x, Math.sin(x))}},
                       {props: {expr: (emit, x) => emit(x, Math.tan(x))}},
                      ]} />

One More Thing…

Images are data. So is audio. That means MathBox 2 is Winamp AVS, Milkdrop and the mythical Fridge all rolled into one. You can replicate your everyday trippy music visualizer with two operators: render-to-texture (rtt) and compose. It acts as an embedded scene, rendering all of its children to an off-screen image, while Compose renders a full-screen pass. This is where the model's expressiveness shines.

Milkdrop equals mathbox.rtt(…).compose(…).….end().compose(…), that is, an image feeding back into itself, but also rendered to the screen. The necessary double buffering and swaps are abstracted away. Drop in shapes and shaders, add transforms, nest as you like. RTTs have a history parameter like arrays, so Turing patterns, self-propagating hypno spirals, and other cool partial diffy eqs are a shader away.

id="1" scale={720}>
  id="render" minFilter="nearest" magFilter="nearest" type="unsignedByte">
    id="3" lookAt={[0, 0, 0]} position=>{(t) => { return [Math.cos(t) * 3, 0, Math.sin(t) * 3] }} />
    id="4" range={[[-2, 2], [-1, 1], [-1, 1]]} scale={[2, 1, 1]}>
      id="5" scale={[7/10, 7/10, 7/10]}>
        id="6" width={5} divideX={2} divideY={2} zBias={10} opacity={1/4} color={16768992} />
      >
    >
  >
  id="rtt1" history={4} type="unsignedByte" minFilter="linear" magFilter="linear">
    id="8" code="
uniform vec3 dataResolution;
uniform vec3 dataSize;
uniform float cosine;
uniform float sine;
vec4 getSample(vec3 xyz);
vec4 getFramesSample(vec3 xyz) {
  vec2 pos = xyz.xy * dataResolution.xy - .5;
  pos = ((pos * dataSize.xy) * mat2(cosine, sine, -sine, cosine) * .999) / dataSize.xy;
  xyz.xy = (pos + .5) * dataSize.xy;
  vec4 c = getSample(xyz + vec3( 0.0, 0.0, 1.0));
  vec3 t = getSample(xyz + vec3( 0.0, 1.5, 0.0)).xyz;
  vec3 b = getSample(xyz + vec3( 0.0,-1.5, 0.0)).xyz;
  vec3 l = getSample(xyz + vec3(-1.5, 0.0, 0.0)).xyz;
  vec3 r = getSample(xyz + vec3( 1.5, 0.0, 0.0)).xyz;
  return vec4((t + b + l + r) / 2.0 - c.xyz, c.w);
}"
 cosine=>{(t) => Math.cos(Math.sin(t * .2) * .005)} sine=>{(t) => Math.sin(Math.sin(t * .2) * .005)} />
    id="resample1" indices={3} channels={4} />
    id="10" />
    id="11" source="#render" blending="add" />
  >
  id="12" minFilter="linear" magFilter="linear" type="unsignedByte">
    id="colormap" code="
uniform float modulate1;
uniform float modulate2;
uniform float modulate3;
uniform float modulate4;
vec4 getSample(vec3 xyz);
vec4 getFramesSample(vec3 xyz) {
  vec4 color = (
    getSample(xyz) +
    getSample(xyz + vec3(0.0, 0.0, 1.0)) +
    getSample(xyz + vec3(0.0, 0.0, 2.0)) +
    getSample(xyz + vec3(0.0, 0.0, 3.0))
  ) / 4.0;
  color = color * color * color * 1.15;
  float v = color.x + color.y + color.z;
  vec3 c = vec3(v*v + color.x * .2, v*v, v*v*v + color.z) * .333;
  c = mix(c, mix(sqrt(c.yzx * c), c.zxy, modulate1), modulate2);
  c = mix(c, mix(c.yzx, c.zxy, modulate1), modulate2);
  c = mix(c, mix(abs(sin(c.yxz * 2.0)), c.zyx, modulate3), modulate4);
  return vec4(c, 1.0);
}"
 modulate1=>{(t) => Math.cos(t * .417) * .5 + .5} modulate2=>{(t) => Math.cos(t * .617 + Math.sin(t * .133)) * .5 + .5} modulate3=>{(t) => Math.cos(t * .217 + 2.0) * .5 + .5} modulate4=>{(t) => Math.cos(t * .117 + 3.0 + Math.sin(t * .133)) * .5 + .5} />
    id="resample2" source="#rtt1" indices={3} channels={4} />
    id="15" />
  >
  id="16" code="
vec4 getSample(vec2 xy);
vec4 getFramesSample(vec2 xy) {
  return getSample(xy + vec2(0.5, 0.5));
}"
 />
  id="resample3" indices={2} channels={4} />
  id="18" source="#resample2" />
>

The difference compared to AVS is that the .rtt() is inert to its container by default. Until you add on a .compose() pass, it's just a dangling data source. Meanwhile the .compose() op offers the necessary GL blend modes, opacity and color tints through style properties. Document order defines drawing order, so the decomposition into render passes is direct. On top, zOrder can be overridden (drawing order), as can zIndex (2D stacking order) and zBias (3D stacking order).

You can nest effects and compose shaders to create recursive visualizations, sampling from themselves or each other:

Open in New Window
(Burn that fillrate, baby.)

Bonus: Endless Visualizer.

id="1" scale={720}>
  id="2" proxy={true} position={[3/10, 1/10, 2]} />
  id="3">
    id="audioTime" data={[]} width={1024} channels={1} />
    id="audioFreq" data={[]} width={512} channels={1} />
  >
  id="render" width={256} height={144} type="unsignedByte" minFilter="nearest" magFilter="nearest">
    id="7" position={[0, 0, 5/2]} />
    id="8">
      id="9" source="#audioTime" order="yx" />
      id="10" width={[861/250, 0, 0, 0]} />
      id="11" code="
vec4 getSample(vec4 xyzw);
vec4 getColor(vec4 xyzw) {
  float h = getSample(xyzw).y;
  return vec4(vec3(h), 1.0);
}"
 />
      id="12" />
      id="13" scale={[1, 3/4, 1]}>
        id="14" points="<<" colors="<" width={5} color={16777215} opacity={2/5} blending="add" />
      >
    >
    id="15" range={[[-2, 2], [-1, 1], [-1, 1]]} scale={[1/2, 1/4, 1/4]} quaternion=>{(t) => {
          c = Math.cos(t / 3);
          s = Math.sin(t / 3);
          c2 = Math.cos(t / 8.71);
          s2 = Math.sin(t / 8.71);
          return [s * s2, s * c2, .2, c];
        }}>
      id="16" divideX={4} divideY={4} zBias={10} opacity={1/10} color={16768992} width={6} />
    >
  >
  id="rtt1" history={4} width={256} height={144} type="unsignedByte">
    id="18" code="#map-rotate" />
    id="resample1" indices={3} channels={4} />
    id="20" color="#ffffff" zWrite={false} />
    id="21" source="#render" blending="add" color="#ffffff" zWrite={false} />
  >
  id="rtt2" width={256} height={144} type="float">
    id="23" position={[0, 0, 5/2]} />
    id="24" seek=>{(t) => audio ? audio.currentTime : t}>
      id="25" code="#map-temporal-blur" time=>{(t) => t * 16.0} modulate=>{(t) => {
            var bang = ((t > 69.229311)  && (t < 88.922656)) ||
                       ((t > 88.922656)  && (t < 148.9143)) ||
                       ((t > 148.9143)   && (t < 158.2)) ||
                       ((t > 168.284427) && (t < 188.00)) ? 1 : 0;
            if ((t > 88.922656)  && (t < 148.9143)) {
              bang *= .5 + .45 * Math.cos(t / 3);
            }
            if ((t > 168)) {
              bang *= .85 + .15 * Math.cos(t);
            }
            modulate = modulate + (bang - modulate) * .1;
            return modulate;
          }} pattern=>{(t) => {
            var bang = ((t > 88.922656) && (t < 148.9143));
            pattern = pattern + (bang - pattern) * .1;
            if ((t > 168)) {
              pattern = .5 + .4 * Math.cos(t * 2.311);
            }
            return pattern;
          }} warp=>{(t) => {
            var bang = (t > 148.9143);
            if ((t > 168)) {
              warp *= 1 + .5 * Math.cos(t * .556);
            }
            if ((t > 148.2) && (t < 158.2)) warp = warp + .75 + .25 * Math.cos((t - 158.2));
            warp = warp + (bang - warp) * .1;
            return warp;
          }} shift=>{(t) => {
            var bang = (t > 168) ? Math.max(0, Math.min(1, .1 * (t - 168))) : 0;
            bang *= .75 + .25 * Math.cos(t * .731);
            warp = warp + (bang - warp) * .1;
            return warp;
          }} />
      id="resample2" source="#rtt1" indices={3} channels={4} />
      id="27" color="#fff" zWrite={false} />
    >
    id="28" scale={[1, 1/4, 1]}>
      id="29" source="#audioTime" order="yx" />
      id="30" width={[861/250, 0, 0, 0]} />
      id="31" code="
vec4 getSample(vec4 xyzw);
vec4 getColor(vec4 xyzw) {
  float h = getSample(xyzw).y;
  return vec4(vec3(h) * .2, 1.0);
}"
 />
      id="32" />
      id="33" points="<<" colors="<" width={50} color={16777215} opacity={1} blending="add" />
    >
  >
  id="34" width={129} height={73} />
  id="lerp" depth={2} />
  id="36" code="#map-xy-to-xyz" />
  id="37" indices={3} channels={3} />
  id="transpose" order="xywz" />
  id="color" source="#lerp" order="xywz" />
  id="40" seek=>{(t) => audio ? audio.currentTime : t}>
    id="disco" speed=>{(t) => {
        var bang = ((t > 69.329311)  && (t < 89.122656)) ||
                   ((t > 148.9143)   && (t < 158.0)) ||
                   ((t > 168.284427) && (t < 188.077772));
        return bang ? 1 : .2;
      }}>
      id="42" code="#map-z-to-color" modulate1=>{(t) => Math.cos((t + 1) * .417) * .5 + .5} modulate2=>{(t) => Math.cos((t + 1) * .617 + Math.sin(t * .133)) * .5 + .5} modulate3=>{(t) => Math.cos((t + 1) * .217 + 2.0) * .5 + .5} modulate4=>{(t) => Math.cos((t + 1) * .117 + 3.0 + Math.sin(t * .133)) * .5 + .5} />
      id="color1" source="#lerp" indices={2} channels={4} />
      id="44" code="#map-z-to-color-2" modulate1=>{(t) => Math.cos((t + 1) * .417) * .5 + .5} modulate2=>{(t) => Math.cos((t + 1) * .617 + Math.sin(t * .133)) * .5 + .5} modulate3=>{(t) => Math.cos((t + 1) * .217 + 2.0) * .5 + .5} modulate4=>{(t) => Math.cos((t + 1) * .117 + 3.0 + Math.sin(t * .133)) * .5 + .5} />
      id="color2" source="#lerp" indices={2} channels={4} />
    >
    id="46" range={[[-1.7788, 1.7788], [-1, 1], [-1, 1]]} scale={[16/9, 1, 1]} quaternion=>{(t) => {
        t = t / 3;
        c = Math.cos(t / 4);
        s = Math.sin(t / 4);
        c2 = Math.cos(t / 11.71) * 1.71;
        s2 = Math.sin(t / 11.71) * 1.71;
        return [s * s2, s * c2, -.2, c];
      }}>
      id="47" source="#transpose" width={33} height={19} />
      id="48" source="#color2" width={33} height={19} />
      id="49">
        id="50" points="<<" colors="<" color="#ffffff" width={2} zBias={5} />
      >
      id="51" script={{19: {position: [0, 0, 0]}, 39: {position: [0, 0, 2]}, 57: {position: [0, 0, 0]}}} />
      id="52" source="<<" order="yxzw" />
      id="53" source="<<" order="yxzw" />
      id="54">
        id="55" points="<<" colors="<" color="#ffffff" width={2} zBias={5} />
      >
      id="56" script={{19: {position: [0, 0, 0]}, 39: {position: [0, 0, -2]}, 57: {position: [0, 0, 0]}}} />
      id="57">
        id="58" points="<<" colors="<" color="#ffffff" size={10} zBias={5} zOrder={1} blending="add" zWrite={false} />
      >
      id="59" script={{19: {position: [0, 0, 0]}, 39: {position: [0, 0, -1]}, 57: {position: [0, 0, 0]}}} />
      id="60">
        id="61" points="#transpose" colors="#color2" color="#ffffff" size={5} zBias={5} zOrder={1} blending="add" zWrite={false} />
      >
      id="62" script={{9: {position: [0, 0, 0]}, 39: {position: [0, 0, 1]}, 57: {position: [0, 0, 0]}}} />
      id="63" points="#transpose" colors="#color1" color="#ffffff" start={false} end={false} width={40} opacity={3/100} blending="add" zWrite={false} zOrder={-2} />
    >
  >
>

Full Stack

What's left is basically kicking the tires and fixing the blind spots. As such this is not MathBox 2.0, this is MathBox 2 Alpha 1. It's still rough in the compatibility department, easily letting you exceed GL limits satisfied by only 70-80% of WebGL implementations in the wild, without warning. My own goal for public release was to be able to make another one of those presentations with it, only this time, 100% idiosyncratic MathBox. Result, The Pixel Factory.

Some people have assumed this talk was another tour-de-force of multi-week autism, but in fact, rebuilding my old slides for v2 was easy and obvious. The RGBA subpixels and their labels are animated lambdas and GLSL. The multi-samples, the depth buffer columns, the tangents and normals, same thing. JavaScript twiddles the knobs while the GPU visualizes the visualizer, and in doing so, itself.

Here the code be.

To give it a whirl in your browser, open the JSBin Sandbox. There is a quick start introduction and a list of legos.

A DOM for Robots

2015-09-27T00:00:00+02:00

Modelling Live Data

I want to render live 3D graphics based on a declarative data model. That means a choice of shapes and transforms, as well as data sources and formats. I also want to combine them and make live changes. Which sounds kind of DOMmy.

3D engines don't have Document Object Models though, they have scene graphs and render trees: minimal data structures optimized for rendering output. In Three.js, each tree node is a JS object with properties and children like itself. Composition only exists in a limited form, with a parent's matrix and visibility combining with that of its children. There is no fancy data binding: the renderer loops over the visible tree leaves every frame, passing in values directly to GL calls. Any geometry is uploaded once to GPU memory and cached. If you put in new parameters or data, it will be used to produce the next frame automatically, aside from a needsUpdate bit here and there for performance reasons.

So Three.js is a thin retained mode layer on top of an immediate mode API. It makes it trivial to draw the same thing over and over again in various configurations. That won't do, I want to draw dynamic things with the same ease. I need a richer model, which means wrapping another retained mode layer around it. That could mean observables, data binding, tree diffing, immutable data, and all the other fun stuff nobody can agree on.

However I mostly feed data in and many parameters will end up as shader properties. These are passed to Three as a dictionary of { type: '…', value: x } objects, each holding a single parameter. Any code that holds a reference to the dictionary will see the same value, as such it acts as a register: you can share it, transparently binding one value to N shaders. This way a single .set('color', 'blue') call on the fringes can instantly affect data structures deep inside the WebGLRenderer, without actually cascading through.

I applied this to build a view tree which retains this property, storing all attributes as shareable registers. The Three.js scene graph is reduced to a single layer of THREE.Mesh objects, flattening the hierarchy. Rather than clumsy CSS3D divs which encode matrices as strings, there's binary arrays, GLSL shaders, and highly optimizable JS lambdas.

As long as you don't go overboard with the numbers, it runs fine even on mobile.

id="1" scale={600} focus={3}>
  id="2" proxy={true} position={[0, 0, 3]} />
  id="3" code="
uniform float time;
uniform float intensity;

vec4 warpVertex(vec4 xyzw, inout vec4 stpq) {
  xyzw +=   0.2 * intensity * (sin(xyzw.yzwx * 1.91 + time + sin(xyzw.wxyz * 1.74 + time)));
  xyzw +=   0.1 * intensity * (sin(xyzw.yzwx * 4.03 + time + sin(xyzw.wxyz * 2.74 + time)));
  xyzw +=  0.05 * intensity * (sin(xyzw.yzwx * 8.39 + time + sin(xyzw.wxyz * 4.18 + time)));
  xyzw += 0.025 * intensity * (sin(xyzw.yzwx * 15.1 + time + sin(xyzw.wxyz * 9.18 + time)));

  return xyzw;
}"
 time=>{(t) => t / 4} intensity=>{(t) => {
        t = t / 4;
        intensity = .5 + .5 * Math.cos(t / 3);
        intensity = 1.0 - Math.pow(intensity, 4);
        return intensity * 2.5;
      }} />
  id="4" stagger={[10, 0, 0, 0]} enter=>{(t) => 1.0 - Math.pow(1.0 - Math.min(1,  (1 + pingpong(t))*2), 2)} exit=>{(t) => 1.0 - Math.pow(1.0 - Math.min(1,  (1 - pingpong(t))*2), 2)}>
    id="5" pass="view">
      id="6" bend={1/4} range={[[-π, π], [0, 1], [-1, 1]]} scale={[2, 1, 1]}>
        id="7" position={[0, 1/2, 0]}>
          id="8" detail={512} />
          id="9" divide={10} unit={π} base={2} />
          id="10" width={3} classes=["foo", "bar"] />
          id="11" divide={5} unit={π} base={2} />
          id="12" expr={(x) => {
        return x ? (x / π).toPrecision(2) + 'π' : 0
      }} />
          id="13" depth={1/2} zIndex={1} />
        >
        id="14" axis={2} detail={128} crossed={true} />
        id="15" position={[π/2, 0, 0]}>
          id="16" axis={2} detail={128} crossed={true} />
        >
        id="17" position={[-π/2, 0, 0]}>
          id="18" axis={2} detail={128} crossed={true} />
        >
        id="19" divideX={40} detailX={512} divideY={20} detailY={128} width={1} opacity={1/2} unitX={π} baseX={2} zBias={-5} />
        id="20" width={512} expr={(emit, x, i, t) => {
        emit(x, .5 + .25 * Math.sin(x + t) + .25 * Math.sin(x * 1.91 + t * 1.81));
      }} channels={2} />
        id="21" width={5} />
        id="22" pace={10} loop={true} to={3} script=[[{color: "rgb(48, 144, 255)"}], [{color: "rgb(100, 180, 60)"}], [{color: "rgb(240, 20, 40)"}], [{color: "rgb(48, 144, 255)"}]] />
      >
    >
  >
>

Note: The JSX is a lie, you define nodes in pure JS.

Keep it Simple

From afar there's a tree of nodes, similar to SVG tags. This is the MathBox library of vector primitives. The basic shapes are all there: points, lines, faces, vectors, surfaces, etc. These nodes are placed inside a shallow hierarchy of views and transforms.

However none of the shapes draw anything by themselves. They only know how to draw data supplied by a linked source. Data can be an array (static or live), a procedural source, custom JS / GLSL code, etc. This is further augmented by data operators which can be sandwiched between source and shape, forming automatic pipelines between siblings.

The current set of components looks like this:

Base

Group
Inherit
Root
Unit

Camera

Camera

Draw

Axis
Face
Grid
Line
Point
Strip
Surface
Ticks
Vector

Data

Area
Array
Interval
Matrix
Scale
Volume
Voxel

Operator

Grow
Join
Lerp
Memo
Resample
Repeat
Slice
Split
Spread
Swizzle
Transpose

Overlay

DOM
HTML

Present

Move
Play
Present
Reveal
Slide
Step

RTT

Compose
RTT

Shader

Shader

Text

Format
Label
Text
Retext

Time

Clock
Now

Transform

Fragment
Layer
Transform
Transform4
Vertex

View

Cartesian
Cartesian4
Polar
Spherical
Stereographic
Stereographic4
View

To make you feel at home, nodes have an id and classes, and you can use CSS selectors to identify them. Nodes link up with preceding siblings and parents by default, but you can select any node in the tree. This allows for arbitrary graphs, including feedback loops. However all of this is optional: you can also pass in direct node objects or MathBox's own jQuery-like selections. What it doesn't have is a notion of detached document fragments: nodes are immediately inserted on creation.

A node's attributes can be .get() and .set(), though there is also a read-only .props dictionary for fashionable reasons. The values are strongly typed as Three.js colors, vectors, matrices, … but accept e.g. CSS colors and ordinary arrays too. The values are normalized for immediate use, the original values are preserved on the side for printing and serialization.

What's unique is the emphasis on time. First, properties can be directly bound to time-dependent expressions, on creation or afterwards. Second, clocks are primitives on their own. This allows for nested timelines, on-demand bullet time, fast forwards and more. It even supports limited time travel, evaluating an expression several frames in the past. This can be used to ensure consistent 60 fps data logging through janky updates, useful for all sorts of things. It's exposed publicly as .bind(key, expr) and .evaluate(key, time) per node. It's also dogfood for declarative animation tracks. The primitives clock/now provide timing, while step and play handle keyframes on tracks.

This is definitely a DOM, but it has only basic features in common with the HTML DOM and does much less. Most of the magic comes from the components themselves. There's no cascade of styles to inherit. Children compose with a parent, they do not inherit from it, only caring about their own attributes. The namespace is clean, with no weird combo styles à la CSS. As much as possible, attributes are unique orthogonal knobs you can turn freely.

Model-View-Projection

On the inside I separate the generic data model from the type-specific View Controller attached to it. The controller's job is to create and manage Three.js objects to display the node (if any). Because a data source and a visible shape have very little in common, the nodes and their controllers are blank slates built and organized around named traits. Each trait is a data mix-in, with associated attributes and helpers for common behavior. Primitives with the same traits can be expected to work the same, as their public facing models are identical.

Controllers can traverse the graph to find each other by matching traits, listening for events and making calls in response. This way only specific events will cascade through cause and effect, often skipping large parts of the hierarchy. The only way to do a "global style recalculation" would be to send a forced change event to every single controller, and there's never a reason to do so.

The controller lifecycle is deliberately kept simple: make(), made(), change(…), unmake(), unmade(). When a model changes, its controller either updates in place, or rebuilds itself, doing an unmake/make cycle. The change handler is invoked on creation as well, to encourage stateless updates. It affords live editing of anything, without having to micro-optimize every possible change scenario. Controllers can also watch bound selectors, retargeting if their matched set changes. This lets primitives link up with elements that have yet to be inserted.

Unlike HTML, the DOM is not forced to contain a render tree as well. Only some of the leaf nodes have styles and create renderables. Siblings and parents are called upon to help, but the effects don't have to be strictly hierarchical. For example, a visual effect can wrap a single leaf but still be applied after all its parents, as transformations are collected and composed in passes.

It'll Do

The result is not so much a document model as it is a computational model inside a presentational model. You can feed it finalized data and draw it directly… or you can build new models within it and reproject them live. Memoization enables feedback and meta-visualization. The line between data viz and demo scene is rarely this blurry.

Here, the notion of a computed style has little meaning. Any value will end up being transformed and processed in arbitrary ways down the pipe. As I've tried to explain before, the kinds of things people do with getComputedStyle() and getClientBoundingRect() are better achieved by having an extensible layout model, one that affords custom constraints and composition on an equal footing. To do otherwise is to admit defeat and embrace a leaky abstraction by design.

The shallow hierarchy with composition between siblings is particularly appealing to me, even if I realize it introduces non-traditional semantics more reminiscent of a command-line. It acts as both a jQuery-style chainable API, and a minimal document model. If it offends your sensibilities, you could always defuse the magic by explicitly wiring up every relationship. In case of confusion, .inspect() will log syntax highlighted JSX, while .debug() will draw the underlying shader graphs.

I've defined a good set of basic primitives and iterated on them a few times. But how to implement it, when WebGL doesn't even fully cover OpenGL ES 2?

MathBox² - PowerPoint Must Die
A DOM for Robots - Modelling Live Data
Yak Shading - Data Driven Geometry
ShaderGraph 2 - Functional GLSL

Zero to Sixty in One Second

2013-08-23T00:00:00+02:00

Fusing WebGL, CSS 3D and HTML

Ladies and gentlemen, this is your captain speaking. Today's flight on WebGL Air will take us high above the cloud. Those of you sitting in Chrome class, on the desktop side of the plane, will have the clearest view. Internet Explorer class passengers may turn to their in-seat entertainment system instead. Passengers are reminded to put away their iPads and iPhones during take off and landing.

Edit: You can now also watch the YouTube video kindly provided by Leland Batey.

Acko.net, the domain, just turned 13, so it's time for a birthday present in the form of a complete front-end rewrite. The last design was entirely based on CSS 3D: it was a fun experiment, but created at a time when browser implementations were still wonky. It ultimately proved impractical: the DOM is not a good place to store complicated geometry. It's too bulky and there is a huge difference between a styled

and a shaded quad. After adding WebGL to the mix with MathBox and typesetting with MathJax, it turned into a catastrophic worst case for loading, and the smoothness was often lost even on fast computers.

Since then, we've seen a big push for rendering performance, both from the native and JS side. Hardware-accelerated DOM compositing is better understood, requestAnimationFrame is now common-place and reliable, and we have excellent profiling tools down to the frame level.

Hence the goal: to fuse 3D elements into the page like before, but with full 60 fps rendering. Plus to use WebGL instead of CSS 3D where possible, and be free of the constraints of the DOM.

Like Voodoo.js I use a fixed full-screen canvas and sync up scrolling with a 3D camera. The scene is mapped to CSS pixels and CSS perspective is locked to the camera. Once HTML, CSS 3D and WebGL are all in sync there's a truckload of linear algebra and easing functions to keep you amused. The code is based on the platform I kludged together for the christmas demo, at times a mess of ad hoc demo formulas and spaghetti, though robust enough in the parts that count.

Daim

Procedural Wildstyle

The so Designers Republic kitsch of last time was starting to grate, and I wanted something more stylish. Taking inspiration from street art and graffiti, in particular long-time favorite Daim, Acko's gone wild, though with a math twist. This design was built procedurally using some nifty vector calculus and more difference equations than you can shake a stick at.

At its heart it's really a game of Snake, albeit a complicated one. Starting from a line-based skeleton of the model, the curves are traced out, smoothed, oriented and finally extruded on the fly into ribbons. The final pose shows 261 lines mapped to 2,783 curve segments, tessellated into 43,168 triangles, though that amount is just a knob that can be tuned. In other words, it's a scalable vector graphic. No images were harmed in the making of this header, or the rest of the window dressing for that matter. These pixels are local, organic and bespoke, though I'm pretty sure they're not vegan.

The ribbons need to animate, so I'm using a slightly exotic set up: a single Three.js BufferGeometry mesh is created with all the ribbons' topology in it. It only contains the connectivity of the triangles that span the vertices, stored as indices. The actual positions and normals are read from a texture that is updated on the fly by JavaScript using gl.texSubImage2D.

As a ribbon follows its path, only its head and tail change, growing new segments or collapsing older ones. Each ribbon scrolls through the texture, which alters only a fraction of the total data per frame. Both the head and tail can add and remove segments at will, which allows me to vary the geometrical detail along each path. As long as the number of drawn segments at any given time never exceeds the texture height, a ribbon could in theory follow an infinitely long trail, wrapping around the edge endlessly. They're basically semi-virtualized meshes, allocated out of a fixed memory space. The arrows are an easy upgrade: I allocate a couple additional segments at the end and shape them to the right widths.

To draw a ribbon, I draw one contiguous subset of the total mesh (or two, if wrapping around the edge) which maps into a rectangular area of the texture. Ribbon color is stored statically as a vertex attribute, which means the entire hero piece can be drawn using a single shader, texture and vertex buffer.

For the background, I added a static BufferGeometry clocking in at 3,344 triangles. It consists of rounded boxes laid out in a randomized hyperboloid pattern. Aside from breaking the uniform whiteness, it's also quite the parallax enhancer down here.

DJ Ambient

All of this is shaded with some custom GLSL. I use a basic Blinn-Phong lighting model tweaked for aesthetics, and there's some edge highlights and fog to top it off. But if I left it there, the result would still look pretty flat, without shadows. A common approach is to use shadow mapping, but that only works for point sources like the sun. The diffuse shadows that you see on an overcast day require an entirely different approach.

What's needed is ambient occlusion, a measure of how much skylight is obscured by the surrounding geometry at every point. The less sky you can see from a point, the darker it should be. This is slow to compute analytically for large scenes and hence is typically either faked or baked in. An easy hack is Screen-Space AO, which is really just crease darkening using the final image's depth buffer. It's clever but expensive, creating only local shadows. Quality can vary a lot between implementations, often creating distracting dark or light halos around silhouettes. They all share the same blind spot: only the currently visible pixels can cast any shadow. So while I added alteredq's SSAO shader, it's disabled by default and only rendered at quarter resolution, upscaled with a bilateral filter to hide this fact.

Instead, I used a technique from NVidia's GPU Gems 2. It uses a disc-based model to calculate occlusion and even indirect lighting on the fly. Rather than go real-time on the GPU, I decided to only do it statically in JS, and just do plain occlusion without light bounces.

The discs for all the ribbons are generated ahead of time, and the disc-to-disc shadowing is done once for the final model in two passes. The first pass overestimates due to overlapping shadows and thus is too dark. The second pass uses the first to assign lesser weights to discs in shadow to compensate, then runs the algorithm anew.

There's another simplification: instead of small one-sided discs based on the real geometry, I use large two-sided discs and treat each ribbon as flat. Each disc occludes in both directions, but accumulates shadow on its front and back separately. This gives a decent approximation of light around and across the entire object.

Below is the actual disc model. In this diagram, the two radii of each disc represent its front and back illumination, with smaller discs receiving more shadow. The crosshairs mark the real disc radius.

When streaming out new pieces of ribbon, the occlusion is baked in on the fly, stored in the unused alpha channel next to the position (RGB). The four nearest discs on the ribbon are interpolated bilinearly, and the vertex's normal and position is used to mix the front and back shadow values appropriately. Thus it's effectively trilinear filtering a 2×2×N voxel grid per ribbon, which snakes and twists along its length.

I'm very happy with how good it looks, even with very coarse divisions like this. Unlike SSAO it costs practically nothing once generated. If you wish, you can view the design entirely in white to examine the lighting—or in black and white, or in the seventies, tone mapping is fun.

To ensure smooth rendering, the resolution is scaled down if it drops below 45 fps for several frames. This is essential on slower GPUs, but can sometimes overcompensate, as WebGL is easily interrupted by other tasks on the machine. The browser doesn't tell you how much you're currently pushing it, so it's impossible to safely scale back up without causing more stutters. To mitigate this problem, I built in some strategic reset points, like when you focus the window or scroll to the top. In the end, 60fps happens more often than not, though it remains a goal rather than a guarantee. If you fullscreen your browser, you are now the bane of my existence, especially if you keep two dozen other windows open on the second monitor at the same time. On an underpowered retina MacBook.

Finally I also turned on MultiSample Anti-Aliasing to avoid the dreaded jaggies and make the vector style shine. It's substituted with Fast Approximate AA if it's unavailable. FXAA is also used if SSAO is enabled, as you can't post-process multisampled images with WebGL.

Preparing all of this would take ~400ms on my 4 year old laptop, but most of that is spent on the occlusion which is static. Hence, I saved the lighting data into a JSON file for speedier loading, which cuts it down to ~100ms. The data gzips into 18KB, about the size of a small PNG, so no worries there. Add to that the time required to fetch and run the rest of the page, and we can call it an even second.

Of course, it only works on a subset of browsers and leaves the vast majority of mobile devices out in the cold. It does work in Chrome Beta on Android, though performance and stability is still pretty crap and more fixes are needed, both in the browser and in my own code. Rather than try and emulate some of the bling for CSS 3D-only environments, it's all or nothing. Without WebGL, you get plain images.

Achievement Unlocked

Reading all that, you might mistake the header as a mere love letter to hackery. But there's a twist, quite a literal one. The twisting of each ribbon is not generated arbitrarily, but mathematically derived. It embodies the differential principle of parallel transport. The up direction changes parallel to each curve, which means the ribbons never rotate in place. They only turn when they naturally want to. Hence, the design kind of has a will of its own. I can set the initial up direction of a ribbon, but only affect it through its curved path. Arranging and tuning all the ribbons was a nice puzzle in itself, and it's a nice nod to the math that's all over the place, even if it is invisible.

As I started playing around with some screensaver-style shots, I realized it would make a pretty neat demo just by itself, so I built that in too, to the dulcet tones of Selah Sue—whose last name I hope is not indicative. Here parallel transport would ensure a perfectly smooth ride, but that's not exciting, so there's some springy exponential easing with adaptive lookahead instead. That's on top of the secondary demo, which features an audio visualizer and the smooth drum and bass of Seba. There's maybe a third one too.

The songs are used here entirely for educational purposes of course. Not that it matters, since they're all on YouTube anyway. Click the Growl-like notifications to find out more about the artists. There's also a handful of achievements to be gathered, eight to be precise. Some of these are experiments that were turned off in the final version. Others are... trickier. Oh hey, did you notice the JS console?

By the way, special mention also goes to Time and how some browsers keep terrible track of it. Like when you're trying to sync a demo to a stuttery clock, Firefox ಠ_ಠ. There's also the matter of what to do when the user switches to another tab, and the answer is.... it'll desync, because I don't want to be sapping resources in a background tab. Minor constraints of putting on a live act, doors close after the show starts.

Turn Right Past The Header

The refresh-less navigation is also back from last time, slightly cleaned up. It still works on the same basic principle of fetching static, full HTML documents. However the transition mechanism now carefully choreographs the necessary DOM manipulation to avoid stutters: any image, iframe or video inserted in the content would mean future paints at an unknown time, so they're postponed until after the transition is complete. I did it the dirty but fast way, with string manipulation before it hits the DOM: such is the luxury of being the only one, person or machine, writing the markup.

Which leaves of course the content. But don't fix what ain't broke: distraction-free publishing is easy enough with Jekyll and a responsive 960 pixel grid with sideburns, so things still look and work mostly the same. I only added a home for my math talks and expanded the front page to highlight some demos and experiments. I aim to keep focusing on quality rather than quantity, and to remind visitors what the web looks like when you take away everything social media's done to it.

It's also fun to observe that after many years, the unified front-end convergence really has arrived. There is little difference between this site and the games that use HTML for their UI, such as the latest Sim City. For dealing with typography, illustration and UI, you want the comfort of DOM and CSS. For real-time graphical content, you'll want to draw it yourself, either in 2D, or 3D. Combine the two, and you get what game developers have been doing for years. Only this lacks all the sharp C bits, which game devs been replacing with Lua for years anyhow. That's Portuguese for "JavaScript" by the way.

So there you have it. A fresh look with a pile of juicy hax, and hopefully not too many bugs in the wild. Thanks go to all those who came before and provided all these toys for me to play with. You can too, for the source is entirely on Github. Like I said, educational. Meanwhile I'll be over here, listening to the wipE'out" soundtrack on repeat in memory of the old one. Comments welcome on Google Plus.

PS: The previous version has been archived with its fallbacks disabled so you can see the current state of CSS 3D in browsers. Still pretty broken. Still a great test case. So is the disembodied head.

On WebGL

2013-03-11T00:00:00+01:00

More than pretty pictures

Like a dragon, WebGL slumbers. But you've seen them, right? Those seemingly magical demos that transform your ordinary browser into a lush 3D world with one click?

While available in Chrome and Firefox on the desktop, WebGL is still not widely supported. So far it's mostly used for demo projects and flashy one-off brochures. On the few mobile devices that support it, you need developer access to enable it. It's certainly nowhere near to being ready for prime time. So why should you care?

AlteredQualia

The Black Sheep

The goal of WebGL is to bring the graphics capabilities of traditional apps and games into the browser, with performance as the main benefit. The graphics hardware does the work directly, leaving the CPU to just coordinate. Yet those developers look on with skepticism: "You mean we have to code in JavaScript?" There's grumbling about the limited capabilities too, which lag a few years behind the latest OpenGL and Direct3D APIs, and there's worries about copyright and modding.

First, we have to be honest: there's no question that native apps and 3D engines will continue to excel, bringing cutting edge graphics and performance. The layers of indirection in both HTML5 and WebGL cannot be circumvented.

But they do serve a purpose: to provide a safe sandbox for untrusted code from the web at large. Even triple-A games still occasionally crash, a result of their complexity, with thread synchronization, memory management and manual context-switching the price to pay. Random phishers shouldn't have that level of access to your system, nor should it be required.

AlteredQualia

WebGL represents a different way of using high-performance graphics: not as a bare metal API with caveats, but as a safe service to be exposed, to be flicked on or off without a second thought. It may not sound like much, but the security implications are big and will only be solved carefully, over time. It's undoubtedly a big reason behind Apple and Microsoft's reluctance to embrace it.

We should also note that this isn't a one-way cross-over. HTML has already snuck into the real-time graphics scene. First we saw in-game web views and browsers, then UIs such as Steam's overlay. In fact, all of Steam is WebKit. The main benefit is familiarity: designers can use the well-known techniques of the web both inside and outside the game. This mirrors the way Adobe Flash entered the gaming space before, being used to drive menus and overlays in many games.

It's been said that the skills required for front-end web development and game development eventually converge on the same thing. The technologies certainly have.

Felix Woitzel

The Procedural Canvas

The web is the world's only universal procedural medium. Content is downloaded in partially assembled form, and you and your browser decide how it should be displayed. The procedural aspect has always been there, and today's practice of responsive design is just another evolution in procedural page layout. It all started with resizable windows and tables.

But when we decide to put a graphic into a page, we still bake it into a grid of pixels and send that down the pipe. This has worked great as a delivery mechanism, but is starting to show its age, due to high DPI displays and adaptive streaming.

It's also pushed the web further towards consumption: YouTube and Tumblr are obvious results. Both sites have a huge asymmetry between content creator and consumer, encouraging sharing rather than creating.

Felix Woitzel

Real-time graphics level the playing field: once built, both creator and consumer have the same degree of control—at least in theory. All the work necessary to produce the end result is ideally being done 60 times per second. The experience of e.g. playing a game is like a sort of benign DRM, which requires you to access the content in a certain way. All native apps implement such 'DRM' by accident: their formats are binary and often proprietary, the code is compiled. Usually modding is supported in theory—that's what Downloadable Content is, an official mod—but the tools simply aren't included.

The web is different. No matter how obfuscated, all code eventually has to talk to an interface that is both completely open and introspective. You can hook into any aspect of it and watch the data. There isn't a serious web developer around who would argue that this is a bad thing, who hasn't spent time deconstructing a site through a web inspector on a whim.

Florian Bösch

This is where WebGL gets interesting. It takes the tools normally reserved for well, the hardcore geeks, and makes them much more open and understandable. I can certainly say from experience that coding with an engine like Three.js is an order of magnitude more productive than e.g. Ogre3D in C++. For most of the things I want to do with it, the performance difference is negligible, but there is much less code. Once you get your dev environment going, creating a new 3D scene is as simple as opening a text file. You can interact with your code live through the console for free.

More so, it integrates with the publishing tools we already know. I wonder for example how many hours of dev time the game industry has spent reinventing the wheel for fonts, menus, option screens, etc. To be fair, they often do so with amazing production value. But guess what: you now have CSS 3D, and soon you'll have CSS shaders. You don't need custom in-house tools when your designers can just use Chrome's Inspector and get the exact same result. Content delivery is easy: you have cloud storage, CDNs and memory caches at your disposal.

There is a missing link however: WebGL is a canvas inside the page, isolated from what's outside. But you could imagine APIs to help bring HTML content into a WebGL texture, taking over responsibility for drawing it. After all, most web browsers already use hardware acceleration to compose 2D web pages on screen. The convergence has already started.

Florian Bösch

The web has a history of transformative changes. CSS gave us real web design, Flash gave us ubiquitous video, Firebug gave us Web Inspectors, jQuery gave us non-painful DOM manipulation, and so on. None of these ideas were new in computing when they debuted, the web merely adapted to fill a need. WebGL is an idea in a similar vein, a base platform for an ecosystem of specialized frameworks on top.

It can help lead to a WolframAlpha-ized LCARS future, where graphics can be interactive and introspective by default. Why shouldn't you be able to click on a news graphic to filter the view, or download the dataset? For sure, this is not something that uniquely requires WebGL, and tools like d3.js are already showing the way with CSS and SVG. As a result, the last mile of interactivity becomes a mere afterthought: everything is live anyway. What WebGL does is raise the limit significantly on what sort of content can be displayed in a browser. It's not until those caps are lifted that we can say with a straight face that web apps can rival native apps.

Still, we shouldn't be aiming to recreate Unreal Engine in HTML / JS / GL, though someone will probably try, and eventually succeed. Rather we should explore what happens when you put a 3D engine inside a web page. Is it web publishing, or demoscene? Does it matter?

Ro.me team

Mr.doob

Chrome Workshop

A Useful Baseline

In this light, WebGL's often lamented limitation becomes its strength. WebGL is not modelled after 'grown-up' OpenGL, but mirrors OpenGL ES (Embedded Systems). It's a suite of functionality supported by most mobile devices, but eclipsed by even the crummiest integrated laptop graphics from 3 years ago.

This needn't be a worry for two reasons. First, WebGL supports extensions, which add to the functionality and continue to be specced out. A WebGL developer can inspect the capabilities of the system and determine an appropriate strategy to use. Many extensions are widely supported, and even without extensions, all GL code is already subject to the platform's size limits on resources. WebGL is no different from other APIs, it just puts the bar a bit lower than usual.

Second of all, it means WebGL is the only 3D API that has a shot at being universal, from desks to laps to pockets to living rooms, and everything in between. Your game console could be an Android computer, handheld or appliance. Your TV might run Linux or iOS. So might your fridge. WebGL fits with where hardware and software is going, and adapting to various devices is nothing new for the web. I imagine we might see a standardized benchmark library pop up, and developer tools to make e.g. desktop Chrome mimic a cellphone's limited capabilities.

For the Christmas demo above, I included a simple benchmark that pre-selects the demo resolution based on the time needed to generate assets up front. Additionally, it was built on a 4 year old laptop GPU, so it should run well for the majority of viewers on first viewing. The same can't be said for cutting-edge demoscene demos, which often only run smoothly on top of the line hardware. I know I'm usually resigned to watching them on YouTube instead. As neat as tomorrow's tech is, for most people it only matters what they have today.

This is the biggest philosophical difference between WebGL and OpenGL. WebGL aims to be a good enough baseline that you can carry in your pocket as well as put on a big screen, and make accessible with a simple link. I don't expect graphics legends like John Carmack to take anything but a cursory glance at it, but then, it's not encroaching on his territory. It is a bit surprising though that the demoscene hasn't taken to the web more quickly. It has never been about having top of the line hardware, only what you use it for. Contests like JS1K continue to demonstrate JavaScript's expressiveness, but we haven't really seen the bigger guns come out yet.

And it really is good enough. Here's 150,000 cubes, made out of 1.8 million triangles:

AlteredQualia

Next up is a fractal raytracer. At 30 frames per second, 512x512 pixels, 40 iterations per pixel, each folding 3D space 18 levels deep… that's 5.6 billion folds per second. This intricate visualization is little more than raw number crunching power. That's just the core loop and excludes set up and lighting. It's all driven by a couple kilobytes of JavaScript wrapped in some HTML, delivered over HTTP.

Why wouldn't you want to play with that? Come try WebGL, the water's fine.

Making MathBox

2012-11-14T00:00:00+01:00

Presentation-Quality Math with Three.js and WebGL

For most of my life, I've found math to be a visual experience. My math scores went from crap to great once I started playing with graphics code, found some demoscene tutorials, and realized I could reason about formulas by picturing the graphs they create. I could apply operators by learning how they morph, shift, turn and fold those graphs and create symmetries. I could remember equations and formulas more easily when I could layer on top the visual relationships they embody. I was less likely to make mistakes when I could augment the boring symbolic manipulation with a mental set of visual cross-checks.

So, when tasked with holding a conference talk on how to make things out of math at Full Frontal—later redone at Web Directions Code—I knew the resulting presentation would have to consist of intricate visualizations as the main draw, with whatever I had to say as mere glue to hold it together.

The problem was, I didn't know of a good tool to do so, and creating animations by hand would probably be too time consuming. With the writings of Paul Lockhart and Bret Victor firmly in mind, I also knew I wanted to start blogging more about mathematical concepts in a non-traditional way, showing the principles of calculus, analysis and algebra the way I learnt to see them in my head, rather than through the obscure symbols served up in engineering school.

So I set out to create that tool, keeping in mind the most important lesson I've picked up as a web developer: one cannot overstate the value in being able to send someone a link and have it just work, right there. It was obvious it would have to be browser-based.

2015 Update: MathBox has evolved, version 2 is now available.

Choose your Poison

Now, when people think of graphs in a browser, the natural thought is vector graphics and SVG, which quickly leads to visualization powerhouse d3.js. It really is an amazing piece of tech with a vast library of useful code to accompany it. When I wrapped my head around how d3's enter/exit selections are implemented and how little it actually does to achieve so much, I was blown away. It's just so elegant and simple.

Unfortunately, d3's core is intricately tied to the DOM through SVG and CSS. And that means ironically that d3 is not really capable of 3D. Additionally, d3 is a power tool that makes no assumptions: it is up to you to choose which visual elements and techniques to use to make your diagrams, and as such it is more like assembly language for graphs than a drop-in tool. These two were show stoppers.

For one, manually designing layouts, grids, axes, etc. every time is tedious. You should be able to drop in a mathematical expression with as little fanfare as possible and have it come out looking right. This includes sane defaults for transitions and animations.

For another, I've found that, when in doubt, adding an extra dimension always helps. The moment I finally realized that every implicit graph in N dimensions is really just a slice of an explicit one in N+1 dimensions, a ridiculous amount of things clicked together. And it took until years after studying signal processing to at long last discover the 4D picture of complex exponentiation that tied the entire thing together (projected into 3D below): it revealed the famous "magic formula" involving e, i and π to be a meaningless symbological distraction, a pinhole view of a much larger, much more beautiful structure, underpinning every Fourier and Z transform I'd ever encountered.

So, WebGL it was, because I needed 3D. Unfortunately that meant the promise of having it just work everywhere was tempered by a lack of browser support, but I would certainly hope that's something we can overcome sooner than later. Dear Apple and Microsoft: get your shit together already. Dear Firefox and Opera: your WebGL performance could be a lot better.

Shady Dealings

These days I don't really touch WebGL without going through Three.js first. Three.js is a wonderful, mature engine that contains tons of useful high-level components. At the same time, it also does a great job in just handling the boilerplate of WebGL while not getting in the way of doing some heavy lifting yourself.

Rendering vector-style graphics with WebGL is not hard, certainly easier than photorealistic 3D. Primitives like lines and points are sized in absolute pixels by default, and with hardware multisampling for anti-aliasing, you get somewhat decent image quality out of it. Though, as is typical for a Web API, we're treated like children and can only cross our fingers and request anti-aliasing politely, hoping it will be available. Meanwhile native developers have full control over speed and quality and can adjust their strategy to the specific hardware's capabilities. The more things change... And then Chrome decided to disable anti-aliasing altogether due to esoteric security issues with buggy drivers. Bah.

Now, when rendering with WebGL, you really have two options. One is to just treat it as a dumb output layer, loading or generating all your geometry in JavaScript and rendering it directly in 3D. With the speed of JS engines today, this can get you pretty far.

The second option is to leverage the GPU's own capabilities as much as possible, doing computations in GLSL through so-called vertex and fragment shader programs. These are run for every vertex in a mesh, every pixel being drawn, and have been the main force driving innovation in real-time graphics for the past decade. With the goal of butter-smooth 60fps graphical goodness, this seemed like the better choice.

Unfortunately, GLSL shaders are rather monolithic things. While you do have the ability to create subroutines, every shader still has to be a stand-alone program with its own main() function. This means you either need to include a shader for every possible combination of operations, or generate shader code dynamically by concatenating pre-made snippets or using #ifdef switches to knock them out. This is the approach taken by Three.js, which results in some very hairy code that is neither easy to read nor easy to maintain.

Having made a prototype, I knew I wanted to show continuous transitions between various coordinate systems (e.g. polar and spherical), knew I needed to render shaded and unshaded geometry, and knew I would need to slot in specific snippets for things like point sprites, bezier curves/surfaces, dynamic tick marks, and more. Sorting this all out Three.js-style would be a nightmare.

So I wrote a library to solve that problem, called ShaderGraph.js. It is best described as a smart code-concatenator, a few steps short of writing a full blown compiler. You feed it snippets of GLSL code, each with one or more inputs and outputs, and these get parsed and turned into lego-like building blocks. Each input/output becomes an outlet, and outlets are wired up in a typical dataflow style. Given a graph of connected snippets, it can be compiled back into a program by assembling the subroutines, assigning intermediate variables and constructing an appropriate main() function to invoke them. It also exports a list of all external variables, i.e. GLSL uniforms and attributes, so you can control the program's behavior easily.

If I'd stopped there however, I'd have just replaced the act of manual code writing with that of manually wiring graphs. So I applied the principle of convention-over-configuration instead: you tell ShaderGraph to connect two snippets, and it will automatically match up outlets by name and type. This is augmented by a chainable factory API, which allows you to pass a partially built graph around. It allows different classes to work together to build shaders, each inserting their own snippets into the processing chain.

For example, to render a Bezier surface, the vertex shader is composed of: cubic interpolation, viewport transform (position + tangents), normal calculation and lighting. When transforming to e.g. a polar viewport, the surface normals are seamlessly recalculated. It really works like magic and I can't wait to use this in my next WebGL projects.

Viewports, Primitives and Renderables

At its core, Three.js matches pretty directly with WebGL. You can insert objects such as a Mesh, Line or ParticleSystem into your scene, which invokes a specific GL drawing command with high efficiency. As such, I certainly didn't want to reinvent the wheel.

Hence, MathBox is set up as a sort of scene-manager-within-a-scene-manager. It's a little sandbox that speaks the language of math, allowing you to insert various primitives like curves, vectors, axes and grids. Each of these primitives then instantiates one or more renderables, which simply wrap a native Three.js object and its associated ShaderGraph material. Thus, once instantiated, MathBox gets out of the way and Three.js does the heavy lifting as normal. You can even insert multiple mathboxen into a Three.js scene if you like, mixed in with other objects.

For example, a vector primitive is rendered as an arrow: it consists of a shaft and an arrowhead, realized as a line segment and a cone. An axis primitive is an arrow as well, but it also has tick marks (specially transformed line segments), and is positioned implicitly just by specifying the axis' direction rather than a start and end point.

To render curves and surfaces, you can either specify an array of data points or a live expression to be evaluated at every point. This turned out to be essential for the kinds of intricate visualizations I wanted to show, my slides being driven by timed clocks, shared arrays of data points, and live formulas and interpolations. I even fed in data from a physics engine, and it worked perfectly.

This is all tied together through Viewport objects, which define a specific mapping from a mathematical coordinate space into the 3D world space of Three.js. For example, the default cartesian viewport has the range [–1, 1] in the X, Y and Z directions. Altering the viewport's extents will shift and scale anything rendered within, as well as reflow grids and tick marks on each axis.

There are two more sophisticated viewport types, polar and spherical, which each apply the relevant coordinate transform, and can transition smoothly to and from cartesian. More viewport types can be added, all that is required is to define an appropriate transformation in JavaScript and GLSL. That said, defining a seamless transition to and from cartesian space is not always easy, particularly if you want to preserve the aspect-ratio through the entire process.

Interpolate all the things!

Finally, I had to tackle the problem of animation, keeping in mind a tip I learnt from the ever so mindbending Vihart: "If I can draw the point of a sentence, I don't actually need to say the sentence." This applies doubly so for animation: every time you replace a "before" and "after" with a smooth transition, your audience implicitly understands the change rather than having to go look for it.

Hence, each primitive can be fully animated. Each has a set of options (controlling behavior) and styles (controlling GLSL shaders), and there is a universal animator that can interpolate between arbitrary data types in a smart fashion.

For example, given a viewport with the XYZ range [[–1, 1], [–1, 1], [–1, 1]], you can tell it to animate to [[0, 2], [0, 1], [–3, 3]], and it just works. The animator will recursively animate each subarray's elements, and any dependent objects like grids and axes will reflow to match the intermediate values. This works for colors, vectors and matrices too. In case of live curves with custom expressions, the animator will invoke both the old and the new, and interpolate between the results.

However, executing animations manually in code is tedious, particularly in a presentation, where you want to be able to step forward and backward. So I added a Director class whose job it is to coordinate things. All you do is feed it a script of steps (add this object, animate that object). Then, as it applies them, it remembers the previous state of each object and generates an automatic rollback script. It also contains logic to detect rapid navigation, and will hurry up animations appropriately. This avoids that agonizing situation of watching someone skip through their slide deck, playing the same cheesy PowerPoint transitions over and over again.

Presenting Naturally

With MathBox's core working, it was time to build my slides for the conference. After a quick survey, I quickly settled on deck.js as an HTML5 slidedeck solution that was clean and flexible enough for my purposes. However, while MathBox can be spawned inside any DOM element, it wouldn't work to insert a dozen live WebGL canvases into the presentation. The entire thing would grind to a halt or at least become very choppy.

So instead, I integrated each MathBox graphic as an IFRAME, and added some logic that only loads each IFRAME one slide before it's needed, and unloads it one slide after it's gone off screen. To sync up with the main presentation, all deck.js navigation events were forwarded into each active IFRAME using window.postMessage. With the MathBox Director running inside, this was very easy to do, and meant that I could skip around freely during the talk, without any worries of desynchronization between MathBox and the associated HTML5 overlays.

In fact, I applied a similar principle to this post. To avoid rendering all diagrams simultaneously and spinning up laptop fans more than necessary, each MathBox IFRAME is started as it scrolls into view and stopped once it's gone.

I've also found that having a handheld clicker makes a huge difference while speaking—as it allows you to gesture freely and move around. So, I grabbed the infrared remote code from VLC and built a simple bridge from to Cocoa to Node.js to WebSocket to allow the remote to work in a browser. It's a shame Apple's decided to discontinue IR ports on their laptops. I guess I'll have to come up with a BlueTooth-based solution when I upgrade my hardware.

Towards MathBox 1.0

In its current state, MathBox is still a bit rough. The selection of primitives and viewports is limited, and only includes the ones I needed for my presentation. That said, it is obvious you can already do quite a lot with it, and I couldn't have been happier to hear that all this effort had the desired response at the conference. I wasn't 100% sure whether other people would have the same a-ha moments that I've had, but I'm convinced more than ever that seeing math in motion is essential for honing our intuition about it. MathBox not only makes animated diagrams much easier to make and share, but it also opens the door to making them interactive in the future.

I plan to continue to evolve MathBox as needed by using it on this site and addressing gaps that come up, though I've already identified a couple of sore points:

I used tQuery as a boilerplate and because I liked the idea of having a chainable API for this. However, this also means it's currently running off an outdated version of Three.js. I need to look into updating and/or dropping tQuery.
MathBox has been updated to Three.js r53.
Numeric or text labels are completely unsupported. It should be possible to use my CSS3D renderer for Three.js to layer on beautifully typeset MathJax formulas, positioning them correctly in 3D on top of the WebGL render.
I've added labeling for axes. I've integrated MathJax, but it's tricky because the typesetting is painfully slow in the middle of a 60fps render. But it's automatically used if MathJax is present.
All styles have to be specified on a per-object basis. Some form of stylesheet, default styles or class mechanism to allow re-use seems like an obvious next step.
There are undoubtedly memory leaks, as I was focused first and foremost on getting it to work.
Expressions that don't change frame-to-frame are still continuously re-evaluated, which is wasteful. There is a live: false flag you can set on objects, but it triggers a few bugs here and there.
There needs to be a predictable, built-in way of running a clock per slide to sync custom expressions off of. In my presentation I used a hack of clocks that start once first invoked, but this lacks repeatability.
I added a director.clock() method that gives you a clock per slide.

Finally, it doesn't take much imagination to imagine a MathBox Editor that would allow you to build diagrams visually rather than having to use code like I did. However, that's a can of worms I'm not going to open by myself, especially because the API is already quite straightforward to use, and the library itself is still a bit in flux. Perhaps this could be done as an extension of the Three.js editor.

You can see what MathBox is really capable of in the conference video. I invite you to play around with MathBox and see what you can make it do. Contributions are welcome, and the architecture is modular enough to allow its functionality to grow for quite some time.

Making Love to WebKit

2012-01-09T00:00:00+01:00

Parallax, GPUs and Technofetishism

If the world is going to end in 2012, Acko.net will at least go out in style: I've redesigned. Those of you reading through RSS readers will want to enter through the front door in a WebKit-browser like Chrome, Safari or even an iPad.

The last design was meant to feel spacious, the new design is spacious, thanks to generous use of CSS 3D transforms.

CSS 3D vs. WebGL

This idea started with an accidental discovery: if you put a CSS perspective on a scrollable

, then 3D elements inside that

will retain their perspective while you scroll. This results in smooth, native parallax effects, and makes objects jump out of the page, particularly when using an analog input device with inertial scrolling.

This raises the obvious question: how far can you take it? Of course, this only works on WebKit browsers, who currently have the only CSS 3D implementation out of beta, so it's not a viable strategy by itself yet. IE10 and Firefox will be the next browsers to offer it. There's WebGL in Chrome and Firefox that can be used to do similar things, but WebGL is its own sandbox: you can't put DOM elements in there, or use native interaction. And any amount of WebGL rendering in response to e.g. scrolling is going to involve some amount of lag. Still, I wasn't going put a lot of effort into making a CSS 3D-only design without some backup.

That's why I actually built the whole thing on top of Three.js, mrdoob's excellent JavaScript 3D engine. Aside from providing a comprehensive standard library for 3D manipulation, it also lets you swap out the rendering component. Out of the box, it can render to a 2D canvas, a WebGL canvas, or SVG.

The DOM Scenegraph

So I augmented it with a CSS 3D renderer (GitHub). It reads out the scene and renders each object using DOM elements, shaped and transformed into the right 3D position, orientation and appearance. They sit ‘in’ the page, and the browser projects and composits them for you. Of course, this only works for simple geometric shapes like lines or rectangles, but luckily that's all I need.

It would be too slow to have to render out new elements for every frame, so the CSS 3D renderer's elements persist. Moving or rotating an object involves just changing a CSS property. Same for the camera: the entire scene is wrapped in a

that has its own 3D transform.

So it's VRML all over again, but this time, it actually sort of performs. With our browsers being actual 3D engines, it's not a huge leap from here to having a tag in HTML6, can-of-worms-factor not withstanding.

Having built a quick prototype, I was satisfied with how well it worked, particularly in Safari on OS X, where the cross-pollination from the iPhone's mature tile-based GPU renderer has clearly paid off and there is no lag at all.

Design Process

Now all that was needed was a design. Last time I drew out a manual perspective drawing in Illustrator, which was tedious, but still basically came down to designing a flat image. This time, it would have to work in 3D. I started with a quick sketch to get a feel for the perspective, now that it no longer needed to double as a flat frame for the site's content.

Simple geometric shapes, parallel lines, consistent angles. Simple enough. But if real perspective was involved, I would have to place items so they would look good from multiple angles, and each would need convincing depth and shading. To do this all by hand, typing out coordinates and perpetually refreshing the page, would take forever.

So instead I built a simple editor to speed up the process. It's super ghetto, and basically just exists to manipulate the colors, positions and orientations of objects in a Three.js scene. It spits out a JSON object describing them, which can then be unserialized again into a scene.

This also helped maintain a consistent palette. The colors are built from a few base tints, brightened or darked in linear RGB—i.e. before gamma correction. This ensures even tones and allowed for easy color adjustments.

The editor is almost entirely keyboard operated, but with its minimum amount of features I was at least able to place items in 3D, copy/paste objects and see it from any angle or position I wanted. To 'save', I just copied the output into a .JS file, where I could make manual tweaks too if necessary.

As for the actual site and content, I wanted to keep it much more sober. Like many others these days, I want to treat blogging more like publishing. That way I can focus on crafting each post more like an article with illustrations and asides rather than just a text blog.

Hence, while there's a big party upstairs, it's all typography down below. The font of choice is Klavika, a humanist/geometric sans-serif with just the right kind of “Dutch Art Museum Signage” meets “Cyberpunk” I was looking for. The layout is a responsive multi-column grid that collapses down for smaller screens and devices. Finally, a strict vertical rhythm is enforced in the lines to keep everything nice and tidy.

Editor

Open editor in new window

Controls

Click+Drag — Orbit camera
Enter — New object
Space — Clone object
Backspace — Delete object
Tab / Shift+Tab
Cycle through objects
WASD QE
Move object
Shift+WASD QE
Resize object
Ctrl+WASD QE
Move camera
[] — Lower/raise units
ZX
Orbit distance
T/T/U
Tag/untag/untag all

She cannae take the power cap'n!

307 objects later it was finished, and not a single image was used. Unfortunately, as you can see there are tons of glitches in the editor—though some objects only have one side by design, and it works a lot better in a separate window. CSS 3D was never meant to do this, and you often see incorrect depth layering and flickering. Luckily most of these are caused by the floating grid markers and aren't a problem in the final view. The rest was resolved by splitting up objects or dual layering problematic surfaces, but some minor problems remain. Also for some reason, the background

's click areas extend beyond their visible area, causing some click layering issues that I had to work around. Text resizing in the browser also leads to breakage, though multi-touch zoom works in Safari.

Performance in Safari is wonderfully smooth too, but Chrome OS X starts to lag a bit. Luckily the effects are turned off as soon as they go off screen, so any lag should be confined to the top of the page. Finally, there's also a random bug where sometimes the page will refuse to scroll if the mouse is over a 3D object, which is unfortunate, but also near-impossible to reproduce reliably.

In theory the iPad would perform second, but it has its own issues. The use of page-in-page scrolling disables inertia, but this is entirely beyond my control. The other issue is that sometimes, the iPad will decide to render the page content at lower resolution, making it hard to read. I guess the CSS wizardry confuses its GPU texture management. A refresh usually fixes this.

I also discovered some funny ways of abusing CSS 3D for weird effects. If you have a WebKit browser, scroll to the top and enter the Konami code for an impressionistic version of the same thing.

I guess I'm now the proud owner of the first unofficial CSS 3D ‘ACID’ test. I'm eager to see how the next browser handles it. If it ends up being a silly idea in the long run, I can always just switch the output to WebGL, but for now I'm willing to run with it. I put in a universal CSS 3D detector and prefixes for all the major browsers.

For non-CSS 3D browsers, I simply rendered the header into a static image. It's not as fun without the shifting perspective, but it adds its own kind of optical illusion as you scroll down.

Putting it all together

To power the site, I got rid of Drupal and replaced it with the nimble Jekyll. Hat tip to James Walker, who did the same thing just a few days earlier and put all the code on GitHub to learn from.

I've been really impressed with Jekyll's simple workflow, and though it's all static HTML, it's a refreshing change of pace. And thanks to client-side JS, it doesn't preclude adding interactive elements at all. I can treat my site as just a database of documents retrievable over HTTP, and wrap the logic around that.

So I created a nice client-side navigator that transitions between pages, using 2D transforms, which also work on Firefox. It uses the HTML5 pushState API and replaces regular links with AJAX requests. Aside from being a faster way to navigate around, it also lets me link up multiple articles in a series elegantly. When you go back to a previous screen, it literally presses the browser's back button, thus avoiding creating a long, useless history trail. You go back exactly the way you came, scrolling back to where you were, just like the real back/forward buttons do. For example, click over to my Making Worlds series of posts. You can come back right away.

I didn't use any libraries or router frameworks for this, simply because I wanted to have done it all myself at least once. As it now says on my About page, quoting Feynman: "What I cannot create, I do not understand". The only way to grok the intricacies of something like browser history state, which we all use every day, is to dive in and replicate it. Otherwise, you'll just take carefully choreographed behavior for granted and your mental model will be incomplete.

To keep code size down, I compiled a custom build of Three.js with only the parts I need. I also used YUI compressor to minify the CSS and JS. However, I don't mean to obfuscate the code: the important bits will make their way onto Github soon enough.

Update: The CSS 3D renderer and editor are now available on GitHub.

And Done?

I migrated over most of the content and did some house cleaning while I was at it. Most things should be back, but further fixes will be made. I also haven't implemented any commenting solution so far, but I'll be adding it back somehow as soon as I figure something out. In the mean time, there's a Google Plus thread.

The final result looks like something that would perhaps once unironically be labeled The Information Superhighway in a magazine from the 90s, though with less neon green. I like it.

Acko.net

Yak Shading

Data-Driven Geometry

Virtual Geometry

Vertex Party

MathBox²

PowerPoint Must Die

PowerPoint Syndrome

Manifold Dreams

Geometry Streaming

(4-in-1)²

MathBox²

Part 2

I-Can't-Believe-It's-Not-React

One More Thing…

Full Stack

A DOM for Robots

Modelling Live Data

Keep it Simple

Base

Camera

Draw

Data

Operator

Overlay

Present

RTT

Shader

Text

Time

Transform

View

Model-View-Projection

It'll Do

Zero to Sixty in One Second

Fusing WebGL, CSS 3D and HTML

Procedural Wildstyle

DJ Ambient

Achievement Unlocked

Turn Right Past The Header

On WebGL

More than pretty pictures

The Black Sheep

The Procedural Canvas

A Useful Baseline

Further reading

Making MathBox

Presentation-Quality Math with Three.js and WebGL

Choose your Poison

Shady Dealings

Viewports, Primitives and Renderables

Interpolate all the things!

Presenting Naturally

Towards MathBox 1.0

Making Love to WebKit

Parallax, GPUs and Technofetishism

CSS 3D vs. WebGL

The DOM Scenegraph

Design Process

Editor

Controls

She cannae take the power cap'n!

Putting it all together

And Done?