Vertex Party

If you squint very hard it looks a bit like React for live geometry. Except instead of a diffing algorithm, there's a few events, some texture uploads, a handful of draw calls and then an idle CPU. It's ideal for drawing thousands of things that look similar and follow the same rules. It can handle not just basic GL primitives like lines or triangles, but higher level shapes like 3D arrows or sprites.

My first prototype of this was my last christmas demo. It was messy and tedious to make, especially the shaders, but it performed excellently: the final scene renders ~200,000 triangles. Despite being a layer around Three.js … around WebGL … around OpenGL … around a driver … around a GPU … performance has far exceeded my expectations. Even complex scenes run great on my Android phone, easily 10x faster than MathBox 1, in some cases more like 1000x.

Of course compared to cutting edge DirectX or OpenCL (not a typo), this is still very limited. In today's GPUs, the charade of attributes, geometries, vertices and samples has mostly been stripped away. What remains is buffers and massive operations on them, exposed raw in new APIs like AMD's Mantle and iOS's Metal. My vertex trickery acts like a polyfill, virtualizing WebGL's capabilities to bring them closer to the present. It goes a bit beyond what geometry shaders can provide, but still lacks many useful things like atomic append queues or stream compaction.

For large geometries, the set up cost can be noticeable though. Shader compilation time also grows with transform complexity, doubly so on Windows where shaders are recompiled to HLSL / Direct3D. This makes drawing ops the heaviest MathBox primitives to spawn and reallocate. You could call this the MathBox version of the dreaded 'paint' of HTML. Once warmed up though, most other properties can be animated instantly, including the data being displayed: this is the opposite of how HTML works. Hence you can mostly spawn things ahead of time, revealing and hiding objects as needed, with minimal overhead and jank at runtime.

This all relies on carefully constructed shaders which have to be wired up in all their individual permutations. This needed to be solved programmatically, which is where we go last.

• MathBox² - PowerPoint Must Die
• A DOM for Robots - Modelling Live Data
• Yak Shading - Data Driven Geometry
• ShaderGraph 2 - Functional GLSL

Instanced Data Flow

Enter ShaderGraph 2. It's a total rewrite using Chris Dickinson's bona fide glsl-parser. It still parses snippets and connects them into a directed graph to be compiled. But a snippet is now a full GLSL program whose main() function can have open inputs and outputs. What's more, it now also links code in the proper sense of the word: linking up module entry points as callbacks.

Basically, snippets can now have inputs and outputs that are themselves functions. These connections don't obey the typical data flow of a directed graph and instead are for function calls. A callback connection provides a path along which calls are made and values are returned.

Snippets can be instanced multiple times, including their uniforms, attributes and varyings (if requested). Uniforms are bound to Three.js-style registers as you build the graph incrementally. So it's a module system, sort of, which enables functional shader building. Using callbacks as micro-interfaces feels very natural in practice, especially with bound parameters. You can decorate existing functions, e.g. turning a texture sampler into a convolution filter.

As GPUs are massively parallel pure function applicators, the resulting mega-shaders are a great fit.

$ cat *.glsl | magic

The process still comes down to concatenating the code in a clever way, with global symbols namespaced to be unique. Function bodies are generated to call snippets in the right order, and the callbacks are linked. In the trivial case it links a callback by #defineing the two symbols to be the same. It can also impedance match compatible signatures like void main(in float, out vec2) and vec2 main(float) by inserting an intermediate call.

precision highp float;
precision highp int;
uniform mat4 modelMatrix;
uniform mat4 modelViewMatrix;
uniform mat4 projectionMatrix;
uniform mat4 viewMatrix;
uniform mat3 normalMatrix;
uniform vec3 cameraPosition;
#define _sn_191_getPosition _pg_103_
#define _sn_190_getPosition _pg_102_
#define _sn_189_getSample _pg_100_
#define _pg_99_ _sn_185_warpVertex
#define _pg_103_ _sn_190_getMeshPosition
#define _pg_100_ _sn_188_getTransitionSDFMask
#define _pg_101_ _sn_189_maskLevel
vec2 _sn_180_truncateVec(vec4 v) { return v.xy; }
uniform vec2 _sn_181_dataResolution;
uniform vec2 _sn_181_dataPointer;

vec2 _sn_181_map2DData(vec2 xy) {
  return fract((xy + _sn_181_dataPointer) * _sn_181_dataResolution);
}

uniform sampler2D _sn_182_dataTexture;

vec4 _sn_182_sample2D(vec2 uv) {
  return texture2D(_sn_182_dataTexture, uv);
}

vec4 _sn_183_swizzle(vec4 xyzw) {
  return vec4(xyzw.x, xyzw.w, 0.0, 0.0);
}
uniform float _sn_184_polarBend;
uniform float _sn_184_polarFocus;
uniform float _sn_184_polarAspect;
uniform float _sn_184_polarHelix;

uniform mat4 _sn_184_viewMatrix;

vec4 _sn_184_getPolarPosition(vec4 position, inout vec4 stpq) {
  if (_sn_184_polarBend > 0.0) {

    if (_sn_184_polarBend < 0.001) {
      
      
      
      
      vec2 pb = position.xy * _sn_184_polarBend;
      float ppbbx = pb.x * pb.x;
      return _sn_184_viewMatrix * vec4(
        position.x * (1.0 - _sn_184_polarBend + (pb.y * _sn_184_polarAspect)),
        position.y * (1.0 - .5 * ppbbx) - (.5 * ppbbx) * _sn_184_polarFocus / _sn_184_polarAspect,
        position.z + position.x * _sn_184_polarHelix * _sn_184_polarBend,
        1.0
      );
    }
    else {
      vec2 xy = position.xy * vec2(_sn_184_polarBend, _sn_184_polarAspect);
      float radius = _sn_184_polarFocus + xy.y;
      return _sn_184_viewMatrix * vec4(
        sin(xy.x) * radius,
        (cos(xy.x) * radius - _sn_184_polarFocus) / _sn_184_polarAspect,
        position.z + position.x * _sn_184_polarHelix * _sn_184_polarBend,
        1.0
      );
    }
  }
  else {
    return _sn_184_viewMatrix * vec4(position.xyz, 1.0);
  }
}
uniform float _sn_185_time;
uniform float _sn_185_intensity;

vec4 _sn_185_warpVertex(vec4 xyzw, inout vec4 stpq) {
  xyzw +=   0.2 * _sn_185_intensity * (sin(xyzw.yzwx * 1.91 + _sn_185_time + sin(xyzw.wxyz * 1.74 + _sn_185_time)));
  xyzw +=   0.1 * _sn_185_intensity * (sin(xyzw.yzwx * 4.03 + _sn_185_time + sin(xyzw.wxyz * 2.74 + _sn_185_time)));
  xyzw +=  0.05 * _sn_185_intensity * (sin(xyzw.yzwx * 8.39 + _sn_185_time + sin(xyzw.wxyz * 4.18 + _sn_185_time)));
  xyzw += 0.025 * _sn_185_intensity * (sin(xyzw.yzwx * 15.1 + _sn_185_time + sin(xyzw.wxyz * 9.18 + _sn_185_time)));

  return xyzw;
}



vec4 _sn_186_getViewPosition(vec4 position, inout vec4 stpq) {
  return (viewMatrix * vec4(position.xyz, 1.0));
}

vec3 _sn_187_getRootPosition(vec4 position, in vec4 stpq) {
  return position.xyz;
}
vec3 _pg_102_(vec4 _io_510_v, in vec4 _io_519_stpq) {
  vec2 _io_509_return;
  vec2 _io_511_return;
  vec4 _io_513_return;
  vec4 _io_515_return;
  vec4 _io_517_return;
  vec4 _io_520_stpq;
  vec4 _io_527_return;
  vec4 _io_528_stpq;
  vec4 _io_529_return;
  vec4 _io_532_stpq;
  vec3 _io_533_return;

  _io_509_return = _sn_180_truncateVec(_io_510_v);
  _io_511_return = _sn_181_map2DData(_io_509_return);
  _io_513_return = _sn_182_sample2D(_io_511_return);
  _io_515_return = _sn_183_swizzle(_io_513_return);
  _io_520_stpq = _io_519_stpq;
  _io_517_return = _sn_184_getPolarPosition(_io_515_return, _io_520_stpq);
  _io_528_stpq = _io_520_stpq;
  _io_527_return = _pg_99_(_io_517_return, _io_528_stpq);
  _io_532_stpq = _io_528_stpq;
  _io_529_return = _sn_186_getViewPosition(_io_527_return, _io_532_stpq);
  _io_533_return = _sn_187_getRootPosition(_io_529_return, _io_532_stpq);
  return _io_533_return;
}
uniform vec4 _sn_190_geometryResolution;

#ifdef POSITION_STPQ
varying vec4 vSTPQ;
#endif
#ifdef POSITION_U
varying float vU;
#endif
#ifdef POSITION_UV
varying vec2 vUV;
#endif
#ifdef POSITION_UVW
varying vec3 vUVW;
#endif
#ifdef POSITION_UVWO
varying vec4 vUVWO;
#endif


vec3 _sn_190_getMeshPosition(vec4 xyzw, float canonical) {
  vec4 stpq = xyzw * _sn_190_geometryResolution;
  vec3 xyz = _sn_190_getPosition(xyzw, stpq);

  #ifdef POSITION_MAP
  if (canonical > 0.5) {
    #ifdef POSITION_STPQ
    vSTPQ = stpq;
    #endif
    #ifdef POSITION_U
    vU = stpq.x;
    #endif
    #ifdef POSITION_UV
    vUV = stpq.xy;
    #endif
    #ifdef POSITION_UVW
    vUVW = stpq.xyz;
    #endif
    #ifdef POSITION_UVWO
    vUVWO = stpq;
    #endif
  }
  #endif
  return xyz;
}

uniform float _sn_188_transitionEnter;
uniform float _sn_188_transitionExit;
uniform vec4  _sn_188_transitionScale;
uniform vec4  _sn_188_transitionBias;
uniform float _sn_188_transitionSkew;
uniform float _sn_188_transitionActive;

float _sn_188_getTransitionSDFMask(vec4 stpq) {
  if (_sn_188_transitionActive < 0.5) return 1.0;

  float enter   = _sn_188_transitionEnter;
  float exit    = _sn_188_transitionExit;
  float skew    = _sn_188_transitionSkew;
  vec4  scale   = _sn_188_transitionScale;
  vec4  bias    = _sn_188_transitionBias;

  float factor  = 1.0 + skew;
  float offset  = dot(vec4(1.0), stpq * scale + bias);

  vec2 d = vec2(enter, exit) * factor + vec2(-offset, offset - skew);
  if (exit  == 1.0) return d.x;
  if (enter == 1.0) return d.y;
  return min(d.x, d.y);
}
uniform float _sn_191_worldUnit;
uniform float _sn_191_lineWidth;
uniform float _sn_191_lineDepth;
uniform float _sn_191_focusDepth;

uniform vec4 _sn_191_geometryClip;
attribute vec2 line;
attribute vec4 position4;

#ifdef LINE_PROXIMITY
uniform float _sn_191_lineProximity;
varying float vClipProximity;
#endif

#ifdef LINE_STROKE
varying float vClipStrokeWidth;
varying float vClipStrokeIndex;
varying vec3  vClipStrokeEven;
varying vec3  vClipStrokeOdd;
varying vec3  vClipStrokePosition;
#endif


#ifdef LINE_CLIP
uniform float _sn_191_clipRange;
uniform vec2  _sn_191_clipStyle;
uniform float _sn_191_clipSpace;

attribute vec2 strip;

varying vec2 vClipEnds;

void _sn_191_clipEnds(vec4 xyzw, vec3 center, vec3 pos) {

  
  vec4 xyzwE = vec4(strip.y, xyzw.yzw);
  vec3 end   = _sn_191_getPosition(xyzwE, 0.0);

  
  vec4 xyzwS = vec4(strip.x, xyzw.yzw);
  vec3 start = _sn_191_getPosition(xyzwS, 0.0);

  
  vec3 diff = end - start;
  float l = length(diff) * _sn_191_clipSpace;

  
  float arrowSize = 1.25 * _sn_191_clipRange * _sn_191_lineWidth * _sn_191_worldUnit;

  vClipEnds = vec2(1.0);

  if (_sn_191_clipStyle.y > 0.0) {
    
    float depth = _sn_191_focusDepth;
    if (_sn_191_lineDepth < 1.0) {
      float z = max(0.00001, -end.z);
      depth = mix(z, _sn_191_focusDepth, _sn_191_lineDepth);
    }
    
    
    float size = arrowSize * depth;

    
    
    float mini = clamp(1.0 - l / size * .333, 0.0, 1.0);
    float scale = 1.0 - mini * mini * mini; 
    float invrange = 1.0 / (size * scale);
  
    
    diff = normalize(end - center);
    float d = dot(end - pos, diff);
    vClipEnds.x = d * invrange - 1.0;
  }

  if (_sn_191_clipStyle.x > 0.0) {
    
    float depth = _sn_191_focusDepth;
    if (_sn_191_lineDepth < 1.0) {
      float z = max(0.00001, -start.z);
      depth = mix(z, _sn_191_focusDepth, _sn_191_lineDepth);
    }
    
    
    float size = arrowSize * depth;

    
    
    float mini = clamp(1.0 - l / size * .333, 0.0, 1.0);
    float scale = 1.0 - mini * mini * mini; 
    float invrange = 1.0 / (size * scale);
  
    
    diff = normalize(center - start);
    float d = dot(pos - start, diff);
    vClipEnds.y = d * invrange - 1.0;
  }


}
#endif

const float _sn_191_epsilon = 1e-5;
void _sn_191_fixCenter(vec3 left, inout vec3 center, vec3 right) {
  if (center.z >= 0.0) {
    if (left.z < 0.0) {
      float d = (center.z - _sn_191_epsilon) / (center.z - left.z);
      center = mix(center, left, d);
    }
    else if (right.z < 0.0) {
      float d = (center.z - _sn_191_epsilon) / (center.z - right.z);
      center = mix(center, right, d);
    }
  }
}


void _sn_191_getLineGeometry(vec4 xyzw, float edge, out vec3 left, out vec3 center, out vec3 right) {
  vec4 delta = vec4(1.0, 0.0, 0.0, 0.0);

  center =                 _sn_191_getPosition(xyzw, 1.0);
  left   = (edge > -0.5) ? _sn_191_getPosition(xyzw - delta, 0.0) : center;
  right  = (edge < 0.5)  ? _sn_191_getPosition(xyzw + delta, 0.0) : center;
}

vec3 _sn_191_getLineJoin(float edge, bool odd, vec3 left, vec3 center, vec3 right, float width) {
  vec2 join = vec2(1.0, 0.0);

  _sn_191_fixCenter(left, center, right);

  vec4 a = vec4(left.xy, right.xy);
  vec4 b = a / vec4(left.zz, right.zz);

  vec2 l = b.xy;
  vec2 r = b.zw;
  vec2 c = center.xy / center.z;

  vec4 d = vec4(l, c) - vec4(c, r);
  float l1 = dot(d.xy, d.xy);
  float l2 = dot(d.zw, d.zw);

  if (l1 + l2 > 0.0) {
    
    if (edge > 0.5 || l2 == 0.0) {
      vec2 nl = normalize(d.xy);
      vec2 tl = vec2(nl.y, -nl.x);

#ifdef LINE_PROXIMITY
      vClipProximity = 1.0;
#endif

#ifdef LINE_STROKE
      vClipStrokeEven = vClipStrokeOdd = normalize(left - center);
#endif
      join = tl;
    }
    else if (edge < -0.5 || l1 == 0.0) {
      vec2 nr = normalize(d.zw);
      vec2 tr = vec2(nr.y, -nr.x);

#ifdef LINE_PROXIMITY
      vClipProximity = 1.0;
#endif

#ifdef LINE_STROKE
      vClipStrokeEven = vClipStrokeOdd = normalize(center - right);
#endif
      join = tr;
    }
    else {
      
      float lmin2 = min(l1, l2) / (width * width);

      
#ifdef LINE_PROXIMITY
      float lr     = l1 / l2;
      float rl     = l2 / l1;
      float ratio  = max(lr, rl);
      float thresh = _sn_191_lineProximity + 1.0;
      vClipProximity = (ratio > thresh * thresh) ? 1.0 : 0.0;
#endif
      
      
      vec2 nl = normalize(d.xy);
      vec2 nr = normalize(d.zw);

      vec2 tl = vec2(nl.y, -nl.x);
      vec2 tr = vec2(nr.y, -nr.x);

#ifdef LINE_PROXIMITY
      
      vec2 tc = normalize(mix(tl, tr, l1/(l1+l2)));
#else
      
      vec2 tc = normalize(tl + tr);
#endif
    
      float cosA   = dot(nl, tc);
      float sinA   = max(0.1, abs(dot(tl, tc)));
      float factor = cosA / sinA;
      float scale  = sqrt(1.0 + min(lmin2, factor * factor));

#ifdef LINE_STROKE
      vec3 stroke1 = normalize(left - center);
      vec3 stroke2 = normalize(center - right);

      if (odd) {
        vClipStrokeEven = stroke1;
        vClipStrokeOdd  = stroke2;
      }
      else {
        vClipStrokeEven = stroke2;
        vClipStrokeOdd  = stroke1;
      }
#endif
      join = tc * scale;
    }
    return vec3(join, 0.0);
  }
  else {
    return vec3(0.0);
  }

}

vec3 _sn_191_getLinePosition() {
  vec3 left, center, right, join;

  float edge = line.x;
  float offset = line.y;

  vec4 p = min(_sn_191_geometryClip, position4);
  edge += max(0.0, position4.x - _sn_191_geometryClip.x);

  
  _sn_191_getLineGeometry(p, edge, left, center, right);

#ifdef LINE_STROKE
  
  vClipStrokePosition = center;
  vClipStrokeIndex = p.x;
  bool odd = mod(p.x, 2.0) >= 1.0;
#else
  bool odd = true;
#endif

  
  float width = _sn_191_lineWidth * 0.5;

  float depth = _sn_191_focusDepth;
  if (_sn_191_lineDepth < 1.0) {
    
    float z = max(0.00001, -center.z);
    depth = mix(z, _sn_191_focusDepth, _sn_191_lineDepth);
  }
  width *= depth;

  
  width *= _sn_191_worldUnit;

  join = _sn_191_getLineJoin(edge, odd, left, center, right, width);

#ifdef LINE_STROKE
  vClipStrokeWidth = width;
#endif
  
  vec3 pos = center + join * offset * width;

#ifdef LINE_CLIP
  _sn_191_clipEnds(p, center, pos);
#endif

  return pos;
}

uniform vec4 _sn_189_geometryResolution;
uniform vec4 _sn_189_geometryClip;
varying float vMask;


void _sn_189_maskLevel() {
  vec4 p = min(_sn_189_geometryClip, position4);
  vMask = _sn_189_getSample(p * _sn_189_geometryResolution);
}

uniform float _sn_192_styleZBias;
uniform float _sn_192_styleZIndex;

void _sn_192_setPosition(vec3 position) {
  vec4 pos = projectionMatrix * vec4(position, 1.0);

  
  float bias  = (1.0 - _sn_192_styleZBias / 32768.0);
  pos.z *= bias;
  
  
  if (_sn_192_styleZIndex > 0.0) {
    float z = pos.z / pos.w;
    pos.z = ((z + 1.0) / (_sn_192_styleZIndex + 1.0) - 1.0) * pos.w;
  }
  
  gl_Position = pos;
}
void main() {
  vec3 _io_546_return;

  _io_546_return = _sn_191_getLinePosition();
  _sn_192_setPosition(_io_546_return);
  _pg_101_();
}

It still does guarded regex manipulation of code too, but those manipulations are now derived from a proper syntax tree. GLSL doesn't have strings and its scope is simple, so this is unusually safe. I'm sure you can still trip it up somehow, but it's worth it for speed. I'm seeing assembly times of ~10-30ms cold, 2-4ms warm, but it depends entirely on the particular shaders.

The assembly process is now properly recursive. Unassembled shaders can be used in factory form, standing in for snippets. Completed graphs form stand-alone programs with no open inputs or outputs. The result can be turned straight into a Three.js ShaderMaterial, but there is no strict Three dependency. It's just a dictionary with code and a list of uniforms, attributes and varyings. Unlike before, building a combined vertex/fragment program is now merely syntactic sugar for a pair of separate graphs.

As it's run-time, you can slot in user-defined or code-generated GLSL just the same. Shaders are fetched by name or passed as inline code, mixed freely as needed. You supply the dictionary or lookup method. You could bundle your GLSL into JS with a build step or include embedded <script> tags.

This is the fragment shader that implements the partial differential equation for this ripple effect (getFramesSample). It samples from a volumetric N×N×2 array, feeding back into itself.

Paging Dr. Hickey

ShaderGraph 2 drives the entirety of MathBox 2. Its shaders are specialized for particular types and dimensions, generating procedural data, clipping geometry, resampling transformed data on the fly, …. The composibility comes out naturally. To do so, I pass a partially built factory by interested parties. This way I build graphs for position, color, normal, mask and more. These are injected as callbacks into a final shader. Shader factories enable ad-hoc contracts, sandwiched between the inner and outer retained layers of Three.js and MathBox, but disappearing entirely in the end result.

Of course, all of this is meta-programming of GLSL, done through a stateful JS lasagna and a ghetto compiler, instead of an idiomatic language. I know this, it's an inner platform effect bathing luxuriously in Turing tar like a rhino in mud. I didn't really see a way around it, given the constraints at play.

While the factory API is designed for making graphs on the spot and then tossing them, you could keep graphs around. There's a full data model underneath. You can always skip the factory entirely.

Plenty of caveats of course. There is no built-in preprocessor, so you can't #define or #ifdef uniforms or attributes and have it make sense. But then the point of ShaderGraph is to formalize exactly that sort of ad-hoc fiddling. Preprocessor directives will just pass through. glsl-parser has gaps too, and it is also exceedingly picky with reserved variable names, so watch out for that.

I did sometimes feel the need for more powerful metaprogramming, but you can work around it. It is easy to dynamically make GLSL one-liner snippets and feed them in. String manipulation of code is always still an option, you just don't need to do it at the macro-level anymore.

ShaderGraph 2 has been in active use now for months, it does the job I need it to very well. In a perfect world, this would be solved at the GPU driver level. Until SPIR-V or WebVulkan gets here, imma stick to my regexes. Don't try this at home, kids.

For docs and more, see the Git repository.

• MathBox² - PowerPoint Must Die
• A DOM for Robots - Modelling Live Data
• Yak Shading - Data Driven Geometry
• ShaderGraph 2 - Functional GLSL

There was one big problem: scenes now consisted of diagrams of diagrams, which meant working around MathBox more than with it. Performance issues arose as complexity grew. Above all there was a total lack of composability in the components. None of this could be fixed without ripping out significant pieces, so doing it incrementally seemed futile. I started from scratch and set off to reinvent all the wheels.

$$ \text{MathBox}^2 = \int_1^2 \text{code}(v) dv $$

MathBox 2 was inevitably going to suffer second-system syndrome, parts would be overengineered. Rather than fight it, I embraced it and effectively wrote a strange vector GPU driver in CoffeeScript. (Such is life, this is a blueprint meant to be simplified and made obsolete over time, not expanded upon.) It's a freight train straight to the heart of a graphics card, combining low-level and high-level in a way that feels novel 🐴 when you use it, squeezing 🐴 through a very small opening.

What was tedious before, now falls out naturally. If I format the scene above as XML/JSX, it becomes:

<root>
  <!-- Place the camera -->
  <camera />
  <!-- Change clock speed -->
  <clock>
    <!-- 4D Stereographic projection -->
    <stereographic4>
      <!-- Custom 4D rotation shader -->
      <shader />
      <!-- Move vertices -->
      <vertex>
        <!-- Sample an area -->
        <!-- Draw a set of lines -->
        <area />
        <line />

        <!-- Sample an area -->
        <!-- Draw a set of lines -->
        <area />
        <line />

        <!-- Sample an area -->
        <!-- Draw a set of lines -->
        <area />
        <line />
      </vertex>
    </stereographic4>
  </clock>
</root>

Phew. That's how you make a 4D diagram with Hopf fibration as far as the eye can see. Except it's not actually JSX, that's just me and my pretty-printer pretending.

Geometry Streaming

The key is the data itself. It's an array of points mostly, but how that data is laid out and interpreted determines how useful it can be.

Most basic primitives come in fixed size chunks. Particles are single points, lines have two points, triangles have three points. Polygons and polylines have N points. So it made sense to have a tuple of N points be the basic logical unit. You can think in logical pieces of geometry, rather than raw points or individual triangles, unlike GL.

Each primitive maps over data in a standard way. Feed an array of points to a line, you get a polyline. Feed a matrix of points to a surface and you get a grid mesh. Simple. But feed a voxel to a vector, and you get a 3D vector field. The general idea is that drawing 1 of something should be as easy as drawing 100×100×100.

(4-in-1)²

GPUs can operate on 4×1 vectors and 4×4 matrices, so working with 4D values is natural. Values can also be referenced by 4D indices. With one dimension reserved for the tuples, that leaves us 3 dimensions XYZ. Hence MathBox arrays are 3+1D. This is for width, height, depth, while the tuple dimension is called items. It does what it says on the tin, creating 1D W, 2D W×H and 3D W×H×D arrays of tuples. Each tuple is made of N vectors of up to 4 channels each.

Thanks to cyclic buffers and partial updates, history also comes baked in. You can use a spare dimension as a free time axis, retaining samples on the go. You can .set('history', N) to record a short log of a whole array over time, indefinitely.

All of this is modular: a data source is something that can be sampled by a 4D pointer from GLSL. Underneath, arrays end up packed into a regular 2D float texture, with "items × width" horizontally and "height × depth" vertically. Each 'pixel' holds a 1/2/3/4D point.

Mapping a 4D 'pointer' to the real 2D UV coordinates is just arithmetic, and so are operators like transpose and repeat. You just swap the XY indices and tell everyone downstream that it's now this big instead. They can't tell the difference.

You can create giant procedural arrays this way, including across rectangular texture size limits, as none of them actually exist except as transient values deep inside a GPU core. Until you materialize them by rendering to a texture using the memo primitive. Add in operators like interpolation and convolution and it's a pretty neat real-time finishing kit for data.

Continued in Part 2.

I-Can't-Believe-It's-Not-React

Underneath sits a large codebase driving it, 200+ files in JS/CS alone, not including dependencies that aren't my own. Much of it is infrastructure necessary to pull off certain tricks consistently: you can draw 2.5D lines with grace, render arbitrary Unicode text in GL, sync HTML to GPU-computed geometry, and do all this with GLSL code composed on the fly, including your own. Nobody needs all of this.

These wildly different strategies are actually all abstracted into the DOM path or the GL path, with a quacks-like-React component as the glue in the DOM path.

Most of the code is for initialization only, building up a reactive machine by combining components. Once assembled it lets the GPU do most of the crunching, while relying on the JS VM to inline and optimize the chewy outside.

At the top level, MathBox is plain old JavaScript, used like this:

The WebGL Canvas bootstrapper is a separate piece though, it's wrapping Threestrap, the little non-framework that could. It lets you spawn a fully functioning GL canvas in one exceedingly configurable call. It takes care of browser support, resizing with retina, CSS alignment, warm-up and more.

If you prefer instead, you can spawn a bare MathBox context in a Three.js scene of your choosing, but you'll have to babysit it:

<root>
  <!-- Present slides -->
  <present index={1}>
    <!-- Slide 1 -->
    <slide steps={2}>
      <!-- Transition effect -->
      <reveal>
        <!-- Sample an interval -->
        <interval />
        <!-- Draw a line -->
        <line />
        <!-- Step through keyframes -->
        <step script={[
            {props: {color: 'red'}},
            {props: {color: 'blue'}},
          ]} >
      </reveal>
    </slide>
    <!-- Slide 2 -->
    <slide>
      <!-- Transition effect -->
      <move>
        <!-- Sample an interval -->
        <interval />
        <!-- Play through keyframes -->
        <play script={[
                       {props: {expr: (emit, x) => emit(x, Math.sin(x))}},
                       {props: {expr: (emit, x) => emit(x, Math.tan(x))}},
                      ]} />
        <!-- Draw a line -->
        <line />

        <!-- Slide 2.1 -->
        <slide>
        </slide>
      </move>
    </slide>
    <!-- Slide 3 -->
    <slide>
      <!-- Transition effect -->
      <reveal>
        <!-- Sample an interval -->
        <interval />
        <!-- Draw a line -->
        <line />
      </reveal>
    </slide>
  </present>
</root>

One More Thing…

Images are data. So is audio. That means MathBox 2 is Winamp AVS, Milkdrop and the mythical Fridge all rolled into one. You can replicate your everyday trippy music visualizer with two operators: render-to-texture (rtt) and compose. It acts as an embedded scene, rendering all of its children to an off-screen image, while Compose renders a full-screen pass. This is where the model's expressiveness shines.

Milkdrop equals mathbox.rtt(…).compose(…).….end().compose(…), that is, an image feeding back into itself, but also rendered to the screen. The necessary double buffering and swaps are abstracted away. Drop in shapes and shaders, add transforms, nest as you like. RTTs have a history parameter like arrays, so Turing patterns, self-propagating hypno spirals, and other cool partial diffy eqs are a shader away.

3D engines don't have Document Object Models though, they have scene graphs and render trees: minimal data structures optimized for rendering output. In Three.js, each tree node is a JS object with properties and children like itself. Composition only exists in a limited form, with a parent's matrix and visibility combining with that of its children. There is no fancy data binding: the renderer loops over the visible tree leaves every frame, passing in values directly to GL calls. Any geometry is uploaded once to GPU memory and cached. If you put in new parameters or data, it will be used to produce the next frame automatically, aside from a needsUpdate bit here and there for performance reasons.

So Three.js is a thin retained mode layer on top of an immediate mode API. It makes it trivial to draw the same thing over and over again in various configurations. That won't do, I want to draw dynamic things with the same ease. I need a richer model, which means wrapping another retained mode layer around it. That could mean observables, data binding, tree diffing, immutable data, and all the other fun stuff nobody can agree on.

However I mostly feed data in and many parameters will end up as shader properties. These are passed to Three as a dictionary of { type: '…', value: x } objects, each holding a single parameter. Any code that holds a reference to the dictionary will see the same value, as such it acts as a register: you can share it, transparently binding one value to N shaders. This way a single .set('color', 'blue') call on the fringes can instantly affect data structures deep inside the WebGLRenderer, without actually cascading through.

I applied this to build a view tree which retains this property, storing all attributes as shareable registers. The Three.js scene graph is reduced to a single layer of THREE.Mesh objects, flattening the hierarchy. Rather than clumsy CSS3D divs which encode matrices as strings, there's binary arrays, GLSL shaders, and highly optimizable JS lambdas.

As long as you don't go overboard with the numbers, it runs fine even on mobile.

Keep it Simple

From afar there's a tree of nodes, similar to SVG tags. This is the MathBox library of vector primitives. The basic shapes are all there: points, lines, faces, vectors, surfaces, etc. These nodes are placed inside a shallow hierarchy of views and transforms.

However none of the shapes draw anything by themselves. They only know how to draw data supplied by a linked source. Data can be an array (static or live), a procedural source, custom JS / GLSL code, etc. This is further augmented by data operators which can be sandwiched between source and shape, forming automatic pipelines between siblings.

The current set of components looks like this:

Fusing WebGL, CSS 3D and HTML

Ladies and gentlemen, this is your captain speaking. Today's flight on WebGL Air will take us high above the cloud. Those of you sitting in Chrome class, on the desktop side of the plane, will have the clearest view. Internet Explorer class passengers may turn to their in-seat entertainment system instead. Passengers are reminded to put away their iPads and iPhones during take off and landing.

Edit: You can now also watch the YouTube video kindly provided by Leland Batey.

Acko.net, the domain, just turned 13, so it's time for a birthday present in the form of a complete front-end rewrite. The last design was entirely based on CSS 3D: it was a fun experiment, but created at a time when browser implementations were still wonky. It ultimately proved impractical: the DOM is not a good place to store complicated geometry. It's too bulky and there is a huge difference between a styled <div> and a shaded quad. After adding WebGL to the mix with MathBox and typesetting with MathJax, it turned into a catastrophic worst case for loading, and the smoothness was often lost even on fast computers.

Since then, we've seen a big push for rendering performance, both from the native and JS side. Hardware-accelerated DOM compositing is better understood, requestAnimationFrame is now common-place and reliable, and we have excellent profiling tools down to the frame level.

Hence the goal: to fuse 3D elements into the page like before, but with full 60 fps rendering. Plus to use WebGL instead of CSS 3D where possible, and be free of the constraints of the DOM.

Like Voodoo.js I use a fixed full-screen canvas and sync up scrolling with a 3D camera. The scene is mapped to CSS pixels and CSS perspective is locked to the camera. Once HTML, CSS 3D and WebGL are all in sync there's a truckload of linear algebra and easing functions to keep you amused. The code is based on the platform I kludged together for the christmas demo, at times a mess of ad hoc demo formulas and spaghetti, though robust enough in the parts that count.

Procedural Wildstyle

The so Designers Republic kitsch of last time was starting to grate, and I wanted something more stylish. Taking inspiration from street art and graffiti, in particular long-time favorite Daim, Acko's gone wild, though with a math twist. This design was built procedurally using some nifty vector calculus and more difference equations than you can shake a stick at.

At its heart it's really a game of Snake, albeit a complicated one. Starting from a line-based skeleton of the model, the curves are traced out, smoothed, oriented and finally extruded on the fly into ribbons. The final pose shows 261 lines mapped to 2,783 curve segments, tessellated into 43,168 triangles, though that amount is just a knob that can be tuned. In other words, it's a scalable vector graphic. No images were harmed in the making of this header, or the rest of the window dressing for that matter. These pixels are local, organic and bespoke, though I'm pretty sure they're not vegan.

The ribbons need to animate, so I'm using a slightly exotic set up: a single Three.js BufferGeometry mesh is created with all the ribbons' topology in it. It only contains the connectivity of the triangles that span the vertices, stored as indices. The actual positions and normals are read from a texture that is updated on the fly by JavaScript using gl.texSubImage2D.

DJ Ambient

All of this is shaded with some custom GLSL. I use a basic Blinn-Phong lighting model tweaked for aesthetics, and there's some edge highlights and fog to top it off. But if I left it there, the result would still look pretty flat, without shadows. A common approach is to use shadow mapping, but that only works for point sources like the sun. The diffuse shadows that you see on an overcast day require an entirely different approach.

What's needed is ambient occlusion, a measure of how much skylight is obscured by the surrounding geometry at every point. The less sky you can see from a point, the darker it should be. This is slow to compute analytically for large scenes and hence is typically either faked or baked in. An easy hack is Screen-Space AO, which is really just crease darkening using the final image's depth buffer. It's clever but expensive, creating only local shadows. Quality can vary a lot between implementations, often creating distracting dark or light halos around silhouettes. They all share the same blind spot: only the currently visible pixels can cast any shadow. So while I added alteredq's SSAO shader, it's disabled by default and only rendered at quarter resolution, upscaled with a bilateral filter to hide this fact.

Instead, I used a technique from NVidia's GPU Gems 2. It uses a disc-based model to calculate occlusion and even indirect lighting on the fly. Rather than go real-time on the GPU, I decided to only do it statically in JS, and just do plain occlusion without light bounces.

The discs for all the ribbons are generated ahead of time, and the disc-to-disc shadowing is done once for the final model in two passes. The first pass overestimates due to overlapping shadows and thus is too dark. The second pass uses the first to assign lesser weights to discs in shadow to compensate, then runs the algorithm anew.

There's another simplification: instead of small one-sided discs based on the real geometry, I use large two-sided discs and treat each ribbon as flat. Each disc occludes in both directions, but accumulates shadow on its front and back separately. This gives a decent approximation of light around and across the entire object.

Below is the actual disc model. In this diagram, the two radii of each disc represent its front and back illumination, with smaller discs receiving more shadow. The crosshairs mark the real disc radius.

When streaming out new pieces of ribbon, the occlusion is baked in on the fly, stored in the unused alpha channel next to the position (RGB). The four nearest discs on the ribbon are interpolated bilinearly, and the vertex's normal and position is used to mix the front and back shadow values appropriately. Thus it's effectively trilinear filtering a 2×2×N voxel grid per ribbon, which snakes and twists along its length.

I'm very happy with how good it looks, even with very coarse divisions like this. Unlike SSAO it costs practically nothing once generated. If you wish, you can view the design entirely in white to examine the lighting—or in black and white, or in the seventies, tone mapping is fun.

To ensure smooth rendering, the resolution is scaled down if it drops below 45 fps for several frames. This is essential on slower GPUs, but can sometimes over­compensate, as WebGL is easily interrupted by other tasks on the machine. The browser doesn't tell you how much you're currently pushing it, so it's impossible to safely scale back up without causing more stutters. To mitigate this problem, I built in some strategic reset points, like when you focus the window or scroll to the top. In the end, 60fps happens more often than not, though it remains a goal rather than a guarantee. If you fullscreen your browser, you are now the bane of my existence, especially if you keep two dozen other windows open on the second monitor at the same time. On an underpowered retina MacBook.

Finally I also turned on MultiSample Anti-Aliasing to avoid the dreaded jaggies and make the vector style shine. It's substituted with Fast Approximate AA if it's unavailable. FXAA is also used if SSAO is enabled, as you can't post-process multisampled images with WebGL.

Preparing all of this would take ~400ms on my 4 year old laptop, but most of that is spent on the occlusion which is static. Hence, I saved the lighting data into a JSON file for speedier loading, which cuts it down to ~100ms. The data gzips into 18KB, about the size of a small PNG, so no worries there. Add to that the time required to fetch and run the rest of the page, and we can call it an even second.

Of course, it only works on a subset of browsers and leaves the vast majority of mobile devices out in the cold. It does work in Chrome Beta on Android, though performance and stability is still pretty crap and more fixes are needed, both in the browser and in my own code. Rather than try and emulate some of the bling for CSS 3D-only environments, it's all or nothing. Without WebGL, you get plain images.

Achievement Unlocked

Reading all that, you might mistake the header as a mere love letter to hackery. But there's a twist, quite a literal one. The twisting of each ribbon is not generated arbitrarily, but mathematically derived. It embodies the differential principle of parallel transport. The up direction changes parallel to each curve, which means the ribbons never rotate in place. They only turn when they naturally want to. Hence, the design kind of has a will of its own. I can set the initial up direction of a ribbon, but only affect it through its curved path. Arranging and tuning all the ribbons was a nice puzzle in itself, and it's a nice nod to the math that's all over the place, even if it is invisible.

As I started playing around with some screensaver-style shots, I realized it would make a pretty neat demo just by itself, so I built that in too, to the dulcet tones of Selah Sue—whose last name I hope is not indicative. Here parallel transport would ensure a perfectly smooth ride, but that's not exciting, so there's some springy exponential easing with adaptive lookahead instead. That's on top of the secondary demo, which features an audio visualizer and the smooth drum and bass of Seba. There's maybe a third one too.

The songs are used here entirely for educational purposes of course. Not that it matters, since they're all on YouTube anyway. Click the Growl-like notifications to find out more about the artists. There's also a handful of achievements to be gathered, eight to be precise. Some of these are experiments that were turned off in the final version. Others are... trickier. Oh hey, did you notice the JS console?

By the way, special mention also goes to Time and how some browsers keep terrible track of it. Like when you're trying to sync a demo to a stuttery <audio> clock, Firefox ಠ_ಠ. There's also the matter of what to do when the user switches to another tab, and the answer is.... it'll desync, because I don't want to be sapping resources in a background tab. Minor constraints of putting on a live act, doors close after the show starts.

Turn Right Past The Header

The refresh-less navigation is also back from last time, slightly cleaned up. It still works on the same basic principle of fetching static, full HTML documents. However the transition mechanism now carefully choreographs the necessary DOM manipulation to avoid stutters: any image, iframe or video inserted in the content would mean future paints at an unknown time, so they're postponed until after the transition is complete. I did it the dirty but fast way, with string manipulation before it hits the DOM: such is the luxury of being the only one, person or machine, writing the markup.

Which leaves of course the content. But don't fix what ain't broke: distraction-free publishing is easy enough with Jekyll and a responsive 960 pixel grid with sideburns, so things still look and work mostly the same. I only added a home for my math talks and expanded the front page to highlight some demos and experiments. I aim to keep focusing on quality rather than quantity, and to remind visitors what the web looks like when you take away everything social media's done to it.

It's also fun to observe that after many years, the unified front-end convergence really has arrived. There is little difference between this site and the games that use HTML for their UI, such as the latest Sim City. For dealing with typography, illustration and UI, you want the comfort of DOM and CSS. For real-time graphical content, you'll want to draw it yourself, either in 2D, or 3D. Combine the two, and you get what game developers have been doing for years. Only this lacks all the sharp C bits, which game devs been replacing with Lua for years anyhow. That's Portuguese for "JavaScript" by the way.

So there you have it. A fresh look with a pile of juicy hax, and hopefully not too many bugs in the wild. Thanks go to all those who came before and provided all these toys for me to play with. You can too, for the source is entirely on Github. Like I said, educational. Meanwhile I'll be over here, listening to the wipE'out" soundtrack on repeat in memory of the old one. Comments welcome on Google Plus.

PS: The previous version has been archived with its fallbacks disabled so you can see the current state of CSS 3D in browsers. Still pretty broken. Still a great test case. So is the disembodied head.

Exploring the outer limits

Instead, we pretend the steel is solid throughout. Rather than being composed of atoms with gaps in between, it's made of some unknown, filled in material with a certain density, expressed e.g. as grams per cubic centimetre. Given any shape, we can determine its volume, and hence its total mass, and go from there. That's much simpler than counting and keeping track of individual atoms, right?

Unfortunately, that's not quite true.

The Shortest Disappearing Trick Ever

Like all choices in mathematics, this one has consequences we cannot avoid. Our beam's density is mass per volume. Individual points in space have zero volume. That would mean that at any given point inside the beam, the amount of mass there is $ 0 $. How can a beam that is entirely composed of nothing be solid and have a non-zero mass?

Bam! No more iron anywhere.

While Douglas Adams was being deliberately obtuse, there's a kernel of truth there, which is a genuine paradox: what exactly is the mass of every atom in our situation?

To make our beam solid and continuous, we had to shrink every atom down to an infinitely small point. To compensate, we had to create infinitely many of them. Dividing the finite mass of the beam between an infinite amount of atoms should result in $ 0 $ mass per atom. Yet all these masses still have to add up to the total mass of the beam. This suggests $ 0 + 0 + 0 + … > 0 $, which seems impossible.

If the mass of every atom were not $ 0 $, and we have infinitely many points inside the beam, then the total mass is infinity times the atomic mass $ m $. Yet the total mass is finite. This suggests $ m + m + m + … < \infty $, which also doesn't seem right.

It seems whatever this number $ m $ is, it can't be $ 0 $ and can't be non-zero. It's definitely not infinite, we only had a finite mass to begin with. It's starting to sound like we'll have to invent a whole new set of numbers again to even find it.

Still, it shows that even if we don't understand the whole picture, we can get a lot done. This article is in no way a formal introduction to infinitesimals. Rather, it's a demonstration of why we might need them.

What is happening when we shrink atoms down to points? Why does it make shapes solid yet seemingly hollow? Is it ever meaningful to write $ x = \infty $? Is there only one infinity, or are there many different kinds?

To answer that, we first have to go back to even simpler times, to Ancient Greece, and start with the works of Zeno.

Achilles and the Tortoise

Zeno of Elea was one of the first mathematicians to pose these sorts of questions, effectively trolling mathematics for the next two millennia. He lived in the 5th century BC in southern Italy, although only second-hand references survive. In his series of paradoxes, he examines the nature of equality, distance, continuity, of time itself.

Because it's the ancient times, our mathematical knowledge is limited. We know about zero, but we're still struggling with the idea of nothing. We've run into negative numbers, but they're clearly absurd and imaginary, unlike the positive numbers we find in geometry. We also know about fractions and ratios, but square roots still confuse us, even though our temples stay up.

Limits are the first tool in our belt for tackling infinity. Given a sequence described by countable steps, we can attempt to extend it not just to the end of the world, but literally forever. If this works we end up with a finite value. If not, the limit is undefined. A limit can be equal to $ \infty $, but that's just shorthand for the sequence has no upper bound. Negative infinity means no lower bound.

Breaking Away From Rationality

Until now we've only encountered fractions, that is, rational numbers. Each of our sums was made of fractions. The limit, if it existed, was also a rational number. We don't know whether this was just a coincidence.

It might seem implausible that a sequence of numbers that is 100% rational and converges, can approach a limit that isn't rational at all. Yet we've already seen similar discrepancies. In our first sequence, every partial sum was less than $ 1 $. Meanwhile the limit of the sum was equal to $ 1 $. Clearly, the limit does not have to share all the properties of its originating sequence.

We also haven't solved our original problem: we've only chopped things up into infinitely many finite pieces. How do we get to infinitely small pieces? To answer that, we need to go looking for continuity.

Generally, continuity is defined by what it is and what its properties are: a noticeable lack of holes, and no paradoxical values. But that's putting the cart before the horse. First, we have to show which holes we're trying to plug.

What we just did was a careful exercise in hiding the obvious, namely the digit-based number systems we are all familiar with. By viewing them not as digits, but as paths on a directed graph, we get a new perspective on just what it means to use them. We've also seen how this means we can construct the rationals and reals using the least possible ingredients required: division by two, and limits.

Drowning By Numbers

In school, we generally work with the decimal representation of numbers. As a result, the popular image of mathematics is that it's the science of digits, not the underlying structures they represent. This permanently skews our perception of what numbers really are, and is easy to demonstrate. You can google to find countless arguments of why $ 0.999… $ is or isn't equal to $ 1 $. Yet nobody's wondering why $ 0.000… = 0 $, though it's practically the same problem: $ 0.1, 0.01, 0.001, 0.0001, … $

Furthermore, in decimal notation, rational numbers and real numbers look incredibly alike: $ 3.3333… $ vs $ 3.1415…\, $ The question of what it actually means to have infinitely many non-repeating digits, and why this results in continuous numbers, is hidden away in those 3 dots at the end. By imagining $ π $ as $ 3.1415…0000… $ or $ 3.1415…1111… $ we can intuitively bridge the gap to the infinitely small. We see how the distance between two neighbouring real numbers must be so small, that it really is equivalent to $ 0 $.

That's not as crazy as it sounds. In the field of hyperreal numbers, every number actually has additional digits 'past infinity': that's its infinitesimal part. You can imagine this to be a multiple of $ \frac{1}{\infty} $, an infinitely small unit greater than $ 0 $, which I'll call $ ε $. You can add $ ε $ to a real number to take an infinitely small step. It represents a difference that can only be revealed with an infinitely strong microscope. Equality is replaced with adequality: being equal aside from an infinitely small difference.

You can explore this hyperreal number line below.

As $ ε $ is a fully functioning hyperreal number, $ ε^2 $ is also infinitesimal. In fact, it's even infinitely smaller than $ ε $, and we can keep doing this for $ ε^3, ε^4, …\,$ To make matters worse, if $ ε $ is infinitesimal, then $ \frac{1}{ε} $ must be infinitely big, and $ \frac{1}{ε^2} $ infinitely bigger than that. So hyperreal numbers don't just have inwardly nested infinitesimal levels, but outward levels of increasing infinity too. They have infinitely many dimensions of infinity both ways.

So it's perfectly possible to say that $ 0.999… $ does not equal $ 1 $, if you mean they differ by an infinitely small amount. The only problem is that in doing so, you get much, much more than you bargained for.

A Tug of War Between the Gods

That means we can finally answer the question we started out with: why did our continuous atoms seemingly all have $ 0 $ mass, when the total mass was not $ 0 $? The answer is that the mass per atom was infinitesimal. So was each atom's volume. The density, mass per volume, was the result of dividing one infinitesimal amount by another, to get a normal sized number again. To create a finite mass in a finite volume, we have to add up infinitely many of these atoms.

These are the underlying principles of calculus, and the final puzzle piece to cover. The funny thing about calculus is, it's conceptually easy, especially if you start with a good example. What is hard is actually working with the formulas, because they can get hairy very quickly. Luckily, your computer will do them for you:

That was differential and integral calculus in a nutshell. We saw how many people actually spend hours every day sitting in front of an integrator: the odometers in their cars, which integrate speed into distance. And the derivative of speed is acceleration—i.e. how hard you're pushing on the gas pedal or brake, combined with forces like drag and friction.

By using these tools in equations, we can describe laws that relate quantities to their rates of change. Drag, also known as air resistance, is a force which gets stronger the faster you go. This is a relationship between the first and second derivatives of position.

In fact, the relaxation procedure we applied to our track is equivalent to another physical phenomenon. If the curve of the coaster represented the temperature along a thin metal rod, then the heat would start to equalize itself in exactly that fashion. Temperature wants to be smooth, eventually averaging out completely into a flat curve.

Whether it's heat distribution, fluid dynamics, wave propagation or a head bobbing in a roller coaster, all of these problems can be naturally expressed as so called differential equations. Solving them is a skill learned over many years, and some solutions come in the form of infinite series. Again, infinity shows up, ever the uninvited guest at the dinner table.

Closing Thoughts

Infinity is a many splendored thing but it does not lift us up where we belong. It boggles our mind with its implications, yet is absolutely essential in math, engineering and science. It grants us the ability to see the impossible and build new ideas within it. That way, we can solve intractable problems and understand the world better.

What a shame then that in pop culture, it only lives as a caricature. Conversations about infinity occupy a certain sphere of it—Pink Floyd has been playing on repeat, and there's usually someone peddling crystals and incense nearby.
"Man, have you ever, like, tried to imagine infinity…?" they mumble, staring off into the distance.

"Funny story, actually. We just came from there…"

Comments, feedback and corrections are welcome on Google Plus. Diagrams powered by MathBox.

More like this: How to Fold a Julia Fractal.

Like Hands on a Clock

What does it mean to let numbers turn? Well, when making mathematical choices, we have to be careful. You could declare that $ 1 + 1 $ should equal $ 3 $, but that only opens up more questions. Does $ 1 + 1 + 1 $ equal $ 4 $ or $ 5 $ or $ 6 $? Can you even do meaningful arithmetic this way? If not, what good are these modified numbers? The most important thing is that our rules need to be consistent for them to work. But if all we do is swap out the symbols for $ 2 $ and $ 3 $, we didn't actually change anything in the underlying mathematics at all.

So we're looking for choices that don't interfere with what already works, but add something new. Just like the negative numbers complemented the positives, and the fractions snugly filled the space between them—and the reals somehow fit in between that—we need to go look for new numbers where there currently aren't any.

Pulling a Dragon out of a Hat

With a basic grasp of what complex numbers are and how they move, we can start making Julia fractals.

At their heart lies the following function:

$$ f(z) = z^2 + c $$

This says: map the complex number $ z $ onto its square, and then add a constant number to it. To generate a Julia fractal, we have to apply this formula repeatedly, feeding the result back into $ f $ every time.

$$ z_{n+1} = (z_n)^2 + c $$

We want to examine how $ z_n $ changes when we plug in different starting values for $ z_1 $ and iterate $ n $ times. So let's try that and see what happens.

So, WebGL it was, because I needed 3D. Unfortunately that meant the promise of having it just work everywhere was tempered by a lack of browser support, but I would certainly hope that's something we can overcome sooner than later. Dear Apple and Microsoft: get your shit together already. Dear Firefox and Opera: your WebGL performance could be a lot better.

Shady Dealings

These days I don't really touch WebGL without going through Three.js first. Three.js is a wonderful, mature engine that contains tons of useful high-level components. At the same time, it also does a great job in just handling the boilerplate of WebGL while not getting in the way of doing some heavy lifting yourself.

Rendering vector-style graphics with WebGL is not hard, certainly easier than photorealistic 3D. Primitives like lines and points are sized in absolute pixels by default, and with hardware multisampling for anti-aliasing, you get somewhat decent image quality out of it. Though, as is typical for a Web API, we're treated like children and can only cross our fingers and request anti-aliasing politely, hoping it will be available. Meanwhile native developers have full control over speed and quality and can adjust their strategy to the specific hardware's capabilities. The more things change... And then Chrome decided to disable anti-aliasing altogether due to esoteric security issues with buggy drivers. Bah.

Now, when rendering with WebGL, you really have two options. One is to just treat it as a dumb output layer, loading or generating all your geometry in JavaScript and rendering it directly in 3D. With the speed of JS engines today, this can get you pretty far.

Viewports, Primitives and Renderables

At its core, Three.js matches pretty directly with WebGL. You can insert objects such as a Mesh, Line or ParticleSystem into your scene, which invokes a specific GL drawing command with high efficiency. As such, I certainly didn't want to reinvent the wheel.

Hence, MathBox is set up as a sort of scene-manager-within-a-scene-manager. It's a little sandbox that speaks the language of math, allowing you to insert various primitives like curves, vectors, axes and grids. Each of these primitives then instantiates one or more renderables, which simply wrap a native Three.js object and its associated ShaderGraph material. Thus, once instantiated, MathBox gets out of the way and Three.js does the heavy lifting as normal. You can even insert multiple mathboxen into a Three.js scene if you like, mixed in with other objects.

For example, a vector primitive is rendered as an arrow: it consists of a shaft and an arrowhead, realized as a line segment and a cone. An axis primitive is an arrow as well, but it also has tick marks (specially transformed line segments), and is positioned implicitly just by specifying the axis' direction rather than a start and end point.

To render curves and surfaces, you can either specify an array of data points or a live expression to be evaluated at every point. This turned out to be essential for the kinds of intricate visualizations I wanted to show, my slides being driven by timed clocks, shared arrays of data points, and live formulas and interpolations. I even fed in data from a physics engine, and it worked perfectly.

This is all tied together through Viewport objects, which define a specific mapping from a mathematical coordinate space into the 3D world space of Three.js. For example, the default cartesian viewport has the range [–1, 1] in the X, Y and Z directions. Altering the viewport's extents will shift and scale anything rendered within, as well as reflow grids and tick marks on each axis.

There are two more sophisticated viewport types, polar and spherical, which each apply the relevant coordinate transform, and can transition smoothly to and from cartesian. More viewport types can be added, all that is required is to define an appropriate transformation in JavaScript and GLSL. That said, defining a seamless transition to and from cartesian space is not always easy, particularly if you want to preserve the aspect-ratio through the entire process.

Interpolate all the things!

Finally, I had to tackle the problem of animation, keeping in mind a tip I learnt from the ever so mindbending Vihart: "If I can draw the point of a sentence, I don't actually need to say the sentence." This applies doubly so for animation: every time you replace a "before" and "after" with a smooth transition, your audience implicitly understands the change rather than having to go look for it.

Hence, each primitive can be fully animated. Each has a set of options (controlling behavior) and styles (controlling GLSL shaders), and there is a universal animator that can interpolate between arbitrary data types in a smart fashion.

For example, given a viewport with the XYZ range [[–1, 1], [–1, 1], [–1, 1]], you can tell it to animate to [[0, 2], [0, 1], [–3, 3]], and it just works. The animator will recursively animate each subarray's elements, and any dependent objects like grids and axes will reflow to match the intermediate values. This works for colors, vectors and matrices too. In case of live curves with custom expressions, the animator will invoke both the old and the new, and interpolate between the results.

However, executing animations manually in code is tedious, particularly in a presentation, where you want to be able to step forward and backward. So I added a Director class whose job it is to coordinate things. All you do is feed it a script of steps (add this object, animate that object). Then, as it applies them, it remembers the previous state of each object and generates an automatic rollback script. It also contains logic to detect rapid navigation, and will hurry up animations appropriately. This avoids that agonizing situation of watching someone skip through their slide deck, playing the same cheesy PowerPoint transitions over and over again.

Presenting Naturally

With MathBox's core working, it was time to build my slides for the conference. After a quick survey, I quickly settled on deck.js as an HTML5 slidedeck solution that was clean and flexible enough for my purposes. However, while MathBox can be spawned inside any DOM element, it wouldn't work to insert a dozen live WebGL canvases into the presentation. The entire thing would grind to a halt or at least become very choppy.

So instead, I integrated each MathBox graphic as an IFRAME, and added some logic that only loads each IFRAME one slide before it's needed, and unloads it one slide after it's gone off screen. To sync up with the main presentation, all deck.js navigation events were forwarded into each active IFRAME using window.postMessage. With the MathBox Director running inside, this was very easy to do, and meant that I could skip around freely during the talk, without any worries of desynchronization between MathBox and the associated HTML5 overlays.

In fact, I applied a similar principle to this post. To avoid rendering all diagrams simultaneously and spinning up laptop fans more than necessary, each MathBox IFRAME is started as it scrolls into view and stopped once it's gone.

I've also found that having a handheld clicker makes a huge difference while speaking—as it allows you to gesture freely and move around. So, I grabbed the infrared remote code from VLC and built a simple bridge from to Cocoa to Node.js to WebSocket to allow the remote to work in a browser. It's a shame Apple's decided to discontinue IR ports on their laptops. I guess I'll have to come up with a BlueTooth-based solution when I upgrade my hardware.

Towards MathBox 1.0

In its current state, MathBox is still a bit rough. The selection of primitives and viewports is limited, and only includes the ones I needed for my presentation. That said, it is obvious you can already do quite a lot with it, and I couldn't have been happier to hear that all this effort had the desired response at the conference. I wasn't 100% sure whether other people would have the same a-ha moments that I've had, but I'm convinced more than ever that seeing math in motion is essential for honing our intuition about it. MathBox not only makes animated diagrams much easier to make and share, but it also opens the door to making them interactive in the future.

I plan to continue to evolve MathBox as needed by using it on this site and addressing gaps that come up, though I've already identified a couple of sore points:

• I used tQuery as a boilerplate and because I liked the idea of having a chainable API for this. However, this also means it's currently running off an outdated version of Three.js. I need to look into updating and/or dropping tQuery.
  MathBox has been updated to Three.js r53.
• Numeric or text labels are completely unsupported. It should be possible to use my CSS3D renderer for Three.js to layer on beautifully typeset MathJax formulas, positioning them correctly in 3D on top of the WebGL render.
  I've added labeling for axes. I've integrated MathJax, but it's tricky because the typesetting is painfully slow in the middle of a 60fps render. But it's automatically used if MathJax is present.
• All styles have to be specified on a per-object basis. Some form of stylesheet, default styles or class mechanism to allow re-use seems like an obvious next step.
• There are undoubtedly memory leaks, as I was focused first and foremost on getting it to work.
• Expressions that don't change frame-to-frame are still continuously re-evaluated, which is wasteful. There is a live: false flag you can set on objects, but it triggers a few bugs here and there.
• There needs to be a predictable, built-in way of running a clock per slide to sync custom expressions off of. In my presentation I used a hack of clocks that start once first invoked, but this lacks repeatability.
  I added a director.clock() method that gives you a clock per slide.

Finally, it doesn't take much imagination to imagine a MathBox Editor that would allow you to build diagrams visually rather than having to use code like I did. However, that's a can of worms I'm not going to open by myself, especially because the API is already quite straightforward to use, and the library itself is still a bit in flux. Perhaps this could be done as an extension of the Three.js editor.

You can see what MathBox is really capable of in the conference video. I invite you to play around with MathBox and see what you can make it do. Contributions are welcome, and the architecture is modular enough to allow its functionality to grow for quite some time.