In this tutorial we’ll learn how to position items in our world relative to the bones of our character’s skeleton. You can use this techinque for a whole host of things, the most common being placing things inside of your rig’s hands.

Our approach will be to implement the interactive demo above, while explaining what we’re doing along the way. We won’t nose-dive into every single aspect of the math that powers this, but you should hopefully walk away with a base understanding of the concepts and terms that power placing objects onto bones, as well as a working reference implementation.

With that, let’s jump right in.

Dependencies

Let’s create the directory for our tutorial and download all of our dependencies. Once you’ve downloaded everything you will not need an internet connection for the remainder of the tutorial.

mkdir webgl-wield-tutorial
cd webgl-wield-tutorial
touch tutorial.js

# Our character model
curl -OL https://github.com/chinedufn/\
webgl-wield-item-tutorial/blob/master/cowboy-model\
.dae > cowboy.dae

# Our character texture
curl -OL https://github.com/chinedufn/\
webgl-wield-item-tutorial/blob/master/cowboy-texture\
.dae > cowboy.dae

# Our monkey and short stick
curl -OL https://github.com/chinedufn/\
webgl-wield-item-tutorial/blob/master/\
short-sword.obj > short-sword.obj

curl -OL https://github.com/chinedufn/\
webgl-wield-item-tutorial/blob/master/\
long-sword.obj > long-sword.obj

# Convert our model from COLLADA to JSON
npm install -g collada-dae-parser@0.12.0 && \
dae2json cowboy-model.dae > cowboy-model.json

# Convert our sticks from OBJ to JSON
npm install -g wavefront-obj-parser@1.0.0
obj2json short-sword.obj > short-sword.json
obj2json long-sword.obj > long-sword.json

# Grab our npm dependencies
npm install keyframes-to-dual-quats@1.0.0 \
change-mat4-coordinate-system@1.0.0 \
gl-mat3@1.0.0 gl-mat4@1.1.4 gl-vec3@1.0.3
load-collada-dae@0.6.6 \
load-wavefront-obj@0.8.0 \
raf-loop@1.1.3 \
skeletal-animation-system@0.6.2

# Install a dev server for viewing our tutorial
npm install -g budo

Alright that’s all of our dependencies out of the way. Time to write some code.

Implementation

Open up the tutorial.js file that you created earlier and add the following code, in order:

We start by importing our matrix math dependencies.

var glMat4 = require('gl-mat4')
var glVec3 = require('gl-vec3')
var glMat3 = require('gl-mat3')

Next we create a canvas and grab the WebGL context from our canvas. We’ll use this WebGL context in order to draw to our canvas’s drawing buffer. This drawing buffer holds what you end up seeing in the page.

var canvas = document.createElement('canvas')
canvas.width = 400
canvas.height = 400
canvas.style.display = 'block'

var gl = canvas.getContext('webgl')
gl.enable(gl.DEPTH_TEST)

Create a span that holds the display showing whether we’re holding the Monkey stick or the Short stick.

var stickLabel = document.createElement('span')
stickLabel.style.marginLeft = '10px'
stickLabel.style.fontFamily = 'Helvetica Neue'
stickLabel.style.fontSize = '23px'
stickLabel.innerHTML = 'Short Stick'

Now we make a button that will toggle us from holding our Monkey stick or our Short stick.

var toggleStickButton = document.createElement('button')
toggleStickButton.style.height = '40px'
toggleStickButton.style.cursor = 'pointer'
toggleStickButton.innerHTML = 'Click to change stick'
var useLongStick = false
toggleStickButton.onclick = function () {
  useLongStick = !useLongStick
  stickLabel.innerHTML = useLongStick ? 'Monkey stick' : 'Short stick'
}

Next up we add all of the controls that we just created to the page.

var demoLocation = document.querySelector('#wield-animation-tutorial') 
|| document.body
demoLocation.appendChild(toggleStickButton)
demoLocation.appendChild(stickLabel)
demoLocation.appendChild(canvas)

Our cowboy model comes with 4x4 matrices that represent all of our joint’s positions at any given keyframe. We convert these matrices into dual quaternions because they’re easier to interpolate and they lead to fewer visual artifacts.

var cowboyJSON = require('./cowboy-model.json')
var keyframesToDualQuats = require('keyframes-to-dual-quats')
cowboyJSON.keyframes = keyframesToDualQuats(cowboyJSON.keyframes)

We create the vertex shader that we’ll use when rendering our two sticks. It takes aVertexPosition, which corresponds to every single vertex in the stick. It then transforms these vertices with uStartOffsetMatrix, which will help to make sure that the stick is positioned precisely in the hand of the hand, since our bone actually happens to be just above the hand.

We then place our item onto the hand bone + our start offset.

After this we transform our item’s vertices with our hand’s uHandMVMatrix, which is the matrix that represents our animated hand’s current location and rotation in the world.

Finally we transform using our perspective matrix. The perspective matrix helps with calculating your field of view and how far away you can see into your world.

function createItemVertexShader (opts) {
  return `
    attribute vec3 aVertexPosition;

    uniform mat4 uStartOffsetMatrix;
    uniform vec3 uHandBindPosition;

    uniform mat4 uHandMVMatrix;
    uniform mat4 uPMatrix;

    void main (void) {
      vec4 vertexPosition = uStartOffsetMatrix * vec4(aVertexPosition, 1.0);

      vertexPosition = vertexPosition + vec4(uHandBindPosition, 1.0);
      vertexPosition.w = 1.0;

      gl_Position = uPMatrix * uHandMVMatrix * vertexPosition;
    }
  `
}

Our fragment shader is a lot simpler. We just pass in the color of our stick. The Monkey stick is blue, and the Short stick is green.

function createItemFragmentShader () {
  return `
    precision mediump float;
    uniform vec4 uVertexColor;
    void main(void) {
      gl_FragColor = uVertexColor;
    }
  `
}

Next we buffer up all of our texture and model data so that we can draw it later.

var cowboyModel
var texture = new window.Image()
var monkeyStick
var shortStick
texture.onload = function () {
  var loadCollada = require('load-collada-dae')
  cowboyModel = loadCollada(gl, cowboyJSON, {texture: texture})

  var loadWFObj = require('load-wavefront-obj')
  shortStick = loadWFObj(gl, require('./short-sword.json'), {
    createVertexShader: createItemVertexShader,
    createFragmentShader: createItemFragmentShader,
    // We aren't using this texture. Ignore it
    textureImage: texture
  })
  monkeyStick = loadWFObj(gl, require('./long-sword.json'), {
    createVertexShader: createItemVertexShader,
    createFragmentShader: createItemFragmentShader,
    // We aren't using this texture. Ignore it
    textureImage: texture
  })
}
texture.src = 'cowboy-texture.png'

We initialize a few variables that we’ll use later in our render loop.

var secondsElapsed = 0
var renderLoop = require('raf-loop')
var animationSystem = require('skeletal-animation-system')
var perspectiveMatrix = require('gl-mat4/perspective')(
  [], Math.PI / 4, 400 / 400, 0.1, 100
)

Now comes the mathematical meat of our tutorial. We first get the index of the bone that we want to use, in this case it’s our model’s Hand_R bone. Bones get named in your 3d modeling software, so it just as well have been named anything else.

Next we grab the inverse bind matrix for our bone. The inverse bind matrix is the matrix that when multiplied by your joint’s default position moves your joint to the origin, in model space.

If we take the inverse of this inverse bind matrix, we have the bind matrix, which tells us the default location of our bone in it’s model’s coordinate space.

Before inverting it we transpose it since Blender uses row major matrices, but we want a column major matrix. WebGL uses column major matrices.

Blender uses a right handed coordinate system, but WebGL uses a left handed coordinate system, so we change the coordinate system of our matrix.

Finally we look at the x, y, z translation components of our matrix in order to know where the bone’s default (bind pose) location is.

And with that, we now have the location of the bone before the model has been animated. We’ll make use of this in our render loop.

var handJointIndex = cowboyJSON.jointNamePositionIndex['Hand_R']
var handRInverseBind = cowboyJSON.jointInverseBindPoses[handJointIndex]
var handRBind = []
require('gl-mat4/transpose')(handRInverseBind, handRInverseBind)
require('gl-mat4/invert')(handRBind, handRInverseBind)

var changeMat4CoordinateSystem = require('change-mat4-coordinate-system')
handRBind = changeMat4CoordinateSystem.rightToLeft(handRBind)

var handBindLocation = [ handRBind[12], handRBind[13], handRBind[14] ]

We’ve already covered most of the code in this next block in our WebGL Skeletal Animation Tutorial. But just to recap, we’re calculating the joint positions based on the amount of time that has elapsed so that we can render our model in a new position.

renderLoop(function (millisecondsSinceLastRender) {
  var lightingDirection = [1, -1, -4]
  glVec3.normalize(lightingDirection, lightingDirection)
  glVec3.scale(lightingDirection, lightingDirection, -1)

  var cowboyUniforms = {
    uUseLighting: true,
    uAmbientColor: [0.9, 0.9, 0.9],
    uLightingDirection: lightingDirection,
    uDirectionalColor: [1, 0, 0],
    uPMatrix: perspectiveMatrix
  }

  if (cowboyModel) {
    secondsElapsed += millisecondsSinceLastRender / 1000
    var interpolatedJoints = animationSystem.interpolateJoints({
      currentTime: secondsElapsed,
      keyframes: cowboyJSON.keyframes,
      jointNums: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17],
      currentAnimation: {
        range: [6, 17],
        startTime: 0
      }
    }).joints

    for (var i = 0; i < 18; i++) {
      cowboyUniforms['boneRotQuaternions' + i] = interpolatedJoints[i]
      .slice(0, 4)
      cowboyUniforms['boneTransQuaternions' + i] = interpolatedJoints[i]
      .slice(4, 8)
    }

We get the matrix that transforms our hand from the bind position to the current animation position. We use this matrix to transform our stick from being placed directly on top of the bind position to being translated and rotated relative to the hand’s bind. By doing this we make our stick track our hand.

var animatedHandRMatrix = changeMat4CoordinateSystem.rightToLeft(
  require('dual-quat-to-mat4')(
    interpolatedJoints[9].concat(interpolatedJoints[9])
  )
)

We combine our hand animation transformation with our model’s current location to get a matrix that will transform our sticks with the model’s current hand position.

We then create an offset matrix to re-orient the stick before putting it into the hand. We move it down a tad and rotate it forward a bit, then we rotate it about the Z axis to prevent it from colliding with the character’s leg.

var mvMatrix = [1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0.0, -1.0, -27.0, 1]

var handModelViewMatrix = []
glMat4.multiply(handModelViewMatrix, mvMatrix, animatedHandRMatrix)

var stickOffsetFromHand = glMat4.create()
glMat4.translate(stickOffsetFromHand, stickOffsetFromHand, [0.0, -1.0, -0.1])
glMat4.rotateX(stickOffsetFromHand, stickOffsetFromHand, Math.PI / 2)
glMat4.rotateZ(stickOffsetFromHand, stickOffsetFromHand, Math.PI / 4)

Lastly we look at our useLongStick variable to determine which stick to render. We then render our cowboy model.

if (useLongStick) {
  gl.useProgram(monkeyStick.shader.program)
    monkeyStick.draw({
      attributes: monkeyStick.attributes,
      uniforms: {
        uVertexColor: [0.0, 0.0, 1.0, 1.0],
        uHandMVMatrix: handModelViewMatrix,
        uPMatrix: perspectiveMatrix,
        uHandBindPosition: handBindLocation,
        uStartOffsetMatrix: stickOffsetFromHand
      }
    })
} else {
  gl.useProgram(shortStick.shader.program)
  shortStick.draw({
    attributes: shortStick.attributes,
    uniforms: {
      uVertexColor: [0.0, 1.0, 0.0, 1.0],
      uHandMVMatrix: handModelViewMatrix,
      uPMatrix: perspectiveMatrix,
      uHandBindPosition: handBindLocation,
      uStartOffsetMatrix: stickOffsetFromHand
    }
  })
}

cowboyUniforms.uMVMatrix = mvMatrix
cowboyUniforms.uNMatrix = glMat3.fromMat4([], mvMatrix)

gl.useProgram(cowboyModel.shaderProgram)
  cowboyModel.draw({
    attributes: cowboyModel.attributes,
    uniforms: cowboyUniforms
})
}
}).start()

You did it!

Great work. If you followed along you’ve successfully implemented item wielding. You may or may not understand all of the pieces, so you can go ahead and dive into the commented source code, or more closely check out some of the small modules that power this tutorial.

Til’ next time

- CFN