Motion analysis
From WiiLi
The Wiimote and Nunchuk controllers contain 3-axis accelerometers (ADXL330) which report back, in arbitrary units, the instantaneous acceleration imparted on the controller by the player holding it (or whatever surface is supporting it). At rest, this acceleration is g, but in the upward direction. In free fall, the controller reports back approximately zero acceleration.
A rigid body in 3D space, however, has 6 degrees of freedom: 3 linear translation directions (X, Y, Z) and 3 rotation angles (pitch, roll, yaw), as shown in the figure at right.
Contents |
[edit] Calibration
If we can assume the sensor response to acceleration is roughly linear and the chip is precisely oriented on the circuit board, then calibrating the Wiimote sensor is a three step procedure. We only need the raw integer force sensor readings for these three positions on a flat, level table:
- Horizontal with the A button facing up: <math>(x_1,\, y_1,\, z_1)</math>
- IR sensor down on the table so the expansion port is facing up: <math>(x_2,\, y_2,\, z_2)</math>
- Laying on its side, so the left side is facing up: <math>(x_3,\, y_3,\, z_3)</math>
In each of these cases, we are measuring g on one axis, and the zero point on the other two axes. We can estimate the zero points of each axis to be:
<math>x_0 = (x_1 + x_2) / 2\,</math>
<math>y_0 = (y_1 + y_3) / 2\,</math>
<math>z_0 = (z_2 + z_3) / 2\,</math>
Starting with any integer reading from the sensor <math>(x_{raw},\,y_{raw},\,z_{raw})</math>, the calibrated force readings are:
<math>x = \frac{x_{raw} - x_0}{x_3 - x_0}</math>
<math>y = \frac{y_{raw} - y_0}{y_2 - y_0}</math>
<math>z = \frac{z_{raw} - z_0}{z_1 - z_0}</math>
The calibrated force is now in units where g = 1.0.
To give you some idea of the size of these constants, for one particular Wiimote they were:
| <math>x_0\,</math> | 7 |
| <math>y_0\,</math> | 10 |
| <math>z_0\,</math> | 9.5 |
| <math>x_3 - x_0\,</math> | 27 |
| <math>y_2 - y_0\,</math> | 27 |
| <math>z_1 - z_0\,</math> | 27 |
Following a 3-step calibration procedure that requires a table might be a hassle for the user, so a less rigorous procedure may be preferable. Based upon the above example, we might conjecture (possibly incorrectly) that the gain of the sensor (size of g, basically) is more uniform between remotes than the zero offset. In that case, we would only need the remote to be at rest briefly in order to determine the zero offsets for all 3 axes at once. Such a procedure might in fact be the only way to calibrate the Nunchuk sensor, as that controller does not have the flat surfaces needed to perform the 3-step calibration.
[edit] Tilt Position Sensing
- If the remote is not accelerating, the tilt of the remote can be calculated directly using the force of gravity. First convert the acceleration signals from the sensors so that they vary between -1 g and +1 g (see Calibration, above). Then
roll = asin(ay/1 g) pitch = asin(ax/1 g)
- Be careful with these equations! If the remote is accelerating due to hand movement, then the normalized acceleration value could be more than +1 g or less than -1 g and asin is no longer defined. One way to check that the remote is not accelerating is to see if the vector sum of the three accelerometer readings is close to g.
- Note that the accelerometers become less sensitive to position when they are turned away from perpendicular to gravity (they lose resolution). For this reason, the z-axis accelerometer is not very helpful for position sensing when the remote is held in the standard position. However, as the remote is tilted away from that position, you need to use the z axis readings in addition to the other two. So:
roll = arctan2(az,ax) pitch = arctan2(az,ay)
- Of course, you can also use all three accelerometers:
roll = arctan2(ax,sqrt(ay2+az2)) pitch = arctan2(ay,sqrt(ax2+az2))
- If the remote is accelerating, you need to double integrate the acceleration readings to get the change in tilt relative to the last known value.
- To get an absolute value for the yaw, you have to use the IR sensor. If the remote is turned away from the sensor bar, it should be possible to estimate the yaw using the most recent sensor bar reading, the last known velocity of the sensor bar points, the distance from the sensor bar (estimated from the distance between the two sensor bar points), and the integrals of the accelerometer readings. However, the longer you do this the more error there will be.
- For Position Sensing without the IR sensor, I suggest using a button on the wiimote to "set pointer to the center". This way you don't even have to point the monitor (eg. pointing in a slideshow or anything using a projector). This could be a "solution" to errors while identifying the controller position (the user sometimes points to the center of the screen and presses a button). Without the IR it is difficult to achieve the effect of "laser pointing".
[edit] Dynamics
Since all known Wiimotes are in use near the surface of the Earth, there is a constant background force due to gravity, even when the player is not accelerating the remote. (Without the extra knowledge that we are on the Earth, this force would be indistinguishable from player-induced acceleration. A nice reminder of general relativity.) We will assume that all axes of rotation are close enough to the motion sensor that a rotation does not linearly accelerate the sensor. (i.e. there is no lever arm)
The acceleration <math>a(t)</math> measured by the sensor (not the true acceleration of the remote) is then:
<math>a(t) = R(t)(g\hat{z} + a_r(t))</math>
<math>\dot{a}(t) = \dot{R}(t)(g\hat{z} + a_r(t)) + R(t)\dot{a}_r(t)</math>
where <math>R\,\!</math> is the rotation matrix which transforms from the "living room frame" to the local frame of the remote, and <math>a_r\,\!</math> is the actual acceleration of the remote in the "living room frame". The rotation matrix is parameterized by 3 Euler angles, and the remote's linear acceleration has 3 components as well. With 6 dynamical variables, but only 3 observables, our system is underdetermined! However, we can extract some information if we make simplifying assumptions.
[edit] Pure Rotation
If we assume the remote experiences no acceleration (most likely at rest in normal usage conditions), then the motion sensor directly tells us which way is "up". From this we can derive two of the Euler angles of the controller, but not the angle of rotation around the z-axis in the living room frame (corresponding to "yaw" in the diagram when the controller is level). This lack of knowledge is one of the reasons it is difficult to control an on-screen pointer with just 3 accelerometers. The most natural motion for a user to indicate pointer movement horizontally is (roughly) to pivot the controller around the z-axis.
However, two angles are sufficient controls for many uses. The "up" vector that the sensor reports can be used directly, or decomposed into angles: <math> Put formula here </math>
[edit] No Rotation
If we assume the remote does not rotate (<math>\dot{R}(t) = 0</math>), then the only terms remaining in the motion equations are:
<math>a(t) = R(0)(g\hat{z} + a_r(t))</math>
<math>\dot{a}(t) = R(0)\dot{a}_r(t)</math>
Thus, to get a good estimate of <math>a_r(t)\,\!</math>, we still need to know the initial orientation of the remote <math>R(0)\,\!</math>. We can get this information while the remote is at rest using a method like that mentioned in the previous section.
[edit] Small Jerk
The amusingly named (but rarely used) term for the rate of change of acceleration is jerk. The jerk term for the remote shows up in the time derivative of the force recorded by the sensor, along with the rotation term that contains the angular velocity of the remote. We can extract both rotation and linear acceleration if we assume a few things:
- We know the "up" direction before the motion starts. This is <math>R(t=0)\hat{z}</math>.
- Throughout the motion, the jerk on the remote perpendicular to the current direction of gravity is small.
Then we can assume the time derivative of the force component which is perpendicular to our current estimate of the up direction is caused by the user rotating the controller only. This allows us to update our estimate of the up direction for the next time step. In each time step, we can also get the linear acceleration of the remote by subtracting our estimate of <math>gR(t)\hat{z}</math> from the current force sensor report. In effect we are integrating up a coupled set of ordinary differential equations. (Note, need to review the math here. Beware.)
The main problem with this technique is error accumulation in our estimate of "up." Since it is unlikely the user can keep the controller in constant linear motion without injuring themselves, the TV, or their opponent, we can look for times when the total reported force is close to g = 1.0 to recenter <math>R(t=0)\,\!</math>. You have to be careful when doing this because it is possible and probably common for the Wiimote to report an acceleration close to g = 1.0 while it is accelerating. When this happens your acceleration vector does not actually point "up" and you will recenter to an incorrect R. This can happen anytime you are accelerating both downward and in the horizontal plane.
[edit] Gestures
For more situations than one might first imagine, it is completely unnecessary to extract precise rotations and/or linear accelerations in the living room reference frame, as has been described above. Many games for the Wii have demonstrated the power of using the remote to issue commands through pre-defined motion patterns. For example, slashing with the remote to simulate a sword strike, or whipping the remote around in a circle like a lasso. Identifying these gesture-like commands in the motion data requires some fuzzy pattern matching, which is also insensitive to small rotations of the controller. AiLive is making a program called LiveMove available to Wii developers which allows them to train the system to recognize different motions by physically moving the controller in the desired pattern several times.
(Links to other published research on this type of motion recognition are welcome here.)
http://www.cybernet.com/~ccohen/gesture.html contains a lot of research papers. I'm currently looking into the neural network approach to create a gesture system for the wiimote. --Beau
Since end of december 2007 there exist a gesture recogition library using the wiimote, called "wiigee". I've developed this lib in a project for my university. It uses discrete hidden markov models instead of neural networks (like Beau wanted to try). As a java lib i've decided to use an event driven software design so it should be quite easy for you to integrate this in your current java/other language software project. In mid-february me and a friend would have a talk about our research paper we've written, called "gesture recognition with a wii controller", on this years TEI conference. This paper should be available after the conference on the wiigee-website. Currently i'm writing a short "howto" use this lib and i'm also working on a simple demo-code for you. The next time I would spend a lot of work in this, but i still need your help: please test the lib and give me further inspirations about what you're missing. Library can be found at sourceforge. Feel free to discuss about this lib in the wiili-forum (admin may create an own subforum?) and also feel free to contact me directly for questions or something. BePo
Here is a video of the gesture recognition process: Video at Youtube.com
[edit] References
Introduction to using accelerometers (A presentation from Analog Devices): http://www.analog.com/Analog_Root/static/library/techArticles/mems/sensor971.pdf
Some of the data sheets for Analog Devices accelerometers have hints for how to analyze the output: http://www.analog.com/en/subCat/0,2879,764%255F800%255F0%255F%255F0%255F,00.html
A paper on calculating orientation with linear accelerometers: http://www.freescale.com/files/sensors/doc/app_note/AN3461.pdf
[edit] See Also
FIELD_MESSAGE_dronrac
