9 DoF Motion Sensor Bakeoff
Since I wrote about Affordable 9 DoF Sensor Fusion and Small Pressure Sensors more than a year ago I have made sufficient progress in my continuing studies of motion sensors and sensor fusion to warrant another article. This time I will compare the performance of several popular pressure sensors and 9 DoF (integrated accelerometers, gyroscopes, and magnetometers) motion sensors and how their differences affect simple open-source sensor fusion results. It might seem strange at first to include pressure sensors in such a study but they offer altimetry which can complement and stabilize height assessments when pressure data is folded into the sensor fusion solution. We will demonstrate this in a subsequent article.
The pressure sensors are Measurement Specialties' MS5637, ST Microelectronics' LPS25H, Bosch's BMP280, and Invensense's ICM20730 new 7-axis (accel/gyro/pressure sensor) motion sensor.
Pressure sensor test board used in this study: (clockwise from top left) MS5637, ICM20730 (evaluation sample), BMP280, and LPS25H.
The 9 DoF motion sensors are Invensense's MPU9250, ST Microelectronics' LSM9DS0, Maxim Integrated's MAX21100 (no magnetometer), and Bosch's BMX055.
Motion sensor test board used in this study: (clockwise from top left) BMX055, MPU9250, LSM9DS0, and MAX21100.
It is useful to have all the sensors in a given category on the same board. That way any variation in power supply noise, board stresses from assembly, and environmental differences during testing are minimized. We can test the response of different sensors to the same stimulus at the same time under the same conditions. We have also attempted to maintain as much commonality in the software as possible and to use similar sample data rates, bandwidths, etc where possible to make the tests fair.
Now, this study is not meant to be an exhaustive characterization of every aspect of performance; the manufacturers publish expected performance of these sensors in the respective data sheets and expensive machines are used to quantify specific sensor performance parameters. Nor are we including every available 9 DoF motion or pressure sensor solution. Notably absent is ST Microelectronics' newer LSM9DS1, and there are a variety of 6 DoF accel/gyro motion sensors that can be paired with a separate magnetometer to provide a 9 DoF solution. Also absent is the MS5611, and a variety of other pressure sensors from OMRON, etc. Rather, our purpose here is to answer two questions: Are there significant differences in performance between available 9 DoF motion sensor solutions and, if so, is there any practical effect on the corresponding sensor fusion solution?
In some sense we are asking, which is the "best" accelerometer, gyro, magnetometer, and altimeter to use for absolute orientation or position reckoning. Remember, here we are limiting the study to a narrow selection of small, inexpensive, MEMS sensors and any differences will only matter if we can detect their effect in the resulting sensor fusion output. We will use Madgwick's simple open-source MARG sensor fusion algorithm so all the sensors are evaluated on the same footing.
The testing here shall be rather simple but practically useful. We want to characterize the random or white noise associated with each sensor at rest, and the drift of the sensor output over time. These are of practical importance since motion sensor data are often integrated to get position or as part of a PID (proportion, integral, differential) error correction algorithm. White noise or jitter limits the inherent accuracy of the sensor measurements while drift will introduce large errors in orientation and position estimates rather quickly.
The sensors do not allow identical settings of sample data rate, filtering, etc., but we tried to get close enough not to bias the comparisons too much. Let's start with the easiest comparisons, the pressure sensors. It is difficult to use the same settings since each of these sensors uses a different strategy to reduce noise. Here is a table of the relevant settings for the test:
Pressure sensor test settings
Most pressure sensors use oversampling to internally average the pressure (and temperature) before it is read out by the microcontroller. This is the simple strategy used by the MS5637 (and MS5611). The LPS25H also uses this strategy but has an additional means to average the data via the use of its 96 byte First-In-First-Out (FIFO) buffer. With it's FIFO mean mode an additional user-specified averaging can be done as data samples are accumulated in the FIFO buffer. This allows the power usage for this device to be very low while maintaining high resolution data output. At least that is the claim. The Bosch BMP280 also uses oversampling but has a low-pass filter in addition to simple averaging which reduces noise quite effectively. The tradeoff is a rather long standby time while the sensor processes the data. For most altimetry, a ~10 Hz rate is probably sufficient, but there is definitely a trade-off between low-noise and sample rate.
The results of a simple test of the jitter or white noise from these four pressure sensors is shown below.
Pressure data from the MS5637, LPS25H, BMP280, and ICM20730
This test spans only about a minute of data (x-axis is milliseconds) where in the middle the pressure sensor test board was raised from the desk, where it starts out, to the ceiling (or at least as high as I could reach) and then back again. We can learn a lot from this simple short test. Firstly, there is a difference between the absolute pressures (here given in millibar) reported by the four sensors. At least they all fall within +/- 2 millibars of the mean (+/- 0.2% absolute accuracy is pretty good). They all do a pretty good job of indicating that the pressure changes when the height changes by ~6 feet or so. But the inherent jitter displayed by the four sensors is different. So different, in fact, that the pressure changes due to height are hard to discern in the worst case. It might be common sense, and we have by no means proved anything here yet, but one might expect that low inherent jitter would be required for the best position and orientation accuracy. The trick is to reduce the jitter by good design, low-pass filtering, averaging or what have you without destroying the ability of the sensor to discriminate small changes, here in pressure. Based on this simple test, we must conclude that the BMP280 is doing a pretty good job at balancing these two requirements, with the MS5637 a close second. In fact, one could argue the MS5637 is the better of the two since it responds quickly to small variations in the pressure due to altitude changes while providing low enough jitter to distinguish the changes.
There is another important requirement for pressure sensor performance---namely long-term drift. This one is tricky because the ambient pressure does change even on relatively short time scales like hours. Here we will simply ask whether there is significant drift over the period of one hour. For most navigation applications this provides a sufficient measure of drift since the baseline from which drift is measured can be periodically zeroed. Yes, there are some applications where pressure sensor drift could be catastrophic. But for everyday navigation, meaning determining how high someone climbed on a hike, or what floor we happen to be on at the mall, we want to know what the relative error over a short period might be. In particular, we want to know if there is any difference between the four pressure sensors being tested here in the way of short-term (~1 hour) stability. Below is a pressure plot from the same four pressure sensors during a ~two hour session where the pressure sensor test board sat on a desk and spewed data to a serial monitor inside my house. The scales are millibars and seconds, not milliseconds this time. So the record is just less than two hours.
Notice that each of the pressure sensors detects the same signal changes. That is, there appears to be two slight peaks in the pressure at ~2000 s and ~3000 s. The drift here is likely due to a real change on the household pressure; it is unlikely four independent pressure sensor would have internal drift so closely matched. Note also that the drift is less than 1 millibar, or less than 0.1 %, for all the pressure sensors over nearly a two hour period. Overall, this is pretty good performance. Of course, the change in altitude corresponding to a 1 millibar pressure change is ~30 feet! So you might not be able to tell if you left your car on tier 3 or tier 5 of the parking garage with pressure data alone. But more importantly, there is really nothing to distinguish the performance of one pressure sensor over another except the jitter as discussed above.
Results of the pressure drift test translated into relative altitude variations over the same ~2 hour period; altitude in cm. The altitude variations are about 1 meter; good enough to find your car at the mall parking garage! (Top, MS5637, Middle, LPS25H, Bottom, BMP280)
There are more or less application-specific tradeoffs with respect to device footprint, cost, and power usage that ought to be considered as well as technical performance. Here, the BMP280 is exceptional. It is 2 mm x 2.5 mm, 20% smaller than the next smallest pressure sensor, the LPS25H at 2.5 mm x 2.5 mm. It can be purchased in quantity as low as $1.69 each, compared to $1.80 for the MS5637 and $2.50 for the LPS25H. The ICM20730 is not yet available and inclusion here is really unfair. The ICM20730 pressure sensor I tested here is an engineering evaluation unit with an embedded 6 DoF motion sensor. It is a unique device not commercially available yet, and certainly not in the pre-production form I tested it in. So apologies to Invensense for even mentioning it here, but the fact is I did include it on my pressure sensor test board and did test it. So truth in advertising---I am just reporting what I found. Power usage is not the highest priority for every application like it is for cellphones, but almost all of these devices take ~10s of microAmps to run at full blast. Hardly worth worrying about next to performance considerations IMHO.
Let's talk about motion sensors. We'll start with perhaps the most mysterious one, the magnetometer. Modern MEMS magnetometers use the Hall effect to measure tiny currents in a moving silicon plate that are proportional to the ambient magnetic field. The motion sensors translate the sensed analog current into raw digital counts via an ADC (analog-to-digital converter) on the embedded ASIC (application-specific integrated circuit) and the microcontroller that reads the raw digital data via I2C or SPI protocol then scales it into the more familiar (milli)Gauss. The utility of the magnetometer as part of a suite of motion sensors for sensor fusion is it provides a second absolute reference frame in addition to gravity since the magnetic fields measured are usually dominated by that of the Earth (about 500 milliGauss oriented along the North Pole).
Let's repeat our simple test of jitter and drift for the three magnetometers on our test board, the AK8963C embedded inside the MPU9250, the BMM150 embedded inside the BMX055, and the LIS3MDL (?) embedded inside the LSM9DS0.
Jitter and drift for the three magnetometers embedded in the 9 DoF motion sensors tested here: the MPU9250 (I), BMX055 (B), and LSM9DS0 (L).
There are couple of interesting things to observe in the data. Firstly, the orientation of the magnetometers is not the same. The MPU9250 and LSM9DS0 (yellow and light blue at the top, respectively) have their positive z-axes pointed down on our test board while the BMX055 (brown at the bottom) has its positive z-axis pointed up. Even though each magnetometer has been bias calibrated by measuring the minimum/maximum excursion on each axis during a figure-eight motion on startup to allow recentering of the respective response surfaces on the origin, the BMX055 and MPU9250 read very different values for the z-axis magnetic field (~450 mG) compared to the average z-axis magnetic field measured by the LSM9DS0 (~330 mG). There could be at least two reasons for this. One, the BMX055 magnetometer doesn't measure magnetic fields like the other two sensors. It measures a 13-bit output on the X/Y axes and a 15-bit output on the z-axis and the resistance of the Hall sensor plate. These four data are combined in software through a wickedly complicated set of formulae to produce 3-axis, temperature-compensated 16-bit magnetic fields. It is possible that when I transcribed the formulae from the Bosch application program I made a mistake. However, the fact that the MPU9250 and BMX055 report similar values makes this unlikely. Secondly, the response of the three axes in each sensor is likely to be different and a second bias calibration is needed to sphere-ize (rescale) the response to make it equal along all three axes. In other words, in addition to being centered off the origin, the response surface is likely to be non-spherical as well with no calibration. This can be seen below:
X-axis magnetic field (in mG) plotted against Z-axis magnetic field for the three magnetometers on our test board when undergoing a figure-eight motion.
The data clouds are more or less centered at the origin as they should be after the recentering bias calibration performed on start up. The small cluster of points for each cloud is the response of the magnetometers at rest (where the data should read Mx = ~+/-200 mG, Mz = ~+/-400 mG). It is interesting to note that response surfaces of the BMX055 and MPU9250 seem to be rotated wrt each other by 90 degrees. Notice that none of these data clouds is perfectly circular in shape; all three magnetometers have some cross-axis skew which for the full 3D response surface requires a 3 x 3 calibration matrix to correct. The correction should take the ellipsoidal response surface and spherize it by scaling the axial responses. It is a straightforward but mathematically hairy set of matrix equations. And we can see that this skew is one reason for the LSM9DS0 underestimating the z-axis magnetic field. But the LSM9DS0 response surface is definitely smaller than the other two. It looks like it needs to be scaled up by about 20%. I am not sure I understand how such an error could arise in the coding where a factor of two error might occur, but not a 20% error. Mysterious indeed!
The jitter in the data is comparable at about 14 mG or ~3% on the z-axis and between 1 and 3% on the x and y axes for this particular test data. I have noticed that the jitter changes from test run to run probably because the sensors were slightly moved during the test. Below is another data set where I took extra care not to move the sensors during the test run. You can see that the general trends are the same but the absolute magnitude of the magnetic fields have changed a bit. The jitter in the second set of data is smaller, being less than 2% for all three sensors on all three axes and a remarkable 1-2 mG for the x- and y-axes of the LSM9DS0. It is very difficult to return the sensors to a fixed orientation to test reproducibility without some sort of fixture (easily done, I just didn't do it). The drift in all cases seems to be acceptably small. Except for the absolute accuracy, there is not much to distinguish these magnetometers.
Another set of data measuring jitter and drift for the three magnetometers embedded in the 9 DoF motion sensors tested here: the MPU9250, BMX055, and LSM9DS0. Every time I measure it is a little bit different! Time now is seconds.
According to the National Oceanic and Atmospheric Administration website, the magnetic fields for my area should be 226 mG North, 55 mG East, and -426 mG down. In this light, the Bosch magnetic sensor seems to be doing very well. The MPU9250 is about 5% too high in the z-direction, while the LSM9DS0 is about 20% low. Unless you plan to use the magnetometer in a Star Trek tricorder it shouldn't matter that the absolute value of the magnetic field is off, but it is a curious result. For absolute orientation it is the relative change in magnetic field that can compensate for gyroscope drift, which is the bane of all IMUs.
So lets talk gyro performance. The gyro is literally the heart of the sensor fusion engine since measured rotation rates are a direct measure of the rate of change of the quaternion representation of the sensor orientation. Any non-zero component of the gyro signal due to offset and/or drift will accumulate as an error in orientation that will rather quickly frustrate any effort to maintain heading without some compensation. Fortunately, even noisy gyro data can be successfully compensated with average accelerometer and magnetometer performance. We will ask how the quality of data input affects the resulting sensor fusion solution soon. But first let's continue with our simple-minded comparison of jitter and drift, this time using the four gyros on our test board.
Drift of the z-component of the four gyros (MPU9250 = Iz, BMX055 = Bz, MAX21100 = Mz, and LSM9DS0 = Lz). The data traces have been separated by 1 dps for clarity. Note the millidps and second scales. Standard deviation for the four data traces are 49, 44, 56, and 215 mdps, respectively.
The plot of the four gyro outputs when motionless shows the inherent jitter of the devices as well as the offset from the ideal 0, 1, 2, and 3 dps for the MPU9250, BMX055, MAX21100, and LSM9DS0, respectively. The jitter is pretty good for three of the four coming in around 50 millidegrees per second or 0.02% of the ~250 dps full scale. The exception is the LSM9DS0 which has almost five times the jitter of the best performer. Still not too shabby at 0.1% of full scale but there is no reason I can think of why the LSM9DS0 should be an outlier here. Of course, noise is undesirable but I find the offset a bit more troubling, and so should you.
Each of these sensors has the average measured bias at rest subtracted from the output data so one would naively think the average deviation for all three would be zero. It is not. Rather it is 18 mdps for the MPU9250, 93 for the BMX055, 358 for the MAX21100, and -48 for the LSM9DS0. The jitter will lead to error in the sensor fusion as the square-root of the standard deviation whereas a non-zero mean deviation will add a linear term to the error growth. That's why removing the average bias is important. Perhaps I should have averaged over a longer period, but I suspect no amount of averaging will eliminate this kind of offset. it is precisely what leads to heading drift and what sensor fusion with the accelerometer and magnetometer data is designed to correct for. The fusion allows a nearly instantaneous bias correction to make sure that the quaternion is sensitive to (mostly) real changes in rotation rate. Just based on these two measures, average and standard deviation, we have to give the nod to the MPU9250. It still remains to be seen whether we can measure the effects of these differences in the sensor fusion results. Soon...
The last item in our 9 DoF sensor bakeoff is the accelerometer. The accelerometer is the first means of correcting the drift from the gyro by imposing the constraint of an absolute reference frame, gravity. The jitter of the accelerometer is as important as that of the other two types of sensor since the gyro drift correction is only as good as the input accelerometer data. Also, we really want a low drift accelerometer since both the white noise and errors in the apparent acceleration caused by accelerometer drift will wrongly compensate the gyro resulting in heading accuracy loss. Like the gyro, we remove the accelerometer bias by subtracting the average of a few data taken with the sensors at rest. The result for an ideal accelerometer would be zero mean deviation, with the jitter characterized by the standard deviation. Representative data is plotted below where only the z-axis accelerometer data from the four sensors on our board is shown with a 20 mg offset between the traces for clarity.
Blown up z-axis traces of the MPU9250 = Iz, BMX055 = Bz, MAX21100 = Mz, and LSM9DS0 = Lz). Ideal values would be 1000, 980, 1020, and 1040 mg, respectively. Plot shows milli g versus seconds with the sensor at rest.
The traces cover about 30 minutes of data, enough to see the main trends. The traces are again offset by 20 mg for clarity. The average deviations from the ideal values of 1000, 980, 1020, and 1040 mg of the z-axis accelerations measured for the MPU9250, BMX055, MAX21100, and LSM9DS0 are 0, -10, +10, and 0 mg, respectively. At worst this deviation is 0.5% of the 2 g full scale. Very impressive! There is almost no drift except in the case of the MAX21100 where a linear fit shows a slope of 0.5 microg per second or ~2 mg per hour; the drift rates of the other three accelerometers is 0.2 microg per second. Despite the disparate drift results, the accelerometers show a standard deviation or jitter of 2, 7, 3, and 3 mg for the MPU9250, BMX055, MAX21100, and LSM9DS0, respectively. The BMX055 has the largest jitter as can plainly be seen in the figure. This result is somewhat disappointing since the BMX055 gyro and magnetometer seem to behave pretty well.
I will sum up the results of our bake off by process of elimination. Firstly, we have shown that the LSM9DS0 has a larger offset from the expected magnetic field than the other two magnetometers, and that the gyro has a large jitter. The BMX055 shows a larger jitter in the accelerometer data by a factor of two compared to the other accelerometers in this test. Lastly, the MAX21100 shows a large offset in the gyro data compared to the other three. Offset can be compensated for but jitter is inherent in the sensor response and should be weighted accordingly. We can't pick and choose (easily) the individual sensor components to make up our 9 DoF motion sensor but it looks like we don't have to. By process of elimination the MPU9250 offers the same or better performance for all three sensor data types in this test than every other sensor. In this case it is possible with some work to pair the MPU6500 gyro/accel of the MPU9250 (or MAX21100) with many other magnetometers if so desired. But for my money, I recommend the MPU9250 for the combination of performance, as discussed above, small 3 mm x 3 mm footprint, availability, and reasonable cost.
Does any of this matter for heading accuracy? Let's find out...
I used the calibrated and scaled motion sensor test board data as input to the Madgwick MARG (magnetic, angular rate and gravity) sensor fusion filter to calculate heading (yaw) pitch and roll. With the sensor board laying motionless and flat on my desk, the roll and pitch are within a degree of zero all the time. This is an easy feat even for 6 DoF sensor fusion without a magnetometer. The hard part is accurate absolute heading with little or no drift. Here is a plot of the heading estimated from the sensor data of the four sensors on the motion sensor test board:
Heading estimated from the MPU9250, BMX055, MAX21100 (using the MPU9250 mag data), and the LSM9DS0.
The MAX21100 has no embedded magnetometer so I used the magnetometer data from the MPU9250. In fact, an interesting experiment would be to use the three different magnetometer data with the MAX21100 accel and gyro data to see more directly if differences in the magnetometer data affect the sensor fusion result (see below). The heading estimate is recorded for about two minutes with the sensor board lying flat on the desk at a fixed orientation. Then at the two minute mark the board is rotated ninety degrees, and this is repeated until a full 360 degree rotation has been achieved. At 800 seconds I moved the board all around for 30 seconds to see if vigorous motion would throw off the heading estimate. For an ideal motion sensor and sensor fusion combination, we should expect to see a steady heading at rest and heading changes of ninety degrees every two minutes, with the final heading matching the beginning heading.
What we actually see is that headings estimated from the four motion sensor data inputs do not agree. I paid particular attention to make sure the sensor data was passed to the Madgwick filter in the proper North, East, Down (NED) convention I chose for my reference frame. The initial heading should be about -45 degrees, since the edge of my desk where the board begins the experiment is about 45 degrees from True North. The LSM9DS0 comes close. The MPU9250 and MAX21100 estimates are about 15 degrees too high and the BMX055 estimate is about 20 degrees too low. As the board is rotated through ninety degrees we see that the headings change by about this value but not quite. Looking at the first three steps in the plot, the difference between them should be 180 degrees. And it is for the LSM9DS0. But it is 220 degrees for the BMX055 and 165 degrees for the MPU9250/MAX21100. It is not surprising that the MPU9250 and MAX21100 track so closely together; they both are pretty low-noise, low-drift accelerometer/gyro sensors and they are using the same magnetometer data! The deviation from 180 degrees here is another manifestation of the ellipsoidal magnetometer response surface. In fact, the gyro and accelerometer usually suffer from this sort of non-uniform response surface too. A proper sensor calibration would record the response of all three sensors in different orientations and correct for the non-uniform response surface as well as the recentering usually done.
The good news is that all four heading estimates return to their original value not only after the full rotation but after some vigorous motion, as expected from previous studies. So the relative orientation estimate using simple sensor fusion is robust, but 9 DoF sensor fusion promises absolute orientation accuracy. Can we devise a simple calibration routine to improve these results?
More to come....