09-29 RMSE calculations

Problem: Method to quantitatively grade accuracy

Currently the only way that I assess the accuracy of the IPS is through qualitative remarks. I want to have a measurable way to say whether my estimates are close to the actual position. Otherwise it is hard to say whether something I am testing is better or worse.

Solutions

In order to solve this issue I implemented a small subroutine that sends every interpolation and raw position array through an error calculation. The user is prompted to enter the actual position of the client as a reference. Then the root mean squared error is calculated based on that.

What is nice about this method is that the visual response is immediate and clear, since the actual position is on the map in blue. On the right the RMSE is displayed for the red interpolation and the raw black positions.

Applying RMSE in tests

I used the RMSE results to see what the difference is between using 4 beacons and 8 beacons at once. The 8 beacon constellation achieved RMSEs of less than 25cm in the lower left area of the grid and was most accurate in the corners, where the beacon density is highest. On average the RMSE was 1.07m for the raw and 0.69m for the interpolated results.

{ width=60% }

For the 4 beacons constellation there was a clear bias towards the middle of the grid, regardless of the placement of the device. This is probably because it was always far from at least some of the beacons, which messed up the distance estimations. The average raw RMSE was 1.56m and the average interpolated RMSE was 1.29m. This confirms quantitatively that the 8 beacon setup is more accurate.

{ width=60% }

Up next

The way that I read out the RMSE was not very clean, since I just estimated what the average was by watching the numbers flicker by for a while. This is not acceptable. Tomorrow I will make this a running average for better precision. Things I will test will be different window sizes for the moving averages in the backend, returning more measurements to the frontend for averaging and if possible using a different sized field. It possibly would be interesting to see what difference it makes if the field is more elongated, like in a corridor.