Overview

Screens are nearly everywhere in our modern life - phones, TVs, computers and more. Although useful, extended use of screens can cause us to lose our imagination and visualization skills, especially in 3D space. For example, can you imagine an elephant in your mind? Can you mentally rotate it around? What does it look like?

My project is essentially a touchscreen that adds back that third dimension. Using a magnet and four magnetic field sensors, I created a platform that tracks the position and orientation of a stylus in real time. It can be easily used as a touchscreen, with the added benefit of working at any height. With this technology, I created a simple painting app that allows you to draw in 3D space and test your 3D imagination capabilities.

The platform itself, with a magnetic field sensor on each edge.

How It Works

On a technical level, the core of this project revolves around detecting the position and orientation of a magnet, which is placed in the center of a stylus made from wood and duct tape.

Using the platform in 2D touchscreen mode.

Given four magnetic field readings, how do we actually figure out where the magnet is? First, make a guess on where the magnet might be. This guess contains the x, y, and z coordinates of the magnet, and some way of representing the orientation of the magnet.

Note that the orientation of the magnet can be represented in many ways! For my project, I used a normal vector because it worked the best.

Once we have a guess, we can use an equation to calculate what the sensors should read if the magnet was really there. Then, we can compare the calculated values with our actual sensor values, and figure out the error, which is the sum of squared errors for each measurement.

We have an error function comparing our calculated values versus measured (or target) values.

At this point, this is an optimization problem. For my optimization algorithm, I chose particle swarm optimization because of three reasons:

It doesn ’t require a gradient of the function.
It can enforce bounds on the guesses.
It works well with initial guesses.

After a few attempts and improvements, I was able to get a fast and accurate tracking system. I then took the results of this algorithm and displayed them in a 3D world, and added a “pen ”feature that allowed for drawing.

My attempt at a 3D cube.

My roommate ’s attempt at a tree.

My attempt at a tree!

Application

With more time, this simple program could turn into a complete 3D painting app, similar to current VR solutions without the need of an expensive headset. When testing it out, we also had a lot of fun trying to not use the screen, and use our imagination instead to remember what we had drawn. This is why I think my project could be the start of a great educational tool that helps bring back visualization and imagination skills in the next generation.