Iris Cursor: driving a computer with your eyes and a webcam
Research assistant work at Thapar Institute of Engineering and Technology, Patiala, February – August 2022.
Where this came from
Eye tracking is a solved problem if you’re allowed to buy the hardware. Dedicated trackers use infrared illuminators and calibrated cameras, they work well, and they cost more than the computer they’re plugged into.
That price is the barrier, and it lands hardest on the people the technology is for. Someone who can’t reliably use a mouse — through injury, motor impairment, or illness — needs eye control more than anyone, and is least likely to have a spare thousand for a peripheral.
So the constraint was set deliberately: a standard webcam, and nothing else. Every laptop already has one. If cursor control could be made to work with only that, the cost of entry drops to zero.
What it does
Mediapipe provides a face mesh with iris landmarks — it will tell you, frame by frame, where the irises are within the eye. That’s the input, and it’s the easy half.
The hard half is that where the iris is isn’t where the user is looking. Turning landmark positions into a point of focus on the screen means reasoning about the geometry: iris position relative to the eye corners, head pose, and the mapping from that to screen coordinates. A webcam gives you a 2D projection of a 3D system, and the answer you want lives in the part that got projected away.
Once a focus point exists, the rest is plumbing:
| Piece | Role |
|---|---|
| Mediapipe | Iris landmarks from the webcam stream |
| Focus estimation | Landmarks + geometry → a point on screen |
PyAutoGUI | Moves the cursor; left and right clicks |
| Flask | API layer and the user-facing interface |
| Text-to-speech | Speaks back, so the loop doesn’t require reading the screen |
Why the interface mattered as much as the model
The tracking is the interesting research problem. The interface is what decides whether any of it is usable.
Two decisions carried most of the weight. Text-to-speech means the system can confirm what it’s doing out loud — if you’re driving a cursor with your eyes, having to also read a status message with those same eyes is a loop that fights itself. And serving the whole thing through Flask rather than a desktop GUI meant the interface was a web page, which is far easier to reshape as you learn what actually helps someone.
The general lesson stuck with me: for assistive tools, the model’s accuracy sets the ceiling, but the interface decides how much of that ceiling anyone can reach.