Pointing Device Mayhem: Bonus Edition
How to solve complex UX+HW+SW challenges quickly and effectively with the good mojo of Open Source Software
Wow, what a week.
Some customers had been experiencing drift issues with the Sprintek 8707-51 trackpoint modules I’ve been shipping for the past 8 months since the launch of Lightly. Basically, the cursor would drift off in the opposite direction of the last movement, especially on slower moves. Smelled like an auto-calibration firmware issue.
What do I mean by that?
A trackpoint sensor is a two-axis strain gage. Slow drift of the baseline signal overtime has to be averaged into the expected output — drift could be caused by changes in temperature, changes in mechanical strain in the device, or the fit of the nub on the device. Recalibration zeros out the sensor at the new value so that it stops drifting, but if this algorithm is too hungry to recalibrate it will take user input and zero it out. Then the user lifts their finger off the sensor it then heads off in the opposite direction according to the new 0 point. Whoops!
I’d tried messing with various settings in the Trackpoint registers, but they don’t fully match the original IBM spec, now almost 40 years old, and we’d kind of hit a wall. I was starting to wonder if there was a fundamental hardware problem in modules themselves. About two months ago I reached out to Sprintek, the module manufacturer, and asked for their input. I sent a couple of units back, and they were able to reproduce the issue but were skeptical that it was really a HW issue.
The hard part of these problems is controllably reproducing issues that customers only occasionally experience and which could be sensitive to customer-specific issues like power supply, etc.
So I developed a torture test protocol:
Sneak up on it.
Verrrrry gentle pressure in one direction only, gradually increasing. A sane calibration algorithm will average it in over time, and you'll be able to press harder and harder without the cursor moving. As soon as you let go, it will launch in the opposite direction.
A sufficiently sensitive device is able to see the slight muscle noise in the signal above the baseline noise of the sensor itself. and decide that it's not drift. If that's lost, it's impossible to avoid -- similarly, if you created a little vise that let you add mechanical bias, that would also be averaged in as long as you added force slowly enough.
I then ran this test on a sample of 12 units out of the remaining 30 or so in stock.
I was able to reproduce on 12/12 units, with varying degrees of sensitivity.
My USB trackpoint reference device was *dramatically* more difficult to fool.
I had a lovely chat on the phone yesterday with Eric Zhang from Sprintek to work through all the possible causes of the issues
calibration problems
physical damage to components to potential
nuances of using taller trackpoint nubs
power supply quality issues
Eric was extremely helpful and helped me understand the differences between the current implementation and the old IBM spec, as well as providing some specific commands that we could use to access registers to turn auto-calibration on and off and tweak the dead zone of the sensor if need be.
There's a whole discipline of baseline calibration algorithm development in input tech. While the specific techniques vary, the general skill set is applied across capacitive touch sensing and for sensing applications like the trackpoint.
My old friend Jora Jacobi was super deep in this at Synaptics back in the day, to the extent that she had built specific subroutines to deal with hand lotions on the fingers of someone using a touchpad. There is so much going on underneath the hood in these devices!
On the other hand, sometimes the technology advances to the point where some techniques become less important. The original IBM Trackpoints were designed to auto-calibrate greedily for relatively noisy sensors and sampling methods. In modern times, Trackpoint sensors have gotten much more drift-resistant against both temperature and other electrical factors, and auto-calibration is actually very rarely necessary.
This is where things got extra wonderful: Because Svalboard is an open-source software project, there are a number of folks who've been making great contributions lately.
Within eight hours of getting the information from Sprintek they implemented and tested a fix, shipped it to me for PR, and I made one minor change before validating that the complete solution resolves the trackpoint drift issue entirely.
It turned out that when we turned off auto-calibration completely, the problem just went away. I suspect my USB reference device is much less aggressive about autocalibration.
Now if/when trackpoint drift occurs you can simply hit a button to recalibrate. I don’t expect this to happen very often, probably on the scale of tens of hours if that, because we can also still trigger recalibration periodically to handle baseline drifts like temperature and mechanical strain.
You can assign this keycode to any key or layer you like, just like everything else in Svalboard, thanks to the magic of Vial-QMK.
I wanna take a moment to note that the only one of the people involved here is really a professional SW engineer — small SW problems are relatively easy for random moderately-savvy people to jump in and help on, especially when it comes to hands-on testing of solutions to problems they’ve been experiencing directly.
This highlights a really important thing about solving complex UX/HW/SW challenges: you really need cross-disciplinary collaboration.
In this case we needed naive fingers (real users), golden fingers (people like me who instinctively break algorithms), and three kinds of engineer — mechanical and electrical expertise to understand the origins of possible hardware issues, and software expertise to identify possible algorithmic failure modes and alternative options.
Pretty awesome that between myself and the Svalboard community (now approaching 700 people on the Discord!) we’ve got all of this in spades.
🙏🙏🙏