This reminds me of a story I read once about how Bell Labs engineers wanted to find the ideal length of a landline telephone chord. They did this by secretly shortening the chord over time on test users' telephones and waited to see how short they could make the chord before they noticed. If the plugtest is hardware-hardware testing Bell Labs engineers were doing human-hardware testing.
I tried to find a source for the story but I couldn't find one. I think I read it in The Innovators by Walter Isaacson, but I can't remember it exactly - maybe I'm misremembering it.
> Telephone company executives wondered whether the standard cord, then about three feet long, might be shortened. Mr. Karlin’s staff stole into colleagues’ offices every three days and covertly shortened their phone cords, an inch at time. No one noticed, they found, until the cords had lost an entire foot.
The issue I've always found with testing on human users is their willingness to ignore deficiencies.
There is a strong benefit for people "just getting on with it" and "working around" a problem; but it seems like many people are not very good at identifying when a problem is worth solving.
An analogous anecdote I'm aware of was a small in-house call-centre team being moved to digital playbooks (vs those A5 ring bound / tabbed sets). While time-to-resolve each issue got lower and time-to-adapt to new workflows a lot faster, many tracked metrics got worse. Suddenly they were picking up fewer calls, time to answer was increasing, job-satisfaction decreased.
It turns out the new system was forcing the team to be sitting at their desk. In a small and busy multi-team office these people were usually doing all kinds of other small things - be it making coffee, liaising with other teams, etc. - whatever it was they weren't always at their desk. They did have wireless handsets however, so they used to answer the phone, start the call from memory, and move to use the nearest playbook.
The big issue with the digital playbooks was needing to "follow along", you can't reasonably start it away from your desk.
The team knew this was an issue very quickly, it was quite fixable, but they sucked it up for months.
Interesting story. I have noticed that certain parameters are often ignored not because they're unimportant, but instead because they're hard to quantify.
Another story is how the first product made by Jobs and Wozniak at Apple was a hacked phone that could be used to make international calls for free. They called the Vatican and pretended to be Henry Kissinger wanting to speak to the pope - they noticed and promptly hung up before they reached him
Microsoft did (does?) Installable File System plugfests. The anti-virus vendors always had a pretty full dance card because they go to great lengths to scan files, which is sometimes at odds with other drivers.
We had a hierarchical storage management driver, and getting files released to remote storage and keeping them released until the users actually wanted them back was helped greatly by actually being able to talk to the AV vendors and understand their criteria for when to scan.
At the time, a lot of AV filter drivers were legacy model drivers, which led to some serious shenanigans getting a second minifilter installed above the AV layer in the stack to distinguish between user and AV generated reads.
I've always wanted an excuse to write a minifilter or similar and solve a real problem and totally not at all hose my system's performance. Windows kernel development calls to me for no good reason.
Similarly the Bluetooth SIG organises "Unplugfests" which are the same, but for a wireless protocol. I attended several back when I was doing Bluetooth stuff full time. You learn a lot about how a very clearly written spec can be interpreted in 5 different ways!
Also in the similarly camp: NFL "RF War Games". Everyone running radio gear at the Superbowl all fires up their radios at once to determine who's interfering with who to resolve interference issues before the show.
This reminds me of a story I read once about how Bell Labs engineers wanted to find the ideal length of a landline telephone chord. They did this by secretly shortening the chord over time on test users' telephones and waited to see how short they could make the chord before they noticed. If the plugtest is hardware-hardware testing Bell Labs engineers were doing human-hardware testing.
I tried to find a source for the story but I couldn't find one. I think I read it in The Innovators by Walter Isaacson, but I can't remember it exactly - maybe I'm misremembering it.
Is this it:
> Telephone company executives wondered whether the standard cord, then about three feet long, might be shortened. Mr. Karlin’s staff stole into colleagues’ offices every three days and covertly shortened their phone cords, an inch at time. No one noticed, they found, until the cords had lost an entire foot.
https://www.nytimes.com/2013/02/09/business/john-e-karlin-wh...
I'd love to read this story.
The issue I've always found with testing on human users is their willingness to ignore deficiencies.
There is a strong benefit for people "just getting on with it" and "working around" a problem; but it seems like many people are not very good at identifying when a problem is worth solving.
An analogous anecdote I'm aware of was a small in-house call-centre team being moved to digital playbooks (vs those A5 ring bound / tabbed sets). While time-to-resolve each issue got lower and time-to-adapt to new workflows a lot faster, many tracked metrics got worse. Suddenly they were picking up fewer calls, time to answer was increasing, job-satisfaction decreased.
It turns out the new system was forcing the team to be sitting at their desk. In a small and busy multi-team office these people were usually doing all kinds of other small things - be it making coffee, liaising with other teams, etc. - whatever it was they weren't always at their desk. They did have wireless handsets however, so they used to answer the phone, start the call from memory, and move to use the nearest playbook.
The big issue with the digital playbooks was needing to "follow along", you can't reasonably start it away from your desk.
The team knew this was an issue very quickly, it was quite fixable, but they sucked it up for months.
Interesting story. I have noticed that certain parameters are often ignored not because they're unimportant, but instead because they're hard to quantify.
Another story is how the first product made by Jobs and Wozniak at Apple was a hacked phone that could be used to make international calls for free. They called the Vatican and pretended to be Henry Kissinger wanting to speak to the pope - they noticed and promptly hung up before they reached him
Microsoft did (does?) Installable File System plugfests. The anti-virus vendors always had a pretty full dance card because they go to great lengths to scan files, which is sometimes at odds with other drivers.
We had a hierarchical storage management driver, and getting files released to remote storage and keeping them released until the users actually wanted them back was helped greatly by actually being able to talk to the AV vendors and understand their criteria for when to scan.
At the time, a lot of AV filter drivers were legacy model drivers, which led to some serious shenanigans getting a second minifilter installed above the AV layer in the stack to distinguish between user and AV generated reads.
This stuff is probably the most esoteric part of windows.
https://learn.microsoft.com/en-us/windows-hardware/drivers/i...
It's like the mod load order from hell.
I've always wanted an excuse to write a minifilter or similar and solve a real problem and totally not at all hose my system's performance. Windows kernel development calls to me for no good reason.
Similarly the Bluetooth SIG organises "Unplugfests" which are the same, but for a wireless protocol. I attended several back when I was doing Bluetooth stuff full time. You learn a lot about how a very clearly written spec can be interpreted in 5 different ways!
Reminds me of the "USB Cart of Death"
https://www.youtube.com/watch?v=6_hm3NzLeO8
Also in the similarly camp: NFL "RF War Games". Everyone running radio gear at the Superbowl all fires up their radios at once to determine who's interfering with who to resolve interference issues before the show.
https://operations.nfl.com/gameday/behind-the-scenes/nfl-eve...
https://www.mixonline.com/live-sound/tackling-rf-for-the-sup...