Strangely, to the developers of intelligent control interfaces, these interfaces appear to work perfectly well. Moreover, when the developers demo these interfaces, the demo comes off without a hitch - and is often quite impressive. This is not the normal result of broken software. This "demo illusion" convinces the developers that the product is ready to ship, although it is not and will never be ready to ship.

Demo illusion is caused, I think, by the same compensation mechanism that allows users to grit their teeth and use a hubristic UI. Again, the user who has no choice but to use such a monster develops her own internal mental model of its algorithm. If you are forced to use a Newton, you can, and this is what you do.

For example, the Newton user may note that when she writes a T with the bar sloping up, it is recognized as a T, whereas when the bar slopes down it has an ugly tendency to come out as a lambda. So she trains herself to slope her Ts upwards, or to always enter "one cup of flour" rather than "two cups of flour" and double the Nutrition Facts herself, or to jump through any other trivial and unnecessary hoop in order to placate the angry god inside the "intelligent" UI. By slow painful effort, she constructs a crude subset of the system's functionality which happens to work for her, and sticks to it thereafter.

But for the actual developers, this compensation mechanism is far more effective. The actual developers (a) have enormous experience with the hubristic UI, (b) have enormous patience with its flaws, and (c) most important, know how it actually works. So their internal model can be, and typically is, orders of magnitude better than that of any naive user. So the product actually seems to work for them, and does. Unfortunately, it's hard to make money by selling a product to yourself.

Mencius Moldbug: Wolfram Alpha and hubristic user interfaces