“Accuracy: Beware of Red Herrings and Black Swans,” a Presentation from Perceive

Steve Teig, CEO of Perceive, presents the “Accuracy: Beware of Red Herrings and Black Swans” tutorial at the September 2020 Embedded Vision Summit.

Machine learning aims to construct models that are predictive: accurate even on data not used during training. But how should we assess accuracy? (Hint: simply computing the average error on a pre-determined test set, while nearly universal, is frequently a bad strategy.) How can we avoid catastrophic errors due to black swans—rare, highly atypical events? Consider that, at 30 frames per second, video presents so many events that even “highly atypical” ones occur every day! How can we avoid overreacting to red herrings—coincidences in the training data that are irrelevant? After all, a model’s entire knowledge of the world is the data used in training.

To build more trustworthy models, we must re-examine how to measure accuracy and how best to achieve it. This talk challenges some widely held assumptions and offers some novel steps forward, occasionally livened by colorful, zoological metaphors.

See here for a PDF of the slides.

Here you’ll find a wealth of practical technical insights and expert advice to help you bring AI and visual intelligence into your products without flying blind.



1646 N. California Blvd.,
Suite 360
Walnut Creek, CA 94596 USA

Phone: +1 (925) 954-1411
Scroll to Top