“A naive assumption is often made that reusing software or using commercial off-the-shelf software will increase safety since the software will have been exercised extensively. Reusing software modules does not guarantee safety in the new system to which it is transferred and
sometimes leads to awkward and dangerous designs. Safety is a quality of the system in
which the software is used; it is not a quality of the software itself.”
https://web.stanford.edu/class/cs240/old/sp2014/readings/therac-25.pdf
Mastodon Source 🐘
Evergreen checklist from Leveson & Turner (1993) from their review of the tragic Therac-25 fatal patient dosages:
“Some basic software engineering principles that apparently were violated
in the case of the Therac-25 include:
• Documentation should not be an afterthought.
• Software quality assurance practices and standards should be established.
• Designs should be kept simple.
…”
Mastodon Source 🐘
“• Ways to get information about errors, i.e, software audit trails, should be designed
into the software from the beginning.
• The software should be subjected to extensive testing and formal analysis at the
module and software level; system testing alone is not adequate. “
This is a great paper that highlights core safety principles stemming from a truly horrible series of events.