Why wouldn't we just turn it off?

For today's systems: yes, turn the server off, done. The interesting case is an AI doing valuable autonomous work across many systems — the more useful it is, the more access we give it. “We'll turn it off if something goes wrong” isn't a solution to the control problem; it's a requirement a solution has to guarantee.