How worried should I actually be?

Nobody can give you a reliable probability, and you should be suspicious of anyone who pretends to. The honest case is weaker and still sufficient: very capable AI is plausible, controlling it is not obviously easy, and the consequences of failure could be enormous. That's enough to make AI safety a serious engineering field rather than a prophecy.