Is this a thing that normally happens during crises? I know Bush's approval rating went through the roof after 9/11, but other than that, I can't think of any examples of this phenomenon.

War often gives politicians a boost because of the patriotism it stirs up but it does seem odd to get a boost in a health crisis.

Yeah, according to fivethirtyeight, his approval is at an all-time high, and his disapproval rate is below 50% for the first time since 2017. He's been handling the health crisis fairly well (especially over the last week or so), but he was basically running on the economy, and that's kinda gone now, so it seems odd that he's doing so well.

People who feel afraid wanting continuity and something familiar so the incumbent gets a boost maybe?

i reckon the boost will be temporary depending on what happens. If the crisis is still going on in a few months and people are really feeling the economic losses then it might fall.