Wohoo, I got it! It's a programming issue. Make sure to normalize (to [-1, 1]) all data you might feed through an LSTM to CTC loss, I guess?