In practice, I expect that you can brute force each layer individually, because each layer also needs its own routing information and meta data. Basically, the middle node will receive a blob of encrypted data which it decodes, and in doing so has some routing information and another blob of encrypted data, which it cannot decode, but which it passes along to the next node
Therefore, you should be able to know when you have brute forced the key for the current layer, and in doing so, you should have the next "packet" of data which you can then start brute forcing