I wrote "
If I fix where $a_j$ goes, then I can consider the given permutation $f$ of $Y$ (the one which sends $a_j \mapsto a_1$) as the union of that function with just the $\{(a_j,a_1)\}$ pair and the restriction of $f$ to the subset $Y \setminus \{a_j\}$. There are $n!$ permutations $f$ of $Y$ with this $j$ which can be built in this way"