Noticed on console there are warnings about "precision may be lost" and so on, so you have to play with it more. But the idea is to solve for C[1] and try to find second C[2] which gives non-trivial solution. But maybe you should look at your mathematical model and find why the BC's you are using do not work in first place than try to patch things afterwords.

basically we have some wave functions with coef. that are complex and we want to build the complex conjugates

But I don't understand what the first line is meant to do

psiCrule = {Complex[a_,b_] :> Complex[a,-b],
            R -> RC,
            T -> TC,
            A -> AC,
            B -> BC}

ABRTruleC = {RC -> R,
             TC -> T,
             AC -> A,
             BC -> B} /. ABRTrule /. psiCrule

So, our 3 wave functions, have in total among them, 4 coef. I don't understand how the syntax make the things work as they should

Anyone can clarify what is going on?

Thanks a lot @Nassar for the pointers. I was trying to create my own BCs based on the fact that u[x] is distribution and it has to be normalized. I am actually want a solution for u[x,t] (figure 2 from biorxiv.org/content/10.1101/867143v2.full.pdf). I will work on it more and see what can I do.