@hichris123 If your main concern is speed, I'd consider
gmp. It's ugly, painful, and generally a horrible design--but it is fast. Closely related is
mpir, which is basically gmp ported to work with a few compilers other than gcc (mostly MSVC). If you want to compile for Windows, the latter is probably somewhat easier to deal with.