Using artguments $P = 5; Ns = 10; G = 3; Kg = 4; gh = RandomReal[\{1, 3\}, {G, Ns}]; \beta_{g,n}$ gives argument error: Argument <>...<> at position 3 should be a machine-size real number

That's probably because RandomReal[1,3,G,Ns] is incorrect syntax. If you want a matrix that has dimensions G x Ns than try RandomReal[{1, 3}, {G, Ns}].

Try using costFxn[5, 10, RandomReal[{1, 3}, {3, 10}], 4, 3, 1] and you'll see that works.

Oh, I see the problem. Actually, I did $ RandomReal[\{1,3\},\{G,Ns\}]$. The copy pasting removed the formatting. Problem is, since I'll be using the resulting equation as cost function, the $\beta_{g,n}$ should remain in the compiled function. If we replace last arg $1$ with $\beta_{g,n}$, same error occurs.

Ok, so there is a bigger reason for that - Compiled functions only accept Integers, Reals, Complex Numbers and Booleans. The documentation is here: reference.wolfram.com/language/ref/Compile.html - look under the "Details & Options" part. What you want to do is symbolic, which won't work.

If you're doing some form of numerical optimization, you can still use a compiled function to pass numerical guesses of $\beta_{g,n}$ to your function.

Exactly, costFxn2 is actually the objective function to the optimization problem where the $\beta_{g,n}$ is the optimization variable, so it should remain. There are also additional constraint equations. Any pointer or suggestion on how to format such problem in an numerical optimization with constraint equations?

Does it need to be compiled? I would start with it uncompiled, and read through the tutorials available from Wolfram here: reference.wolfram.com/language/tutorial/… to get you started

Only then, if the performance isn't fast enough for you, should you start thinking about compilation/performance-tuning.

Then, if you still have any questions or problems, come back and post a new question on the site to ask about improving the speed of your code, and people might be able to help you there.

As it stands, I suggest you read the tutorial and work out how to implement it uncompiled

Currently, the uncompiled (modified version) takes almost a full day to complete iteration when used in a numerical computation. That's why I'm exploring compilation.

What optimization method are you using? Local? Global?

And what size matrix are we talking for gh?

If it takes a full day, then a more useful thing might be to post a small working example of the optimization code, rather than just the function, and ask for help improving it.

NMaximize. (Global). G = 12, Ns = 32. I will explore the option of posting a working code. Thanks for your help.

NMaximize is of course much slower than FindMaximum, but also the method you use within NMaximize can be key depending on the problem (e.g. SimulatedAnnealing vs NelderMead etc.) which is why posting that here might be helpful as plenty of people here have good experience with optimization.