$\int_0^1 5040 x^3(1-x)^5\,\mathrm d x=10$ so the provided $f(x)$ is not a pdf. There may be a typo.

@GrahamKemp and what is my p.d.f.?

Should I calculate the derivative?

@Student Well that is for you and your question to say. If you mean that your $X$ has a $\beta(4,6)$ distribution (Beta distribution ) then the constant $5040$ has to be replaced by $\frac{1}{\text{B}(4,6)}$ where $\text{B}(\cdot,\cdot)$ is the beta function

The distribution isn't given. I suppose since x is from (0,1), X can have a Beta distribution.

@Student Your question (i.e. the book/notes you are using) should say that $f_{X}(x)=504\cdot x^{3}(1-x)^{5}\mathbf{1}_{(0,1)}$

Now with that out of the way, have you not yet learned of transformation of random variables? @Student

When you say that it is your "idea" to integrate/sum over $(0,1)$ or differentiate it shows that you lack the understanding of what a pdf actually is. What is it that you are trying to do? You are required to find the distribution of $Y$. So what good would integrating do? It's more important that you clear out these confusions first. Integrating a pdf over a set $A$ gives the probability $P(X\in A)$. How at all will this say anything about $Y$? @Student

@Mr.GandalfSauron You are right. I don't actually understand what a pdf is. I see there is a formula for the pdf of Beta distribution but how can I relate this to Y?

Once you have cleared these confusions, I would suggest you read up on the transformations of random variables. There you will find the Jacobian change of variables formula/method (which is essentially the change of variables method for integration that you have been using since high school) . Then I guess that you will be able to solve the problem on your own. Give it a try and then post your efforts in the question body. If you are still unable, but you have shown enough effort, I (or some other community member) will post an answer. @Student

@Student here is a beginner friendly introduction which does not go into too much rigourous details. Perhaps you'll find this helpful. Follow the steps of the change of variables and you will end up with a pdf for $Y$. To verify if you did it correctly or not, you can then integrate over $(0,1)$ and see if it gives $1$.

Thank you so so much! @Mr.GandalfSauron

And this is to the community members. Instead of blindly posting close votes, why don't you post a helpful comment so that the op can atleast begin to try out the problem. Clearly the op has problems understanding the concept. A mere downvote and closure will only reduce their confidence. Instead why don't you provide an useful link . And no this is not a "no context" problem . There's plenty of context for the educators here to understand and help out the op. And if this question still gets closed, it will further prove my point that people don't read the comments and are too quick to judge.

Thank you for your comment, @Mr.GandalfSauron. I totally agree. It wouldn't have been the first time my question would have gotten closed. I thought the problem was because of me but then I thought everyone deserves a chance to learn, beginner or not. I have posted my solution.

@Student Excellent work. Now what you need to do is know how to differentiate under the integral sign. Do it, and you'll find the pdf of $Y$.

Just one more additional thing. $Y$ does not have a "mass function". $Y$ has a continuous distrubiton and not a discrete one. Probability mass functions are for discrete random variables say taking values in a countable set $\{x_{1},x_{2},...\}$. Then the pmf is the function $f(x_{i})=P(Y=x_{i})$. You see that $P(Y=t)=0$ for any real number $t$ simply because the integral of a function over a single point has to be $0$. So the analogous version of pmf is the pdf, the density function.

I understand. Thank you!

Just like you get the probability by "integrating" over the pdf in case of continuous random variables, you get the probability of a set by summing the mass function over the elements of that set. So here the old saying again should reverberate that "integration is the limit of a summation". Once you get to know measure theory, then things will clear up in full rigour and context and you will also learn that "a summation is nothing but an integral". After that, there'll be no more confusions. But that is way ahead in the future.

@Student Note: $X^5\leqslant y$ implies $X\leqslant \sqrt[5]y$ not $\sqrt y^5$

Also if you only want the pdf, you may skip the integration step: $$\begin{align} f_Y(y) &= \dfrac{\mathrm d ~~}{\mathrm d y}\mathsf P(Y\leqslant y) \\ &= \dfrac{\mathrm d ~~}{\mathrm d y}\mathsf P(X\leqslant y^{1/5}) \\ &= \dfrac{\mathrm d ~~}{\mathrm d y}\int_0^{y^{1/5}} f_X(x)\,\mathrm d x \\ &= f_X(y^{1/5})~\dfrac{\mathrm d y^{1/5}}{\mathrm d y\quad~} \\ &= f_X(y^{1/5})~y^{-4/5}/5 \end{align}$$