Which cantrip are they rolling to see if it lands? Is it like Firebolt or something similar with an attack roll, or Frost bite with a saving throw, or something else? Any extra details you can give would be helpful in helping solve this problem.

It was a cantrip called "mend" and he was trying to use it to repair his armor which had been broken. I made him roll to see how well he repaired it and he got a 2 so it didnt repair much. He went behind my back and spoke to another player about how it was bullshit and "thats not how cantrips work" although he was able to cast the spell just not well

No that's great. I feel that was important information and I am open to any comment or help

Why do you feel the need to make them roll for cantrips? If you and a couple of the players are new to D&D 5e then why confuse things even more by changing the rules? Of course DMs are free to do what they like and the rules don't have to be set in stone but I'm curious as to the reason behind the change. Also, did you have a session 0 with your group to discuss any changes and did everyone agree to them? If not then TBH I can kinda see why some players might be upset by this.

I don't make them roll to activate or prepare the cantrip. I make them roll to see if they can land a hit or in the case of repairing armor to see how well they can use mend. I don't know the rules and even in the Players Handbook it doesn't specify exactly how cantrips work. I feel that the cantrip is not an 100% going to hit a target or do exactly what the player expects. That you still have to be able to hit the target.

"I don't know the rules" - I suggest you learn the rules as written before you try to make up your own... Spells do exactly what they say. If they don't mention a roll, then there isn't one. Most combat spells have either an attack roll or a saving throw (each spell tells you). Most non-combat spells don't.

When I say "I don't know the rules" I mean that as down to the very detail. I count find the spell exactly because the spell can only mend pieces of items like chain or a torn cloak. It says nothing about armor. So I was more making him roll to see how well he repaired his already broken armor that had been torn to shreds. His armor rating was a 23 and it was down to 2. Yes if he chose to use it more then once he could have gotten it fully repaired but he didn't. He decided to question the rules and me as the dm.

This question is very confusing. It sounds like you and your players are trying to play entirely different games. Do your players know the rules of the game you're playing? Have they agreed to those rules? Do you know the rules of the game you're playing?

Well the problem is that its homebrew. I followed the lead of another player who ran a campaign before me. He changed a lot of the rules and when I tried to introduce the actual rules I kept getting "well its homebrew" so I am stuck in an impasse.

Did you introduce the house rules in advance, or did you make them up on the spot? Did the players agree to (or at least acknowledge) these house rules before play?

It might be useful to add to the question text, that he tried to repair a homebrew super(?) armor... (This based on comments under my answer.)

As I said I was following the lead of another dm who is now a player. He made the rules up as he went and no one questioned it. I'm starting to realize that my party thinks I'm a push over and if they don't like something they can just bully me till they get their way. I don't want to be the bad guy and put my foot down. I want to find a common ground as we were just making it up and pulling from the source material when we didn't know. But honestly I might just stop dming.

@WakiNadiVellir He is a druid and currently has his powers dampened because he murdered all life in a swamp by striking it with lightning twice. He wanted to mend his armor which is homebrew supped up leather armor. I made him roll to see if he could do the cantrip successfully because of these factors. But he wont listen he just gets mad and claims that's not how cantrips work.

These comments are becoming a discussion. Can these be moved to a separate chat room?

I literally made this account today. I have no idea how to do that. But I'm open to it

It honestly sounds like you're playing a totally different game that's just "based on" D&D, but customized to the point it's not really the same game anymore. You might want to restart the whole game, actually using the rules this time, to establish a baseline. (I'm pretty sure it was in the 1980's the last time bringing characters from one campaign to another was a common thing, and that was when they were expected to die before reaching level 7...)

Comments are for clarification, not for extended discussion. Please put answers in answers only. I've flagged for moderators to create a new chat room where this back-and-forth discussion can continue.

Hi Taylor. I see you're a new to the stack, so please be welcome. If you haven't already seen them, we have a tour which will help explain how this site differs from traditional discussions forums, as well as a help center for new users. Also, it's okay if you don't know how to make or get to the chat rooms. I can't speak for him, but I don't think @MikeQ was suggesting you should do it, but calling for the moderators to help us out.

Also, I'm seeing a lot of comments discussion that are variations on the theme of, "That's not how that works," followed by, "Well how does it work?" When that point is reached, it's okay-- it is preferred!-- to ask a separate and specific question about a particular spell or rule. As long as you make a good faith effort to understand the rule and maybe search (with a search engine) the stack for similar questions. The reason we do that is the assumption that if you're confused, others might be too, and we want them to be able to find the question and the answers, too.

And the attitude expressed above really has no place here. That comment is unfriendly and unhelpful.

@NautArch Which comment are you referring to?

@MikeQ The one directly above yours about making up new rules.

Eh, reading the rules is probably good advice here. For example the rules say what the DM's role is, which seems to be a point of confusion in the asker's group

....I now see that the asker has deleted their account, so I'm not sure how useful this chat discussion will be.

@MikeQ Knowing the rules, someone coming in and saying it like that? I don't think that tone is helpful.

ANd given OP removed their account...it may not have been.

@MikeQ They haven't removed their account. They just didn't have 20 rep at the time so it shows up weird (I think they'd still be able to chat though)

@user68991 "I don't want to be the bad guy and put my foot down" - That's kinda literally the definition of being the DM. Not necessarily being a "bad guy", but stating "these are the rules". As others have said, while you may have read the rules, you don't understand them yet. It's not uncommon, I've been playing RPG's since D&D 1e, and I have trouble with 5e because I haven't played D&D for a long time.