What would YOU do in this situation? CLICK!?

You didn't mention your age, but I'm guessing teenager. I wouldn't leave someone just because my friends "say stuff" about him. If they witnessed something they think I should know, I'd take the information and try to find out what's really going on. Your friends may not like him because you're spending less time with them, so they resent him for that reason. You should let your friends know that you really care about this person and you'd appreciate it if they keep their opinions to themselves when you're hanging out with them. Although, don't dis your friends completely, nobody's ever worth losing lifelong friends over. If both sides really care, they'll understand and appreciate the fact you're able to balance both relationships.