Keeping Magic From Taking Over Your Story
First, see Phlebotinum-Development Questions. This will help you work out from the very start what your magic could potentially do in the story.
Ask yourself what problems you don't want magic to be able to solve, then think up reasons why it can't be used for that. For example, let's say we don't want magic to be able to bring people back from the dead. There are plenty of possible explanations; here are a few suggestions:
- Reviving from actual death is simply impossible. You can save someone from mortal injury, but once they've crossed the line to actual death, it's a done deal.
- Bringing someone back requires components that are hard to get, or are very expensive.
- Magically bringing someone back from the dead is really, really, really hard. Think of it a little like how certain surgical procedures are difficult to perform or are risky to the patient.
- You could bring them back to life, but odds are you won't like what you get - maybe they'll be angry at you, or maybe they'll come back in a monstrous form.
When I played a certain Marvel (movie-verse) sorcerer in an RP, I didn't want him to be able to solve all of his problems magically or be casting spells right and left, so I came up with some rules and limitations to keep him under control:
- All spells take a certain amount of concentration to perform - if something breaks his concentration or if he's too tired or upset to focus, he can't cast properly.
- The bigger or more dramatic the spell's effects are, the longer it generally takes to perform them. This means that someone might be able to interrupt (or even injure!) him before he finishes. Some spells require components that are extremely difficult to obtain.
- Transforming someone into something else requires him to make physical contact with the target after at least a minute of magical buildup, OR somehow get the target to ingest a potion.
- A rule of thumb I keep to is that if it can be done with ordinary household appliances, it's easier and/or more practical to do with ordinary household appliances.
- Another rule is that learning magic takes time and effort. Hypothetically, anyone could try to learn magic, but if they're mutants with certain sets of powers or inventor geniuses they'd probably be better off honing their natural abilities and/or sticking with what they're already good at.
- Different people have different magical knacks or talents. Just because someone is good at one area of magic doesn't mean that person is going to be good at all of them. For example, someone might be naturally good at shapeshifting, but have little to no talent for scrying - and vice-versa.
Magic can also have a physical cost - using it might be tiring. Learning new spells might take significant time and practice. Some works of fiction have a possibility of a magical backlash (particularly if magic is used “improperly” per the rules of the universe), or magic comes at a great personal cost. Not only do these provide good reasons for your characters not to use magic willy-nilly; they're also good mechanisms for providing dramatic potential - eg, a character uses magic improperly or frivolously and has to face the consequences.
Circumstances and conditions can also make the use of magic impractical or even impossible. Perhaps a spell only works on a certain day or when the moon is at a certain phase. Perhaps too many people crowded in an area disrupt the flow of magic.
Another thing I've found effective at curbing the overuse in magic is to make people outside the circle of protagonists react realistically toward it. EG, if one of your protagonists can magically scry into the villains' lair, then the villains should eventually figure out a way to block it. Their solution doesn't have to be "magical" per se - it might be electromagnetic, even - but it sets things up so that magical solutions aren't fix-all solutions.
That said, when it comes to RPGs, I don't like tying magic up in so many rules and potential consequences that players feel punished for using it. I'm all for setting caps and limitations on what it can and can't do - but it's my philosophy that at the end of the day, it should still be fun to use and have rewarding outcomes.
Setting Rules & Limitations In Your World: Why & How You Need To Do This
So You Wanna Mix Science And Magic?
Keeping Shapeshifters From Getting Overpowered
Tips & Ideas To Write More Believable Masquerades
Tips & Ideas To Create More Believable Sword 'n Sorcery Worlds
Magical & Supernatural Tropes To Reconsider (And Tips To Build Up Your Magical/Supernatural Settings!)