How do you handle Magic--or super-science inventor types? Or are those the same things?
Rules Only The Character Knows
One way to make magic "magical" in fiction is to have rules to it that the reader never really sees. This is the case of, for example, Gandalf (at least to my reading of Lord of the Rings years ago) where I wasn't really sure what he was capable of--but man, did he have some tricks up his sleeve. He was mysterious.
In something like Harry Potter, there are clearly some hard-core rules* around how magic works--but we never really learn them. As a result, the characters can sometimes pull a surprise on us--a new spell or some new effect or whatever.
With super-science gadgeteers / mad-scientists, it's much the same: we don't know exactly what Doc Brown is capable of building (besides a time machine) but while we wouldn't believe he could put together and launch an International Space Station using household goods, we'd probably believe he could turn a microwave oven into some kind of ray-gun. Again: there are fictional rules or guidelines that we don't know the limits of--but presumably the characters themselves do.
But in role-playing we are the characters. So what do you do?
The problem with modeling magic or super-science in fiction is two-fold. The first issue is that magical / mad-scientist characters can do a vast variety of things given the "right conditions." The second issue is that they are (generally) fully aware of their limitations and know a great deal more about the domain that we, the players / readers are going to.
One solution to this is the Noun-Verb method of magic where characters learn "component pieces" of magic (this could work for technology too) and then combine them in ways that are (mostly) sensible. If you know 'Burn' and 'Person' you can probably immolate someone or maybe cast body of fire on yourself. Maybe both?
Another solution is Hero's Variable Point Power Pool which allows you to "spend points" at will on a large variety of effects. Hero's excellent list of generic effects works for this--although it can result in some fairly bland results depending on exactly how the spending is allocated and works.
Are There Other Solutions?
There are. In the original JAGS magic book we opted for a huge (300+) list of spells that was, really, pretty darn 'complete.' A mage might not have many of them (compared to the total) but you could have a fair number--and the rules around them made it so that you could be pretty versatile for "not that many points." We were explicitly trying to 'simulate' computer games (and AD&D, to a degree--which is also what those computer games were simulating) so that was okay.
JAGS Wonderland's magic system is even better: It's a magic setting. The rules about different levels of reality are fixed / explained and then your degree of training allows you to bend them. You can do Truth or Place / Person by channeling your Shadow a few chessboards down (if this makes no sense go download the free PDF for the Book of Knots and look in the back).
The fact that a few chessboards down places and people display their "essence" is established: allowing you to 'see it' while being in reality is just a matter of training. To someone who doesn't know the rules, a player using those abilities would seem kind of random and plenty mysterious: for someone who has read the book, though, it makes a lot of sense.
JAGS does not have a list of mechanical effects--instead we have something like 400+ individual powers. The degree of coverage with Hero is not all that different (although, to be fair, you can build powers with Hero that we don't cover--we do have power modification rules, though so, you know, maybe ...).
How might a super-science system work in JAGS? Like this: you spend APs on "Super Science gadgets" and, immediately, take both a reduction in effectiveness (having points that can be re-allocated during play to almost anything is an advantage) but you get some back because it takes time and requires a lab (presumably).
We then have a Super Science Drama: this is where you make your science skill rolls (three rolls) and try to beat a target number (each point you make your roll by counts--so if you have a 13- skill and roll three 10's in a row, that's 9 points). If you meet your number you get a power boost of some sort--so if you are going for a very powerful device you might want a Target Number of 20 (so you'd have three rolls to try to get a sum-total of 20 success points--good luck if your roll is 13 or less).
But we have a way to have you mitigate that: Between rolls, if things aren't going well you can roll on "drawback tables." These tables lower your Target Number (making you more likely to succeed) but give you some random drawback to the device (such as the Freeze Ray Gun has a 6-second charge-up time).
This produces some uncertainty about what the final result will be--but allows the player to have a decent amount of control over the process.
We can also allow for Mad Science effects where a roll can tell you things like "You need a human brain to complete the device."
For magic, the rules would be similar--but you could roll on the Black Magic table and get things like corruption of the self or 'causes mutation.' The point would be to have magicians who are exceeding their power-limits be able to take risks or simply degrade their spell in some meaningful way (such as, again, charge up time) to accomplish their goal.
Things Man Was Not Meant To Know
If I had infinite time and patience I would have a book of "drawback charts" that players would only get to read some of (and note, this would just be for some games where everyone was onboard and thought this was cool)**. The chapter the characters would read would be the "in-game rules" for the charts.
Such as: a chapter on aligning your mage with extra-dimensional entities. A set of chapters could be about these shadowy, untrustworthy entities who you could call on for help. If you do, you get your power-boost--but you get a roll on the hidden 'cost chart.' The GM then describes to you (maybe in private) what the effect was.
To the other players this is VERY mysterious (they may not have read the chapter at all--and they don't know what the effect was). To you it's still somewhat mysterious (unless you are very experienced with the hidden chapters)--but you do know a lot of the basic terrain.
We could do the same thing for super-science: have a chapter that describes the break-through and then if the character learns that break-through, they can then experiment with the hidden tables ("I'll use dimensional gateways technology to power my super-car ... oh, crap: Red Spider Invasion!!"). This would be a way for players, during the act of actual play, to explore the system and the world ...
* In Harry Potter, in a wizard battle, it is possible to connect with a punch or push when you flat out can't hit the target with a ranged attack spell. Presumably either protective wards don't stop "attacks that won't really hurt you" or else magical duelists are really missing out not having death-touch spells they can fire up.
Also, in HP, what exactly can you do without a wand? Clearly powerful wizards can do something without a wand--but it's never really described how that works.
And how does Quiddich scoring work anyway!?
** Warhammer Fantasy did this wonderfully with their bestiary: the book (the first part) is wonderful old-style text and illustrations that describe the monster. The back of the book then has the stats. This allows players to go through a bunch of it and learn cool things--but not entirely look behind the curtain when it comes to the monster's performance in game-mechanics terms. When we do a fantasy monster's book, I'd like to (at least partially) mimic that.