Verified answer

I'm not quite clear on the ruling your DM made, but here's my take with what we've got:

1. I would guess you were denied your called shot even with a critical hit, and since you mentioned false life, I will assume that played some role. False Life should play no role in the effectiveness of a critical hit. However, if you're doing 10 damage on a hit, your DM may be saying the damage came out of these 'temporary' hit points granted by the spell. While true that damage is first taken from these hit points, there is no basis in the rules to say these extra hit points could ever deny a critical hit or effect. Temporary hit points are not a magical shield or force field that soaks damage. They are a part of a creature's life force, allowing him to fight longer, much like a burst of adrenalin might allow someone to run faster. If you get stabbed in the gut or bashed in the head, your body is still taking damage, even if you are more hyped up than usual.

2. Under the normal rules, instant kills and decapitations are not part of the game, with the exception of the Vorpal weapon, which must be on a "slashing" weapon (DMG 226). A rapier is a piercing weapon and would not have the weight nor sharpness to actually lop off someone's head. Your DM may be denying you the right to instant kill regardless of what you say.

3. However, there is a "variant" rule in the DMG for 'Damage to Specific Areas.' This is not a rule for instant kills, but it would allow a called shot to the head, which would disorient a humanoid target to the point they suffer -2 to all attacks, saves, and checks (DMG 27) until magically healed or resting (or the eyes, which has a list of its own penalties on the same page). If you want to take advantage of this, remember, your DM may do the same with his bad guys and start shooting for the eyes.

4. As a DM, I would not allow called shot instant kills, primarily to protect my players. Otherwise, the next dragon they see that has high attack rolls is going to take called shots to kill the players and wipe your party out in one shot. (Since monsters outnumber players in most campaigns, any feature like this generally helps your monsters, not your players). Your DM may have done this.

5. On a side note, critical hits, have been playtested for balance. Maximum damage on a Natural 20 severely curtails the critical damage on heavier weapons, like axes, which do x3 damage. There would be little incentive to arm one's self with any weapon except long swords and x2 critical weapons, and even then, you're being shorted of a lot of damage. It appears your rapier could only do 10 damage on a crit. Normally, your damage output should vary from 10-20 on a critical hit (1d6x2) + (4x2).

In an actual fight, I would not allow a insta-kill possibility from a rapier (or most other weapons, for that matter). 3.5 had plenty of rules built-in to cover situations where an insta-kill MIGHT happen (Vorpal weapon, death from massive damage and coup de grace on a helpless foe the usual methods.)