I decided to use my knowledge of game design and Brazilian jiu-jitsu to create better combat rules for Pathfinder. Now martial characters can more sensibly take an opponent to the ground, cause debilitating conditions, and cooperate to take out a tough opponent.
These rules replace how the Pathfinder Core Rulebook treats grappling and tripping with something more sensible and interesting.
You can discuss these rules at this Paizo forum topic.
The combat manevers disarm and sunder are unaffected by these new rules. The combat maneuvers bull rush and overrun are less needed but might still have a place.
Note that other Pathfinder publications have added further grappling rules through other feats and class archetypes. Those will need further house-ruling to be made compatible with these rules.
I hope the "wrestle golem" blend of Pathfinder and Gracie Jiu-Jitsu trademarks is humorous enough that neither business will complain. I will, of course, remove it if requested.
The attacker calculates a Wrestling Attack Bonus (WAB). The WAB is used instead of CMB when wrestling.
When characters are created they choose whether their style of wrestling focuses on brute force, leverage, or a mix. This choice determine whether they will use their Strength Modifier, Dexterity Modifier, or the average of those. Once this choice is made it cannot be changed.
The attacker's Armor Check Penalty is applied because bulkier armor makes it more difficult to wrestle.
WAB = BAB + Armor Check Penalty + [either Strength Modifier, Dexterity Modifier, or the average of those]
Ability score damage to Strength always penalizes WAB. If a character chose to have WAB affected by Dexterity then ability score damage to Dexterity would also decrease WAB.
The feat Agile Maneuvers seldom aids wrestling. Most young wizards-to-be, when roughhousing with their siblings, do not rely on brute strength.
Example of Averaging Strength and Dexterity
Silas the Sorceror has Strength 8 and Dexterity 8 and no plans to increase either as he adventures. He does not expect to wrestle much. He choses averaging Strength and Dexterity so that bonuses to either ability score will aid him escaping from a wrestler, although only half as much.
Wrestling attempts are opposed rolls comparing the attacker's WAB with a DC that depends upon the defender's WAB and the difference in size between the attacker and defender.
Recall the list of size category CMD modifiers.
fine −8, diminutive −4, tiny −2, small −1, medium +0, large +1, huge +2, gargantuan +4, colossal +8
The the single greatest size category modifier difference when shifting from the attacker's size category to the defender's size category applies to the wrestling attempt DC.
Calculating Wrestling DC
Wrestling DC = 10 + defender's WAB + greatest size category modifier difference
Wrestling happens by moving along this flowchart. Each arrow involves a standard action opposed roll of attacker's WAB + d20 versus defender's Wrestling DC, with appropriate modifiers.
Green arrows are easier maneuvers and grant a +4 bonus to the roll. Blue arrows are medium difficulty. Red arrows are difficult maneuvers and impose a −4 penalty to the roll.
The attacker and defender do not trigger attacks of opportunity from each other. However, all wrestling actions trigger an attack of opportunity from any opponents other than the defender who can reach the attacker.
Prone wrestlers have their Armor Class change as usual.
Wrestlers who are not prone and within reach of each other receive cover from their opponent.
None of the flowchart stages use the pinned condition. All wrestlers may attempt to move. Prone wrestlers are restricted, as usual, to crawling five feet as a move action. The Rogue talent Fast Crawl allows a prone wrestler to move at half speed: the opponent scoots along if on top or is dragged along if on bottom.
If the attacker and defender are more than one size category apart the smaller wrestler may not attempt the flowchart's lowest three stages.
Example Involving Large and Small
An ogre (size large) is wrestling with a gnome (size small).
The ogre's Wresling DC equals 10 + its WAB + 1. The gnome's Wresling DC equals 10 + its WAB − 1.
Only the ogre may attempt the three most debilitating wrestling stages.
Example Involving Small and Diminutive
The gnome (size small) next wrestles with a housecat (size diminutive).
The gnome's Wresling DC now equals 10 + its WAB + 2. The housecat's Wresling DC equals 10 + its WAB − 2.
Only the gnome may attempt the three most debilitating wrestling stages.
A combatant may follow the flowchart backwards. This can allow the defender to become the attacker.
Examples of Intereting Backwards Flowchart Movement
Moving from "Grab On" to "Start" might representing escaping from a wrist grab.
Moving from "Dominant Wrestling Position" to "Throw" might represent tossing my opponent off me.
Moving from "Very Dominant Wrestling Position" to "Dominant Wrestling Position" might represent rolling my opponent off me and then kneeling over him.
A character following the flowchart backwards horizontally suffers a −4 penalty instead of the bonus or penalty due to the arrow's color.
A character following the flowchart backwards upward suffers a −8 penalty instead of the bonus or penalty due to the arrow's color.
When following the flowchart backwards an Escape Artist die roll can be used instead of the WAB die roll.
The blue arrow leading from "Dominant Wrestling Position" back to "Dominant Wrestling Position" represents the character who is currently losing managing to gain the upper hand. This is the only way to do what the Core Rulebook calls reversing the grapple.
Note that even standing up after being thrown requires a die roll as the flowchart is followed backwards fom "Throw" to "Start". Your opponent might push you back down as you try to stand up!
Allied attackes can used readied actions to build off each other's flowchart stages. The next attacker may act as if he or she was in the previous ally's flowchart stage.
Characters may also use the Aid Another action as usual to help attackers or defenders.
Examples of Teamwork Using Readied Actions to Share the Flowchart
One ally uses "Throw" to allow a readied ally to more easily obtain "Dominant Wrestling Position". This simulates one ally forcing the opponent to the ground while the other ally waits to get on top of that opponent.
One ally uses "Very Dominant Wrestling Position" to allow a readied ally to more easily obtain "Tear or Break". This simulates one holding down an opponent so another other ally can really clobber that opponent.
Normal combat. Either wrestler might be prone.
This represents the attacker grabbing one of the defender's wrists, ankles, arms, legs, tails, etc. It might even represent grabbing an enemy's torso while the enemy remains standing, although this is usually unwise during group combat.
If successful both attacker and defender gain the grappled condition. The defender suffers nonlethal damage equal to the half attacker's Strength Modifier from being yanked around and from its own fatiguing exertion.
The defender might also suffer damage from a weapon or natural attack, if the grab was caused by a bite, whip, etc.
Also, the attacker can pick which limb is grabbed. This might immobilize the limb or provide other benefit. For example, grabbing a wrist could make a knife-wielding defender unable to use the knife. Grabbing a tail might prevent a giant scorpion from stinging.
(Note that this is the only stage in the flowchart that causes the grappled condition.)
A grappled creature is restrained by a creature, trap, or effect. Grappled creatures cannot move and take a –4 penalty to Dexterity. A grappled creature takes a –2 penalty on all attack rolls and combat maneuver checks, except those made to grapple or escape a grapple. In addition, grappled creatures can take no action that requires two hands to perform. A grappled character who attempts to cast a spell or use a spell-like ability must make a concentration check (DC 10 + grappler's CMB + spell level), or lose the spell. Grappled creatures cannot make attacks of opportunity.
A grappled creature cannot use Stealth to hide from the creature grappling it, even if a special ability, such as hide in plain sight, would normally allow it to do so. If a grappled creature becomes invisible, through a spell or other ability, it gains a +2 circumstance bonus on its CMD to avoid being grappled, but receives no other benefit.
This represents any movement that puts the defender on the ground. It could be a bull rush that knocks over the defender. It could be a flying kick that knocks know the defender but the attacker lands on his or her feet. It could be a judo throw. It could be tripping the enemy with a staff. It could be a jiu-jitsu tackle that puts the attacker kneeling above a sprawled defender.
If successful the attacker choses a map square within its reach or one square farther. The defender becomes prone in that map square. The defender suffers falling damage equal to twice the attacker's Strength Modifier.
The attacker choses to retain a grip on the defender and also apply the effects of "Grab On" flowchart stage, or to release the defender and return them to the "Start" flowchart stage.
The character is lying on the ground. A prone attacker has a –4 penalty on melee attack rolls and cannot use a ranged weapon (except for a crossbow). A prone defender gains a +4 bonus to Armor Class against ranged attacks, but takes a –4 penalty to AC against melee attacks.
This state represents positions such as "side control" and "guard" from which it is more difficult to debilitate the opponent. It might also represent might represent standing over an enemy who is on the ground.
Moving from "Start" to "Dominant Wrestling Position" represents tackling or tripping the opponent with subsequent maneuvering into a favorable position.
If successful the defender becomes prone and the attacker moves into the defender's map square without being considered prone. The defender's Strength Modifier is halved. The defender suffers nonlethal damage equal to the attacker's Strength Modifier from being squashed and from its own fatiguing exertion.
The reduction to Strength Modifier only lasts while the defender remains in the unfavorable position.
An attacker in an enemy's map square does not normally count as a flanking partner with any ally. The defender need not focus attention in a certain map direction to see the attacker just above. The defender is indeed hampered in movement and visibility by being on the ground, but the prone condition already describes those penalties.
For the lower three flowchart stages the attacker and defender are always both considered to be prone.
This represents any armlock or other move that can tear ligaments or break bones. It also might represent a punch that breaks ribs, or wringing an imp's neck.
If successful the defender becomes disabled despite still having positive hit points and without loss of its movement causing injury. The defender also suffers ability score damage to Strength equal to the attacker's Strength Modifier.
Furthermore, the attacker retains his or her position of "Grab On", "Dominant Wrestling Position", or "Very Dominant Wrestling Position" and again causes nonlethal damage as appropriate.
A disabled character may take a single move action or standard action each round (but not both, nor can he take full-round actions, but he can still take swift, immediate, and free actions). He moves at half speed.
This state represents positions such as "mount" and "take the back" from which it is easier to debilitate the opponent.
If successful the defender becomes staggered and the attacker moves into the defender's map square (if not there already). The defender's Strength Modifier is reduced to zero. The defender is denied his or her Dexterity bonus to AC. The defender suffers nonlethal damage equal to the twice attacker's Strength Modifier from being squashed and from its own fatiguing exertion.
The staggered condition, the reduction to Strength Modifier, and the denial of Dexterity bonus to AC all only last while the defender remains in the unfavorable position. (Unless the defender was staggered for another reason.)
A staggered creature may take a single move action or standard action each round (but not both, nor can he take full-round actions). A staggered creature can still take free, swift, and immediate actions
This represents an attacker's attempt to restrict the defender's airway to cause unconsciousness.
If successful the defender, if wearing armor with an Armor Check Penalty of zero or −1, becomes unconscious.
All armor with Armor Check Penalty of −2 or more contains a gorget that completely protects the wearer from being choked out.
The releveant Armor Check Penalty is what is intrinsic to the armor. Abilities such as a Fighter's armor training do not remove a gorget.
(Yes, in my mind that masterwork mithral breastplate has given up its gorget's effectiveness to achieve a zero Armor Check Penalty. Try tumbling while wearing a still metal collar.)
Unconscious / Helpless
Unconscious creatures are knocked out and helpless.
A helpless target is treated as having a Dexterity of 0 (–5 modifier). Melee attacks against a helpless target get a +4 bonus (equivalent to attacking a prone target). Ranged attacks get no special bonus against helpless targets. Rogues can sneak attack helpless targets. As a full-round action, an enemy can use a melee weapon to deliver a coup de grace to a helpless foe.
The heroes are a hearty Fighter, a mischevious halfling, and a stern Ranger with loyal wolf companion. They are ambushed by three ogres!
Fighter: STR 18 (+4 mod), HP 30, AC 23, WAB +3, magic longsword +9 to hit (1d8+5 damage), composite longbow +4 to hit (1d8+4 damage)
Halfling: STR 10 (+0 mod), HP 20, AC 17, WAB +6, masterwork rapier +7 to hit (1d6 damage)
Ranger: STR 16 (+3 mod), HP 21, AC 17, WAB +3, masterwork longsword +9 to hit (1d8+4 damage)
Wolf: STR 14 (+2 mod), HP 16, AC 15, WAB +3, bite +3 to hit (1d6+3 damage)
Ogres: STR 21 (+5 mod), HP 30, AC 17, WAB +5, greatclub +7 to hit (2d8+7 damage), two javelins +1 to hit (1d8+5 damage)
The ogres throw javelins. The Fighter and Ranger are injured!
Fighter: STR 18 (+4 mod), HP 22, AC 23, WAB +3, magic longsword +9 to hit (1d8+5 damage), composite longbow +4 to hit (1d8+4 damage)
Ranger: STR 16 (+3 mod), HP 12, AC 17, WAB +3, masterwork longsword +9 to hit (1d8+4 damage)
The ogres throw javelins again. The Fighter and Halfling are injured!
Fighter: STR 18 (+4 mod), HP 13, AC 23, WAB +3, magic longsword +9 to hit (1d8+5 damage), composite longbow +4 to hit (1d8+4 damage)
Halfling: STR 10 (+0 mod), HP 10, AC 17, WAB +6, masterwork rapier +7 to hit (1d6 damage)
The Ranger shouts and points (and readies his action). The brave wolf runs at the nearest ogre and clamps sharp teeth into its leg (going from "Start" to "Grab On"). The ogre screams and sways, but remains standing.
Ogre #1 [grappled]: STR 21 (+5 mod), HP 22, AC 17, WAB +5, greatclub +7 to hit (2d8+7 damage)
The Ranger takes advantage of his foe being off balance by charging forward to land a tremendous blow that cracks the ogre's ribs (going from "Grab On" to "Tear or Break"). The ogre bellows! It doubles over, then straightens, grimicing in pain with fresh malevolence shining in its eyes.
Ogre #1 [grappled, disabled]: STR 18 (+4 mod), HP 21, AC 17, WAB +4, greatclub +6 to hit (2d8+6 damage)
Meanwhile, the Halfling charges a second ogre and after tripping it leaps onto its back and grabs its shirt collar (going from "Start" to "Dominant Wrestling Position"). The Fighter tries to take advantage of this ogre's unfavorable position but fails (did not succeed in going from "Dominant Wrestling Position" to "Tear or Break").
Ogre #2 [prone]: STR 21 (+2 mod), HP 30, AC 17, WAB +2, greatclub +4 to hit (2d8+4 damage)
The ogre with broken ribs reaches down and pulls the wolf from his leg. The wolf writhes free from its grasp. (They move from "Grab On" to "Start" and both lose the grappled condition.)
The second ogre, with the halfling on its back, rolls over and pins the small hero underneath its huge bulk (going from "Dominant Wrestling Position" back to "Dominant Wrestling Position"). The halfling squeaks and gasps.
Halfling [prone]: STR 10 (+0 mod), HP 6, AC 17, WAB +6, masterwork rapier +7 to hit (1d6 damage)
The third ogre was waiting for the halfling to be pinned, and rushes forward to deliver a ruthless blow to the small hero's head with his club. The halfling goes limp.
Halfling [prone, bleeding out]: STR 10 (+0 mod), HP -1, AC 17, WAB +6, masterwork rapier +7 to hit (1d6 damage)
The wolf leaps at its ogre and bites fiercely.
Ogre #1 [disabled]: STR 18 (+4 mod), HP 14, AC 17, WAB +4, greatclub +6 to hit (2d8+6 damage)
The Fighter calls to the Ranger (and readies his action). The Ranger sees an ally at death's door, and drops his sword to grapple the third ogre. He knocks it down and kneels over it (going from "Start" to "Dominant Wrestling Position"). The third ogre swings his club in vain, failing to interrupt the Ranger's manuever.
Ogre #2 [prone again]: STR 21 (+2 mod again), HP 27, AC 17, WAB +2, greatclub +4 to hit (2d8+4 damage)
The Fighter smiles grimly as he puts an armored knee onto that ogre's neck, choking it (going from "Dominant Wrestling Position" to "Choke Out"). The ogre struggles and falls unconscious.
Ogre #2 [prone, unconscious]: STR 21 (+2 mod), HP 27, AC 17, WAB +2, greatclub +4 to hit (2d8+4 damage)
The ogre near the wolf manages to grab the wolf again, and this time throws the wolf against a tree (going from "Start" to "Throw"). The wolf is on the ground, but still in fighting shape.
Wolf [prone]: STR 14 (+2 mod), HP 8, AC 15, WAB +3, bite +3 to hit (1d6+3 damage)
The third ogre swings his club heavily into the Ranger. The Range reels but remains kneeling.
Ranger: STR 16 (+3 mod), HP 2, AC 17, WAB +3, masterwork longsword +9 to hit (1d8+4 damage)
The wolf gets to its feet, sees its master in trouble, and comes to help.
The Ranger signals to the Fighter. The Fighter trips the third ogre with his sword and stands above the fallen foe (going from "Start" to "Dominant Wrestling Position").
Ogre #3 [prone]: STR 21 (+2 mod), HP 26, AC 17, WAB +2, greatclub +4 to hit (2d8+4 damage)
The Ranger quickly draws a dagger and plunges it deep into the fallen ogre's hip (going from "Dominant Wrestling Position" to "Tear or Break".
Ogre #3 [prone, disabled]: STR 18 (+4 mod), HP 21, AC 17, WAB +4, greatclub +6 to hit (2d8+6 damage)
The first ogre stumbles towards its ally, but is too hurt to be able to also attack.
The third ogre punches the Ranger, who goes limp.
Ranger: STR 16 (+3 mod), HP -2, AC 17, WAB +3, masterwork longsword +9 to hit (1d8+4 damage)
The Fighter backs up and draws his bow. He signals for the wolf to stay back at a safe distance.
Rounds Five and Greater
The Fighter keeps backing up and shooting the ogres, who at first stumble after him but then attempt to flee. The Fighter kills them both.
Then, just in time, the Fighter gets two healing potions from his backpack and administers them to his companions.
Ta da! These new rules allow combat to feel more like an exciting fight.