So they don't arm until the fins open? Hmm... didn't know that. That definitely changes things... If you can get it out of the shell, patched, and rammed down into the Brown Bess without opening the fins... it just might work.
That said, if it doesn't, well... I knew a guy who had an M80 go off in his hand whilst he was getting ready to throw it. IIRC, he lost 3 fingers, part of his earlobe, had a big hole blown in his cheek, and I think he lost a few teeth too. Plus a blown-out eardrum and permanent hearing damage his right ear. Now, replace M80 with FRAG 12 and you're in for a world of hurt.
Possible consequences aside, if it did work it'd be freakin' cool!
BTW, I looked up the muzzle diameter of a 12-gauge, and I think the FRAG-12 would be just small enough to fit in a Brown Bess.