MathML stands for
mathematical markup language.
It is based on XML to be HTML-friendly.
Firefox and Seamonkey can render MathML embedded in XHTML. I have compiled a MathML-capable version of Camino.
MathPlayer is a plugin for Internet Explorer on Windows.
You need to install mathematical fonts (TEX and Mathematica fonts) to display special characters such as root glyph and long braces.
Note that Bakoma fonts available from Mozilla site are not compatible with Gecko engine on Mac.
itex2MML is an authoring tool for MathML. It reads itex embedded html files to to generate xhtml+mathml files. Jacques Distler distributes binaries for Mac OS X and Linux. MathZilla provides web service based on itex2MML.
Amaya is an XHTML/MathML authoring tool from w3c.
I am (or was) working on these bugs at bugzilla.mozilla.org.