The OS detection is done with some clunky user agent sniffing (boo!) and is written up as a Modernizr test.
Modernizr.addTest("macosx", function () {
var strIndex = navigator.userAgent.toLowerCase().indexOf("mac");
return (strIndex > -1);
});
From here, it really depends on your plugin library, your theme etc. The Kiandra theme I'm using for this demo has an outdented close button, similar to Growl for Mac. Instead of positioning it to the right, you just lock it to the left, and switch the button floats. Super simple stuff.
.macosx .ui-dialog-titlebar-close { right: inherit; left:-32px; }
.macosx .ui-dialog-buttonset { float: none; }
.macosx .ui-dialog-buttonset button { float: right; }