Quality MSAA Resolve

When MSAA is done right I avoid all morphological AA's like FXAA because they add artifacts. Here are some hints on how to do MSAA right. These techniques work in HLSL or GLSL, using the HLSL terms here.

When using MSAA with alpha test, you can run at the standard pixel rate, but manually use EvaluateAttributeSnapped() to super-sample the alpha test inside the shader and then manually set SV_Coverage to get an MSAAed edge. There are other tricks for soft particle blending, but out of time right now...

Describing the 4xMSAA version of a proper filtered resolve below, it is trivial to extend to 8xMSAA. Read samples in a 2 pixel diameter (minimum suggested, larger can improve quality at more cost). That means start with the 3x3 pixel neighborhood,

. N . .  . N . .  . N . . 
. . . E  . . . E  . . . E
W . . .  W . . .  W . . . 
. . S .  . . S .  . . S .

. N . .  . N . .  . N . . 
. . . E  . . . E  . . . E
W . . .  W . . .  W . . . 
. . S .  . . S .  . . S .

. N . .  . N . .  . N . . 
. . . E  . . . E  . . . E
W . . .  W . . .  W . . . 
. . S .  . . S .  . . S .

And read the following 16 samples (make sure to have linear data after the read for the filtering process),

. . . .  . . . .  . . . . 
. . . .  . . . .  . . . .
. . . .  W . . .  W . . . 
. . S .  . . S .  . . . .

. . . .  . N . .  . N . . 
. . . E  . . . E  . . . .
. . . .  W . . .  W . . . 
. . S .  . . S .  . . . .

. . . .  . N . .  . N . . 
. . . E  . . . E  . . . .
. . . .  . . . .  . . . . 
. . . .  . . . .  . . . .

Filter weights can be any filter you want (see Reconstruction Filters in Computer Graphics for many options, but note negative lobes with only 4xMSAA tends to not work well, and play with the radius until it looks right). Each of the four following lettered sample groups shares the same filter weight (same distance from center of the pixel),

. . . .  . . . .  . . . . 
. . . .  . . . .  . . . .
. . . .  c . . .  d . . . 
. . d .  . . b .  . . . .

. . . .  . a . .  . c . . 
. . . b  . . . a  . . . .
. . . .  a . . .  b . . . 
. . c .  . . a .  . . . .

. . . .  . b . .  . d . . 
. . . d  . . . c  . . . .
. . . .  . . . .  . . . . 
. . . .  . . . .  . . . .

Filter weights should be pre-computed. You end up with a resolve filter which does the following math where {a,b,c,d} are pre-computed filter weights,

outColor.rgb =
sample0 * a +
sample1 * a +
sample2 * a +
sample3 * a +
sample4 * b +
sample5 * b +
sample6 * b +
sample7 * b +
sample8 * c +
sample9 * c +
sampleA * c +
sampleB * c +
sampleC * d +
sampleD * d +
sampleE * d +
sampleF * d;

I'd highly suggest re-ordering the math and loads so that samples are processed from pixels in this order,

0 1 2
5 4 3
6 7 8

Fetching all samples needed for the pixel and FMAing them to the weighted average output. This way the compiler is pre-warmed to the best fetch ordering.