Brief overview of object textures and their roles by Inge
Mask
Color used as 3 seperate channels. Red channel = patternA, Green = patternB, Blue = patternC. Alpha channel used for patternD when there is one. For each channel, its pattern will show up to some extent anywhere the pixel is not pure black. The usual is to have the pixel either completely black or completely white for any individual channel. Variations can be used to interesting effect for those objects that support it. (depends on TXTC)
Some objects are set up so that the channels can blend together where they overlap. For those that are not set that way, alpha overlays blue, blue overlays green, green overlays red.
Save as DXT5 if object has 4 patterns, otherwise DXT1
Multiplier
Greyscale. Alpha channel can be used for semi-transparency when the shader supports it. Most shaders support a complete cutout when alpha channel pixel is black. The rgb channels are used together to provide the grey scale. Controls brightness of applied patterns. Darker grey is used to suggest shadowy areas, brighter grey for lighter, like the top of ledges.
Save as DXT5 if you needed alpha channel transparency, else DXT1 without alpha channel.
Overlay
The overlay is applied over the top of any CASt patterns, but underneath any stencils. The alpha channel has white pixels anywhere you will your overlay to appear, and black pixels everywhere else. Just use the color channels to create a normal color image in the area where your overlay is required. It doesn't matter what you do with the areas under black alpha pixels, they will never be seen.
If you do not want an overlay at all, save file size by using a tiny image with black alpha and black picture, or link to EA's default blank overlay.
Save as DXT5
Specular
Controls overall shininess of object. Amount of shine is controlled by alpha channel. The whiter the pixel, the shinier that area. Black alpha means no shine, white alpha means glaring shine - probably obscuring any patterns you apply! Color in color channels influences color of shine.
Shine of the specular is not a true reflection, nor is it actual light being reflected or emitted. It is a type of graphic effect.
Some shaders do not support specular at all, and some objects don't even have a specular compositor, so you'd waste your time making a specular image.
Save as DXT5
Stencils
Mainly pointless. Work like overlay, and would lay over the top of an overlay. Not all TXTCs support all 4 stencils, and some only support one enabled at a time.
How the images are made into a composite texture
Mask comes first. All images are subsequently stretched or shrunk to fit mask size. If you set it as -1 it will take the actual mask image size, else it will obey your stated mask size and, mask will be stretched or shrunk to fit that.
Next the patterns from the default preset (the one in the main xml) are applied to mask areas.
Then the multiplier greyscale (ignoring alpha channel) is applied to modify the brightness.
Then overlay is applied, respecting its alpha channel.
Then stencils are applied, respecting their alpha channels.
Finally, any area that has a black alpha pixel in overlay, multiplier and all enabled stencils will be cut away (made invisible) for objects that support it (floor tiles don't - you will just see the grey floor mesh through the pattern if you try it that is why we can't make proper glass floors)