Depth-Stencil State

This document details the configuration and usage of depth-stencil states within the Windows graphics API. Depth-stencil state is a crucial part of the rendering pipeline that controls how depth and stencil operations are performed, enabling effects like hidden surface removal and advanced rendering techniques.

Introduction

Depth-testing, also known as Z-buffering, is used to determine which fragments are visible to the camera. Fragments closer to the camera overwrite fragments farther away. Stencil operations provide a mask for rendering, allowing for complex effects such as shadows, decals, and silhouette outlines.

The depth-stencil state object encapsulates the configuration for both depth and stencil testing and operations. It's typically created once and bound to the graphics pipeline as needed.

Depth Testing

Depth testing compares the depth value of a new fragment with the depth value already stored in the depth buffer. The outcome of this comparison determines whether the fragment is discarded or written to the depth and color buffers.

• Depth Test Enable: A boolean flag to enable or disable depth testing.
• Depth Function: Specifies the comparison function used. Common values include D3D11_COMPARISON_LESS (default), D3D11_COMPARISON_EQUAL, D3D11_COMPARISON_LESS_EQUAL, etc.
• Depth Write Mask: Controls whether the depth buffer is written to. D3D11_DEPTH_WRITE_MASK_ALL (default) allows writing, while D3D11_DEPTH_WRITE_MASK_ZERO prevents writing.

Stencil Testing

Stencil testing uses a stencil buffer, which stores integer values per pixel. These values can be modified and tested against during rendering to create masking effects.

• Stencil Test Enable: A boolean flag to enable or disable stencil testing.
• Stencil Read Mask: A mask applied to the stencil buffer values before comparison.
• Stencil Write Mask: A mask applied to the stencil buffer values when they are written.
• Stencil Front Face: Defines stencil operations for faces oriented towards the camera.
• Stencil Back Face: Defines stencil operations for faces oriented away from the camera.

Stencil Operations

Each face (front and back) can have separate stencil operations defined for the following conditions:

• Stencil Fail: Operation to perform if the stencil test fails.
• Stencil Depth Fail: Operation to perform if the stencil test passes but the depth test fails.
• Stencil Pass: Operation to perform if both stencil and depth tests pass.

Common stencil operations include:

• D3D11_STENCIL_OP_KEEP (default): Keep the current stencil buffer value.
• D3D11_STENCIL_OP_ZERO: Set the stencil buffer value to zero.
• D3D11_STENCIL_OP_REPLACE: Replace the stencil buffer value with the reference value.
• D3D11_STENCIL_OP_INCR_SAT: Increment the stencil buffer value, saturating at the maximum value.
• D3D11_STENCIL_OP_DECR_SAT: Decrement the stencil buffer value, saturating at zero.
• D3D11_STENCIL_OP_INVERT: Invert the bits of the stencil buffer value.
• D3D11_STENCIL_OP_INCR: Increment the stencil buffer value.
• D3D11_STENCIL_OP_DECR: Decrement the stencil buffer value.

Creating a Depth-Stencil State

Depth-stencil states are created using the ID3D11Device::CreateDepthStencilState method. This method takes a D3D11_DEPTH_STENCIL_DESC structure as input, which specifies all the configuration parameters.

D3D11_DEPTH_STENCIL_DESC Structure

The primary structure used for defining the state:

typedef struct D3D11_DEPTH_STENCIL_DESC {
    BOOL DepthEnable;
    D3D11_DEPTH_WRITE_MASK DepthWriteMask;
    D3D11_COMPARISON_FUNC DepthFunc;
    BOOL StencilEnable;
    UINT8 StencilReadMask;
    UINT8 StencilWriteMask;
    D3D11_DEPTH_STENCIL_OP FrontFace;
    D3D11_DEPTH_STENCIL_OP BackFace;
} D3D11_DEPTH_STENCIL_DESC;
            

D3D11_DEPTH_STENCIL_OP Structure

Defines stencil operations for a face:

typedef struct D3D11_DEPTH_STENCIL_OP {
    D3D11_STENCIL_OP StencilFailOp;
    D3D11_STENCIL_OP StencilDepthFailOp;
    D3D11_STENCIL_OP StencilPassOp;
} D3D11_DEPTH_STENCIL_OP;
            

Example: Default Depth-Stencil State

Here's how to create a typical depth-stencil state with depth testing enabled:

D3D11_DEPTH_STENCIL_DESC dsDesc = {}; // Initialize to zeros

dsDesc.DepthEnable = TRUE;
dsDesc.DepthWriteMask = D3D11_DEPTH_WRITE_MASK_ALL;
dsDesc.DepthFunc = D3D11_COMPARISON_LESS;

dsDesc.StencilEnable = FALSE; // Stencil test disabled by default

// Other members are default initialized if StencilEnable is FALSE

ID3D11DepthStencilState* pDefaultDSState = nullptr;
HRESULT hr = pDevice->CreateDepthStencilState(&dsDesc, &pDefaultDSState);
if (FAILED(hr)) {
    // Handle error
}
            

Example: Stencil Buffer for Shadow Mapping

A common use case for stencil buffers is shadow mapping. The following configuration might be used:

D3D11_DEPTH_STENCIL_DESC shadowPassDesc = {};

// Depth test disabled for the shadow caster pass
shadowPassDesc.DepthEnable = FALSE; 
shadowPassDesc.StencilEnable = TRUE;
shadowPassDesc.StencilReadMask = 0xFF;
shadowPassDesc.StencilWriteMask = 0xFF;

// Increment stencil buffer for shadow casters
shadowPassDesc.FrontFace.StencilFailOp = D3D11_STENCIL_OP_KEEP;
shadowPassDesc.FrontFace.StencilDepthFailOp = D3D11_STENCIL_OP_KEEP;
shadowPassDesc.FrontFace.StencilPassOp = D3D11_STENCIL_OP_INCR_SAT;

// Back face operations might be different or similar depending on complexity
shadowPassDesc.BackFace = shadowPassDesc.FrontFace; 

ID3D11DepthStencilState* pShadowPassDSState = nullptr;
hr = pDevice->CreateDepthStencilState(&shadowPassDesc, &pShadowPassDSState);
// ... handle hr ...
            

Binding the Depth-Stencil State

Once created, the depth-stencil state is bound to the graphics pipeline using ID3D11DeviceContext::OMSetDepthStencilState:

ID3D11DepthStencilState* pCurrentDSState = nullptr; // To get current state if needed
UINT stencilRef = 0; // Stencil reference value

// Bind the created depth-stencil state
pImmediateContext->OMSetDepthStencilState(pDefaultDSState, 1); // Using stencilRef = 1
            

API Reference Summary

Function/Structure	Description
ID3D11Device::CreateDepthStencilState	Creates a depth-stencil state object.
ID3D11DeviceContext::OMSetDepthStencilState	Sets the depth-stencil state for the output merger stage.
D3D11_DEPTH_STENCIL_DESC	Structure describing depth-stencil state properties.
D3D11_DEPTH_STENCIL_OP	Structure describing stencil operations.
D3D11_COMPARISON_FUNC	Enumeration for depth/stencil comparison functions.
D3D11_STENCIL_OP	Enumeration for stencil operations.
D3D11_DEPTH_WRITE_MASK	Enumeration for controlling depth buffer writes.