Stencil Test

The stencil test compares the stencil attachment value sa in the depth/stencil attachment at each sample’s framebuffer coordinates (xf, yf) and sample index i against a stencil reference value.

If the stencil test is not enabled, as specified by AfxCmdEnableStencilTest or VkPipelineDepthStencilStateCreateInfo::stencilTestEnable, or if there is no stencil attachment, the coverage mask is unmodified by this operation.

The stencil test is controlled by one of two sets of stencil-related state, the front stencil state and the back stencil state. Stencil tests and writes use the back stencil state when processing fragments generated by back-facing polygons, and the front stencil state when processing fragments generated by front-facing polygons or any other primitives.

The comparison operation performed is determined by the afxCompareOp value set by AfxCmdSetStencilOp::compareOp, or by afxStencilOpState::compareOp during pipeline creation.

The compare mask sc and stencil reference value sr of the front or the back stencil state set determine arguments of the comparison operation. sc is set by the VkPipelineDepthStencilStateCreateInfo structure during pipeline creation, or by the AfxCmdSetStencilCompareMask command. sr is set by VkPipelineDepthStencilStateCreateInfo or by AfxCmdSetStencilReference.

sr and sa are each independently combined with sc using a bitwise AND operation to create masked reference and attachment values s’r and s’a. s’r and s’a are used as the reference and test values, respectively, in the operation specified by the afxCompareOp.

If the comparison evaluates to false, the coverage for the sample is set to 0.

A new stencil value sg is generated according to a stencil operation defined by VkStencilOp parameters set by AfxCmdSetStencilOp or VkPipelineDepthStencilStateCreateInfo. If the stencil test fails, failOp defines the stencil operation used. If the stencil test passes however, the stencil op used is based on the depth test – if it passes, VkPipelineDepthStencilStateCreateInfo::passOp is used, otherwise VkPipelineDepthStencilStateCreateInfo::depthFailOp is used.

The stencil attachment value sa is then updated with the generated stencil value sg according to the write mask sw defined by writeMask in VkPipelineDepthStencilStateCreateInfo::front and VkPipelineDepthStencilStateCreateInfo::back as:

sa = (sa & ¬sw) | (sg & sw)

To dynamically enable or disable the stencil test, call:

void AfxCmdEnableStencilTest
(
    afxDrawScript dscr, /// is the command buffer into which the command will be recorded.
    afxBool enabled /// specifies if the stencil test is enabled.
);

This command sets the stencil test enable for subsequent drawing commands when the graphics pipeline is created with VK_DYNAMIC_STATE_STENCIL_TEST_ENABLE set in VkPipelineDynamicStateCreateInfo::pDynamicStates. Otherwise, this state is specified by the VkPipelineDepthStencilStateCreateInfo::stencilTestEnable value used to create the currently active pipeline.

To dynamically set the stencil operation, call:

void vkCmdSetStencilOp
(
    afxDrawScript dscr, /// is the command buffer into which the command will be recorded.
    afxFlags faceMask, /// is a bitmask of bits specifying the set of stencil state for which to
update the stencil operation.
    afxStencilOp failOp, /// is a value specifying the action performed on samples that fail the stencil
test.
    afxStencilOp passOp, /// is avalue specifying the action performed on samples that pass both the
depth and stencil tests.
    afxStencilOp depthFailOp, /// is a value specifying the action performed on samples that pass the
stencil test and fail the depth test.
    afxCompareOp compareOp /// is a value specifying the comparison operator used in the stencil test.
);

This command sets the stencil operation for subsequent drawing commands when when the graphics pipeline is created with VK_DYNAMIC_STATE_STENCIL_OP set in VkPipelineDynamicStateCreateInfo::pDynamicStates. Otherwise, this state is specified by the corresponding VkPipelineDepthStencilStateCreateInfo::failOp, passOp, depthFailOp, and compareOp values used to create the currently active pipeline, for both front and back faces.

The afxStencilOpState structure is defined as:

AFX_DEFINE_STRUCT(afxStencilOpState)
{
    afxStencilOp failOp; ///  is a VkStencilOp value specifying the action performed on samples that fail the stencil test.
    afxStencilOp passOp; ///  is a VkStencilOp value specifying the action performed on samples that pass both the depth and stencil tests.
    afxStencilOp depthFailOp; ///  is a VkStencilOp value specifying the action performed on samples that pass the stencil test and fail the depth test.
    afxCompareOp compareOp; ///  is a VkCompareOp value specifying the comparison operator used in the stencil test.
    afxNat32 compareMask; ///  selects the bits of the unsigned integer stencil values participating in the stencil test.
    afxNat32 writeMask; ///  selects the bits of the unsigned integer stencil values updated by the stencil test in the stencil framebuffer attachment.
    afxNat32 reference; /// is an integer stencil reference value that is used in the unsigned stencil comparison.
};

To dynamically set the stencil compare mask, call:

void vkCmdSetStencilCompareMask
(
    afxDrawScript dscr, /// is the command buffer into which the command will be recorded.
    afxFlags faceMask, /// is a bitmask of (0/FRONT, 1/BACK, 2/BOTH) bits specifying the set of stencil state for which to update the compare mask.
    afxNat32 compareMask /// is the new value to use as the stencil compare mask.
);

This command sets the stencil compare mask for subsequent drawing commands when the graphics pipeline is created with VK_DYNAMIC_STATE_STENCIL_COMPARE_MASK set in VkPipelineDynamicStateCreateInfo::pDynamicStates. Otherwise, this state is specified by the VkStencilOpState::compareMask value used to create the currently active pipeline, for both front and back faces.

To dynamically set the stencil write mask, call:

void AfxCmdSetStencilWriteMask
(
    afxDrawScript dscr, /// is the command buffer into which the command will be recorded.
    afxFlags faceMask, /// is a bitmask of (0/FRONT, 1/BACK, 2/BOTH) bits specifying the set of stencil state for which to update the write mask, as described above for vkCmdSetStencilCompareMask.
    afxNat32 writeMask /// is the new value to use as the stencil write mask.
);

This command sets the stencil write mask for subsequent drawing commands when the graphics pipeline is created with VK_DYNAMIC_STATE_STENCIL_WRITE_MASK set in VkPipelineDynamicStateCreateInfo::pDynamicStates. Otherwise, this state is specified by the writeMask value used to create the currently active pipeline, for both VkPipelineDepthStencilStateCreateInfo::front and VkPipelineDepthStencilStateCreateInfo::back faces.

To dynamically set the stencil reference value, call:

void AfxCmdSetStencilReference
(
    afxDrawScript dscr, /// is the command buffer into which the command will be recorded.
    afxFlags faceMask, /// is a bitmask of (0/FRONT, 1/BACK, 2/BOTH) bits specifying the set of stencil state for which to
    afxNat32 reference /// is the new value to use as the stencil reference value.
);

This command sets the stencil reference value for subsequent drawing commands when the graphics pipeline is created with VK_DYNAMIC_STATE_STENCIL_REFERENCE set in VkPipelineDynamicStateCreateInfo::pDynamicStates. Otherwise, this state is specified by the VkPipelineDepthStencilStateCreateInfo::reference value used to create the currently active pipeline, for both front and back faces.

Possible values of the failOp, passOp, and depthFailOp members of VkStencilOpState, specifying what happens to the stored stencil value if this or certain subsequent tests fail or pass, are:

typedef enum afxStencilOp
{
    afxStencilOp_KEEP, /// keeps the current value.
    afxStencilOp_ZERO, /// sets the value to 0.
    afxStencilOp_REPLACE, /// sets the value to reference.
    afxStencilOp_INC_N_CLAMP, /// increments the current value and clamps to the maximum representable unsigned value.
    afxStencilOp_DEC_N_CLAMP, /// decrements the current value and clamps to 0.
    afxStencilOp_INV, /// bitwise-inverts the current value.
    afxStencilOp_INC_N_WRAP, /// increments the current value and wraps to 0 when the maximum value would have been exceeded.
    afxStencilOp_DEC_N_WRAP, /// decrements the current value and wraps to the maximum possible value when the value would go below 0.
} afxStencilOp;

For purposes of increment and decrement, the stencil bits are considered as an unsigned integer.