Custom shaders

A custom shader is a surface that looks the way you write it: cel shading, a hologram, glowing lava, stylised water. You write it in HLSL, in one .hlsl file, and materials use it as they use the engine’s own shaders. It comes in two kinds:

• A surface shader says what the surface is: its colour, shininess, bumps and glow, and optionally how its vertices move. The engine does the rest: every light, shadow and reflection probe, the shadows cast by the moved vertices, and picking in the editor. This is the kind to start with.
• A low-level shader is the whole program, vertex and pixel stages, for shading of your own, such as a cel shader that bands its light. It has to keep the engine’s pass contract itself, and the engine’s shader library helps it do so.

Making one

New > Materials has three shaders: Lit Surface Shader (glowing lava cracks, lit by the scene), Unlit Surface Shader (a colour and a texture, unaffected by light) and Low-Level Shader (shaded the engine’s way, ready to change). Each makes a shader in the Shaders folder, with its .hlsl file in the project’s Sources/, and opens it. You can also drop an .hlsl file into the Assets panel; the shader is named after the file.

The shader page

The code is on the left. A moment after you stop typing, the shader is checked: errors are marked on their lines and listed under the code. On the right is a sphere drawn with the shader as it stands, at its parameters’ defaults. While the code does not compile, the sphere keeps the last version that did and says so. Below the sphere are the shader’s settings:

• Name: the name materials find the shader by. It cannot be the name of one of the engine’s own shaders (forward, unlit and the rest); a shader with such a name fails to cook.
• Vertex Entry Point and Fragment Entry Point: the functions a low-level shader’s stages start in, VSMain and PSMain at first. A surface shader uses the engine’s.

The check as you type compiles the base version of each stage. Save compiles every version of the shader (skinned, instanced, masked, and a surface shader’s shadow and pick passes) and reports what fails. A shader that fails keeps its last good version in your scenes until it compiles again.

Parameters: the properties block

A shader declares the parameters its materials set in a properties block, one declaration per line, each ending with ;: a type, a name, an optional default and, for a float or an int, an optional [min, max] range, which the material page shows as a slider. A default is one number, or several in parentheses, one for each component; a colour may leave out its alpha, which is then 1.

properties
{
    Color    BaseColor = (1, 0.4, 0.1, 1);
    ColorHdr Glow      = (1, 0.3, 0, 4);     // a colour with an intensity
    float    Speed     = 0.2 [0, 2];
    int      Steps     = 3   [1, 8];
    float2   Tiling    = (1, 1);
    Texture  Cracks;
    Sampler  MainSampler;
}

Type
float, float2, float3, float4, int	Numbers. 0 unless given.
Color	A colour, picked as you see it; the shader reads it linear. White unless given.
ColorHdr	A colour and an intensity (the fourth value), for glowing things. Black at intensity 1 unless given.
Texture	A 2D texture. It takes no default.
TextureCube	A cube map. It takes no default.
Sampler	How the textures are sampled. Declare one to sample with. It takes no default.

The engine turns the block into the HLSL declarations, so the shader uses each parameter by its name, and a material’s settings are exactly these. Lines after the block keep their numbers, so errors point at the line you wrote.

Surface shaders

A surface shader starts with a line saying how it is lit, then its parameters, then Surface():

// surface: lit
properties
{
    Color   CrackColor = (1.0, 0.35, 0.05, 1.0);
    float   Speed      = 0.2 [0, 2];
    float   Glow       = 4.0 [0, 20];
    Texture Cracks;
    Sampler MainSampler;
}

void Surface(SurfaceInput input, inout SurfaceOutput surface)
{
    float2 uv = input.uv + float2(0.0, Time() * Speed);
    float crack = Cracks.Sample(MainSampler, uv).r;
    surface.albedo = lerp(float3(0.08, 0.05, 0.04), CrackColor.rgb, crack);
    surface.roughness = 0.9;
    surface.emissive = CrackColor.rgb * crack * Glow;
}

The // surface: lit line, on a line of its own, makes the file a surface shader that is shaded with every light, shadow and reflection probe; // surface: unlit shows its colour as it is. Any other word after surface: is an error. A surface shader uses the engine’s variants, so it has no // variants lines; the check reports one as an error.

What Surface() is given (SurfaceInput), in world space: worldPos, normal, tangent (with the bitangent’s sign in w), uv, color (the vertex colour times the mesh’s tint, or for an instanced mesh, the Instance Member’s Tint), viewDir (toward the camera) and screenPos (in pixels).

What it decides (SurfaceOutput), starting as a plain white surface:

Field		Starts at
albedo	its colour	white
alpha	opacity; a Masked material cuts out below its cutoff	1
normal	a bump, in tangent space (a normal map’s value)	(0, 0, 1), flat
metallic	0 for most things, 1 for bare metal	0
roughness	0 is a mirror, 1 matte	0.5
occlusion	darkening in creases, 1 = none	1
emissive	light it gives off, unlit; above 1 it glows	black

Moving the vertices. Add Vertex() to move them, in world space, before they are drawn:

void Vertex(inout VertexInput v)
{
    v.position.y += sin(v.position.x * 2.0 + Time()) * 0.1;   // waves
}

VertexInput has position, normal and tangent, which you may change, and localPosition (the mesh’s own position, before it is placed), uv and color, which are read only. The shadow, the depth and the editor’s picking all move with it.

Time() is the scene’s clock in seconds, in both functions.

Cut-outs. On a Masked material, the surface’s alpha cuts it out, and its shadow too. The cut is below the value of a float parameter named AlphaCutoff when the shader declares one, and below a half otherwise.

What a surface shader cannot use. It reads the camera and the clock through its input and Time(), not the frame’s own constants, because each of its passes has different ones. A parameter cannot take a name the engine’s shaders already use (Tint, for one): the shader page says so at the parameter’s line.

Low-level shaders

A low-level shader draws in the engine’s forward pass, so it keeps its contract: it takes the mesh’s vertex in, and its pixel stage writes what the pass expects. The engine’s shader library does most of that for you, and the Low-Level Shader starter shows how to use it:

// variants(vertex): SKINNED INSTANCED WIND
// variants(fragment): ALPHA_TEST GBUFFER DITHER
// preserve-interface

#define FORWARD_VS_ENTRY VSMain
#include "forward_vs.hlsli"     // the whole vertex stage, as VSMain
#include "lighting.hlsli"       // the frame's lights, shadows and image-based light
#include "forward_ps.hlsli"     // what the pixel stage writes

• The vertex stage. forward_vs.hlsli is the engine’s own: skinning, instancing and wind, writing a VSOutput (world position, normal, UV, vertex colour) the pixel stage reads.

• Variants. The variants lines say which versions of each stage to compile: // variants(vertex): for the vertex stage and // variants(fragment): for the pixel stage (a plain // variants: line covers a stage that has no line of its own). A skinned mesh uses SKINNED, an instanced one INSTANCED and a swaying one WIND (see Wind); ALPHA_TEST is a masked material, DITHER a fading mesh, and GBUFFER the opaque pass, which writes the extra targets the post-processing reads. Test them with #ifdef. A flag the shader tests but does not declare is an error.

• // preserve-interface keeps every input the pixel stage declares, even one a version does not read. The starter needs it because its pixel stage reads the fade input only under DITHER.

• Lighting the engine’s way. DirectLight adds up every light reaching a pixel, shaded and shadowed; AmbientLight adds the sky’s light, reflection probes and the scene’s ambient.

• Lighting your own way. Walk the lights yourself, as a toon shader does:

  LightList lights = LightsAt(input.clip.xy, viewDepth);
  for (uint i = 0; i < lights.count; ++i)
  {
      GpuLight L = LightAt(lights, i);
      float atten;
      float3 toLight = LightDirection(L, input.worldPos, atten);
      float shadow = ShadowOf(L, input.worldPos, N, viewDepth);
      // ... your response to it
  }

• What the pixel stage writes. Under GBUFFER, a GBufferOutput: the colour, the normal (OctEncode), the motion vector (MotionVector), roughness and metallic, and the albedo; otherwise only the colour. Under DITHER, call ScreenDoorFade first.

#include reaches the engine’s shader library only; a shader cannot include files of the project’s own.

Materials on it

On a material’s page, the Shader row picks the shader: drag one onto it or pick it. A shader must have been saved without errors before a material can use it. The material’s settings become the shader’s parameters, keeping every value whose name and type still match. Clear the row to go back to the built-in PBR shader, or choose PBR, Unlit or Phong in the Built-in row. When you change the shader’s parameters and save it, its materials follow.

Wind

A shader’s materials can sway in the wind, as the engine’s own do: each has a Wind section on its page (Strength, Speed and Height; see Meshes and materials). A shader takes part by declaring WIND in its vertex variants. Every surface shader does, since the engine writes its vertex stage, and the Low-Level Shader starter’s // variants(vertex): line lists it; leave WIND out of that line and the shader’s materials have no Wind section and never sway. A shader on the engine’s vertex stage (every surface shader, and a low-level one that includes forward_vs.hlsli, as the starter does) needs nothing more: that stage applies the sway, and the shadow and the picking move with it.

The values are the material’s own, not parameters of the shader: they need no line in its properties block, and a parameter of its own may still be named Speed or Height. A shader that wants them reads them as Wind.Strength, Wind.Speed and Wind.Height, in its vertex stage under #ifdef WIND, once wind.hlsli is included (forward_vs.hlsli includes it). That suits a low-level shader that writes its own vertex stage:

#ifdef WIND
#include "wind.hlsli"
#endif

// ... in VSMain
#ifdef WIND
    float sway = Wind.Strength;   // 0 when the material's wind is off
#endif

In an exported game

Exporting compiles every shader the game uses for its platform, so an exported game, the web included, needs no shader compiler. The .hlsl source is left out; turn on the export preset’s Shader source to ship it beside the compiled shaders, for players who mod the game to read.

Learning by example

The Shader Gallery sample (Data/SampleProjects/ShaderGallery) is a course in custom shaders: a hall of plinths with one shader on each, and a list that glides the camera to each lesson. Every .hlsl is a short lesson that builds on an earlier one and comments what is new in it, so read them in order, or open one alone and copy it. Open the project in the editor to edit a lesson on its shader page with its preview beside it, or to drag its material’s sliders.

#	Lesson	Kind	What it teaches
1	Unlit Colour	unlit surface	The smallest shader: a properties block and Surface()
2	Textured	lit surface	Sampling a texture, the uv and the vertex colour, lit by the scene
3	Lava	lit surface	Time() to make it move, and emission to make it glow
4	Hologram	unlit surface	viewDir for a fresnel rim, screenPos for scanlines, alpha on a blended material
5	Triplanar Rock	lit surface	Mapping by worldPos with no uv, and a normal map turned into tangent space
6	Dissolve	lit surface	Alpha on a masked material cutting holes, its shadow cut too, with a burning edge
7	Waves	lit surface	Vertex() moving the vertices, with the normal from the slope
8	Wind	lit surface	Vertex() by localPosition: a flag still at the pole, flapping at the end
9	Built-in PBR	lit surface	The engine’s own material written as a surface shader: the one to start from
10	Toon	low-level	Lighting of its own: light in bands, a shadow colour and a rim
11	Custom Lighting	low-level	Walking the lights with LightsAt, LightAt and ShadowOf: half-Lambert and Blinn-Phong