Tiller = top limb gap − bottom limb gap. Each gap is measured with a bow square at a right angle to the string, from the string to the limb’s belly right where it leaves the riser (Lancaster Archery; Improve Your Archery).
What it should be depends on your hook (Olympian Jake Kaminski; Bow International; Glasgow Archers): split finger, the top 1/8–1/4″ (3–6 mm) more than the bottom; three under, about even or the bottom up to 1/8″ more; string walking, the bottom 1/8″ or more, more than the top. Tighten a limb bolt and that limb’s gap shrinks; loosen it and the gap grows. A quarter turn at a time.
A worked example
Top gap 7 1/4″, bottom gap 7 1/16″: 7 1/4 − 7 1/16 = +3/16″ (4.8 mm). For split finger that’s inside 1/8–1/4″: nothing to fix. Shooting three under, it’s set for split finger, so a quarter turn tighter on the top bolt starts evening it out.
The full explanation: Tiller in the glossary.