AST Explorer

Source code is written as text, saved as text and reviewed as text, so it is natural to think of a program as a long string of characters. No compiler agrees. Before anything runs, the text is taken apart and rebuilt as a tree, and from that moment the tree is the program. Every later decision, from what a name refers to down to the order in which two operations happen, is made by walking that tree rather than by rereading the characters.

That gap between what we see and what the machine uses is where many small mysteries live. Why does a minus sign in front of a power behave so oddly? Why does an indented line belong to one block and not another? Why does a formatter sometimes move a comment you thought was attached to a statement? Each of these questions has a definite answer, and the answer is written in the tree.

From characters to tokens to nodes

Parsing, in the textbook sense, is the analysis of a string of symbols against the rules of a formal grammar, breaking it into parts. In practice it happens in two passes. The first pass, lexical analysis, cuts the stream of characters into tokens: names, numbers, strings, operators, brackets. The second pass checks that the tokens form an allowed arrangement and records that arrangement as structure.

Tokens and nodes are easy to confuse, but they are different things. A token is a flat piece of text with a type: the characters rate, a NAME. A node is a place in the structure: the same rate as the right operand of a multiplication that is itself the right operand of an addition. One token can produce a node, several tokens can produce one node, and some tokens produce no node at all.

The parentheses are the clearest case. An abstract syntax tree is called abstract because it keeps the structural details of the code and drops the rest; grouping parentheses are implicit in the shape of the tree, so they need no node of their own. Write (a + b) * c and the tree has a multiplication whose left child is an addition. The brackets did their job by deciding that shape, and then they vanished.

Commas, colons, semicolons and the keyword then that some languages require go the same way. What survives is meaning: an if statement becomes a single node with branches for the condition, the body and the alternative. This is what makes the tree useful to programs that read programs. A tool that wants every function call does not have to know how many spaces or line breaks a programmer put between the name and the bracket.

Spans: the tree remembers the text

Throwing away punctuation could make the tree hard to connect back to the file, and that would be a problem, because almost every tool that reads code eventually has to talk to a person about a place in the source. So parsers store positions. A syntax tree commonly records where each element sat in the source code, which is what lets a compiler print an error message that points at the right line.

Python’s ast module is precise about this. Expression and statement nodes carry four numbers: lineno and end_lineno, counted from 1, and col_offset and end_col_offset, which are UTF-8 byte offsets of the first and last tokens that produced the node. The end offset sits just after the last symbol, so slicing a one-line source line from col_offset to end_col_offset gives back exactly the text of the node. Bytes, not characters, are counted because the parser works in UTF-8 internally.

That detail matters. On a line with an accented letter or an emoji, byte and character columns disagree, and a tool that mixes them underlines the wrong code. On plain ASCII the bug stays invisible.

The TypeScript compiler keeps two starting points for each node. Its parser, the createSourceFile function in the compiler source, gives every node a pos and an end, and pos is the start of the node including the whitespace and comments in front of it, the leading trivia. The start of the node’s own first token comes from skipping that trivia. Paste /* c */ foo; into an explorer and the statement reports a pos of 0 but a start of 8 once the comment and its space are skipped.

Program analysis and program transformation systems are built on these spans. A linter flags a line because the offending node told it where it lives. A refactoring tool renames a variable by editing only the spans of the matching name nodes. A formatter has to decide which node owns a comment that sits in the trivia between two statements, which is why comments sometimes move when code is reformatted: in the tree they belong to no node at all.

Reading precedence off the tree

Precedence tables are easy to misremember. The tree shows the grouping the parser actually chose. In Python, operators of equal precedence group left to right, except exponentiation and conditional expressions, which group right to left.

Here is a worked example you can reproduce. Parse print(2 ** 3 ** 2) as Python. The argument is a BinOp with the Pow operator spanning columns 6 to 17. Its left child is the constant 2 at columns 6 to 7. Its right child is not a constant but another BinOp, 3 ** 2, at columns 11 to 17. The inner node is evaluated first, giving 9, and 2 to the ninth is 512. A left-grouping reading would have given 8 squared, which is 64. Select the 3 and its parent path reads Module, Expr, Call, BinOp, BinOp, Constant.

The same tree answers the minus sign puzzle. The reference states that the power operator binds more tightly than unary operators on its left and less tightly than unary operators on its right, and gives the example that -1**2 results in -1. In the tree, the UnaryOp with USub sits above the BinOp, so the square is taken first and the sign is applied afterwards.

JavaScript made a different choice. In the ECMAScript 2025 grammar, the left side of ** must be an update expression, which excludes a unary minus, while the right side is again an exponentiation expression. So 2 ** 3 ** 2 also groups to the right, but -2 ** 2 is not a valid expression at all. The TypeScript parser still builds a tree for it, as it does for most broken input, and reports that a unary minus is not allowed on the left of an exponentiation, suggesting parentheses.

Ordinary arithmetic reads the same way. In the TypeScript expression BASE + p.weightKg * km / 100 * rate, the top node is the addition. Its right child is a multiplication whose left child is a division whose left child is another multiplication, because multiplication and division share one level and group to the left. With BASE 4.5, a weight of 2, 350 km and a rate of 2, the subtrees give 700, then 7, then 14, and the sum is 18.5.

How Python turns indentation into structure

Most languages mark blocks with braces, which become brackets in the token stream and vanish into the tree like any other punctuation. Python marks blocks with indentation, so its tokenizer has to turn white space into tokens before the parser can see any structure.

The rule is a stack. Before the first line, a zero is pushed. At the start of each logical line, the indentation is compared with the top of the stack: if it is larger it is pushed and one INDENT token is produced, if it is smaller every larger level is popped and each pop produces a DEDENT token. A smaller indentation that matches no level on the stack is an error. At the end of the file, one DEDENT is produced for every level still open. To the parser, INDENT and DEDENT are simply an opening and a closing bracket.

Measuring the indentation needs a rule for tabs. Python replaces each tab by one to eight spaces so that the column reaches the next multiple of eight, and it refuses, with a TabError, a file that mixes tabs and spaces in a way that makes the meaning depend on the width of a tab. Inside parentheses, square brackets and curly braces, line breaks are joined implicitly, so the indentation of continuation lines never reaches the parser at all.

CPython has parsed this token stream with a parsing expression grammar since Python 3.9, when PEP 617 replaced the older LL(1) parser. A parsing expression grammar uses ordered choice: if the first alternative succeeds, the rest are ignored, which removes ambiguity by definition. Such a grammar maps directly onto a recursive descent parser, a set of mutually recursive procedures with one procedure for each rule. With memoization it becomes a packrat parser, which runs in linear time at the cost of more memory.

Python 3.12 went further. PEP 701 brought f-strings into the grammar, so the expressions inside the braces are tokenized and parsed like any other code, and quotes of the same kind may now be reused inside them.

What a tree cannot tell you

A syntax tree records shape and position, and nothing more. It does not know what a name refers to, what type a value has, or whether a function exists. A Python file with a return outside any function produces a perfectly good tree, because that mistake is caught later by the compiler, not by the parser.

Error recovery is another limit. A parser that keeps going after a mistake, as the TypeScript parser does, has to invent missing nodes of zero width to complete the shape, so the tree of a broken file is a guess about what was meant. And the tree depends on the grammar you ask for: the same text read as JavaScript or as TypeScript, with or without JSX, can produce different nodes, which is why the file kind is part of the question.

Within those limits, the tree is the most honest account of a program there is. Not a guess at what the code means, but the structure the parser built, tied to the text.