Introduction

When Reddit complains that a comment is too long, it’s time to write a blog post.

The backstory is that I was having a discussion with someone over a previous post.

That person, username Qweesdy, had a vision of what future programming languages should be like. This person had a few good ideas, like permissions and distributing binaries in a general form, to be specialized by the machine that would use it.

But the rest of Qweesdy’s vision will not work.

And here’s why.

Source Code as AST

The core of Qweesdy’s vision is that source code will be stored in Abstract Syntax Tree (AST) form, in binary files.

This is something a lot of people have wanted, including myself.

It sounds attractive, like Communism. But also like Communism, it does not work in practice.

No Parsing?

Qweesdy claims that there won’t be any parsing when reading in such an AST file.

I beg to differ.

Files are byte streams. To turn a byte stream into an AST, you need to parse. It doesn’t matter if that byte stream is a binary stream or a text stream; you need to parse because there has to be something to convert plain bytes to a tree.

Take Lisp: it’s canonical form is its AST, but its AST is still text that has to be parsed. Even Protobufs need to be parsed.

Text Is the Universal Interface

When describing the Unix Philosophy, Douglas McIlroy said, “Write programs that handle text streams, because that is a universal interface.”

This remains true.

Despite the good design of Protobufs, JSON (a text format) still reigns king for data transfer. Why? Because text is universal.

Despite the attempts to non-text IPC, pipes and text are still on top.

Sometimes, I need to do a whole-project transformation on my code. Maybe I rename a function or something.

Sure, that can be done with an AST-based language if you write a specific program for that language to do that task.

With text? I just write a sed script and run it on all of the files in the project. Done.

It doesn’t matter what language the files are, either; I’ve used sed with code in C, shell, Python, and others at the same time!

You could do that with AST-based languages, but you would probably need more than one tool. And if one tool sufficed, I shudder to think of the complexity of that tool.

Not only that, but text is what allows interoperability.

Interoperability has to do with languages working together. A lot of software is written in multiple languages. How do we get them to interoperate with AST forms?

Without intentional work towards interoperability, languages would come up with their own forms of AST.

“Well, just create a universal AST!”

The problem is that there have been attempts to create a universal AST, and they have all failed.

Well, not all. LLVM succeeded. And it is universal, more or less.

So why don’t languages use it? Well, they do. But the kicker is that they don’t use it for their canonical forms.

There are a couple of reasons for this.

First, LLVM itself needs passes for “canonicalization”; these are passes that simply transform the AST into a more canonical form. In other words, universality means that the AST can have non-canonical forms.

Second, LLVM is too low-level. Just imagine having LLVM as your AST and trying to generate a canonical text version of your language from it. It certainly wouldn’t be a high-level language, or the transformer would be terribly complex.

But LLVM has to be low-level to be universal. This is because all languages compile down to the same stuff, hardware, but every high-level language is different in its own way. The only way to make something common is to work at the level where they have stuff in common. Thus, LLVM is low-level.

And the final nail in the coffin: LLVM also has a text form, almost like its designers knew it would be necessary.

You might argue that LLVM’s text form is LLVM, thus it is not an AST. This is not true. The most common use of LLVM is to use an in-memory AST. Every time you compile C, C++, Swift, Rust, Zig, and others from source to binary, you used the in-memory LLVM without ever going to text.

In fact, the LLVM language reference says:

The LLVM code representation is designed to be used in three different forms: as an in-memory compiler IR, …and as a human readable assembly language representation.

The in-memory version is first-class. It is an AST.

And no one uses it for their languages’ file formats.

Always Bet on Text

Graydon Hoare wrote a post that I would consider a classic of Computer Science.

It’s called “Always Bet on Text.”

He lays out four reasons text is best:

• It’s the most stable.
• It’s the most flexible.
• It’s the most efficient.
• It’s the most socially useful.

Text Is Stable

Graydon wrote:

Text is the oldest and most stable communication technology (assuming we treat speech/signing as natural phenomenon – there are no human societies without it – whereas textual capability has to be transmitted, taught, acquired) and it’s incredibly durable. We can read texts from five thousand years ago, almost the moment they started being produced. It’s (literally) “rock solid” – you can readily inscribe it in granite that will likely outlast the human species.

In a world where we lost all software, but still had our files, we could read text files. We could even compile text files by hand and bootstrap our software again that way.

Even if we lost our files, we could start with text again.

AST-based languages? Not a chance.

But that’s not all. There might be a reason to change an AST form. The compiler for the new AST has to be able to process two different forms of AST. This happens with text too, but not nearly as bad because text can use markers, and its flexibility usually helps with backwards compatibility.

And that’s to say nothing of the other tools that you needed to replace my sed scripts.

For example, Qweesdy talks about a sort of version control for the AST form of a language. What if the form of that AST changed? The VCS would have to track both forms of the AST, for different files, at the same time.

Text Is Flexible

Graydon wrote:

Text is the most flexible communication technology. Pictures may be worth a thousand words, when there’s a picture to match what you’re trying to say. But let’s hit the random button on wikipedia and pick a sentence, see if you can draw a picture to convey it, mm? Here:

Human rights are moral principles or norms that describe certain standards of human behaviour, and are regularly protected as legal rights in national and international law.

Not a chance. Text can convey ideas with a precisely controlled level of ambiguity and precision, implied context and elaborated content, unmatched by anything else. It is not a coincidence that all of literature and poetry, history and philosophy, mathematics, logic, programming and engineering rely on textual encodings for their ideas.

I can’t say it any better.

Text Is Efficient

Graydon wrote:

Text is the most efficient communication technology. By orders of magnitude….At every step of communication technology, textual encoding comes first, everything else after. Because it’s vastly cheaper on a symbol-by-symbol basis. You have a working optical telegraph network running in 1790 in France. You [have] the better part of a century of electrical telegraphy, trans-oceanic cables and everything, before anyone bothers with trying to carry voice. You have decades of teleprinter and text-only computer networking, mail and news, chat and publishing, editing and diagnostics, before bandwidth gets cheap enough for images, voice and video. You have pagers, SMS, WAP, USSD and blackberries before iPhones. You have Teletext and BBSs, netnews and gopher before the web. And today many of the best, and certainly the most efficient parts of the web remain text-centric. I can download all of wikipedia and carry it around on the average smartphone.

Text Is Socially Useful

Graydon wrote:

Text is the most socially useful communication technology. It works well in 1:1, 1:N, and M:N modes. It can be indexed and searched efficiently, even by hand. It can be translated. It can be produced and consumed at variable speeds. It is asynchronous. It can be compared, diffed, clustered, corrected, summarized and filtered algorithmically. It permits multiparty editing. It permits branching conversations, lurking, annotation, quoting, reviewing, summarizing, structured responses, exegesis, even fan fic. The breadth, scale and depth of ways people use text is unmatched by anything. There is no equivalent in any other communication technology for the social, communicative, cognitive and reflective complexity of a library full of books or an internet full of postings. Nothing else comes close.

I just want to add to that.

Every modern programming language, besides esoteric or joke ones, include one common feature: comments.

Not everyone uses them, but when used well, they are indispensible.

How would an AST-based language do comments? They are text after all, but where on the AST would they be attached?

I have software with comments in the middle of nowhere, not attached to anything. Why? Because they didn’t fit anywhere. Where would I put those in an AST-based language?

We could solve that, but it would make the AST even more complicated.

And anyway, comments are text, so why not just use text?

The Dream

Qweesdy’s vision is a common dream may best be articulated by Bret Victor in his seminal talk The Future of Programming.

Bret talks about contrast four pairs of things:

• Coding vs direct manipulation of data.
• Procedures vs goals and constraints.
• Text dump vs spatial representations.
• Sequential vs concurrent.

I want to focus on the third pair.

He talks about programmers drawing boxes. That takes too much time. Just write text.

He talks about programmers writing things by hand. I can type faster.

He talks about Smalltalk’s browser. Smalltalk failed in the marketplace of ideas.

In theory, it was great. In practice, people didn’t like it. Software is meant to serve people, so it was a failure.

In Inventing on Principle, Bret Victor goes even further.

His principle is that creators need an immediate connection with what they create:

When you’re making something, if you make a change or make a decision, you need to see the effect of that immediately.

I agree. And he shows many wonderful demos of how immediate feedback lets him think thoughts he could not have otherwise had.

Yet…

Smalltalk existed, Bret Victor’s demos exist, but text is still king.

Text is just too useful, too flexible, too communicative and universal to not use. Even Bret Victor in his demonstrations uses text as the base! He just shows tools he built to see the immediate results of it.

Have you ever seen level editors for games? Or Blender’s ability to do a full render in real-time?

The tools exist.

And we are programmers; we can give ourselves the tools. We can build visualizations. We can add all sorts of manipulation controls. We can add a way for IDE’s (or the Language Server Protocol) to compile and run programs on every keypress.

We just have to do it.

Game studios know this and give their people the tools.

Bret Victor’s vision is good, but it depends on people choosing that path and making the tools for themselves or others. You can’t build a general tool to visualize everything.

Conclusion

But regardless, text is still the base.

No-code and like ideas will exist, of course, but they will never replace text.

Text is king. And it will always be.