Thoughts on Whatnot
A blog about .
Introducing WDTE

This last semester, I took a class on compiler design. Towards the end of the semester, I suddenly realized that something I’d been mildly interested in working on for a while was in fact quite a bit easier than I had originally expected: Writing my own scripting language. I began work on it and had a working prototype in just a week or two. I worked on it pretty much non-stop for a bit, and I had originally intended to write this introduction back a few months ago, but some stuff came up and I was busy and… Either way, here it is.

Why?

But why, you may be thinking, did I want to write a new scripting language? Aren’t there enough of them out there already? An excellent question. In fact, that question is the namesake of the language: Why Does This Exist, or WDTE. In the design of WDTE, I had three overriding goals: It had to be written in Go, it had to be very simple in its implementation, and it had to have a focus primarily on ability to embed it.

Writing WDTE in Go is primarily a complement to the third goal, although it should be quite possible to reimplement it in another language. I wanted it to be embeddable from Go code without the need for Cgo. Not only every type, but every single syntactic structure of the language map directly to Go types. This allows for a number of neat things that I’ll cover when I get to embeddability.

The goal of simplicity refers more to that of the implementation, not the actual language itself. The language is fairly simple, but that wasn’t itself a design goal. For example, the language is designed to be LL(1) parsable simply because an LL(1) parser is extremely easy to write. The entire grammar, as it currently stands, is described in 34 lines of EBNF.

WDTE’s primary goal, above both the others, was to be easy to embed as a scripting language. WDTE ships with not only no command-line interpreter, but it doesn’t even know how to import its own standard library by default. The main part of the implementation is a Go package which provides the ability load WDTE modules and run functions in those modules. Speaking of functions…

How?

While I didn’t really have any particular design goals originally for the actual language itself, I did have a number of vague ideas. WDTE is a function-ish, lazily evaluated language with a syntax extremely loosely based on the limited amount of Haskell that I’ve written.

By functional-is, I mean that in WDTE, everything is a function. This is similar to some OO languages, such as Ruby, in which everything is an object. Numbers are functions, strings are functions, arrays are functions… Even expressions are functions. Every single thing in the language can be passed arguments and called, and each returns another function. Function names are also very loose. Provided a name doesn’t conflict with a keyword, a number, or other scanner token, any combination of any non-whitespace Unicode characters are allowed. For example, the standard library includes a summation function called +. This function takes any number of arguments and returns their sum, or, if given no arguments, it simply returns itself, which allows it to be passed around. So, for example, if I wanted to add 3 and 5, I’d just use + 3 5. And no, there’s no infix notation. Sorry.

As previously stated, a WDTE module is accessed by loading it via the Go package and then calling a function. This is what triggers the lazy evaluation. Each function in WDTE is converted into a tree of Go types, each of which implements a Func interface. When one of these functions is called, the whole tree gets evaluated in a cascade. In other words, nothing is evaluated unless called from Go code. Functions are passed unevaluated expressions as functions, and these must be called in order for them to be evaluated. For example, when the above + function is called, it gets the functions for 3 and 5, but it has no way to know if it was given those or an unevaluated expression which, when evaluated, will return a number, so the first thing it does is evaluate the expressions it’s been given by calling them as functions with no arguments. Conveniently, numbers simply return themselves when called, so everything works after this is done. If this sounds weird, hopefully it’ll make a bit more sense after a few examples.

What does it look like?

Before I start getting into some examples, you should probably know that there is a WDTE playground which allows you to try the language in your browser. The playground was made with GopherJS, meaning that the entire thing, including the WDTE scanner and parser, run on the client-side in your browser. I think that it’s a pretty decent example of WDTE’s embeddability.

Every WDTE module is made up of a chain of semicolon-separated import statements and function declarations. These can be in any order and there can be however many of them as you want.

Important statements have the form

'path' => name;

This is similar to a Go import statement in terms of the pieces that it contains. The string at the beginning, which, by the way, can be either single or double quoted, is the import path, while the identifier at the end, name, in this case, is the identifier to be used to reference the imported module elsewhere in the code. In Go, this second part is often omitted, as Go package source files start with a package statement that sets a default for this when importing, but WDTE has no equivalent for these, so it must be specified every time. The above WDTE import statement would be equivalent to the following Go import statement:

import name "path"

By default, like with everything else, WDTE doesn’t know how to import anything. When a module is loaded by the Go package, it can be provided with an Importer which is used to handle every import statement encountered.

A function declaration is, as you can probably guess, a bit more complicated, so I’ll use several examples to try to illustrate a variety of different features. For a number of these, I will simply assume the existence of certain functions, such as a print function, in order to be able to demonstrate the syntax.

First off, a very simple function that just returns the number 3:

example => 3;

At first glance, this looks a lot like an import statement, but there’s one crucial difference. In a function declaration, the identifier is on the left side of the =>.

Next up, a simple max function:

max v1 v2 => switch v1 {
  >= v2 => v1;
  default => v2;
};

First of all, this function takes two arguments, v1 and v2. This works pretty much how you would expect, but there is one little oddity. If a function is defined as having arguments and is called with less arguments than it takes, it will return a new function that takes the remaining arguments. For example, max 3 will return a function that takes one argument and returns either 3 or whatever argument it’s given if that argument is greater than 3.

Next up, this function contains a switch expression. Duh. switch expressions are WDTE’s only conditional syntactic construct. They take an expression, v1 in this case, and a series of semicolon-separated branches. Each branch looks somewhat similar to a function declaration, but an expression is placed on the left side, rather than an identifier. If the arm is chosen, the right side is returned from the entire switch expression. Each arm is evaluated in turn, and whichever is true first is chosen. The keyword default on the left instead of an expression simply means that that arm will always trigger if reached.

When a switch is being evaluated, the first thing that happens is the conditional expression, which, again, is v1 in this case, is evaluated. Then, the left-hand side of each arm is evaluated. The conditional expression is then passed to the result of that evaluation, and if that evaluation returns a boolean true then that arm is chosen. If not, then the process is repeated with the next arm. This takes advantage of the way that calling functions with less parameters than they were defined with works. For example, >= v2 returns a function that returns true if the argument that that function is given is greater than or equal to v2.

This is great and all, but what if you want to do multiple things in a row?

main => (
  print 'Each';
  print 'of';
  print 'these';
  print 'will';
  print 'get';
  print 'evaluated';
  print 'in';
  print 'turn';
  print '.';
);

A compound is denoted by parentheses. It may contain one or more semicolon-separated expressions and, when evaluated, it evaluates each expression in turn. The result of the final expression in the compound is returned from the whole compound. That’s about it, really. The last semicolon is actually optional, as the scanner will automatically insert it before the final parentheses. Because of this, compounds are also useful for passing ‘compound’ expressions as arguments to other functions. Which leads nicely into…

main => (
  print (max 3 5);
  max 3 5 -> print;
);

Both of the lines in the main function above have the exact same result, but the way in which evaluation works is quite different. The first one simply passes the expression max 3 5 to the print function as its first argument. The second one makes use of one of WDTE’s most unusual features: Chains.

Chains are based loosely on the Unix shell’s well-known pipe system. Each piece of the chain is evaluated and the result of previous parts is passed to the results of later parts. So, for the above example, both the expression max 3 5 and the expression print are evaluated separately, and then the result of max 3 5 is passed to the result of print. Since print returns itself, this results in the expression print 5 being evaluated, essentially.

Anything else?

Yep, there is. WDTE has a number of other features, including automatically memoized functions, but this post is already getting a bit long so I think I’ll end it here. For more information on the internal working, take a look at the Godoc, or you could also read through the code. The standard library modules demonstrate how to implement WDTE-callable functions in Go, for example. If you have any questions, feel free to leave a comment or contact me via one of the methods listed on the About page, or, since I’ll probably be posting this on Reddit, post a comment on the Reddit thread. If you find the project interesting, feel free to submit a bug report, or even a pull request. WDTE is mostly usable for basic things now, but there’s still a lot more to do.

comments powered by Disqus