Forth Inside Out part 2

In which we internalise the read-execute loop, or die in the attempt.

In Forth: Inside Out I did some refactoring on the read-execute loop. In this part I and going to re-engineer it entirely and implement it in Forth. The reason for doing this is that it will play better with some of the core words that I haven’t yet implemented. These include abort and quit. Also, it will make Forth style exception handling work better.

The refactoring I have done so far is essential for this part since we now have a model where the compiler expects to be able to consume input “inline” rather than in an external event driven manner.

At a high level, the loop needs to do this:

while there is input
  read a line into the input buffer
  if we got a line
    while there are words in the input buffer
      *read a word* and find its execution token
      if we are interpreting
        execute the word
      else
        compile the word

I would structure it slightly differently though:

attempt to read a line into the input buffer
while last read was successful
  
  while there are words in the input buffer
    *read a word* and find its execution token
    if we are interpreting
      execute the word
    else
      compile the word
    attempt to read a line into the input buffer

Some of the work that’s needed above already exists. For example, the asterisked “read a word” operation already has a defined implementation in Forth: word. The “attempt to read a line into the input buffer” operation is the Forth refill word, which I have not yet implemented (or had not when I wrote this paragraph).

Implementing Refill

refill refills the input buffer from the current input source. At the start of this blog post, there was no concept of a current input source. We just passed a sequence of characters to interpret(input:, output:). In the given function, input is just a generic sequence of characters. We need a type erased version of that sequence as an instance variable. Also, we need to consume the sequence, which precludes using AnySequence because we don’t want to be constantly rereading the same line if, for example, a String is used as the input source. The choice is between using AnyIterator with perhaps an extension method to consume the iterator in chunks of one line, or create a new type that wraps the iterator and provides various methods related to input sources. I chose the latter option because it gives me the option to associate diagnostic information with the input source e.g. a file name and line and character numbers.

NB: I started the development of refill on the inside-out branch, but I realised, at this point that I’ve finished that bit. So I created a new branch called refill for the rest of the development of refill.

The implementation of refill is tagged blog-1849-1