Skip to main content

Generators

A generator is a function that produces a sequence lazily, suspending between values instead of computing them all up front. Verum has two flavours:

  • fn* — sync generator, returns Iterator<T>.
  • async fn* — async generator, returns Stream<T> (consume with for await).

For the coinductive (observation-oriented) counterpart, see language/copatterns. Generators are imperative — you drive them with yield; cofix functions are observational — consumers drive them by asking for .head / .tail.

Sync generator — fn*​

fn* fibonacci() -> Int {
    let (mut a, mut b) = (0, 1);
    loop {
        yield a;
        (a, b) = (b, a + b);
    }
}

for n in fibonacci().take(10) {
    print(f"{n} ");
}
// 0 1 1 2 3 5 8 13 21 34

fn* name() -> T declares a generator. yield value produces the next item and suspends; execution resumes on the next call to the iterator's .next().

With state​

fn* parser_tokens(src: &Text) -> Token {
    let mut i = 0;
    while i < src.len() {
        let tok = lex_one_token(src, &mut i);
        yield tok;
    }
}

Internally, the compiler compiles the generator to a state machine — each yield is a suspension point. The generator's local variables become fields of the state.

Parameterised​

fn* take_from(n: Int, iter: impl Iterator<Int>) -> Int {
    let mut i = 0;
    for x in iter {
        if i >= n { return; }
        yield x;
        i += 1;
    }
}

Generators can take arguments and interact with other iterators.

Consuming sync generators​

Drive one with for, or pull values with next() until it is spent:

fn* three() -> Int { yield 1; yield 2; yield 3; }

for n in three() {
    print(f"{n} ");            // 1 2 3
}

let mut g = three();
while let Maybe.Some(n) = g.next() {
    print(f"{n} ");            // 1 2 3
}

Generators are iterators, so the adapter chain applies as well — here on a generator expression, materialised with collect:

let ten: List<Int> = gen{ x * 2 for x in 0..1_000_000 }
    .take(10)
    .collect();
// [0, 2, 4, 6, 8, 10, 12, 14, 16, 18]

Bound an unbounded sequence at the source — a while with an exit condition in the generator body, or a bounded range in a gen{…} — rather than relying on a downstream adapter to stop it.

The entire Iterator protocol applies — .map, .filter, .flat_map, .zip, .chain, .cycle, .step_by, …. See stdlib/base.

Async generator — async fn*​

For sequences involving suspension (network, files, streams):

async fn* lines_from(path: &Path) -> Result<Text, IoError>
    using [FileSystem]
{
    // `AsyncFile.open` takes the path as `&Text`, not `&Path`.
    let file = AsyncFile.open(&path.to_text()).await?;
    let mut reader = BufReader.new(file);

    loop {
        // `next_line_async` answers `IoResult<Maybe<Text>>` — EOF is
        // `Maybe.None`, not a zero byte count. There is no
        // `read_line_async`.
        match reader.next_line_async().await {
            Result.Ok(Maybe.None)       => return,                  // EOF
            Result.Ok(Maybe.Some(line)) => yield Result.Ok(line.trim_end().to_text()),
            Result.Err(e)               => yield Result.Err(e),
        }
    }
}

Consume with for await​

async fn print_file(path: &Path) using [FileSystem, IO] {
    for await result in lines_from(path) {
        match result {
            Result.Ok(line) => print(line),
            Result.Err(e)   => { eprint(f"{e:?}"); break; }
        }
    }
}

for await is the async analogue of for. Each iteration awaits the next item, so the consumer and producer take turns cooperatively.

Streams are combinator-friendly​

An async fn* returns Stream<T>; the Stream protocol has:

adapteradapteradapter
.map(|x| f(x)).filter(|x| pred(x)).take(n)
.skip(n).chunks(size).window(size)
.throttle(duration).timeout(duration).flatten()
.flat_map(|x| stream_expr).buffer(max_in_flight).buffer_unordered(max_in_flight)
.merge(other).zip(other).scan(init, |acc, x| update)
.fold_async(init, |acc, x| future)

Example: concurrent HTTP fetches with bounded parallelism:

async fn fetch_all(urls: &List<Url>) -> List<Bytes>
    using [Http]
{
    stream.iter(urls.iter())
        .map(|u| Http.get(u.clone()))
        .buffer_unordered(16)                   // 16 concurrent, order undefined
        .filter_map(|r| async { r.ok() })
        .map(|resp| resp.body().await.unwrap())
        .collect<List<_>>()
        .await
}

Closing a generator​

Generators that hold file handles / sockets / channel receivers release them in their Drop impl. Explicit close() is rarely needed but is available:

let mut gen = fibonacci();
print(f"{gen.next().unwrap()}");       // 0
print(f"{gen.next().unwrap()}");       // 1
gen.close();                             // release captured resources

Dropping gen without close() has the same effect, unless you want deterministic cleanup at a specific point.

Error propagation from generators​

An async fn* that yields Result<T, E> is common. A helper:

async fn* try_lines(path: &Path) throws(IoError) -> Text
    using [FileSystem]
{
    let file = AsyncFile.open(&path.to_text()).await?;
    let mut reader = BufReader.new(file);
    loop {
        match reader.next_line_async().await? {
            Maybe.None       => return,                 // EOF
            Maybe.Some(line) => yield line.trim_end().to_text(),
        }
    }
}

With throws(E) on an async fn*, the generator yields T values; on error, propagation short-circuits the for await loop:

for await line in try_lines(path) {
    process(line);
}
// If the generator threw, the `for await` returns the error.

Equivalent to a generator that yields Result<T, E>, but more ergonomic.

Backpressure​

Async generators suspend on .await points. A consumer that pulls slowly throttles a producer that produces fast — built-in backpressure. No channel needed.

async fn* source() -> Int { loop { yield fetch().await; } }

async fn consumer() {
    for await n in source() {
        slow_process(n).await;                  // source waits for us
    }
}

Generator expression — gen { ... for ... in ... }​

For one-shot lazy sequences, use the generator expression form — no need to define a function:

let nums = gen{ x * 2 for x in 0..1_000_000 };
let first_ten: List<Int> = nums.take(10).collect();
// [0, 2, 4, 6, 8, 10, 12, 14, 16, 18]

See language/comprehensions — gen is one of five comprehension forms.

Gotchas​

yield outside a fn* body​

fn main() {
    yield 42;     // COMPILE ERROR: yield only valid in fn* / async fn*
}

return value; in a generator​

Generators cannot return a value — only a bare return to end the sequence. If you need a final value, yield it and then return.

fn* until_zero(xs: &List<Int>) -> Int {
    for &x in xs {
        if x == 0 { yield 0; return; }
        yield x;
    }
}

Infinite generators on a .collect()​

fibonacci().collect<List<Int>>() runs forever. Add a .take(n) before .collect() if you want a finite prefix.

Cancellation only at .await points​

Async generators honour cancellation only between .awaits. A long synchronous block inside an async fn* delays cancellation. Insert yield_now().await to add cancellation checkpoints.

Compile-time vs runtime — generators are VBC​

Generators compile to a state machine in VBC. The generator's captured locals become fields; yield becomes a return-plus-resume. The resulting code is fast — the overhead is roughly the same as a hand-written iterator record.

See also​