Very cool! I have been working on scripting in Rust recently on a Lua project [1].
When you made Roto what kind of workloads were you optimizing for? How are you guys benchmarking performance?
I ran a quick benchmark based on my recent work (Used AI for the code here):
```
fn sum_scalar(n: u64) -> u64 {
let total = 0;
let i = 0;
while i < n { total = total + i; i = i + 1; }
total
}
fn sum_list(xs: List[u64]) -> u64 {
let total = 0;
for x in xs { total = total + x; }
total
}
```
Rust benchmark.rs
```
use std::time::Instant;
use roto::{List, Runtime};
fn main() {
let rt = Runtime::new();
let mut pkg = rt.compile("bench.roto").unwrap();
let sum_list = pkg.get_function::<fn(List<u64>) -> u64>("sum_list").unwrap();
let n = 1024;
let iters = 50_000;
let xs: List<u64> = (0..n).collect();
let t = Instant::now();
for _ in 0..iters { sum_scalar.call(n); } // adds 0..n with a counter
let scalar = t.elapsed();
let t = Instant::now();
for _ in 0..iters { sum_list.call(xs.clone()); } // adds the SAME 0..n from a List
let list = t.elapsed();
println!("sum_scalar (counter): {scalar:?}");
println!("sum_list (List[u64]): {list:?}");
println!("-> {:.0}x slower", list.as_secs_f64() / scalar.as_secs_f64());
}
I'm happy to cut a PR against your repo with some of the benchmarks I run on every commit in my own language projects if that would be helpful!
[1]. https://github.com/ianm199/lua-rs/tree/main
> When you made Roto what kind of workloads were you optimizing for?
We're building a BGP collector with custom filters written in Roto. Imagine a database that constantly receives updates and we want to filter (or transform) those messages based on a script.
> How are you guys benchmarking performance?
Actually, we haven't done that much as feature work has been more important than optimization. There's a lot of opportunities for optimization left on the table.
There are a few benchmarks that we have done:
- A very naive fibonacci computation, where we were faster than Lua,
- There's this benchmark with a lot of string manipulation made by somebody else where we roughly match Lua: https://github.com/khvzak/script-bench-rs
- There's the testing done with Iocaine, where Roto is apparently much faster than Lua. The scripts there do a lot of inspection of fairly simple types.
So the nuanced take is that Roto is fast with numbers and other cases which don't involve complex data structures that some other languages have really optimized for.
> I'm happy to cut a PR against your repo with some of the benchmarks I run on every commit in my own language projects if that would be helpful!
That would be very helpful! A proper benchmark suite is long overdue. (but do note that we don't accept AI contributions)
I think the list is so much slower it's calling out to Rust a lot to get items from the list. Lists currently also have a mutex inside, which would need to be locked for each access.
The syntax is of course attractive (coming from Rust), and I'd love to replace more of my posix scripts with something saner. I struggle understanding whether the utility of having language literals for IP addresses, IP prefixes, and AS numbers is worth it though [0]. It seems like the confusion added by having custom built-ins like this for one particular domain, in addition to the unclear scoping (what could later also deserve being a language literal), combined with special-case errors as famous in e.g. the YAML Norway problem, makes it seem like such features are better left as some general extension / macro / library capability.
Nix is a language with built-in support for URI literals typed as strings [1], which is a source of confusion and edge-cases, and I believe the feature is now discouraged in general use.
Hi! Author here. We are actually planning on removing those literals and allowing applications to extend Roto with their own literals [0]. They should do so with care of course, because indeed adding more literals adds some edge cases. Most applications should be able to get by without any special literals though.
Does anyone know if the Roto runtime is serde-able?
A big problem I encountered in using Lua in Rust for my game engine was that I wasn't able to serde the Lua runtime such that I can snapshot a game session and save it in a file, and retrieve it in another context.
Hi! Author here. What we call the `Runtime` in Roto is not a state of the program, it's only the set of functions, types and constants that are available to the script. Roto scripts cannot really keep state at the moment. The advantage of that is that it allows you to run scripts in parallel. We're thinking about how we can keep that property while also having some state.
Very cool! I have been working on scripting in Rust recently on a Lua project [1].
When you made Roto what kind of workloads were you optimizing for? How are you guys benchmarking performance?
I ran a quick benchmark based on my recent work (Used AI for the code here): ``` fn sum_scalar(n: u64) -> u64 { let total = 0; let i = 0; while i < n { total = total + i; i = i + 1; } total }
```Rust benchmark.rs ```
``` I'm happy to cut a PR against your repo with some of the benchmarks I run on every commit in my own language projects if that would be helpful! [1]. https://github.com/ianm199/lua-rs/tree/mainThanks!
> When you made Roto what kind of workloads were you optimizing for?
We're building a BGP collector with custom filters written in Roto. Imagine a database that constantly receives updates and we want to filter (or transform) those messages based on a script.
> How are you guys benchmarking performance?
Actually, we haven't done that much as feature work has been more important than optimization. There's a lot of opportunities for optimization left on the table.
There are a few benchmarks that we have done: - A very naive fibonacci computation, where we were faster than Lua, - There's this benchmark with a lot of string manipulation made by somebody else where we roughly match Lua: https://github.com/khvzak/script-bench-rs - There's the testing done with Iocaine, where Roto is apparently much faster than Lua. The scripts there do a lot of inspection of fairly simple types.
So the nuanced take is that Roto is fast with numbers and other cases which don't involve complex data structures that some other languages have really optimized for.
> I'm happy to cut a PR against your repo with some of the benchmarks I run on every commit in my own language projects if that would be helpful!
That would be very helpful! A proper benchmark suite is long overdue. (but do note that we don't accept AI contributions)
> sum_scalar (counter): 28.56ms > sum_list (List[u64]): 590.48ms > -> 21x slower
I think the list is so much slower it's calling out to Rust a lot to get items from the list. Lists currently also have a mutex inside, which would need to be locked for each access.
The syntax is of course attractive (coming from Rust), and I'd love to replace more of my posix scripts with something saner. I struggle understanding whether the utility of having language literals for IP addresses, IP prefixes, and AS numbers is worth it though [0]. It seems like the confusion added by having custom built-ins like this for one particular domain, in addition to the unclear scoping (what could later also deserve being a language literal), combined with special-case errors as famous in e.g. the YAML Norway problem, makes it seem like such features are better left as some general extension / macro / library capability.
Nix is a language with built-in support for URI literals typed as strings [1], which is a source of confusion and edge-cases, and I believe the feature is now discouraged in general use.
[0] https://roto.docs.nlnetlabs.nl/en/stable/reference/language_...
[1] https://nix.dev/manual/nix/2.34/language/string-literals
Hi! Author here. We are actually planning on removing those literals and allowing applications to extend Roto with their own literals [0]. They should do so with care of course, because indeed adding more literals adds some edge cases. Most applications should be able to get by without any special literals though.
[0]: https://codeberg.org/NLnetLabs/roto/pulls/358
Nice! That sounds like a good change. I'll try to dive a bit deeper through docs once I find some time :)
Does anyone know if the Roto runtime is serde-able?
A big problem I encountered in using Lua in Rust for my game engine was that I wasn't able to serde the Lua runtime such that I can snapshot a game session and save it in a file, and retrieve it in another context.
Hi! Author here. What we call the `Runtime` in Roto is not a state of the program, it's only the set of functions, types and constants that are available to the script. Roto scripts cannot really keep state at the moment. The advantage of that is that it allows you to run scripts in parallel. We're thinking about how we can keep that property while also having some state.