Nice. That latency is tough though. On that topic, I wrote an equivalent "magic mirror" type tool that lets you write to the mirror/oracle/VLM on the remarkable and get an answer back.
A few possible solutions I explored:
1) You can try and consume (and possibly write to) the frame buffer directly. https://github.com/ddvk/remarkable2-framebuffer was my starting point. This gets you instant updates about what's going on. I guess you could pair this with speculative decoding to get a much faster output.
2) You can use the streaming API on the device to stream the screen to a beefy server, possibly over Tailscale, letting you do everything off device.
3) You can write your own Qt app; ddvk's repos are a good starting point here.
I ultimately instructed claude to write me my own app. Which worked enough to scratch my itch, and I never use it. But this was five months ago, an eternity in vibe hobby projects, so perhaps modern tooling would let me get it in shape and be more usable. Basically, it worked, but VLMs weren't great at what I wanted -- decorating a blank page with a grimoire-style answer to a written question while leaving the original text alone -- and getting it started / stopped from the pen UI is difficult.
Where my mind goes for your project is that I think it'd be nicest to keep a sort of Jupyter notebook somewhere that's the canonical representation, that would have your handwritten blocks and an interpretation, and then the output. then a render layer to get it back onto the screen. At that point, I don't think I'd care very much if it's stored as a PDF on the device, which points back to having this be an app.
Either way it's fun to tinker! the RM line is very hackable, and I still wish it were even easier, the hardware just makes you think of so many possibilities.
OP here. See [0] – for this post, I’ve moved from PNGs to SVGs and found an easier way to determine link coordinates, but otherwise the process is mostly the same.
A fun experiment, and cute if not for the lag. A full 14 seconds between the pen stopping writing and the result,
What I'd find interesting is the trace of that 14 seconds. How much is the Remarkable processing, how much is the claude transcription, how much is the let-go start-up / processing, etc.
I haven't used the latest reMarkable devices, but the first/older ones takes multiple seconds to save notes to disk. I remember having similar issues for my own file syncing daemon I tried to make for my reMarkable, where it took many seconds before the filesystem event actually was fired away, it might still be the same issue today I suppose.
I find getting stuff up on the rM2 screen a hassle.
The challenge comes from the way the e-paper works. To turn a pixel from white to black (or vice versa) it needs multiple actual frames. The pixel data must also be packed in a specific format. Instructions for how many frames a single operation requires are coded in a wbf[0] file, which comes included with the OEM firmware.
The most commonly used approach is hooking into xochitl, since it handles all of the user facing stuff like the notebooks but also the actual drawing. This is somewhat brittle and tends to break with software updates, because all of the actual function addresses have to be updated as well.
I was excited to find waved[1], a C++ library that allowed to drive the display directly using a sane API. Although it's not been updated for quite a while it still works and you can compile and run it yourself.
Since I was interested in driving the display myself, I tried to rewriting waved in Zig. It works - I can now get pixels up on the screen. Unfortunately the code is a mess and its only redeeming quality (stemming from being written in Zig) is that I can cross compile a statically linked binary that 'just werks' using just the Zig toolchain as the only dependency. For debugging purposes, I also implemented a SDL emulator for the display. [2]
While writing this, I stumbled upon a recent Rust implementation of the same ideas. Nice. [3]
Since the xochitl display implementation is the most optimized, that's probably the reason why it's being used most commonly*, even though it might be 'ugly'.
Don't. The USB C port is poorly designed and will break at the drop of the hat, leaving you unable to charge the thing. And the company won't do anything to help you.
Nice. That latency is tough though. On that topic, I wrote an equivalent "magic mirror" type tool that lets you write to the mirror/oracle/VLM on the remarkable and get an answer back.
A few possible solutions I explored:
1) You can try and consume (and possibly write to) the frame buffer directly. https://github.com/ddvk/remarkable2-framebuffer was my starting point. This gets you instant updates about what's going on. I guess you could pair this with speculative decoding to get a much faster output.
2) You can use the streaming API on the device to stream the screen to a beefy server, possibly over Tailscale, letting you do everything off device.
3) You can write your own Qt app; ddvk's repos are a good starting point here.
I ultimately instructed claude to write me my own app. Which worked enough to scratch my itch, and I never use it. But this was five months ago, an eternity in vibe hobby projects, so perhaps modern tooling would let me get it in shape and be more usable. Basically, it worked, but VLMs weren't great at what I wanted -- decorating a blank page with a grimoire-style answer to a written question while leaving the original text alone -- and getting it started / stopped from the pen UI is difficult.
Where my mind goes for your project is that I think it'd be nicest to keep a sort of Jupyter notebook somewhere that's the canonical representation, that would have your handwritten blocks and an interpretation, and then the output. then a render layer to get it back onto the screen. At that point, I don't think I'd care very much if it's stored as a PDF on the device, which points back to having this be an app.
Either way it's fun to tinker! the RM line is very hackable, and I still wish it were even easier, the hardware just makes you think of so many possibilities.
This awesome! I actually had no idea I could ssh into my Remarkable and do neat stuff like this!
> Why do it? It's so impractical!
Because you can and it's fun is always a perfectly valid answer here!
I’m actually impressed by the handwritten blog - really cool concept.
Was it exported by writing on remarkable? How did he include link into the text that he wrote?
OP here. See [0] – for this post, I’ve moved from PNGs to SVGs and found an easier way to determine link coordinates, but otherwise the process is mostly the same.
[0]: https://handwritten.danieljanus.pl/2022-10-01-hyperlinks-in-...
Really clever!
A fun experiment, and cute if not for the lag. A full 14 seconds between the pen stopping writing and the result,
What I'd find interesting is the trace of that 14 seconds. How much is the Remarkable processing, how much is the claude transcription, how much is the let-go start-up / processing, etc.
I haven't used the latest reMarkable devices, but the first/older ones takes multiple seconds to save notes to disk. I remember having similar issues for my own file syncing daemon I tried to make for my reMarkable, where it took many seconds before the filesystem event actually was fired away, it might still be the same issue today I suppose.
The author mentions that in the blog. The majority is the delay of the updated file actually being written to disk by the device.
The author did not find a solution to trigger file write earlier/more frequently.
Great work.
The experience programming with handwritten code in tablets keeps being a pain, when it would be ideal as digital executable paper.
I wonder if this would be easier on the Supernote Manta with the new plugins API that they've introduced recently…
I find getting stuff up on the rM2 screen a hassle.
The challenge comes from the way the e-paper works. To turn a pixel from white to black (or vice versa) it needs multiple actual frames. The pixel data must also be packed in a specific format. Instructions for how many frames a single operation requires are coded in a wbf[0] file, which comes included with the OEM firmware.
The most commonly used approach is hooking into xochitl, since it handles all of the user facing stuff like the notebooks but also the actual drawing. This is somewhat brittle and tends to break with software updates, because all of the actual function addresses have to be updated as well.
I was excited to find waved[1], a C++ library that allowed to drive the display directly using a sane API. Although it's not been updated for quite a while it still works and you can compile and run it yourself.
Since I was interested in driving the display myself, I tried to rewriting waved in Zig. It works - I can now get pixels up on the screen. Unfortunately the code is a mess and its only redeeming quality (stemming from being written in Zig) is that I can cross compile a statically linked binary that 'just werks' using just the Zig toolchain as the only dependency. For debugging purposes, I also implemented a SDL emulator for the display. [2]
While writing this, I stumbled upon a recent Rust implementation of the same ideas. Nice. [3]
Since the xochitl display implementation is the most optimized, that's probably the reason why it's being used most commonly*, even though it might be 'ugly'.
* Citation needed
[0] https://gitlab.com/zephray/glider#understanding-waveform
[1] https://github.com/matteodelabre/waved
[2] https://github.com/jakubvf/dazed
[3] https://github.com/yobert/swtcon
How do you handle latency between handwriting recognition and REPL evaluation is there a noticeable delay before you see the output?
Did foo(42) just output 44 ?
(+ x 2)
Now I have a reason to get reMarkable :)
Don't. The USB C port is poorly designed and will break at the drop of the hat, leaving you unable to charge the thing. And the company won't do anything to help you.
Very nice
impressive
y tho