Building a shell is a great exercise, but honestly having to deal with string parsing is such a bother that it robs like 2/3 of the joy along the way. I once built a very simple one in Go [0] as a learning exercise and I stopped once I started getting frustrated with all the corner cases.
Author here, and yeah, I agree. I skipped writing a parser altogether and just split on whitespace and `|` so that I could get to the interesting bits.
For side-projects, I have to ask myself if I'm writing a parser, or if I'm building something else; e.g. for a toy programming language, it's way more fun to start with an AST and play around, and come back to the parser if you really fall in love with it.
The pipe implementation section is really clean. Working through fork/exec/dup2 by hand like this is one of those exercises that makes you appreciate how much composability Unix got right. Processes that know nothing about each other just work together because they read stdin and write stdout. I built something similar years ago and the moment pipes actually worked felt like unlocking a cheat code.
Very cool. Currently working on the beginning of a small text editor so this part seemed interesting and was curious of any overlap. Thanks for the interesting post!
Bit of pedantry but I don't think traditional unix shell (like this) follows repl model; the shell is not usually doing printing of the result of evaluation. Instead the printing happens more as a side effect of the commands.
I remember my first shell programming I ever did was batch in windows back in the 3.11/95 days.
The first line was always to turn off echo, and I've always wondered why that was a decision for batch script. Or I'm misremembering. 30 years of separation makes it hard to remember the details.
The corner cases are exactly where the learning is though. Every time you hit one unquoted spaces, escaped characters, subshell expansion — you understand something deeper about how the OS actually works. The frustration is the point. Most developers never have to think about what the shell is actually doing between their keystrokes and the kernel.
Somebody blamed this comment on LLMs, and maybe/probably it is, but I think the first sentence is spot-on so I thought it was worth replying to.
Dealing with the corner cases ends up teaching you a lot about a language and for an ancient language like the shell, dealing with the corner cases also takes you through the thinking process of the original authors and the constraints they were subject to. I found myself in this situation while writing EndBASIC and wrote an article with the surprises I encountered, because I found the journey fascinating: https://www.endbasic.dev/2023/01/endbasic-parsing-difficulti...
Not sure it tells all that much about 'how the OS works'. This is a historical abstraction that happened to look how it looks today with all its numerous warts and shortcomings.
We can easily imagine it done a better way - for all the criticism of Windows, PowerShell gives a glimpse into this hypothetical future.
Some time ago I've written an article about a particular aspect of shells, job control: https://emersion.fr/blog/2019/job-control/
Building a shell is a great exercise, but honestly having to deal with string parsing is such a bother that it robs like 2/3 of the joy along the way. I once built a very simple one in Go [0] as a learning exercise and I stopped once I started getting frustrated with all the corner cases.
[0] https://github.com/lourencovales/codecrafters/blob/master/sh...
Author here, and yeah, I agree. I skipped writing a parser altogether and just split on whitespace and `|` so that I could get to the interesting bits.
For side-projects, I have to ask myself if I'm writing a parser, or if I'm building something else; e.g. for a toy programming language, it's way more fun to start with an AST and play around, and come back to the parser if you really fall in love with it.
Had an assignment to build a shell in a week, how hard could it be?
The parser was easy in comparison.The pipe implementation section is really clean. Working through fork/exec/dup2 by hand like this is one of those exercises that makes you appreciate how much composability Unix got right. Processes that know nothing about each other just work together because they read stdin and write stdout. I built something similar years ago and the moment pipes actually worked felt like unlocking a cheat code.
Fun read. Wonder if you are able to edit text in the shell, or if you need to implement a gap buffer to allow it?
Editing the current line works because I brought in https://man7.org/linux/man-pages/man3/readline.3.html towards the end so I could support editing, tab completion, and history.
IIRC readline uses a `char *` internally since the length of a user-edited line is fairly bounded.
Very cool. Currently working on the beginning of a small text editor so this part seemed interesting and was curious of any overlap. Thanks for the interesting post!
Bit of pedantry but I don't think traditional unix shell (like this) follows repl model; the shell is not usually doing printing of the result of evaluation. Instead the printing happens more as a side effect of the commands.
I remember my first shell programming I ever did was batch in windows back in the 3.11/95 days.
The first line was always to turn off echo, and I've always wondered why that was a decision for batch script. Or I'm misremembering. 30 years of separation makes it hard to remember the details.
Echo in that case prints command lines before executing them. Its analog is `set -x` rather than `echo`.
It’s a shell, not the whole thing. The whole thing is the shell+kernel+programs.
The corner cases are exactly where the learning is though. Every time you hit one unquoted spaces, escaped characters, subshell expansion — you understand something deeper about how the OS actually works. The frustration is the point. Most developers never have to think about what the shell is actually doing between their keystrokes and the kernel.
Somebody blamed this comment on LLMs, and maybe/probably it is, but I think the first sentence is spot-on so I thought it was worth replying to.
Dealing with the corner cases ends up teaching you a lot about a language and for an ancient language like the shell, dealing with the corner cases also takes you through the thinking process of the original authors and the constraints they were subject to. I found myself in this situation while writing EndBASIC and wrote an article with the surprises I encountered, because I found the journey fascinating: https://www.endbasic.dev/2023/01/endbasic-parsing-difficulti...
Not sure it tells all that much about 'how the OS works'. This is a historical abstraction that happened to look how it looks today with all its numerous warts and shortcomings.
We can easily imagine it done a better way - for all the criticism of Windows, PowerShell gives a glimpse into this hypothetical future.
Fascinating that you resurrected an account from 2014 just for LLM spam, were the credentials compromised or something?
Maybe the author had it logged into something that their claw had access to