I find it easier to understand in terms of the Unix syscall API. `2>&1` literally translates as `dup2(1, 2)`, and indeed that's exactly how it works. In the classic unix shells that's all that happens; in more modern shells there may be some additional internal bookkeeping to remember state. Understanding it as dup2 means it's easier to understand how successive redirections work, though you also have to know that redirection operators are executed left-to-right, and traditionally each operator was executed immediately as it was parsed, left-to-right. The pipe operator works similarly, though it's a combination of fork and dup'ing, with the command being forked off from the shell as a child before processing the remainder of the line.
Though, understanding it this way makes the direction of the angled bracket a little odd; at least for me it's more natural to understand dup2(2, 1) as 2<1, as in make fd 2 a duplicate of fd 1, but in terms of abstract I/O semantics that would be misleading.
Another fun consequence of this is that you can initialize otherwise-unset file descriptors this way:
$ cat foo.sh
#!/usr/bin/env bash
>&1 echo "will print on stdout"
>&2 echo "will print on stderr"
>&3 echo "will print on fd 3"
$ ./foo.sh 3>&1 1>/dev/null 2>/dev/null
will print on fd 3
It's a trick you can use if you've got a super chatty script or set of scripts, you want to silence or slurp up all of their output, but you still want to allow some mechanism for printing directly to the terminal.
The danger is that if you don't open it before running the script, you'll get an error:
$ ./foo.sh
will print on stdout
will print on stderr
./foo.sh: line 5: 3: Bad file descriptor
And just like dup2 allows you to duplicate into a brand new file descriptor, shells also allow you to specify bigger numbers so you aren’t restricted to 1 and 2. This can be useful for things like communication between different parts of the same shell script.
> At least allow us to use names instead of numbers.
You can for the destination. That's the whole reason you need the "&": to tell the shell the destination is not a named file (which itself could be a pipe or socket). And by default you don't need to specify the source fd at all. The intent is that stdout is piped along but stderr goes directly to your tty. That's one reason they are separate.
And for those saying "<" would have been better: that is used to read from the RHS and feed it as input to the LHS so it was taken.
This is misleading because you use plural for both and I'm sure most of these UX missteps were _each_ made by a _single_ person, and there were >1 users even at the time.
Yeah but now they're using npm to install a million packages to do things like tell if a number is greater than 10000. The chances of the programmer wanting to understand the underlying system they are using is essentially nil.
You're not wrong, but there's fairly easy ways to deal with filenames containing spaces - usually just enclosing any variable use within double quotes will be sufficient. It's tricker to deal with filenames that contain things such as line breaks as that usually involves using null terminated filenames (null being the only character that is not allowed in filenames). e.g find . -type f -print0
Nushell, Powershell, Python, Ruby, heck even Perl is better. Shell scripting is literally the worst language I've ever seen in common use. Any realistic alternative is going to be better.
Trying to be language agnostic: it should be as self-explanatory as possible. 2>&1 is all but.
Why is there a 2 on the left, when the numbers are usually on the right. What's the relationship between 2 and 1? Is the 2 for std err? Is that `&` to mean "reference"? The fact you only grok it if you know POSIX sys calls means it's far from self explanatory. And given the proportion of people that know POSIX sys calls among those that use Bash, I think it's a bit of an elitist syntax.
Is it more sane, or is it just what you are used to?
Python doesn't really have much that makes it a sensible choice for scripting.
Its got some basic data structures and a std-lib, but it comes at a non-trivial performance cost, a massive barrier to getting out of the single thread, and non-trivial overhead when managing downstream processes. It doesn't protect you from any runtime errors (no types, no compile checks). And I wouldn't call python in practice particularly portable...
Laughably, NodeJS is genuinely a better choice - while you don't get multithreading easily, at least you aren't trivially blocked on IO. NodeJS also has pretty great compatibility for portability; and can be easily compiled/transformed to get your types and compile checks if you want. I'd still rather avoid managing downstream processes with it - but at least you know your JSON parsing and manipulation is trivial.
Go is my goto when I'm reaching for more; but (ba)sh is king. You're scripting on the shell because you're mainly gluing other processes together, and this is what (ba)sh is designed to do. There is a learning curve, and there are footguns.
Humans used this combination extensively for decades too. I'm no aware of any other simple way to grep both stdout and stderr from a process. (grep, or save to file, or pipe in any other way).
"not humans" are using this extensively precisely because humans used this combination extensively for decades. It's muscle-memory for me. And so is it for LLMs.
I found the explanation useful, about "why" it is that way. I didn't realize the & before the 1 means to tell it is the filedescriptor 1 and not a file named 1.
Well sure, but surely this takes some inspiration from both `&` as the "address of" operator in C as well as the `>` operator which (apart from being the greater-than operator) very much implies "into" in many circumstances.
So `>&1` is "into the file descriptor pointed to by 1", and at the time any reasonable programmer would have known that fd 1 == STDOUT.
I've also found llms seem to love it when calling out to tools, I suppose for them having stderr interspersed messaged in their input doesn't make much difference
I know the underlying call, but I always see the redirect symbols as indicating that "everything" on the big side of the operator fits into a small bit of what is on the small side of the operator. Like a funnel for data. I don't know the origin, but I'm believing my fiction is right regardless. It makes <(...) make intuitive sense.
The comment about "why not &2>&1" is probably the best one on the page, with the answer essentially being that it would complicate the parser too much / add an unnecessary byte to scripts.
It means redirect file descriptor 2 to the same destination as file descriptor 1.
Which actually means that an undelrying dup2 operation happens in this direction:
2 <- 1 // dup2(2, 1)
The file description at [1] is duplicated into [2], thereby [2] points to the same object. Anything written to stderr goes to the same device that stdout is sending to.
The notation follows I/O redirections: cmd > file actually means that a descriptor [n] is first created for the open file, and then that descriptor's decription is duplicated into [1]:
So if i happen to know the numbers of other file descriptors of the process (listed in /proc), i can redirect to other files opened in the current process? 2>&1234? Or is it restricted to 0/1/2 by the shell?
Would probably be hard to guess since the process may not have opened any file once it started.
> I am thinking that they are using & like it is used in c style programming languages. As a pointer address-of operator. [...] 2>&1 would represent 'direct file 2 to the address of file 1'.
I had never made the connection of the & symbol in this context. I think I never really understood the operation before, treating it just as a magic incantation but reading this just made it click for me.
No, the shell author needed some way to distinguish file descriptor 1 from a file named "1" (note that 2>1 means to write stderr to the file named "1"), and '&' was one of the few available characters. It's not the address of anything.
To be consistent, it would be &2>&1, but that makes it more verbose than necessary and actually means something else -- the first & means that the command before it runs asynchronously.
I enjoyed the commenter asking “Why did they pick such arcane stuff as this?” - I don’t think I touch more arcane stuff than shell, so asking why shell used something that is arcane relative to itself is to me arcane squared.
I love myself a little bit of C++. A good proprietary C++ codebase will remind you that people just want to be wizards, solving their key problem with a little bit of magic.
I've only ever been tricked into working on C++...
As someone who use LLMs to generate, among others, Bash script I recommend shellcheck too. Shellcheck catches lots of things and shall really make your Bash scripts better. And if for whatever reason there's an idiom you use all the time that shellcheck doesn't like, you can simply configure shellcheck to ignore that one.
I always wondered if there ever was a standard stream for stdlog which seems useful, and comes up in various places but usually just as an alias to stderr
The point is that the order in which that is processed is not left to right.
First the | pipe is established as fd [1]. And then 2>&1 duplicates that pipe into [2]. I.e. right to left: opposite to left-to-right processing of redirections.
When you need to capture both standard error and standard output to a file, you must have them in this order:
bob > file 2>&1
It cannot be:
bob 2>&1 > file
Because then the 2>&1 redirection is performed first (and usually does nothing because stderr and stdout are already the same, pointing to your terminal). Then > file redirects only stdout.
But if you change > file to | process, then it's fine! process gets the combined error and regular output.
I understand how this works, but wouldn’t a more clear syntax be:
command &2>&1
Since the use of & signifies a file descriptor. I get what this ACTUALLY does is run command in the background and then run 2 sending its stout to stdout. That’s completely not obvious by the way.
I find it easier to understand in terms of the Unix syscall API. `2>&1` literally translates as `dup2(1, 2)`, and indeed that's exactly how it works. In the classic unix shells that's all that happens; in more modern shells there may be some additional internal bookkeeping to remember state. Understanding it as dup2 means it's easier to understand how successive redirections work, though you also have to know that redirection operators are executed left-to-right, and traditionally each operator was executed immediately as it was parsed, left-to-right. The pipe operator works similarly, though it's a combination of fork and dup'ing, with the command being forked off from the shell as a child before processing the remainder of the line.
Though, understanding it this way makes the direction of the angled bracket a little odd; at least for me it's more natural to understand dup2(2, 1) as 2<1, as in make fd 2 a duplicate of fd 1, but in terms of abstract I/O semantics that would be misleading.
Another fun consequence of this is that you can initialize otherwise-unset file descriptors this way:
It's a trick you can use if you've got a super chatty script or set of scripts, you want to silence or slurp up all of their output, but you still want to allow some mechanism for printing directly to the terminal.The danger is that if you don't open it before running the script, you'll get an error:
Interesting. Is this just literally “fun”, or do you see real world use cases?
Multiple levels of logging, all of which you want to capture but not all in the same place.
Yep, there's a strong unifying feel between the Unix api, C, the shell, and also say Perl.
Which is lost when using more modern or languages foreign to Unix.
Python too under the hood, a lot of its core is still from how it started as a quick way to do unixy/C things.
And just like dup2 allows you to duplicate into a brand new file descriptor, shells also allow you to specify bigger numbers so you aren’t restricted to 1 and 2. This can be useful for things like communication between different parts of the same shell script.
Haha, I'm even more confused now. I have no idea what dup is...
There are a couple of ways to figure out.
open a terminal (OSX/Linux) and type:
open a browser window and search for: Both will bring up the man page for the function call.To get recursive, you can try:
(the unix is important, otherwise it gives you manly men)otherwise it gives you manly men
That's only just after midnight [1][2]
[1] - https://www.youtube.com/watch?v=XEjLoHdbVeE
[2] - https://unix.stackexchange.com/questions/405783/why-does-man...
It's a reminder of how archaic the systems we use are.
File descriptors are like handing pointers to the users of your software. At least allow us to use names instead of numbers.
And sh/bash's syntax is so weird because the programmer at the time thought it was convenient to do it like that. Nobody ever asked a user.
Who do you imagine the users were back when it was being developed?
It should be a lesson to learn on how simple, logical and reliable tools can last decades.
… Or how hard it is to replace archaic software that’s extremely prevalent.
> At least allow us to use names instead of numbers.
You can for the destination. That's the whole reason you need the "&": to tell the shell the destination is not a named file (which itself could be a pipe or socket). And by default you don't need to specify the source fd at all. The intent is that stdout is piped along but stderr goes directly to your tty. That's one reason they are separate.
And for those saying "<" would have been better: that is used to read from the RHS and feed it as input to the LHS so it was taken.
At the time, the users were the programmers.
This is misleading because you use plural for both and I'm sure most of these UX missteps were _each_ made by a _single_ person, and there were >1 users even at the time.
I think he meant that at that time all users were programmers. Yes, _all_ .
> and there were >1 users even at the time.
Are you sure there wasn't >&1 users... Sorry I'll get my coat.
programmers are people too! bash syntax just sucks
arguably if you're using the CLI they still are
Yeah but now they're using npm to install a million packages to do things like tell if a number is greater than 10000. The chances of the programmer wanting to understand the underlying system they are using is essentially nil.
Yea, they are just much higher level programmers… most programmers don’t know the low level syscall apis.
nah, we have long had other disciplines using the CLI who do not write their own software, e.g. sysadmins
The conveniences also mean that there is more than ~one~ ~two~ several ways to do something.
Which means that reading someone else's shell script (or awk, or perl, or regex) is INCREDIBLY inconvenient.
Yes. There are many reasons why one shouldn't use sh/bash for scripting.
But my main reason is that most scripts break when you call them with filenames that contain spaces. And they break spectacularly.
You're not wrong, but there's fairly easy ways to deal with filenames containing spaces - usually just enclosing any variable use within double quotes will be sufficient. It's tricker to deal with filenames that contain things such as line breaks as that usually involves using null terminated filenames (null being the only character that is not allowed in filenames). e.g find . -type f -print0
> bash's syntax is so weird
What should be the syntax according to contemporary IT people? JSON? YAML? Or just LLM prompt?
Nushell, Powershell, Python, Ruby, heck even Perl is better. Shell scripting is literally the worst language I've ever seen in common use. Any realistic alternative is going to be better.
Trying to be language agnostic: it should be as self-explanatory as possible. 2>&1 is all but.
Why is there a 2 on the left, when the numbers are usually on the right. What's the relationship between 2 and 1? Is the 2 for std err? Is that `&` to mean "reference"? The fact you only grok it if you know POSIX sys calls means it's far from self explanatory. And given the proportion of people that know POSIX sys calls among those that use Bash, I think it's a bit of an elitist syntax.
There's a movement to write JSON to fd 3, as a machine-parsable alternative to rickety fd 1.
Nushell! Or powershell, but I much prefer nushell!
Haskell
Honestly, Python with the "sh" module is a lot more sane.
Is it more sane, or is it just what you are used to?
Python doesn't really have much that makes it a sensible choice for scripting.
Its got some basic data structures and a std-lib, but it comes at a non-trivial performance cost, a massive barrier to getting out of the single thread, and non-trivial overhead when managing downstream processes. It doesn't protect you from any runtime errors (no types, no compile checks). And I wouldn't call python in practice particularly portable...
Laughably, NodeJS is genuinely a better choice - while you don't get multithreading easily, at least you aren't trivially blocked on IO. NodeJS also has pretty great compatibility for portability; and can be easily compiled/transformed to get your types and compile checks if you want. I'd still rather avoid managing downstream processes with it - but at least you know your JSON parsing and manipulation is trivial.
Go is my goto when I'm reaching for more; but (ba)sh is king. You're scripting on the shell because you're mainly gluing other processes together, and this is what (ba)sh is designed to do. There is a learning curve, and there are footguns.
It means someone did not bother to name their variables properly, reminding you to use a shell from this century.
Better: Understanding Linux's File Descriptors: A Deep Dive Into '2>&1' and Redirection https://news.ycombinator.com/item?id=41384919 https://news.ycombinator.com/item?id=39095755
Not sure why this link and/or question is here, except to say LLMs like this incantation.
It redirects STDERR (2) to where STDOUT is piped already (&1). Good for dealing with random CLI tools if you're not a human.
Humans used this combination extensively for decades too. I'm no aware of any other simple way to grep both stdout and stderr from a process. (grep, or save to file, or pipe in any other way).
"not humans" are using this extensively precisely because humans used this combination extensively for decades. It's muscle-memory for me. And so is it for LLMs.
I found the explanation useful, about "why" it is that way. I didn't realize the & before the 1 means to tell it is the filedescriptor 1 and not a file named 1.
I get the ocassional file named `1` lying around.
It's an operator called ">&", the 1 is the parameter.
Well sure, but surely this takes some inspiration from both `&` as the "address of" operator in C as well as the `>` operator which (apart from being the greater-than operator) very much implies "into" in many circumstances.
So `>&1` is "into the file descriptor pointed to by 1", and at the time any reasonable programmer would have known that fd 1 == STDOUT.
I've also found llms seem to love it when calling out to tools, I suppose for them having stderr interspersed messaged in their input doesn't make much difference
I know the underlying call, but I always see the redirect symbols as indicating that "everything" on the big side of the operator fits into a small bit of what is on the small side of the operator. Like a funnel for data. I don't know the origin, but I'm believing my fiction is right regardless. It makes <(...) make intuitive sense.
The comment about "why not &2>&1" is probably the best one on the page, with the answer essentially being that it would complicate the parser too much / add an unnecessary byte to scripts.
Man I miss stack overflow. It feels so much better to ask humans a question then the machine, but it feels impossible to put the lid back on the box.
It means redirect file descriptor 2 to the same destination as file descriptor 1.
Which actually means that an undelrying dup2 operation happens in this direction:
The file description at [1] is duplicated into [2], thereby [2] points to the same object. Anything written to stderr goes to the same device that stdout is sending to.The notation follows I/O redirections: cmd > file actually means that a descriptor [n] is first created for the open file, and then that descriptor's decription is duplicated into [1]:
So if i happen to know the numbers of other file descriptors of the process (listed in /proc), i can redirect to other files opened in the current process? 2>&1234? Or is it restricted to 0/1/2 by the shell?
Would probably be hard to guess since the process may not have opened any file once it started.
Redirects are fun but there are way more than I actually routinely use. One thing I do is the file redirects.
I do that with diff <(xxd -r file.bin) <(xxd -r otherfile.bin) sometimes when I should expect things to line up and want to see where things break.I've almost never needed any of these, but there's all sorts of weird redirections you can do in GNU Bash: https://www.gnu.org/software/bash/manual/bash.html#Redirecti...
agentic ai tends to use it ALL the time.
> I am thinking that they are using & like it is used in c style programming languages. As a pointer address-of operator. [...] 2>&1 would represent 'direct file 2 to the address of file 1'.
I had never made the connection of the & symbol in this context. I think I never really understood the operation before, treating it just as a magic incantation but reading this just made it click for me.
No, the shell author needed some way to distinguish file descriptor 1 from a file named "1" (note that 2>1 means to write stderr to the file named "1"), and '&' was one of the few available characters. It's not the address of anything.
To be consistent, it would be &2>&1, but that makes it more verbose than necessary and actually means something else -- the first & means that the command before it runs asynchronously.
It's not inconsistent. The & is attached to the redirection operator, not to the 1 token. The file descriptor being redirected is also attached:
Thus you cannot write:
You also cannot write However you may write The n>& is one clump.back when stackoverflow was still good and useful, I asked about some stderr manipulation[0] and learnt a lot from the replies
[0] https://stackoverflow.com/questions/3618078/pipe-only-stderr...
I enjoyed the commenter asking “Why did they pick such arcane stuff as this?” - I don’t think I touch more arcane stuff than shell, so asking why shell used something that is arcane relative to itself is to me arcane squared.
I love myself a little bit of C++. A good proprietary C++ codebase will remind you that people just want to be wizards, solving their key problem with a little bit of magic.
I've only ever been tricked into working on C++...
If you need to know what 2>&1 means, then I would recommend shellcheck
It's very, very easy to get shell scripts wrong; for instance the location of the file redirect operator in a pipeline is easy to get wrong.
As someone who use LLMs to generate, among others, Bash script I recommend shellcheck too. Shellcheck catches lots of things and shall really make your Bash scripts better. And if for whatever reason there's an idiom you use all the time that shellcheck doesn't like, you can simply configure shellcheck to ignore that one.
I saw this newer bash syntax for redirecting all output some years ago on irc
I didn't know about |&, not sure if it was introduced at the same time. So I'd always use &> for redirection to file and 2>&1 for piping
I think the "|&" is the most intuitive syntax - you can just amend an existing pipe to also include STDERR
I always wondered if there ever was a standard stream for stdlog which seems useful, and comes up in various places but usually just as an alias to stderr
/dev/stderr on Linux
Powershell has ”stdprogress”
A gotcha for me originally and perhaps others is that while using ordering like
sends stdout and stderr to blah, imitating the order with pipe instead does not. This is because | is not a mere redirector but a statement terminator.Useless use of cat error/award
But also | isnt a redirection, it takes stdout and pipes it to another program.
So, if you want stderr to go to stdout, so you can pipe it, you need to do it in order.
bob 2>&1 | prog
You usually dont want to do this though.
The point is that the order in which that is processed is not left to right.
First the | pipe is established as fd [1]. And then 2>&1 duplicates that pipe into [2]. I.e. right to left: opposite to left-to-right processing of redirections.
When you need to capture both standard error and standard output to a file, you must have them in this order:
It cannot be: Because then the 2>&1 redirection is performed first (and usually does nothing because stderr and stdout are already the same, pointing to your terminal). Then > file redirects only stdout.But if you change > file to | process, then it's fine! process gets the combined error and regular output.
Why would that second one be expected to work?
I understand how this works, but wouldn’t a more clear syntax be:
command &2>&1
Since the use of & signifies a file descriptor. I get what this ACTUALLY does is run command in the background and then run 2 sending its stout to stdout. That’s completely not obvious by the way.
even clearer syntax:
command &stderr>&stdout