I wonder, why not use a string buffer paired with its length? For example, maybe use struct that has char pointer, and 2 ints (occupied length + total buffer length). Almost like c++'s std::string. This null terminator thing really sucks, it's potentially insecure and often unperformant.
Zero terminated string is a special case of sentinel value termination.
And sentinel value terminations make a lot of sense when you have punch cards and fixed length records that you need to carve into pieces.
Nobody expected any decisions they were making in the 1960s and 1970s to have any bearing on computing a half-century later. They all expected to have their mistakes long papered over by smarter people at some point.
But we ALL make the mistake of underestimating inertia.
I think it was NULL itself. It was a long way until we realised we don't want invalid values and could use the type system to help us use special values safely.
Compared to scripting languages with actual tagged types, C doesn't really have a type system, and that's readily apparent to anyone who has written C in the last 43 years and debugged a program written in it.
C pretends types exist with you, but once bytes hit the road, it's all real-life and segmentation faults.
Meh, I think NULL is fine in C. It's an extra, valid state to represent pointers at no cost. Unlike the more hand holdy languages, it's quite rare for a pointer in C to have the ability to be NULL since, more often than not, it's pointing at something known. It's actually quite rare to see NULL checks unless it's API code or something like that. I can see this being more of a problem in a managed language where anything can be NULL at any time.
NULL as a concept is fine. Inability to declare something as non-null is not.
There is a huge gap between developer expectation "it's pointing at something known" and hard reality confirmed by zillions of CVE. That's the reason optionality is prevalent in modern languages and type checkers (python, typescript), nowdays even Java has sane non-nullable types.
This precise mindset is why the world has suffered for decades (wrt security/integrity/availability) at the hands of what can only be described as an industry led by completely unjustified male confidence.
Why are there still people fighting the “it’s not that bad, guys! you’ve just got to be a good developer like ME!” fight?
It was definitely an interesting way to allocate pointers.
I did once have a very large project where devs didnt understand this and resolved hundreds or more off by one and memory overwrites in C due to this feature.
But at the same time, I think blaming the software was kind of a cop out. Devs were in a hurry and simply didnt respect the rules. Given todays software engineer at large. Nerfing programming languages so they cant destroy things might not be a bad idea. But AI will nerf everything.
why is AI gonna nerf everything? sure it could be used as the easy button, but I just spent two hours this morning learning about the neuroscience of how memory works in the brain that I didn't mean to and now I want to run studies on how memory works.
Why do you assume that AI is gonna nerf everything?
In addition to having to pick a size for the length counter and then, later, having to differentiate between lengths in bytes, codepoints, and glyphs, you can't subdivide a Pascal string using pointer arithmetic. To pass just the end of a string into a function, you have to either copy the tail of one Pascal-style string to another with a smaller size value, or your string has to be a struct with an integer and a pointer to the actual data instead of just an integer stuck on the beginning of the string. The first is a lot of copying in some cases, the second raises the specter of structs with invalid pointers. That's not to mention the potential problems that would cause with caches.
The third option is to have a variable width length: the top most bit signals whether the next byte corresponds to the length or to the start of the string.
The limitations were brutal. Initially you could only have 255 bytes in a string. The length of a string and the size of the allocation are now separate and you may need to think about that unused memory in your design. The problem now doubles with the introduction of UTF-8. Your string size is in bytes and you need to track characters separately.
If you want to create an array of strings you either need to specify the length of all strings and accept the memory overhead or have an array of pointers to strings. If you use an array of pointers you may end up choosing to use the 'nil' value as a sentinel that means "end of list." So we're right back where we started.
--
Because someone decided to downvote this HN has limited the speed at which I can reply. This site is tragic and I'm fully done with it now. You can spread propaganda and poorly sourced zeitgeist and be among friends but if you try to have a genuine conversation about programming languages you are made to be unwelcome immediately. Screw this.
--
> No other data structure works like this.
The linked list.
> You can't mess this up in an array
C happily decomposes arrays into pointers. You can erase your length information from the type. This was an intentional decision.
> Strings are the only data structure that assume there will be a NULL at end.
Which is why almost every string API has a version that allows you to specify the maximum length. The fact that you can use a NUL doesn't mean you have to. Which is why the concept of "sentinel values" is broadly used in many types of applications you haven't considered here.
>The problem now doubles with the introduction of UTF-8. Your string size is in bytes and you need to track characters separately.
That isn't really a problem.
The problem with null-terminated strings is specifically what happens when you reach the end of the allocated array and there ISN'T a NULL character.
Every string function is designed to keep going until it finds the NULL character, so if a hacker gets rid of the NULL character, he can exploit pretty much any standard string manipulation function being used elsewhere in the program to manipulate whatever memory comes AFTER the string data structure.
No other data structure works like this. You can't mess this up in an array, because no function that manipulates arrays is just going to keep going until there is a null. That would be stupid because it would require users of the function to add a NULL to the end of their arrays before passing it to the function, so instead we just pass the size of the array to everything. Strings are the only data structure that assume there will be a NULL at end.
By the way, I read once that if you use UTF-32 every code point will be 4 bytes, constantly, but even then a single code point isn't necessarily a single character. Text is just complicated.
In C most data structures work like this, you keep going until you find NUL (character) or NULL (pointer). E.g. Strings, array of pointers, linked lists, etc. Of course you can add length to most of those, but it isn't the canonical/traditional way of doing things.
I worked on a Win32 app that used space-padded strings, i.e. the destination string was padded with spaces, but there was still a null on the last byte. You had to use special versions of the string functions for length, copy etc.
I’m not sure why this was - the source base was so old it might have had its origins in Pascal struct behaviour.
A reminder that we've had strlcpy[1] for ~ 30 years but it was never accepted into the Linux world because of typical petty open source bullshit. This is why we can't have nice things.
The Linux kernel had strlcpy over 20 years ago. It was removed in favor of strscpy because the latter was judged a better interface. Here's a 2022 article: https://lwn.net/Articles/905777/
why would you start there instead of creating something from scratch ?if you can port drivers just as easily meaning you don't especially give a shit about hardware you're running on in the first place, why even deal with linux? The battle tested LRU cache system?
I've seen several workalike kernels in various stages of completion. at least one of them was able to run some pretty substantial applications (Postgres, nginx, that kind of thing), and that is still I guess around 250kloc. but it only really has drivers to support hypervisor devices.
unfortunately as time goes by, the linux api surface gets larger and more convoluted. so there's going to be some coverage you're just never going to get.
but in the abstract, definitely. linux is so bloated at this point that its not clear that it can ever be 'made safe'.
I wonder, why not use a string buffer paired with its length? For example, maybe use struct that has char pointer, and 2 ints (occupied length + total buffer length). Almost like c++'s std::string. This null terminator thing really sucks, it's potentially insecure and often unperformant.
the zero terminated string is I think is computing's biggest mistake. Pascal style strings were much safer.
Zero terminated string is a special case of sentinel value termination.
And sentinel value terminations make a lot of sense when you have punch cards and fixed length records that you need to carve into pieces.
Nobody expected any decisions they were making in the 1960s and 1970s to have any bearing on computing a half-century later. They all expected to have their mistakes long papered over by smarter people at some point.
But we ALL make the mistake of underestimating inertia.
I think it was NULL itself. It was a long way until we realised we don't want invalid values and could use the type system to help us use special values safely.
Compared to scripting languages with actual tagged types, C doesn't really have a type system, and that's readily apparent to anyone who has written C in the last 43 years and debugged a program written in it.
C pretends types exist with you, but once bytes hit the road, it's all real-life and segmentation faults.
Meh, I think NULL is fine in C. It's an extra, valid state to represent pointers at no cost. Unlike the more hand holdy languages, it's quite rare for a pointer in C to have the ability to be NULL since, more often than not, it's pointing at something known. It's actually quite rare to see NULL checks unless it's API code or something like that. I can see this being more of a problem in a managed language where anything can be NULL at any time.
NULL as a concept is fine. Inability to declare something as non-null is not.
There is a huge gap between developer expectation "it's pointing at something known" and hard reality confirmed by zillions of CVE. That's the reason optionality is prevalent in modern languages and type checkers (python, typescript), nowdays even Java has sane non-nullable types.
This precise mindset is why the world has suffered for decades (wrt security/integrity/availability) at the hands of what can only be described as an industry led by completely unjustified male confidence. Why are there still people fighting the “it’s not that bad, guys! you’ve just got to be a good developer like ME!” fight?
> to represent pointers at no cost
I wouldn't call "cause of bugs and security issues" "no cost".
> it's quite rare for a pointer in C to have the ability to be NULL
As a C programmer for more than 25 years, that is the exact opposite of my experience.
It was definitely an interesting way to allocate pointers. I did once have a very large project where devs didnt understand this and resolved hundreds or more off by one and memory overwrites in C due to this feature.
But at the same time, I think blaming the software was kind of a cop out. Devs were in a hurry and simply didnt respect the rules. Given todays software engineer at large. Nerfing programming languages so they cant destroy things might not be a bad idea. But AI will nerf everything.
why is AI gonna nerf everything? sure it could be used as the easy button, but I just spent two hours this morning learning about the neuroscience of how memory works in the brain that I didn't mean to and now I want to run studies on how memory works.
Why do you assume that AI is gonna nerf everything?
In addition to having to pick a size for the length counter and then, later, having to differentiate between lengths in bytes, codepoints, and glyphs, you can't subdivide a Pascal string using pointer arithmetic. To pass just the end of a string into a function, you have to either copy the tail of one Pascal-style string to another with a smaller size value, or your string has to be a struct with an integer and a pointer to the actual data instead of just an integer stuck on the beginning of the string. The first is a lot of copying in some cases, the second raises the specter of structs with invalid pointers. That's not to mention the potential problems that would cause with caches.
The third option is to have a variable width length: the top most bit signals whether the next byte corresponds to the length or to the start of the string.
> Pascal style strings were much safer.
The limitations were brutal. Initially you could only have 255 bytes in a string. The length of a string and the size of the allocation are now separate and you may need to think about that unused memory in your design. The problem now doubles with the introduction of UTF-8. Your string size is in bytes and you need to track characters separately.
If you want to create an array of strings you either need to specify the length of all strings and accept the memory overhead or have an array of pointers to strings. If you use an array of pointers you may end up choosing to use the 'nil' value as a sentinel that means "end of list." So we're right back where we started.
--
Because someone decided to downvote this HN has limited the speed at which I can reply. This site is tragic and I'm fully done with it now. You can spread propaganda and poorly sourced zeitgeist and be among friends but if you try to have a genuine conversation about programming languages you are made to be unwelcome immediately. Screw this.
--
> No other data structure works like this.
The linked list.
> You can't mess this up in an array
C happily decomposes arrays into pointers. You can erase your length information from the type. This was an intentional decision.
> Strings are the only data structure that assume there will be a NULL at end.
Which is why almost every string API has a version that allows you to specify the maximum length. The fact that you can use a NUL doesn't mean you have to. Which is why the concept of "sentinel values" is broadly used in many types of applications you haven't considered here.
>The problem now doubles with the introduction of UTF-8. Your string size is in bytes and you need to track characters separately.
That isn't really a problem.
The problem with null-terminated strings is specifically what happens when you reach the end of the allocated array and there ISN'T a NULL character.
Every string function is designed to keep going until it finds the NULL character, so if a hacker gets rid of the NULL character, he can exploit pretty much any standard string manipulation function being used elsewhere in the program to manipulate whatever memory comes AFTER the string data structure.
No other data structure works like this. You can't mess this up in an array, because no function that manipulates arrays is just going to keep going until there is a null. That would be stupid because it would require users of the function to add a NULL to the end of their arrays before passing it to the function, so instead we just pass the size of the array to everything. Strings are the only data structure that assume there will be a NULL at end.
By the way, I read once that if you use UTF-32 every code point will be 4 bytes, constantly, but even then a single code point isn't necessarily a single character. Text is just complicated.
> No other data structure works like this.
In C most data structures work like this, you keep going until you find NUL (character) or NULL (pointer). E.g. Strings, array of pointers, linked lists, etc. Of course you can add length to most of those, but it isn't the canonical/traditional way of doing things.
I worked on a Win32 app that used space-padded strings, i.e. the destination string was padded with spaces, but there was still a null on the last byte. You had to use special versions of the string functions for length, copy etc.
I’m not sure why this was - the source base was so old it might have had its origins in Pascal struct behaviour.
It can perhaps be due to the string originating from a sql database ”char” field, I.e. not ”varchar”. Char fields in databases are space padded.
Perhaps prevent realocation when string size changes? Or aligning cpu cache lines?
I think this behavior has its roots in COBOL, not pascal.
Which has its roots in punch cards, where pre-computer hardware operated on fixed-sized fields and an unpunched column is equivalent to a space.
A reminder that we've had strlcpy[1] for ~ 30 years but it was never accepted into the Linux world because of typical petty open source bullshit. This is why we can't have nice things.
[1] https://man.openbsd.org/strlcpy
The Linux kernel had strlcpy over 20 years ago. It was removed in favor of strscpy because the latter was judged a better interface. Here's a 2022 article: https://lwn.net/Articles/905777/
Actually, glibc 2.38 has it.
Wow it only took them 26 years to import a 30 line C function, a third of which is comments?
I should have sent them a nice fruit basket to commemorate the occasion.
Wonder when is someone going to brave and fork the linux kernel and try to ffwd it with automatic programming.
why would you start there instead of creating something from scratch ?if you can port drivers just as easily meaning you don't especially give a shit about hardware you're running on in the first place, why even deal with linux? The battle tested LRU cache system?
It's much easier to use something with all the edge cases already handled as a starting point.
I've seen several workalike kernels in various stages of completion. at least one of them was able to run some pretty substantial applications (Postgres, nginx, that kind of thing), and that is still I guess around 250kloc. but it only really has drivers to support hypervisor devices.
unfortunately as time goes by, the linux api surface gets larger and more convoluted. so there's going to be some coverage you're just never going to get.
but in the abstract, definitely. linux is so bloated at this point that its not clear that it can ever be 'made safe'.