>None of the three can easily be captured in code, but are trivial to capture as documentation.
Posts like this frustrate me. Not because of what they ask, but because of what they incorrectly assume. They assume that documentation can provide enough context, and that human knowledge is not needed.
Every bit of written documentation can and will be misinterpreted. And perfect clarity is impossible. A well-written ADR does not eliminate all ambiguity, because there is too much cultural context around the writing of the ADR that attempting to read it from some other cultural vantage point leads to bad assumptions. We can find this basic lesson from reading law (2nd, 14th amendments to the constitution), history (what did happen after Muhammad died?), philosophy (what in the world is Plato's cave talking about?), or theology (how should we translate Ephesians 5:22-33 and what does that mean) outside its original context with other people.
Just writing things down and thinking an AI is going to later perfectly understand what the intent of the author is... patently ridiculous. I do not intend to dismiss the idea that we should probably document more, but the idea that the AI can just take our documentation and competently understand all the decisions represented in them is ludicrous.
Having worked at enterprise with 100+ engineers across multiple teams with complex reporting lines, I definitely agree with this. There's a lot of nuance behind decisions that docs simply can't capture. I mean in theory you can probably write every considerations that led you to make a certain decision, e.g.:
- I'm writing this service even though team X has built the same thing, because my team lead doesn't trust team X since the last time we depended on their service 3 years ago, they had a major downtime that screwed us up big time
- This service is using AWS Lambda simply because I think it's cool, despite the fact that the company has a dedicated team running k8s stack with argocd, argo rollouts, KEDA, etc for the entire company
- Service Y is written in this particular way because it's a service that is shared with another team that came from a company that was acquired, and they wouldn't use it unless we write it this particular way, and making the top execs happy is more important than dealing with a small tech debt (this is probably true)
But no one is going to write these in their RFC. But Sarah knows.
People would often ask me why I would just read the code for a project rather than ask whoever wrote it. At the end of 30 minutes talking to someone, I may not have learned anything. Or I may have learned the wrong thing because they wrote it 3 years ago and forgot about it.
At the end of 30 minutes I either understand how the project works at a high level or know enough to know I'm not going to be improving anything with this project, ever.
Pulling histograms from my dictation software, it's apparent that I speak for sixty minutes at a time multiple times a day at 200-350 wpm aggregate just... trying to communicate all the nuance about a tiny, _tiny_ subject area, for an agent to implement or write in plans/docs... and then despite the best efforts of both me and LLMs, that can be and usually still _is_ misinterpreted.
Whether bulk or terse, highly precise with words or highly nuanced in how it's communicated... I don't think any of that gets you to a place where the documentation is a substitute for asking the person.
People are trying to do this with meetings as well and certainly they help but code that's written plus meetings plus an architect like person yammering on endlessly about the nuance that went into it still is often not enough to capture the detail in earnest – and especially not in a way that won't be misinterpreted.
Perhaps if AI becomes truly superhuman in all of the relevant areas to the point that it makes the decisions just as well as the person in the chair... _then_ we might solve this by having it instantly pattern recognize the solution and the why, but until and unless we reach that day, I think what you're saying is very true.
>Every bit of written documentation can and will be misinterpreted.
Yes, humans (and human languages) are flawed and lossy.
>A well-written ADR does not eliminate all ambiguity,
True: no docs can ever eliminate all ambiguity (on a decent sized project at least).
But this entire argument seems to be "letting perfect be the enemy of the good". Documentation doesn't have to be perfect or 100% unambiguous to be useful.
> They assume that documentation can provide enough context, and that human knowledge is not needed.
It's funny actually, because I fully agree with your reasoning. The only part were we differ is whether that's assumed, or even implied.
No documentation means running fully on tribal knowledge, or institutional knowledge if you prefer. Even if you capture your intent, imperfect and incomplete, in as little as 2 paragraphs, you'll get durable recorded memory, and intent you'll be able to reference. It does not eliminate ambiguity, but it adds framing, direction, and friction.
The examples are great, and they serve really well to prove another point that I intentionally left out: writing is not a one-shot activity. Documentation is living and should be treated as such. Unless it receives proper care continuously, it will wither and die. That could very well be the topic of a future post!
Thank you for reading and for providing thoughtful feedback!
I do agree with you both, but we need a wider view of what documentation means. I'm spending time this days reading the OpenBSD code and while the latter is really well written, I spent even more time reading the mailing list archives, the revision notes, the BSD4.4 book, some OS textbooks (Three Easy Pieces is great BTW), and various datasheets (Intel, USB,...). I would have a much easier time if I could find a dev to be my mentor and walk me through it.
All of this to say, tribal knowledge is valuable and rarely well captured. Even experts organize conferences and lectures instead of just sending each other papers. Written words is better than nothing, but knowledge is better transferred from the person that has it. If you lose that person, the only way forward is to train another one to replace it (either intentionally or not).
As the 'sarah' for an org I'm golden handcuffed to, avoid becoming the 'sarah' at all costs it will cost you so much in your career. I am a founding engineer self-taught from it/software security to full stack dev, and I pushed so we aimed to higher better engineers than myself. When they came in it was a full rewrite, a new abstraction and going down the same paths I learned the hard way not to go down. I was pushed out of enforcing hard-earned business logic, and we are still paying for it. Everywhere from not accepting the inherent complexity of the problem and over simplifying software to trying to get same features we had from 3 years ago. This then has made me the scapegoat for why the new engineers made xyz decision, and was the one in charge of fixing the shortcuts, bugs and workarounds. I have received promotions for this to be in charge of the veterans that didn't listen to me, whom still don't until it's a fire drill. Joke among colleges is I was the first 'agent' our company had, endless work, just enough authority to do current task, not enough respect/authority to solve the symptom. I understand this is also a failure of my office politics and am improving, however its hard to balance blunt productivity, slow careful positioning and letting people struggle with items I don't recommend until circle back.
This is such a toxic combination, becuse it requires significant people skills to get out of the "the kid who learned everything here and is grateful for it" and get proper respect as a professional. At some point the only option is changing jobs. I've seen companies matching your offer, finally realizing you actually have value in the market, but don't count on that, don't bluff.
I too ended up as the "go to guy", partly because I had a lot of enthusiamsm for my new job, and partly because the talent pool there wasn't very deep (or maybe they were smarter than me). It's fulfilling until it becomes unrewarding, I had to move on after almost 5 years. Still did consulting for them ocassionally for a couple more years.
That feels more familiar than I'd care to admit - definitely been there, done that. I think what most fail to realize is that it eventually turns into a ball and chain, decreasing your mobility within the organization. So the consequences of running an org on tribal knowledge and Sarahs is far worse, and direct, than that it won't work very well with agents.
It’s very easy to turn into the Sarah - or the Brent if you prefer the Phoenix Project analogy. As exciting as it might initially be to be the go-to person, it’s also, as you so elegantly put it, “endless work, just enough authority to do current task, not enough respect/authority to solve the symptom”.
Much appreciated! Thanks for the blog around ADR, I didn't know there was a full ecosystem for the approach and the AI tie in will help me sell the process
All organizations above a certain size have a Sarah. This I've learned first and second hand over decades (the second hand was a spouse whose job at one point was finding, interviewing, and collecting the knowledge of her's org's Sarahs).
Very, very few of these organizations have ever known, and fewer still have ever cared, about their Sarahs.
This isn't the end of Sarahs. Sarahs have never had their time or place beyond immediate teams, many of which have used Fight Club rules when it came to their Sarah: Never talk about Sarah, especially not to the boss. Other, non Fight Club rules: When Sarah is away, cover as best you can. Change jobs before Sarah retires. It is not the end, because the time of Sarahs never began.
So I agree with ";dr" comment, but it would apply had this been written by a human, by AI, by a super-intelligent shade of blue, or a small furry creature from Alpha Centauri.
>None of the three can easily be captured in code, but are trivial to capture as documentation.
Posts like this frustrate me. Not because of what they ask, but because of what they incorrectly assume. They assume that documentation can provide enough context, and that human knowledge is not needed.
Every bit of written documentation can and will be misinterpreted. And perfect clarity is impossible. A well-written ADR does not eliminate all ambiguity, because there is too much cultural context around the writing of the ADR that attempting to read it from some other cultural vantage point leads to bad assumptions. We can find this basic lesson from reading law (2nd, 14th amendments to the constitution), history (what did happen after Muhammad died?), philosophy (what in the world is Plato's cave talking about?), or theology (how should we translate Ephesians 5:22-33 and what does that mean) outside its original context with other people.
Just writing things down and thinking an AI is going to later perfectly understand what the intent of the author is... patently ridiculous. I do not intend to dismiss the idea that we should probably document more, but the idea that the AI can just take our documentation and competently understand all the decisions represented in them is ludicrous.
Having worked at enterprise with 100+ engineers across multiple teams with complex reporting lines, I definitely agree with this. There's a lot of nuance behind decisions that docs simply can't capture. I mean in theory you can probably write every considerations that led you to make a certain decision, e.g.:
- I'm writing this service even though team X has built the same thing, because my team lead doesn't trust team X since the last time we depended on their service 3 years ago, they had a major downtime that screwed us up big time
- This service is using AWS Lambda simply because I think it's cool, despite the fact that the company has a dedicated team running k8s stack with argocd, argo rollouts, KEDA, etc for the entire company
- Service Y is written in this particular way because it's a service that is shared with another team that came from a company that was acquired, and they wouldn't use it unless we write it this particular way, and making the top execs happy is more important than dealing with a small tech debt (this is probably true)
But no one is going to write these in their RFC. But Sarah knows.
People would often ask me why I would just read the code for a project rather than ask whoever wrote it. At the end of 30 minutes talking to someone, I may not have learned anything. Or I may have learned the wrong thing because they wrote it 3 years ago and forgot about it.
At the end of 30 minutes I either understand how the project works at a high level or know enough to know I'm not going to be improving anything with this project, ever.
Pulling histograms from my dictation software, it's apparent that I speak for sixty minutes at a time multiple times a day at 200-350 wpm aggregate just... trying to communicate all the nuance about a tiny, _tiny_ subject area, for an agent to implement or write in plans/docs... and then despite the best efforts of both me and LLMs, that can be and usually still _is_ misinterpreted.
Whether bulk or terse, highly precise with words or highly nuanced in how it's communicated... I don't think any of that gets you to a place where the documentation is a substitute for asking the person.
People are trying to do this with meetings as well and certainly they help but code that's written plus meetings plus an architect like person yammering on endlessly about the nuance that went into it still is often not enough to capture the detail in earnest – and especially not in a way that won't be misinterpreted.
Perhaps if AI becomes truly superhuman in all of the relevant areas to the point that it makes the decisions just as well as the person in the chair... _then_ we might solve this by having it instantly pattern recognize the solution and the why, but until and unless we reach that day, I think what you're saying is very true.
>Every bit of written documentation can and will be misinterpreted.
Yes, humans (and human languages) are flawed and lossy.
>A well-written ADR does not eliminate all ambiguity,
True: no docs can ever eliminate all ambiguity (on a decent sized project at least).
But this entire argument seems to be "letting perfect be the enemy of the good". Documentation doesn't have to be perfect or 100% unambiguous to be useful.
> They assume that documentation can provide enough context, and that human knowledge is not needed.
It's funny actually, because I fully agree with your reasoning. The only part were we differ is whether that's assumed, or even implied.
No documentation means running fully on tribal knowledge, or institutional knowledge if you prefer. Even if you capture your intent, imperfect and incomplete, in as little as 2 paragraphs, you'll get durable recorded memory, and intent you'll be able to reference. It does not eliminate ambiguity, but it adds framing, direction, and friction.
The examples are great, and they serve really well to prove another point that I intentionally left out: writing is not a one-shot activity. Documentation is living and should be treated as such. Unless it receives proper care continuously, it will wither and die. That could very well be the topic of a future post!
Thank you for reading and for providing thoughtful feedback!
I do agree with you both, but we need a wider view of what documentation means. I'm spending time this days reading the OpenBSD code and while the latter is really well written, I spent even more time reading the mailing list archives, the revision notes, the BSD4.4 book, some OS textbooks (Three Easy Pieces is great BTW), and various datasheets (Intel, USB,...). I would have a much easier time if I could find a dev to be my mentor and walk me through it.
All of this to say, tribal knowledge is valuable and rarely well captured. Even experts organize conferences and lectures instead of just sending each other papers. Written words is better than nothing, but knowledge is better transferred from the person that has it. If you lose that person, the only way forward is to train another one to replace it (either intentionally or not).
As the 'sarah' for an org I'm golden handcuffed to, avoid becoming the 'sarah' at all costs it will cost you so much in your career. I am a founding engineer self-taught from it/software security to full stack dev, and I pushed so we aimed to higher better engineers than myself. When they came in it was a full rewrite, a new abstraction and going down the same paths I learned the hard way not to go down. I was pushed out of enforcing hard-earned business logic, and we are still paying for it. Everywhere from not accepting the inherent complexity of the problem and over simplifying software to trying to get same features we had from 3 years ago. This then has made me the scapegoat for why the new engineers made xyz decision, and was the one in charge of fixing the shortcuts, bugs and workarounds. I have received promotions for this to be in charge of the veterans that didn't listen to me, whom still don't until it's a fire drill. Joke among colleges is I was the first 'agent' our company had, endless work, just enough authority to do current task, not enough respect/authority to solve the symptom. I understand this is also a failure of my office politics and am improving, however its hard to balance blunt productivity, slow careful positioning and letting people struggle with items I don't recommend until circle back.
> founding engineer self-taught
This is such a toxic combination, becuse it requires significant people skills to get out of the "the kid who learned everything here and is grateful for it" and get proper respect as a professional. At some point the only option is changing jobs. I've seen companies matching your offer, finally realizing you actually have value in the market, but don't count on that, don't bluff.
I too ended up as the "go to guy", partly because I had a lot of enthusiamsm for my new job, and partly because the talent pool there wasn't very deep (or maybe they were smarter than me). It's fulfilling until it becomes unrewarding, I had to move on after almost 5 years. Still did consulting for them ocassionally for a couple more years.
That feels more familiar than I'd care to admit - definitely been there, done that. I think what most fail to realize is that it eventually turns into a ball and chain, decreasing your mobility within the organization. So the consequences of running an org on tribal knowledge and Sarahs is far worse, and direct, than that it won't work very well with agents.
I am Sarah, and it has been incredibly profitable for me: good ratings, promotions up to director-level IC, extra retention RSU grants.
If you have golden handcuffs and multiple promotions out of being Sarah, what is it costing you?
Author here.
That sounds exhausting to say the least.
It’s very easy to turn into the Sarah - or the Brent if you prefer the Phoenix Project analogy. As exciting as it might initially be to be the go-to person, it’s also, as you so elegantly put it, “endless work, just enough authority to do current task, not enough respect/authority to solve the symptom”.
Best wishes! I hope you manage to turn it around.
Much appreciated! Thanks for the blog around ADR, I didn't know there was a full ecosystem for the approach and the AI tie in will help me sell the process
All organizations above a certain size have a Sarah. This I've learned first and second hand over decades (the second hand was a spouse whose job at one point was finding, interviewing, and collecting the knowledge of her's org's Sarahs).
Very, very few of these organizations have ever known, and fewer still have ever cared, about their Sarahs.
This isn't the end of Sarahs. Sarahs have never had their time or place beyond immediate teams, many of which have used Fight Club rules when it came to their Sarah: Never talk about Sarah, especially not to the boss. Other, non Fight Club rules: When Sarah is away, cover as best you can. Change jobs before Sarah retires. It is not the end, because the time of Sarahs never began.
So I agree with ";dr" comment, but it would apply had this been written by a human, by AI, by a super-intelligent shade of blue, or a small furry creature from Alpha Centauri.
It's already quite feasible to record meetings and get AI summaries. As that works better, it will become more widespread.
Simon should ask Sarah how to write an article without relying on an LLM
This article fails to mention Sarah burned out and is now a barista.
Besides documentation, you can commit your chat sessions in git notes e.g. with https://usegitai.com/
ai;dr