It seems like this is measuring algorithms against the disparate impact standard of all demographic groups needing to have the same aggregate results.
Which is extra weird because the samples to this are applications, not humans, so this is subject to bias in how people apply to these positions. So if a demographic group is more likely to apply to some jobs they are not qualified for, this paper would say they are being discriminated against.
On top of all this, there isn't even really a claim that the algorithms are picking up on anything demographic related. One of the vendors they look at pymetrics, which makes players play abstract games and uses that to pre-screen people.
In the abstract, it makes sense that monocultures are problematic since ML bias alone (in the bias vs variance sense) would just randomly harm folks in a fairly persistent way. But it's also not immediately clear that this even applies to the pymetrics example where I think they have a large assortment of games they make people play for different positions?
It's also not clear that these systems breed monocultures if the inputs into them are firm/position-specific, e.g. job descriptions.
Though honestly I would be far more interested in the validity of these measures at predicting actual on the job measures like performance reviews, etc.
> So if a demographic group is more likely to apply to some jobs they are not qualified for, this paper would say they are being discriminated against.
Your understanding appears to be incorrect.
> Our research also found that this pattern does not appear to be the case in other circumstances. We analyzed data from the largest prior study of hiring decisions, which sent 83,000 applications to 108 Fortune 500 firms during the same time period as our study and did not focus on whether AI was used to make decisions. We found that the rate at which applicants were rejected from every firm they applied to in this data was no higher than what you’d expect if each company decided independently of the others.
If it were what you were asserting, then this behavior and results would persist even without AI being used. Instead when they remove the filter for AI decisions (and AI mono-culture in decisions) the effect is no longer present.
This seems to strongly support they argument that effectively a single AI makes a single decision for a candidate across "all" positions they apply for rather than independently assessing them for each position.
Essentially it's more or less saying they're is one hiring manager for the entire industry and if they have a random reason they don't like you, you won't be hired for any job in the industry.
There is a single evaluation function for the industry and if it puts you a negative for any reason in the model's distribution, every job that uses it will do so.
A few years ago the same professor did an analysis of racism in police traffic stops. Compared to this current study on job applications, the conclusions from the police study were more convincing because they actually observed the words spoken by the cops in each interaction.
This sort of thing needs to be illegal. We saw a similar thing wtih RealPage. So many corporate landlords use it that it essentially becomes anticompetitive price-fixing.
I've heard a claim that an issue with these ATS AI Systems is that your CV gets scored and that score is cached for some period from 3 to 12 months. So any application with a completely different company with your name will just yield the exact same score. If true, it means that if you score badly for whatever reason, you're going to get auto-rejected by every company that uses that system before ever being seen by a human.
This seems to fit anecdotal data where people have applied for hundresd of jobs and never gotten anything other than an automated rejection. But obviously that's not proof or confirmation. But if it is, it's almost like being a voncicted felon. It greatly limits your ability to find a job and that's a huge problem.
I don't know what the solution is but I hope these companies get sued for states for issueslike this where actual discrimination occurs.
Recruitment and hiring has been a mess for decades, though, especially in tech.
It's like all the leetcode bullshit. We know that is not a valid measurement for actual performance in a dev job, but that doesn't stop managers from using it.
What they need is a number, a rating, on how much of a fit each candidate is, through some process that can be described as objective and fair. The algorithms provide that.
If we make this illegal, they'll just come up with some other bullshit.
In an ideal world, companies would assess each and every candidate individually and on their merits. But no-one has time or patience for that, so we have these bullshit systems.
Lots of monocultures exist in hiring even without an algorithmic scoring system. That's roughly how every stupid hiring fad works, and how it's always worked, because most employers have no idea how to identify great potential employees.
Hiring managers and companies choose algorithms and hiring fads because they don't know how to be really certain of who to hire, so they'll settle for either assuming someone else's expertise will save them, or for some rubric that "everyone is doing" so it "can't be that bad".
When I first became a hiring manager, I was working for a public university. Our salaries were limited, being staff rather than faculty and being public servants, to between 1/3 and 1/2 the going salary for equivalent cybersecurity professionals in the private sector. I did not have the option to hire the people everyone else was trying to hire. I also faced one of the key risks of working in a public institution: once you keep someone past their probationary period, it is very, very hard to fire them. So, it's important not to get it wrong. I learned some things that I have carried forward into every hiring manager or senior leadership role since:
1. I base hiring practices on Manager Tools behavioral interviewing systems (https://manager-tools.com). No affiliation, they've just made my work life better.
2. I became really good at understanding what my team or organization really needs. Most hirers focus way too much on "years of experience" and specific technologies than is usually wise. As my favorite former supervisor said, "I can teach a smart person cybersecurity, but I can't teach a dumb [or unmotivated] cybersecurity person to be smart."
3. I became really good at developing people, and ensuring that the managers under me were as well. We couldn't lay someone off just because their technical specialty became irrelevant, so we couldn't afford to hire people who weren't lifelong learners, or to fail as coaches to ensure that learning was always taking place.
4. I cast as wide a net as my HR and regulatory overlords would let me (and now, as a business leader, I cast a huge net). I look for things that aren't just useful at the moment, but are useful long term, in my candidates. I don't care about pedigree.
I end up paying less for good employees due to simple supply and demand: I often go for the diamonds in the rough that don't have 10 competing offers.
I end up having really good employees who generally stay with me long term, because I apply long-term thinking in hiring, and structure my teams around constant learning and development.
I dodge a LOT of bullets... people who have just the right pedigree to look like great hires worth a lot of money, but who'll disappoint me until the day they leave.
When it's a tight labor market -- too few candidates for roles I care about -- I'm tapping a hiring market that other managers aren't aware enough of, and still finding talent while they have roles that sit open for months.
It seems like this is measuring algorithms against the disparate impact standard of all demographic groups needing to have the same aggregate results.
Which is extra weird because the samples to this are applications, not humans, so this is subject to bias in how people apply to these positions. So if a demographic group is more likely to apply to some jobs they are not qualified for, this paper would say they are being discriminated against.
On top of all this, there isn't even really a claim that the algorithms are picking up on anything demographic related. One of the vendors they look at pymetrics, which makes players play abstract games and uses that to pre-screen people.
In the abstract, it makes sense that monocultures are problematic since ML bias alone (in the bias vs variance sense) would just randomly harm folks in a fairly persistent way. But it's also not immediately clear that this even applies to the pymetrics example where I think they have a large assortment of games they make people play for different positions?
It's also not clear that these systems breed monocultures if the inputs into them are firm/position-specific, e.g. job descriptions.
Though honestly I would be far more interested in the validity of these measures at predicting actual on the job measures like performance reviews, etc.
> So if a demographic group is more likely to apply to some jobs they are not qualified for, this paper would say they are being discriminated against.
Your understanding appears to be incorrect.
> Our research also found that this pattern does not appear to be the case in other circumstances. We analyzed data from the largest prior study of hiring decisions, which sent 83,000 applications to 108 Fortune 500 firms during the same time period as our study and did not focus on whether AI was used to make decisions. We found that the rate at which applicants were rejected from every firm they applied to in this data was no higher than what you’d expect if each company decided independently of the others.
If it were what you were asserting, then this behavior and results would persist even without AI being used. Instead when they remove the filter for AI decisions (and AI mono-culture in decisions) the effect is no longer present.
This seems to strongly support they argument that effectively a single AI makes a single decision for a candidate across "all" positions they apply for rather than independently assessing them for each position.
Essentially it's more or less saying they're is one hiring manager for the entire industry and if they have a random reason they don't like you, you won't be hired for any job in the industry.
There is a single evaluation function for the industry and if it puts you a negative for any reason in the model's distribution, every job that uses it will do so.
A few years ago the same professor did an analysis of racism in police traffic stops. Compared to this current study on job applications, the conclusions from the police study were more convincing because they actually observed the words spoken by the cops in each interaction.
https://www.pnas.org/doi/10.1073/pnas.1702413114
This is just one of many reasons why my current job is likely to be my last. I feel like so much of modern life is just irredeemably broken right now.
This sort of thing needs to be illegal. We saw a similar thing wtih RealPage. So many corporate landlords use it that it essentially becomes anticompetitive price-fixing.
I've heard a claim that an issue with these ATS AI Systems is that your CV gets scored and that score is cached for some period from 3 to 12 months. So any application with a completely different company with your name will just yield the exact same score. If true, it means that if you score badly for whatever reason, you're going to get auto-rejected by every company that uses that system before ever being seen by a human.
This seems to fit anecdotal data where people have applied for hundresd of jobs and never gotten anything other than an automated rejection. But obviously that's not proof or confirmation. But if it is, it's almost like being a voncicted felon. It greatly limits your ability to find a job and that's a huge problem.
I don't know what the solution is but I hope these companies get sued for states for issueslike this where actual discrimination occurs.
Recruitment and hiring has been a mess for decades, though, especially in tech.
It's like all the leetcode bullshit. We know that is not a valid measurement for actual performance in a dev job, but that doesn't stop managers from using it.
What they need is a number, a rating, on how much of a fit each candidate is, through some process that can be described as objective and fair. The algorithms provide that.
If we make this illegal, they'll just come up with some other bullshit.
In an ideal world, companies would assess each and every candidate individually and on their merits. But no-one has time or patience for that, so we have these bullshit systems.
Lots of monocultures exist in hiring even without an algorithmic scoring system. That's roughly how every stupid hiring fad works, and how it's always worked, because most employers have no idea how to identify great potential employees.
Hiring managers and companies choose algorithms and hiring fads because they don't know how to be really certain of who to hire, so they'll settle for either assuming someone else's expertise will save them, or for some rubric that "everyone is doing" so it "can't be that bad".
When I first became a hiring manager, I was working for a public university. Our salaries were limited, being staff rather than faculty and being public servants, to between 1/3 and 1/2 the going salary for equivalent cybersecurity professionals in the private sector. I did not have the option to hire the people everyone else was trying to hire. I also faced one of the key risks of working in a public institution: once you keep someone past their probationary period, it is very, very hard to fire them. So, it's important not to get it wrong. I learned some things that I have carried forward into every hiring manager or senior leadership role since:
1. I base hiring practices on Manager Tools behavioral interviewing systems (https://manager-tools.com). No affiliation, they've just made my work life better.
2. I became really good at understanding what my team or organization really needs. Most hirers focus way too much on "years of experience" and specific technologies than is usually wise. As my favorite former supervisor said, "I can teach a smart person cybersecurity, but I can't teach a dumb [or unmotivated] cybersecurity person to be smart."
3. I became really good at developing people, and ensuring that the managers under me were as well. We couldn't lay someone off just because their technical specialty became irrelevant, so we couldn't afford to hire people who weren't lifelong learners, or to fail as coaches to ensure that learning was always taking place.
4. I cast as wide a net as my HR and regulatory overlords would let me (and now, as a business leader, I cast a huge net). I look for things that aren't just useful at the moment, but are useful long term, in my candidates. I don't care about pedigree.
I end up paying less for good employees due to simple supply and demand: I often go for the diamonds in the rough that don't have 10 competing offers.
I end up having really good employees who generally stay with me long term, because I apply long-term thinking in hiring, and structure my teams around constant learning and development.
I dodge a LOT of bullets... people who have just the right pedigree to look like great hires worth a lot of money, but who'll disappoint me until the day they leave.
When it's a tight labor market -- too few candidates for roles I care about -- I'm tapping a hiring market that other managers aren't aware enough of, and still finding talent while they have roles that sit open for months.