Source: a small wiki I help manage, for an obscure game with <10k players, recently had to disable new signups, because the spam was so bad (and it was stuck on an old version of MediaWiki, which didn't have CAPTCHA-support).
On a popular wiki, and it sounds like this one was fairly popular, I imagine even CAPTCHA's won't be enough to stop wiki spammers. If those spammers were posting more than just "buy my penis pill" garbage (e.g. they were putting links to malware sites), Google probably, and somewhat legitimately, saw them as a source of such malware.
I imagine the fix for the OP is a thorough audit/cleansing of all malicious content on the wiki, followed by some sort of appeal to Google (which will no doubt take months, if they even respond at all, because ... Google).
Really OP's only hope is that the Google team responsible for this has an Italian Pokemon fan; otherwise they are probably screwed.
We have very good anti-bot system set up with a good number of Cloudflare fine-tuned rules, limited permissions for newly created accounts, and a very dedicated team of volunteers that patrol the recent edits constantly. I cannot exclude that somewhere on a rarely visited page (out of 37k+) there is a spam link, but I doubt it’s the reason for the deindexing. I think this would also appear on the Google Search Console.
I'd still recommend doing searches for common spam topics to see if you have "bad" stuff. On our wiki everything looked fine until you searched for (say) "finance" (which most users never would) ... and then you'd find a mess of spam finance stuff.
As for whether it's responsible or not, obviously I don't know. What I do know is that, without all the info, "Google saw malicious content on your wiki" is a far more logical theory than "Google just decided to hate us out of the blue".
Do you have any basis for saying that this wiki is overrun with spam, or are you just hand-waving? They were explicit in their Twitter thread about not being full of AI slop, and that they checked their list of pages that were marked as 'crawled but not indexed' and found no abuse.
I understand that you were taken aback by spam attacks on the wiki you help manage, but it's not reasonable to generalize from yours to theirs.
AI slop wouldn’t be on my top ten list of annoying wiki spam, having been the one dealing with such things in the past. You can be free of slop and still overflowing with spam.
Social sites should have all have a tree-based invite system. This would allow wiping out spammers and their enablers in a single hit. It would allow vetting of good actors too.
I feel like the dream solution is more like tree-based content: you see content that is vouched for by people you vouch for; if someone's account is compromised then their vouches get updated to not matter anymore, cutting their whole tree off at the root to make it invisible. Spammers should end up in largely disconnected components of the trees.
Pretty much what xwitter / bsky is on the following page.
The algorithm layer atop twitter was pretty good connecting me with people/content before daddy elon came along. And this algorithm layer is actually needed (in my view) to make the social network thing work, otherwise there is no critical mass
Thankfully bsky is not that good, so I don't get hooked by it at all. But i miss it
- You can't vouch for downstream invites, so the tree aspect isn't useful.
- It's not your fault if someone's account gets taken over by a spammer.
- Just because you vouched for someone once doesn't mean you vouch for them in the future.
- What should the punishment be if you accidentally invite a bad actor?
- Your community has to be large and desirable enough for people to bother. The vast majority of sites will die before anyone cares about jumping through hoops.
Addressing issues like these ends up kinda defeating the ideals of the proposal and regresses it into a mechanic that simply makes it harder to register. Which might be useful wrt anti-spam, but it has its own issues, like people having to constantly grovel for invites, shutting out earnest contributors, etc.
You still need criteria to handle reputation: does an account invited years ago and now spamming affects the reputation of the inviter, how much? What about the hacked accounts?
For small platforms it makes a lot of sense, for larger the potential for abuse is still there in different forms.
Yes Lobste.rs is great but much more limited in conversational scope. I don’t think the content on each site is directly comparable. The sites are not equal in audience and intention. For example, Lobste.rs doesn’t allow rampant evangelism or want to attract start-ups and thus doesn’t attract a more “spammy” crowd.
An organization I'm involved with has had to add Anubis (https://github.com/TecharoHQ/anubis) because of the recent wiki attacks from LLM scrapers. It's finally fixed our outages.
Why would Google need to direct traffic to the website when they've already scraped and trained their models on the data? Content creators and legitimate websites were wham-bammed and thank-you-ma’amed.
Personifying Google in this way is not realistic. The search team alone at Google is made of thousands of people who are all working on different things with an over-arching mission of making the web MORE accessible, not less. Any release from any of those people could have created a side effect of this kind. Is there a chance it was an intentional policy implementation? Sure. But the odds are heavily against it.
This seems akin to saying that humans shouldn't be personified because their brains are made of millions of neurons that are all doing different things. But the actions or motives of individual processing units are hardly relevant, especially at the scale of The Google. We don't need to speculate how non-malevolent individuals cause harmful side effects. It doesn't even matter what The Google "thinks". The system is what it does, and what it does is consistently operate in ways that are not for the mere benefit of users of the Web. The conceptual model that The Google hates (or is callously indifferent to) us makes far better predictions than a model presuming thousands of people make mistakes while trying to make the Web more accessible. It doesn't matter if the former model isn't a technically perfect reflection of reality. We are less likely to be victim to The Google when we act as if it is a hostile force. Diffusing the results of its actions across thousands of nameless humans increases the risk that one finds themself posting on HN or X about how The Google spontaneously locked them out of their entire life.
So same thing ad-block users have been doing for 20 years now?
Edit: You can downvote, but you can't tell me the difference, can you?
Edit 2: Funny how when you call out ad block users for denying creators revenue, they go on about how the internet was fine in '96, how no one should expect anything for putting content online, or how it's their computer so they can chose what loads on it. Where did those arguments go?
Users take part and improve wikis, it's the whole model. If they don't take the adverts, they still can contribute. Googlebot isn't making edits, not even giving signal to the site about what is useful allowing the owners to hone the site.
Two ways in which issues who have adblock are better than bots.
Users will promote organically, which can win more credence than even a higher listing in SERPs. Depends if your wiki is part of building a community.
Because while your argument sounds nice, if you break out the numbers, it becomes largely meaningless. In fact you find that the average internet user, especially in the tech/gaming space, usually contributes nothing, while watching/loading no ads and self congratulates themselves for doing so while encouraging others to do the same.
The difference is that I am not preventing anyone else from finding their content. I whitelist ads on sites that have good ad policies, like limiting ad size, labeling ads, and not allowing animated ads.
Advertisers only care about attention, if you don't impose editorial standards they'll contaminate your entire site.
On air reads. Lift a finger for your ads. When I spent more time producing podcasts I categorically rejected (and discouraged my clients from doing) injected ads by 3rd parties. They scream “idgaf” and actual on air reads convert better anyway by huge margins in comparison.
Ublock origin et al can’t block those so there’s your solution. Don’t lazily monetize your content.
well I didn't downvote but there is an obvious difference in thousands of uncoordinated people doing something whenever it benefits vs. a large organization with automated resources doing things at the kinds of speeds and volumes that automation allows.
The same thing happened with my blog a few weeks ago. It was well referenced for years and suddenly almost all of my entries are not indexed anymore. The Search Console indicates that the URLs were crawled but are currently not indexed, and contrary to technical problems, there nothing I can do to fix it, I just have to accept that most of my articles cannot be found via Google anymore.
EDIT: I don't actually think it is related, but now that I think of it, the timing corresponds with when I started setting up TDMRep to forbid using my content to train LLMs.
Same. I've been running a personal blog for over 20 years. Last year, I couldn't find any links to my blog on Google. Went to Google Search Console to find all my links are "Crawled by not indexed", with no reason given.
If Google already slurped up any training data from your site, then not indexing it probably gives them something of a moat over anyone using Google search for site discovery.
To be honest it's probably just jank on Google's end.
There's a lot of delayed cause and effect in search, and it's much easier to make a minor mistake that excludes 0.1% of websites from crawling or indexing than it is to detect that it's happened except from affected websites telling you about it.
Like in marginalia I've had a bug that affected websites in the condition that if the root path didn't support HEAD, but did support GET with a `Range` header, and it correctly responded with a HTTP 206, then the website wouldn't be indexed because some code that was testing the root document for issues as an initial probe handled that as an error state. Most websites that support range requests also support HEAD (as this usually means the document isn't generated). Except a handful of Caddy-based configurations, about 0.3% of servers.
Google neither hates nor loves any of us, the only thing it cares about as an institution is cramming as many advertisements in front of as many people as it can get away with to generate increasingly ridiculous piles of money.
This is not meant to be a defense of Google, which is (like virtually every large corporation) completely sociopathic.
Public corporations have historically been multi-cellular biological organisms made up of individual cells working toward the collective goal of continuing the organism's existence. They're probably most analogous to bee hives.
Each cell receives nourishment from the corporation in the form of monetary compensation (and other benefits). Some cells have a more direct role in the "reasoning" process of the organism than others, depending on their logical position within the corporation.
The corporations aren't sentient in the collective, though it can be argued many of their constituent cells are. The corporations are able to influence their environment using individual constituent conscious calls to communicate with similar cells in other organisms.
Ultimately, the corporation itself has the goal of producing value for its owners, since its owners provide the working capital necessary for the corporation to function.
The methods corporations use to achieve their goal of returning value can be opaque to the owners and potentially inscrutable to the individual constituent cells. Their "reasoning" is a manifest property coming from the interaction of the cells with their environment, each other (both within and outside the corporation), and other organisms.
If the owners stop receiving value and withdraw their working capital the corporation becomes unable to nourish its cells and it dies.
Recently these organisms have become biological / technological hybrids, incorporating unconscious computational models in their reasoning process. This change increases the inscrutability and opacity of the reasoning process. It's likely the unconscious computational models will eventually be tasked with communicating with similar models in other organisms, at which point the inscrutability will probably increase by an even greater amount.
I think this analogy is flawed. Corporations cannot exist without laws pertaining to them. They're made up of _laws_. The individual components all have actions dictated to them by these laws.
> If the owners stop receiving value the organism becomes unable to nourish its cells and it dies.
Owners are people. They're vulnerable to sentiment. There's plenty of failing businesses with their doors open for this reason.
You're attempting to rationalize something in biological terms that's somewhat irrational in logical terms.
> You're attempting to rationalize something in biological terms that's somewhat irrational in logical terms.
I'm mainly riffing for fun. Corporations are irrational because all the actors are irrational.
Their irrationality, unpredictability, and adaptability to regulation, particularly when they're hulking daikaiju-like monstrosities shambling thru society wantonly smashing their tails into other institutions and social infrastructure (or mating with other entities to create super-monstrosities), is what's troubling to me.
> I think this analogy is flawed. Corporations cannot exist without laws pertaining to them. They're made up of _laws_. The individual components all have actions dictated to them by these laws.
The law is a component of the environment. The law binds the corporation together, but it also constrains and shapes how it can act.
Sometimes corporations allocate resources to get the law changed (altering their environment to suit their manifest desires), other times they adapt (changing business processes to achieve legal compliance), and sometimes they act counter to the law, likely because some individual cells working in a reasoning capacity will have significant individual gain and very little individual risk (Dieselgate, or maybe the subprime crisis of 2008).
> Owners are people. They're vulnerable to sentiment.
Some owners are people, and some of those people are vulnerable to sentiment.
Some owners are themselves inscrutable manifest organisms (other corporations, pension funds, insurance companies, nationalized interests, and myriad other forms of semi-autonomous masses capital). Individual people own those entities, but I'd argue ownership by-proxy dilutes individual sentimentality.
Increasingly I see marketing as akin to LLM training. We are all being trained (by activating and reinforcing neural pathways in our meaty heads) to respond to certain stimuli in a certain way (e.g.: at the store, select _this_ brand of soap).
I'm not sure most of those calling the shots at Google realizes they even have paying customers other than advertisers. Notably though, website publishers and consumers of their massive products like Search, Gmail, and Android are not really customers.
A bit outside of HN's typical domain, but I wonder the treatment that top ad buyers get from Google.
Like if you are the VP of advertising for Procter & Gamble, and your nephews gmail account gets banned, are these guys treated so specially that they can get a white glove unban just like that? I wouldn't be surprised for Googles golden geese if they can
I wonder about that. Part of me thinks maybe they might not bother, because what are they gonna do, just not advertise? The entire online ad market is basically Google and Meta now. You'd get fired if you pulled all your ads because Google didn't do you a favor.
But on the other hand, I think they might definitely do that kind of thing as more of an old-boys-network type thing.
Google's always adjusting its search rankings, but it's rare for a legitimate site to suffer such a sudden massive hit without reason.
My first thought would be that they accidentally blocked Google's crawler (maybe through some kind of anti-AI setting?) or that Google believes that the site is serving malware or spam. Either scenario can have that kind of effect. I can see that their forum at least appears to have strong Cloudflare anti-bot rules in place, so that might be the case.
They're also using a subdomain for both their wiki and forum, which Google has been observed to punish. They might consider moving each of those to their own separate .com domain.
But aside from that usual stuff, there's one more possible reason that's specific to this site. In November of last year, the Pokemon Company rebranded their "Pokemon Trainer Club" to "Pokemon Trainer Central", which is the first result that comes up when you search for "Pokemon Central".
That change was made a few months before the sudden drop in traffic, but could still be a viable explanation here. Google does routine re-ranking on a daily basis along with occasional major re-ranking, which happens maybe a few times a year, so the delayed hit that they saw could have come from Google finally recognizing that most people who search for "Pokemon Central" are no longer looking for the wiki like was once true in the past.
A blocked Googlebot would’ve caused a report of 403s in the Search Console, which isn’t the case. And the subdomain has worked perfectly fine for the last 15 years.
You may have a point with the Trainer Central rebranding, but please consider this in the context of Italian language results. It’s not about reaching the home of the wiki (which is pretty much the only page that’s still indexed), it’s all the other search queries (Pokémon names, moves, games, etc - without adding “pokemon central” even) that usually returned our wiki’s dedicated page as first result (or top 5 at least) and now those specific pages are not even indexed anymore.
> They're also using a subdomain for both their wiki and forum, which Google has been observed to punish. They might consider moving each of those to their own separate .com domain.
Any sources for this? AFAIK, Google treats websites on a subdomain as a separate entity.
> Things we believe you shouldn't focus on: As SEO has evolved, so have the ideas and practices (and at times, misconceptions) related to it. What was considered best practice or top priority in the past may no longer be relevant or effective due to the way search engines (and the internet) have developed over time.
> Subdomains versus subdirectories: From a business point of view, do whatever makes sense for your business. For example, it might be easier to manage the site if it's segmented by subdirectories, but other times it might make sense to partition topics into subdomains, depending on your site's topic or industry.
Doesn't quite explicitly say it treats them the same, but it kinda implies it.
Google have also blatantly lied in the past about SEO. My familiarity with the subject is about a decade old, but I don't expect Google to have improved in that regard, quite the contrary. I don't think you can take anything in their guidelines as 'truth that will help for SEO'.
After yesterday's keynote and the changes to Search, it became clear in the near future, Google will cease to direct any traffic to websites and the search results will just become a footnote in Gemini's response.
Wikis are just high-risk for SEO. Getting my own personal wiki to be indexed was such a challenge that I'd just about given up when a friend who is more acquainted with the whole thing helped me make sure I had all the bits and bobs in the right place. If you're not careful, people can easily put spam all over your site and then it'll really ruin your presence on a search engine.
Google is really big, though. Really really big. They're so big that not even all the people inside Google are trustworthy to them on a subject like this.
But they don't universally hate wikis and so on. It's just you have to do a lot of work and make sure you don't have spam on your wiki, and then fill in all of the information in your meta tags, and have a sitemap.xml, and all that. Here's my wiki for example: https://wiki.roshangeorge.dev/w/images/8/89/Screenshot_-_Goo...
Tangential: this would be an excellent time for the Kagi folks to do a Mullvad style campaign around "Remember Google before the apocalypse? Meet Kagi."
I still have 42k page indexed, but previously I had 20k impressions per day, past week impressions started to change. And now I have 399 impressions per 24 hours :/
I suspect this is a cloudflare thing since the other search engines are doing fine. I'd look closer into your cloudflare settings and see what you can relax.
One of the sad things about this story is that everyone has to read tea leaves to guess what reasoning might be going on at Google's end. Tech companies have normalized the practice of cutting people off with no explanation, or saying 'we investigated and found a violation' without articulating what the violation is. Naturally, they want to secure themselves against abuse and people trying to game their system, but refusing to provide any information does not achieve that.
It does infuriate legitimate users, enables other kind of abuse and scamming (eg immunize yourself against delisting with this one weird trick!', link farming etc), and act as a fig leaf for abusive behavior by platform operators. Effectively, we've allowed large teach companies to act as digital dictatorships with no accountability to their customers. Yes I consider users to be 'customers' even if they're uploading content or doing searches 'for free'. If you're monetizing their activity on your platform, they are your customers whether or not you call them that to avoid legal liability.
It appears for me when I search for it. Even Gemini is cool with looking for it.
Here is a part of the Gemini result I got which was directly above the regular result link.
"Pokémon Central is a major community network and independent Italian encyclopedia for everything Pokémon-related"
Honestly, the title is super clickbait and it doesn't even reflect reality.
Its so easy picking some giant entity far away and create some drama about it. Dont get me wrong, I am not a google fan, but I also dislike clickbaits and whiney dramatic claims, moreover if unverified.
My guess is that the combination of Wiki and Pokemon is highly suspect for Google.
The Pokemon Industrial Complex has advanced astroturfing especially on YouTube/Twitch, where streamers mention the damn things in any second episode, they "accidentally" meet people going to Pokemon conventions in live streams and so on.
I think it's become increasingly clear that search engines, in their most basic form, aren't cut out for the internet of today. The open internet is basically dead and I see the world returning to old yahoo-like web page indexes that are manually verified and sorted
Kagi has been mentioned already, just to provide anecdotal reference: searching for "pokemon wiki" with the country set to Italy shows OP's website as first result.
Pokemon Central runs ads (Google AdSense at that!), which is probably how they pay for everything.
Google is likely their biggest inbound source of traffic, so they're probably experiencing a marked revenue drop as well.
It's unfortunate that so many livelihoods are subject to the capricious whims of a single company. A company that is increasingly seeking to keep users on their engine without sending eyeballs or revenue to any third parties at all.
We're watching Google's "embrace-extend-extinguish" arc for the web. It's not over by a long shot, but they absolutely intend to finish the job.
Why is it a lot of work? Could you specify some
off the more difficult effort? Wouldn’t LLMs help speed this up? This is the one area where I’d think Llms could really take Google down by empowering in house ad platforms.
Depends on how in-house you want to go. If you go full in-house, you'll need sales staff to make deals with advertisers. You'll then need a way of hosting the media provided. You'll need a way to deal with media that does not match what you've requested. You'll need a system to allocate ad space accordingly to contracts with ad clients. It's like a whole new department in your company.
I don’t have a sales staff. I just call a company, tell them why I’m calling and what’s the opportunity, crack some jokes, get serious and make them an offer. It helps that I’m in a niche, thoroughly know the sector and that they most likely already know my websites. As long as I can get to the owner of the company, then I’m golden.
We would like the wiki to be free of ads, but hosting costs at our scale are real. Since we don’t like ads either, we compromise like this: users can register for free and never see an ad (they are only served to anonymous visitors); they can also use an ad blocker and we won’t bug them about it.
No, I think "us" is apt, considering this will eventually affect all sites that rely on traffic from Google search, which is basically every text heavy site.
All we can hope for is that people will stop using search (after eventually having enough of the AI wave) for these sort of niche sites and will bookmark and access them directly in future. I don't have much hope.
I spend a lot of time wondering about the true role of search in people's lives in 2026. When I watch people use the Internet, it seems like most of them perform searches simply as fuzzy-matching navigation to the websites they use. Like the way many people use Spotlight to launch desktop apps.
I think this is because
(A) bookmarks lists are inconvenient - scrolling to find a bookmark is slower than typing "youtube" or (cringe) "bank of america" in the URL bar
(B) typing URLs directly requires precision of memory with TLDs being numerous and even things that were once predictable are now mere suggestions (e.g. is your city or town at cityofwhatever.com? city.org? city.gov? Could be anything!)
(C) related to (B) if you screw up a full URL you may well end up at a phishing site that looks like the site you wanted.
I really believe that 90% of Google and Bing searches today are probably for the names (or misspelled or partial names) of the top 100 websites.
If the dominant browsers weren't Google Chrome and Mobile Safari (who gets paid by Google for every search) browsers would build bookmarks for you of your frequently-used sites, and ordered by frequency of visits, present those for direct navigation when you type a word in the search bar, and not send any query to a search engine if you chose one of those. But all incentives point very strongly against doing that and toward sending you to a SERP with 13 ads and an "AI Overview" above the organic results.
Perhaps they will investigate why 541,000 pages aren’t being indexed. In my experience, Google provides adequate tools for identifying and resolving indexing issues.
Google won’t serve pages it hasn’t indexed. Seems they left a lot of relevant details out of that tweet.
Edit: and the most likely answer would be that their current robots.txt disallows virtually all indexing. I’m no SEO expert but entries like this seem like footguns:
User-agent: Google-Extended
Disallow: /
Edit 2: there’s more info in the full thread but that was only viewable via the xcancel link someone else shared (despite having the X app installed - deeplinks don’t work today). A helpful example of why X is not the best platform for sharing multi-post threads. Seems robots.txt was considered but ruled out.
In the first image you can see how indexed pages go from 40k+ to 11 in the matter of days. Further down the thread I show how 114k+ pages are marked as “crawled but not index” and we can’t understand why. The rest is stuff that is (correctly) blocked by robots.txt.
They're a wiki. Wiki spammers are relentless now.
Source: a small wiki I help manage, for an obscure game with <10k players, recently had to disable new signups, because the spam was so bad (and it was stuck on an old version of MediaWiki, which didn't have CAPTCHA-support).
On a popular wiki, and it sounds like this one was fairly popular, I imagine even CAPTCHA's won't be enough to stop wiki spammers. If those spammers were posting more than just "buy my penis pill" garbage (e.g. they were putting links to malware sites), Google probably, and somewhat legitimately, saw them as a source of such malware.
I imagine the fix for the OP is a thorough audit/cleansing of all malicious content on the wiki, followed by some sort of appeal to Google (which will no doubt take months, if they even respond at all, because ... Google).
Really OP's only hope is that the Google team responsible for this has an Italian Pokemon fan; otherwise they are probably screwed.
We have very good anti-bot system set up with a good number of Cloudflare fine-tuned rules, limited permissions for newly created accounts, and a very dedicated team of volunteers that patrol the recent edits constantly. I cannot exclude that somewhere on a rarely visited page (out of 37k+) there is a spam link, but I doubt it’s the reason for the deindexing. I think this would also appear on the Google Search Console.
I'd still recommend doing searches for common spam topics to see if you have "bad" stuff. On our wiki everything looked fine until you searched for (say) "finance" (which most users never would) ... and then you'd find a mess of spam finance stuff.
As for whether it's responsible or not, obviously I don't know. What I do know is that, without all the info, "Google saw malicious content on your wiki" is a far more logical theory than "Google just decided to hate us out of the blue".
Do you have any basis for saying that this wiki is overrun with spam, or are you just hand-waving? They were explicit in their Twitter thread about not being full of AI slop, and that they checked their list of pages that were marked as 'crawled but not indexed' and found no abuse.
I understand that you were taken aback by spam attacks on the wiki you help manage, but it's not reasonable to generalize from yours to theirs.
AI slop wouldn’t be on my top ten list of annoying wiki spam, having been the one dealing with such things in the past. You can be free of slop and still overflowing with spam.
Weird Gloop (wiki host, started with runescape but now has dozens) has blogged about this https://weirdgloop.org/blog/clankers
Social sites should have all have a tree-based invite system. This would allow wiping out spammers and their enablers in a single hit. It would allow vetting of good actors too.
I feel like the dream solution is more like tree-based content: you see content that is vouched for by people you vouch for; if someone's account is compromised then their vouches get updated to not matter anymore, cutting their whole tree off at the root to make it invisible. Spammers should end up in largely disconnected components of the trees.
Pretty much what xwitter / bsky is on the following page. The algorithm layer atop twitter was pretty good connecting me with people/content before daddy elon came along. And this algorithm layer is actually needed (in my view) to make the social network thing work, otherwise there is no critical mass
Thankfully bsky is not that good, so I don't get hooked by it at all. But i miss it
How does new content or content from new accounts get seen by anyone?
People search for a thing, find said thing, then share it themselves under their own name. You know, like how conversations work in real life
iykyk rules
It doesn't solve as much as it sounds.
- You can't vouch for downstream invites, so the tree aspect isn't useful.
- It's not your fault if someone's account gets taken over by a spammer.
- Just because you vouched for someone once doesn't mean you vouch for them in the future.
- What should the punishment be if you accidentally invite a bad actor?
- Your community has to be large and desirable enough for people to bother. The vast majority of sites will die before anyone cares about jumping through hoops.
Addressing issues like these ends up kinda defeating the ideals of the proposal and regresses it into a mechanic that simply makes it harder to register. Which might be useful wrt anti-spam, but it has its own issues, like people having to constantly grovel for invites, shutting out earnest contributors, etc.
You still need criteria to handle reputation: does an account invited years ago and now spamming affects the reputation of the inviter, how much? What about the hacked accounts?
For small platforms it makes a lot of sense, for larger the potential for abuse is still there in different forms.
That’s literally how Facebook started
I remember begging my older step brother for an invite since he had the college email to get in
Now you just created a market for farmed "legit" accounts.
Yes, but the site operator can significantly increase the market price for such an account. This makes spamming more expensive.
Interesting to compare this site and lobste.rs for that
Yes Lobste.rs is great but much more limited in conversational scope. I don’t think the content on each site is directly comparable. The sites are not equal in audience and intention. For example, Lobste.rs doesn’t allow rampant evangelism or want to attract start-ups and thus doesn’t attract a more “spammy” crowd.
Lobste.rs has an invite based system however.
Both from safety and volume perspectives, I’d imagine. Openness has value.
An organization I'm involved with has had to add Anubis (https://github.com/TecharoHQ/anubis) because of the recent wiki attacks from LLM scrapers. It's finally fixed our outages.
Why would Google need to direct traffic to the website when they've already scraped and trained their models on the data? Content creators and legitimate websites were wham-bammed and thank-you-ma’amed.
Personifying Google in this way is not realistic. The search team alone at Google is made of thousands of people who are all working on different things with an over-arching mission of making the web MORE accessible, not less. Any release from any of those people could have created a side effect of this kind. Is there a chance it was an intentional policy implementation? Sure. But the odds are heavily against it.
This seems akin to saying that humans shouldn't be personified because their brains are made of millions of neurons that are all doing different things. But the actions or motives of individual processing units are hardly relevant, especially at the scale of The Google. We don't need to speculate how non-malevolent individuals cause harmful side effects. It doesn't even matter what The Google "thinks". The system is what it does, and what it does is consistently operate in ways that are not for the mere benefit of users of the Web. The conceptual model that The Google hates (or is callously indifferent to) us makes far better predictions than a model presuming thousands of people make mistakes while trying to make the Web more accessible. It doesn't matter if the former model isn't a technically perfect reflection of reality. We are less likely to be victim to The Google when we act as if it is a hostile force. Diffusing the results of its actions across thousands of nameless humans increases the risk that one finds themself posting on HN or X about how The Google spontaneously locked them out of their entire life.
lol.
>wham-bammed and thank-you-ma’amed.
So same thing ad-block users have been doing for 20 years now?
Edit: You can downvote, but you can't tell me the difference, can you?
Edit 2: Funny how when you call out ad block users for denying creators revenue, they go on about how the internet was fine in '96, how no one should expect anything for putting content online, or how it's their computer so they can chose what loads on it. Where did those arguments go?
Users take part and improve wikis, it's the whole model. If they don't take the adverts, they still can contribute. Googlebot isn't making edits, not even giving signal to the site about what is useful allowing the owners to hone the site.
Two ways in which issues who have adblock are better than bots.
Users will promote organically, which can win more credence than even a higher listing in SERPs. Depends if your wiki is part of building a community.
Does "users" refer to 100% of users or 0.05%?
Because while your argument sounds nice, if you break out the numbers, it becomes largely meaningless. In fact you find that the average internet user, especially in the tech/gaming space, usually contributes nothing, while watching/loading no ads and self congratulates themselves for doing so while encouraging others to do the same.
Users, ad-block users, and scrapers all consume the publicly-available content whether you like it or not.
I expect the difference is that the scrapers are the most likely to regurgitate the content one way or the other.
If there is any model on the internet that has proven you don't need to monetize through ads for a working business model, it's Wikipedia.
The difference is that I am not preventing anyone else from finding their content. I whitelist ads on sites that have good ad policies, like limiting ad size, labeling ads, and not allowing animated ads.
Advertisers only care about attention, if you don't impose editorial standards they'll contaminate your entire site.
In the tech space, using youtube as an example, tech youtubers, who are widely lauded, still have about 40-50% of users ad-blocking and <1% donating.
So thank you, but you are one of about 14 people on the internet who actually use a whitelist.
On air reads. Lift a finger for your ads. When I spent more time producing podcasts I categorically rejected (and discouraged my clients from doing) injected ads by 3rd parties. They scream “idgaf” and actual on air reads convert better anyway by huge margins in comparison.
Ublock origin et al can’t block those so there’s your solution. Don’t lazily monetize your content.
The edits are likely why you’re getting downvoted so much tbh.
You can run unblockable ads on your site.
You just have to not use third party integrations that run untrusted code on your visitors computers.
well I didn't downvote but there is an obvious difference in thousands of uncoordinated people doing something whenever it benefits vs. a large organization with automated resources doing things at the kinds of speeds and volumes that automation allows.
I was listening to David Bowie's Suffragette City as I read your comment (Apparently Bowie was a popularizer of 'wham bam, tym' usage)
The same thing happened with my blog a few weeks ago. It was well referenced for years and suddenly almost all of my entries are not indexed anymore. The Search Console indicates that the URLs were crawled but are currently not indexed, and contrary to technical problems, there nothing I can do to fix it, I just have to accept that most of my articles cannot be found via Google anymore.
EDIT: I don't actually think it is related, but now that I think of it, the timing corresponds with when I started setting up TDMRep to forbid using my content to train LLMs.
Same. I've been running a personal blog for over 20 years. Last year, I couldn't find any links to my blog on Google. Went to Google Search Console to find all my links are "Crawled by not indexed", with no reason given.
If Google already slurped up any training data from your site, then not indexing it probably gives them something of a moat over anyone using Google search for site discovery.
To be honest it's probably just jank on Google's end.
There's a lot of delayed cause and effect in search, and it's much easier to make a minor mistake that excludes 0.1% of websites from crawling or indexing than it is to detect that it's happened except from affected websites telling you about it.
Like in marginalia I've had a bug that affected websites in the condition that if the root path didn't support HEAD, but did support GET with a `Range` header, and it correctly responded with a HTTP 206, then the website wouldn't be indexed because some code that was testing the root document for issues as an initial probe handled that as an error state. Most websites that support range requests also support HEAD (as this usually means the document isn't generated). Except a handful of Caddy-based configurations, about 0.3% of servers.
You guys made the classic SEO mistake of building a real community site instead of a Reddit thread, a coupon subfolder, or an AI summary.
Scherzi a parte, spero che possiate recuperare presto…
Grazie! Speriamo anche noi.
I can even tell you that Google hates us all
Google neither hates nor loves any of us, the only thing it cares about as an institution is cramming as many advertisements in front of as many people as it can get away with to generate increasingly ridiculous piles of money.
This is not meant to be a defense of Google, which is (like virtually every large corporation) completely sociopathic.
Public corporations have historically been multi-cellular biological organisms made up of individual cells working toward the collective goal of continuing the organism's existence. They're probably most analogous to bee hives.
Each cell receives nourishment from the corporation in the form of monetary compensation (and other benefits). Some cells have a more direct role in the "reasoning" process of the organism than others, depending on their logical position within the corporation.
The corporations aren't sentient in the collective, though it can be argued many of their constituent cells are. The corporations are able to influence their environment using individual constituent conscious calls to communicate with similar cells in other organisms.
Ultimately, the corporation itself has the goal of producing value for its owners, since its owners provide the working capital necessary for the corporation to function.
The methods corporations use to achieve their goal of returning value can be opaque to the owners and potentially inscrutable to the individual constituent cells. Their "reasoning" is a manifest property coming from the interaction of the cells with their environment, each other (both within and outside the corporation), and other organisms.
If the owners stop receiving value and withdraw their working capital the corporation becomes unable to nourish its cells and it dies.
Recently these organisms have become biological / technological hybrids, incorporating unconscious computational models in their reasoning process. This change increases the inscrutability and opacity of the reasoning process. It's likely the unconscious computational models will eventually be tasked with communicating with similar models in other organisms, at which point the inscrutability will probably increase by an even greater amount.
> made up of individual cells
I think this analogy is flawed. Corporations cannot exist without laws pertaining to them. They're made up of _laws_. The individual components all have actions dictated to them by these laws.
> If the owners stop receiving value the organism becomes unable to nourish its cells and it dies.
Owners are people. They're vulnerable to sentiment. There's plenty of failing businesses with their doors open for this reason.
You're attempting to rationalize something in biological terms that's somewhat irrational in logical terms.
Bottom-line-up-front:
> You're attempting to rationalize something in biological terms that's somewhat irrational in logical terms.
I'm mainly riffing for fun. Corporations are irrational because all the actors are irrational.
Their irrationality, unpredictability, and adaptability to regulation, particularly when they're hulking daikaiju-like monstrosities shambling thru society wantonly smashing their tails into other institutions and social infrastructure (or mating with other entities to create super-monstrosities), is what's troubling to me.
> I think this analogy is flawed. Corporations cannot exist without laws pertaining to them. They're made up of _laws_. The individual components all have actions dictated to them by these laws.
The law is a component of the environment. The law binds the corporation together, but it also constrains and shapes how it can act.
Sometimes corporations allocate resources to get the law changed (altering their environment to suit their manifest desires), other times they adapt (changing business processes to achieve legal compliance), and sometimes they act counter to the law, likely because some individual cells working in a reasoning capacity will have significant individual gain and very little individual risk (Dieselgate, or maybe the subprime crisis of 2008).
> Owners are people. They're vulnerable to sentiment.
Some owners are people, and some of those people are vulnerable to sentiment.
Some owners are themselves inscrutable manifest organisms (other corporations, pension funds, insurance companies, nationalized interests, and myriad other forms of semi-autonomous masses capital). Individual people own those entities, but I'd argue ownership by-proxy dilutes individual sentimentality.
Increasingly I see marketing as akin to LLM training. We are all being trained (by activating and reinforcing neural pathways in our meaty heads) to respond to certain stimuli in a certain way (e.g.: at the store, select _this_ brand of soap).
Don't make the mistake of anthropomorphizing Google, as a wise man once said.
All large companies are sociopaths, but few tech companies treat their paying customers with the level of contempt that Google does.
I'm not sure most of those calling the shots at Google realizes they even have paying customers other than advertisers. Notably though, website publishers and consumers of their massive products like Search, Gmail, and Android are not really customers.
A bit outside of HN's typical domain, but I wonder the treatment that top ad buyers get from Google.
Like if you are the VP of advertising for Procter & Gamble, and your nephews gmail account gets banned, are these guys treated so specially that they can get a white glove unban just like that? I wouldn't be surprised for Googles golden geese if they can
I wonder about that. Part of me thinks maybe they might not bother, because what are they gonna do, just not advertise? The entire online ad market is basically Google and Meta now. You'd get fired if you pulled all your ads because Google didn't do you a favor.
But on the other hand, I think they might definitely do that kind of thing as more of an old-boys-network type thing.
I don't think that's necessarily true - some large companies are coöps, they don't seem to be sociopathic?
Google's always adjusting its search rankings, but it's rare for a legitimate site to suffer such a sudden massive hit without reason.
My first thought would be that they accidentally blocked Google's crawler (maybe through some kind of anti-AI setting?) or that Google believes that the site is serving malware or spam. Either scenario can have that kind of effect. I can see that their forum at least appears to have strong Cloudflare anti-bot rules in place, so that might be the case.
They're also using a subdomain for both their wiki and forum, which Google has been observed to punish. They might consider moving each of those to their own separate .com domain.
But aside from that usual stuff, there's one more possible reason that's specific to this site. In November of last year, the Pokemon Company rebranded their "Pokemon Trainer Club" to "Pokemon Trainer Central", which is the first result that comes up when you search for "Pokemon Central".
That change was made a few months before the sudden drop in traffic, but could still be a viable explanation here. Google does routine re-ranking on a daily basis along with occasional major re-ranking, which happens maybe a few times a year, so the delayed hit that they saw could have come from Google finally recognizing that most people who search for "Pokemon Central" are no longer looking for the wiki like was once true in the past.
https://gonintendo.com/contents/54863-pokemon-trainer-club-r...
A blocked Googlebot would’ve caused a report of 403s in the Search Console, which isn’t the case. And the subdomain has worked perfectly fine for the last 15 years.
You may have a point with the Trainer Central rebranding, but please consider this in the context of Italian language results. It’s not about reaching the home of the wiki (which is pretty much the only page that’s still indexed), it’s all the other search queries (Pokémon names, moves, games, etc - without adding “pokemon central” even) that usually returned our wiki’s dedicated page as first result (or top 5 at least) and now those specific pages are not even indexed anymore.
> They're also using a subdomain for both their wiki and forum, which Google has been observed to punish. They might consider moving each of those to their own separate .com domain.
Any sources for this? AFAIK, Google treats websites on a subdomain as a separate entity.
https://developers.google.com/search/docs/fundamentals/seo-s...
> Things we believe you shouldn't focus on: As SEO has evolved, so have the ideas and practices (and at times, misconceptions) related to it. What was considered best practice or top priority in the past may no longer be relevant or effective due to the way search engines (and the internet) have developed over time.
> Subdomains versus subdirectories: From a business point of view, do whatever makes sense for your business. For example, it might be easier to manage the site if it's segmented by subdirectories, but other times it might make sense to partition topics into subdomains, depending on your site's topic or industry.
Doesn't quite explicitly say it treats them the same, but it kinda implies it.
Google have also blatantly lied in the past about SEO. My familiarity with the subject is about a decade old, but I don't expect Google to have improved in that regard, quite the contrary. I don't think you can take anything in their guidelines as 'truth that will help for SEO'.
After yesterday's keynote and the changes to Search, it became clear in the near future, Google will cease to direct any traffic to websites and the search results will just become a footnote in Gemini's response.
Wikis are just high-risk for SEO. Getting my own personal wiki to be indexed was such a challenge that I'd just about given up when a friend who is more acquainted with the whole thing helped me make sure I had all the bits and bobs in the right place. If you're not careful, people can easily put spam all over your site and then it'll really ruin your presence on a search engine.
Google is really big, though. Really really big. They're so big that not even all the people inside Google are trustworthy to them on a subject like this.
But they don't universally hate wikis and so on. It's just you have to do a lot of work and make sure you don't have spam on your wiki, and then fill in all of the information in your meta tags, and have a sitemap.xml, and all that. Here's my wiki for example: https://wiki.roshangeorge.dev/w/images/8/89/Screenshot_-_Goo...
Tangential: this would be an excellent time for the Kagi folks to do a Mullvad style campaign around "Remember Google before the apocalypse? Meet Kagi."
I still have 42k page indexed, but previously I had 20k impressions per day, past week impressions started to change. And now I have 399 impressions per 24 hours :/
They sucked out all the content and then pulled the plug.
I suspect this is a cloudflare thing since the other search engines are doing fine. I'd look closer into your cloudflare settings and see what you can relax.
Interesting. My small game wiki was also affected ~3 weeks ago. It doesn't even show up on Google anymore even if you directly search for the URL.
We don't get any spam since there's no public signups for editing access.
oof that sucks; i really wish there was more info on why google decides to crawl or not crawl a page
One of the sad things about this story is that everyone has to read tea leaves to guess what reasoning might be going on at Google's end. Tech companies have normalized the practice of cutting people off with no explanation, or saying 'we investigated and found a violation' without articulating what the violation is. Naturally, they want to secure themselves against abuse and people trying to game their system, but refusing to provide any information does not achieve that.
It does infuriate legitimate users, enables other kind of abuse and scamming (eg immunize yourself against delisting with this one weird trick!', link farming etc), and act as a fig leaf for abusive behavior by platform operators. Effectively, we've allowed large teach companies to act as digital dictatorships with no accountability to their customers. Yes I consider users to be 'customers' even if they're uploading content or doing searches 'for free'. If you're monetizing their activity on your platform, they are your customers whether or not you call them that to avoid legal liability.
It appears for me when I search for it. Even Gemini is cool with looking for it.
Here is a part of the Gemini result I got which was directly above the regular result link.
"Pokémon Central is a major community network and independent Italian encyclopedia for everything Pokémon-related"
Honestly, the title is super clickbait and it doesn't even reflect reality. Its so easy picking some giant entity far away and create some drama about it. Dont get me wrong, I am not a google fan, but I also dislike clickbaits and whiney dramatic claims, moreover if unverified.
Maybe the claims in the Twitter thread weren’t clear enough; I’ve added some clarification especially for you: https://x.com/pokemoncentral/status/2057157781611024764
Thanks for the clarification, the title is still a whiney clickbait.
Do you suspect they have faked the search console screenshots as well? Make your accusation explicit.
My guess is that the combination of Wiki and Pokemon is highly suspect for Google.
The Pokemon Industrial Complex has advanced astroturfing especially on YouTube/Twitch, where streamers mention the damn things in any second episode, they "accidentally" meet people going to Pokemon conventions in live streams and so on.
Try to audit the Wiki if anyone abused it.
Now? Google has hated us since at least the DoubleClick aquistion in 2008. That's when people became the product
https://xcancel.com/pokemoncentral/status/205712380740463825...
Thank you
This might be useful: https://addons.mozilla.org/fr/firefox/addon/toxcancel (Redirects to xcancel.com (a mirror of x) when the browser is about to load an x.com page).
Can someone start a new Google, please? Just search, nothing more. I'm willing to pay 10 USD a month for that. API access included.
https://kagi.com/pricing
I think it's become increasingly clear that search engines, in their most basic form, aren't cut out for the internet of today. The open internet is basically dead and I see the world returning to old yahoo-like web page indexes that are manually verified and sorted
Kagi has been mentioned already, just to provide anecdotal reference: searching for "pokemon wiki" with the country set to Italy shows OP's website as first result.
Yeah Kagi already exists luckily, it’s extremely good and worth the money.
Perhaps they're decommissioning search in favor of LLM:s.
That's only supposed to happen later this week.
This aligns with their Google Zero doctrine, keep all info internal and make the goal for the user to hit 0 external websites.
Pokemon Central runs ads (Google AdSense at that!), which is probably how they pay for everything.
Google is likely their biggest inbound source of traffic, so they're probably experiencing a marked revenue drop as well.
It's unfortunate that so many livelihoods are subject to the capricious whims of a single company. A company that is increasingly seeking to keep users on their engine without sending eyeballs or revenue to any third parties at all.
We're watching Google's "embrace-extend-extinguish" arc for the web. It's not over by a long shot, but they absolutely intend to finish the job.
Hi EU. How about one of those lovely anti-trust cases?
I really hope eu can extinguish Google before they extinguish all websites. Will be an exponential death of website very soon once they lose traffic.
I'm worried about the Trump/Tech-Oligarch axis interfering with legal processes in the EU and preventing needed regulation.
We should've gotten out of US dependence decades ago.
It’s why I moved to in-house advertising. It’s a lot of work, but I hope it is the right decision.
Why is it a lot of work? Could you specify some off the more difficult effort? Wouldn’t LLMs help speed this up? This is the one area where I’d think Llms could really take Google down by empowering in house ad platforms.
Depends on how in-house you want to go. If you go full in-house, you'll need sales staff to make deals with advertisers. You'll then need a way of hosting the media provided. You'll need a way to deal with media that does not match what you've requested. You'll need a system to allocate ad space accordingly to contracts with ad clients. It's like a whole new department in your company.
I don’t have a sales staff. I just call a company, tell them why I’m calling and what’s the opportunity, crack some jokes, get serious and make them an offer. It helps that I’m in a niche, thoroughly know the sector and that they most likely already know my websites. As long as I can get to the owner of the company, then I’m golden.
Spiders Georg runs his company from a cave and buys 10,000 spiders every day; he is an outlier and should not be counted.
Its better off if ads go away. Just use ad blockers.
We would like the wiki to be free of ads, but hosting costs at our scale are real. Since we don’t like ads either, we compromise like this: users can register for free and never see an ad (they are only served to anonymous visitors); they can also use an ad blocker and we won’t bug them about it.
there are many laudable places such as yours, most auspiscious, deserving of reparations, alas such favourable anachronisms are unduly burdened .
i like to mail a fiver or two to a P.O.B.
Title could be: Apparently Google hates Pokémon Central Wiki now
(to be clearer what the source of the post is)
No, I think "us" is apt, considering this will eventually affect all sites that rely on traffic from Google search, which is basically every text heavy site.
All we can hope for is that people will stop using search (after eventually having enough of the AI wave) for these sort of niche sites and will bookmark and access them directly in future. I don't have much hope.
I spend a lot of time wondering about the true role of search in people's lives in 2026. When I watch people use the Internet, it seems like most of them perform searches simply as fuzzy-matching navigation to the websites they use. Like the way many people use Spotlight to launch desktop apps.
I think this is because
(A) bookmarks lists are inconvenient - scrolling to find a bookmark is slower than typing "youtube" or (cringe) "bank of america" in the URL bar
(B) typing URLs directly requires precision of memory with TLDs being numerous and even things that were once predictable are now mere suggestions (e.g. is your city or town at cityofwhatever.com? city.org? city.gov? Could be anything!)
(C) related to (B) if you screw up a full URL you may well end up at a phishing site that looks like the site you wanted.
I really believe that 90% of Google and Bing searches today are probably for the names (or misspelled or partial names) of the top 100 websites.
If the dominant browsers weren't Google Chrome and Mobile Safari (who gets paid by Google for every search) browsers would build bookmarks for you of your frequently-used sites, and ordered by frequency of visits, present those for direct navigation when you type a word in the search bar, and not send any query to a search engine if you chose one of those. But all incentives point very strongly against doing that and toward sending you to a SERP with 13 ads and an "AI Overview" above the organic results.
A wiki with only 11 pages?
Perhaps they will investigate why 541,000 pages aren’t being indexed. In my experience, Google provides adequate tools for identifying and resolving indexing issues.
Google won’t serve pages it hasn’t indexed. Seems they left a lot of relevant details out of that tweet.
Edit: and the most likely answer would be that their current robots.txt disallows virtually all indexing. I’m no SEO expert but entries like this seem like footguns:
Edit 2: there’s more info in the full thread but that was only viewable via the xcancel link someone else shared (despite having the X app installed - deeplinks don’t work today). A helpful example of why X is not the best platform for sharing multi-post threads. Seems robots.txt was considered but ruled out.In the first image you can see how indexed pages go from 40k+ to 11 in the matter of days. Further down the thread I show how 114k+ pages are marked as “crawled but not index” and we can’t understand why. The rest is stuff that is (correctly) blocked by robots.txt.