We made the jobs public, and the hard part was never the scraping

A job link should just open

Until today, a Remoet job link mostly did not work for the person you sent it to. They would click, hit our signup form, and never see the job. Right role, right company, wall first. (If you have ever closed a tab the instant a signup form appeared, you know exactly how that story ends.)

That is the whole problem in one click. We had built a catalogue of tech job listings pulled straight from companies' hiring infrastructure, and then we put a door in front of it. Every job board does this. Hold the inventory, trade it for an email address. Here is the uncomfortable bit: I never actually decided to do it. It was a default, and defaults have a way of quietly becoming policy while you are looking somewhere else.

So I made a real decision and wrote it down: as much of our job inventory as possible should be public, unless there is a clear legal reason not to. Public meaning no account, no login, readable by a person, by a search engine, and by an AI agent.

That ships to production today. What follows is the part nobody warns you about.

Scraping was never the hard question

I expected to spend this project arguing about scraping. I did not. Job postings are published so they will be found. A company puts a role on their careers page precisely so a candidate, or a candidate's agent, will read it. Reading it is what it is for.

Re-publishing is a different act entirely. Putting someone else's job posting on your own site, under your own layout, next to your own navigation, is not reading. That is the thing that needed an actual answer, and "well, it was on the internet" is not one.

What we landed on is an output contract, deliberately narrow. We publish the facts of a role (title, location, salary when it is stated, the tech stack), a summary we wrote ourselves, our own enrichment on top, and a canonical link to the employer's own posting.

We do not publish the employer's prose. Not a paragraph of it. The employer keeps the canonical page and we send people (and agents) there. Facts about a job are not the employer's copy, and a link back is not competition, it is traffic. That is the entire argument. Everything else in this project was the work of making it true host by host.

Platforms first, because platforms are legible

The easy tier is applicant tracking systems. When a company posts through an ATS, the terms and the robots signals belong to the platform, not to the company, and they sit in one place you can actually read. So we read them.

Five are cleared for publishing: Lever, Greenhouse, Ashby, Recruitee, Workable. Everything else stays dark.

Dark by default is the refrain for the rest of this post, so it is worth naming what it costs. Adding a platform is not a config change and we refuse to treat it like one (no six line pull request, no "looks fine, shipping it"). It is a legal decision that happens to end in a config change.

Then the pile of custom career pages

Companies running their own careers pages are the interesting half, because every host is its own jurisdiction. There is no single document to read once and apply to a thousand companies. There is one site, its own robots.txt, its own terms, its own opinions.

So each host earns a clearance report before it publishes anything, built from three pieces of evidence: robots.txt parsed properly per RFC 9309 (the robots exclusion standard), the actual terms-of-service text, and Content-Signal declarations, a newer robots.txt extension where a site states machine-readable preferences like search=yes or ai-train=no.

Content-Signal turned out to be the most useful thing in that pile, because it lets a site say the specific thing it means. Plenty of sites want to be in search results and do not want to be training data. Those are different asks, and it is a relief to have a vocabulary that separates them. Our policy: if a site permits search and only reserves AI training, that is compatible with what we do (we index and we link, we do not train on anyone's content). An explicit search=no or ai-input=no and the host stays dark.

Then a person reads the report. The report never clears anyone. It only recommends.

The robots.txt bug we wrote ourselves

Our own scanner was wrong, and wrong in the direction that feels safe.

It ignored Allow-rule precedence. Under RFC 9309 the longest matching rule wins, which means a specific Allow can beat a broad Disallow. A site saying "disallow everything, except allow the careers path" is telling you plainly what it wants. Our scanner heard the first half and kept the host dark.

The instinct when you find that is to shrug, because being stricter than the standard sounds harmless. It is still a bug. We were ignoring what sites explicitly told us was permitted, and a policy that cannot hear yes is not caution, it is a broken parser with good marketing. Fixed.

The false positives were genuinely funny

We keyword-scan terms-of-service pages to decide what needs a careful read. Keyword scanning terms of service is a comedy generator.

An ATS vendor's marketing page tripped on "automated resume parsing", which is a feature they are proud of. A form tool tripped on its own product copy about "automated workflows", also a feature, also on the pricing page. A game studio flagged on an anti-cheat clause about autoclickers (I would pay money to watch someone argue that reading a careers page is autoclicking).

None of those were prohibitions aimed at anything like a job index. Every one of them needed a human to open the page and read it, which is the part you cannot automate away. Separating product copy from a real prohibition takes reading.

How the pile broke down

47 hosts reviewed. Roughly 30 cleared and can publish. The rest:

9 stay dark for genuine legal signals. Real anti-scraping clauses in the terms, or a robots.txt disallowing the careers paths. These are the easy no. Someone said no in the place you are supposed to say it, and we listened.

5 stay dark because we could not fetch their robots.txt at all. Timeouts, strange redirects, hosts that simply would not answer. If you cannot rule out a Disallow, you stay dark. Absence of evidence is not consent. I do not know whether those five are hostile, misconfigured, or just briefly asleep, and until the file fetches I am not going to guess.

4 were not legal problems at all. Dead job sources. One "board" whose jobs were links to a PDF sitting on a file host. One company that had quietly shut down and left the lights on. Nothing to adjudicate, just data rot wearing a legal costume.

One more lesson cost us more time than any clause. An ATS host serves a single-page app shell that returns a plausible page for any slug you invent, real or not. Type nonsense, get a 200 and a tidy layout. So a resolving URL proves nothing at all. Validate the board, not the URL: prove there is data behind it, or all you have proven is that you can spell.

What is public, and what a star is actually for

The gate in code is boring, which is how you want a gate. A job is publicly readable only if its platform is on the cleared list, or its host carries an explicit per-host cleared flag. Everything else needs an account and a star, exactly as before.

It went to staging first, where roughly 190 additional jobs from cleared company pages joined the public board overnight and nothing caught fire. Production is today, which puts more than 8,000 roles in front of anyone with a browser and no account.

If you already have an account, nothing you had gets worse. A star was never about access. It is about delivery: new roles from a company you starred come to you, in your feed and through your agent, including from the platforms and hosts that are not publicly readable. Roughly 40 percent of roles sit on platforms that stay dark, so starring still unlocks real inventory, and it still does the only job it ever had, which is bringing you roles from the companies you love without you going and looking.

Tell me where this is wrong

I am fairly confident about the output contract. I am much less confident that our reading of Content-Signal is the one the ecosystem will converge on. It is a young standard, we are interpreting it in good faith, and interpretations drift. If you run a careers page and we have read your signals wrong, I would rather hear it from you than keep guessing.

So go send someone a job link that just opens, then come tell me what you think. Is default-open with a legal gate the right instinct, or does hiding inventory behind a wall protect something I am not seeing? I read everything in Discord. Reply here too, I answer those.