Talent marketplace matching

What you are designing

A company posts a role: skills, hours, rate, location constraints. You return a page of vetted experts who fit, best first, in under a second. Assume on the order of ten million profiles, edits every day, and that ‘best’ eventually means ‘likely to be hired,’ not ‘text looks similar.’ New experts have no hire history.

Requirements

• Filter by hard constraints (skill, level, rate ceiling, work authorization).
• Rank the survivors.
• Reflect a profile edit or a ban within a short, stated lag.
• Explain a result well enough that a recruiter trusts it (matched skills, not a mystery score).

A missing great candidate is bad. Showing a banned expert, or the wrong person’s private data, is worse.

Entities and API

Expert, Skill(expert_id, skill, level), Role, Impression(role_id, expert_id, features, position), Outcome(impression, hired).

POST /v1/roles/{id}/search     { filters, cursor } -> ranked experts
PATCH /v1/experts/{id}         profile edits

Search is a POST because the filter body does not belong in a cacheable query string, and it is still safe to retry. The cursor is opaque and tied to one ranking snapshot. Offset pagination will show the same expert twice when someone edits a profile mid-scroll. A banned expert is absent even if the index still contains them, so the serve path checks the ban and does not wait for the next index build. Profile PATCH authorizes the caller against that expert id. Another person’s id is 403.

High-level design

flowchart LR
  Edit[Profile write] --> PG[(Postgres)]
  PG --> Out[Outbox]
  Out --> Lex[(Lexical index)]
  Out --> Vec[(Vector index)]
  Q[Search] --> Ret[Retrieve and filter]
  Ret --> PG
  Ret --> Lex
  Ret --> Vec
  Ret --> Rank[Rerank top N]
  Rank --> Log[Impressions]

Hard filters hit Postgres or the lexical index, not the vector index alone. You merge that set with a vector neighborhood and a BM25 hit list, dedupe, take a few hundred, and rerank with features you can name: skill overlap, rate fit, recent activity, past outcome rate with shrinkage toward the mean. The page returns experts plus the reasons.

Data model

CREATE TABLE expert_skills (
  expert_id BIGINT NOT NULL,
  skill     TEXT NOT NULL,
  level     SMALLINT NOT NULL,
  PRIMARY KEY (expert_id, skill)
);
CREATE INDEX expert_skills_lookup ON expert_skills (skill, level DESC, expert_id);

SELECT e.id
FROM expert_skills s
JOIN experts e ON e.id = s.expert_id
WHERE s.skill = 'rust' AND s.level >= 4 AND e.status = 'active'
ORDER BY e.quality DESC
LIMIT 200;

That index is the answer to the structured part. Embeddings live beside it, versioned by model, not instead of it.

Deep dives

Filtered retrieval. If the SQL predicate is already selective, do not ANN-search ten million vectors and then filter. Search inside the filtered set, or filter first and only embed-rank the survivors. If the filter is broad (“any backend engineer”), ANN plus BM25 is how you recall people whose resume wording does not match the query tokens.

Freshness. Profile writes and bans commit in Postgres with an outbox row. Consumers update both indexes. A ban is a tombstone checked again at serve time against Postgres or a cache, so a lagging index cannot show a banned person. Say the lag for ordinary edits.

Cold start and loops. New experts have no hires. Give them a prior, and reserve a slice of the page for exploration, or they will never collect the outcomes that would rank them. Log impressions and downstream hires. Without the log you cannot train or even argue that a change helped.

Injection and privacy. Resume text is data. A reranker prompt must not follow instructions inside it. Do not return emails or government ids in the search payload. Rate-limit and audit searches; the corpus is a target.

Levels

• Mid-level. Tables for experts and skills, an index for the filter, a simple rank, a cursor.
• Senior. Derived indexes via outbox, hybrid retrieval, filtered-ANN choice, ban checked at serve time.
• Staff. Outcome evaluation, exploration versus exploitation, model-version cutover, and tenant isolation if each customer brings private talent pools.