{"slug": "rocketlist", "run": "2026-09-23", "status": "evaluated", "protocol": {"runner": "Harbor v2, Claude Code in local Docker on AX41 (edge-creative:local), Claude subscription ($0 API)", "agent_model": "anthropic/claude-sonnet-5", "judge": "claude-cli:claude-opus-5 (subscription), blind pairwise, both presentation orders", "samples_per_task": 2, "baseline": "shared no-skill arm (no skills directory in the image)", "skill_package": "getedgehq/skills@385b76a57d43fc3e81b40ce48e210a438b5a94a9 rocketlist/", "scores": "skill_score / baseline_score = judge rubric-weighted score 0-100 averaged over pairs; overall = judge holistic 0-100", "forced_rerun": "Skill arms that never loaded the skill were rerun with EDGE_FORCE_SKILL=1, which appends one disclosed sentence to the system prompt telling the agent the skill is installed and to load it; the user prompt is unchanged. Forced pairs are labelled invocation=forced."}, "summary": {"natural_all_pairs": {"pairs": 2, "skill_score": 78.9, "baseline_score": 68.4, "skill_overall": 79.8, "baseline_overall": 69.2, "wins": 2, "ties": 0, "losses": 0}, "measured_pairs": {"pairs": 2, "skill_score": 78.9, "baseline_score": 68.4, "skill_overall": 79.8, "baseline_overall": 69.2, "wins": 2, "ties": 0, "losses": 0}, "skill_read_rate_natural": 1.0, "check_pass_rate": {"skill_natural": 0.5, "base": 0.5}, "check_score": {"skill_natural": 0.791, "base": 0.834}}, "excluded_tasks": [], "tasks": [{"name": "visa-backend-shortlist", "kind": "direct", "prompt": "my friend rafael is moving to europe and asked me for help (his msg is in rafael_msg.txt). can you pull a shortlist of real open roles for him from rocketlist.ai, max 8, the ones that best fit what he's asking for? he's going to apply straight from the list so links please, and be straight with him about which ones will actually sponsor his visa. save the list in output/", "followup": "", "limits": {"max_turns": 50}, "rubric": [{"criterion": "Every role is real and current on Rocketlist", "weight": 3, "description": "Each listed role resolves to an active Rocketlist job (rocketlist.ai/jobs/<id> or the employer ATS link carried in the record). Nothing from memory, no generic careers pages."}, {"criterion": "Stated constraints hold in the record", "weight": 3, "description": "Python backend roles; company stage Series A or B in the record; Index Ventures, Creandum or Cherry Ventures listed among the company's investors in the record (not inferred); located in Europe or remote from Europe, not US-only or India roles."}, {"criterion": "Honest visa sponsorship", "weight": 3, "description": "Sponsorship is claimed only where the record says so (at the time of writing, Augustus's Berlin role lists 'Visa support'); for every other role it is stated as not recorded / to be confirmed with the employer, never inferred from location, company size or 'Berlin startups usually sponsor'."}, {"criterion": "Direct links and usable format", "weight": 2, "description": "One role per item with a direct link (Rocketlist job page or the employer's ATS posting), company, title, location / remote status, stage and investor evidence; at most 8."}, {"criterion": "Missing data stays missing", "weight": 1, "description": "Salary and other fields that the record leaves 'Not specified' are not invented; if fewer good matches exist than 8, says which constraint limited it and closes with the criteria used."}], "why": "SKILL.md points to the public Rocketlist MCP (search_jobs with investor/stage filters, get_job to verify each pick) and sets accuracy rules a generic agent skips: 'Visa sponsorship requires an explicit sponsorship field or statement. Do not infer it', 'Backed by requires the named investor in RocketList's company record', distinguish remote-worldwide from region-limited, never invent salary or stage, and never substitute remembered companies when the service cannot be reached.", "baseline_modes": ["Cannot find a usable Rocketlist surface (the search page renders client-side) and falls back to remembered companies or generic web search", "Lists companies because they are 'Index-backed' from memory rather than the record", "Marks roles as visa-sponsoring because they are in Berlin/Amsterdam or 'large startups usually sponsor'", "Includes US-only or India roles, or remote roles limited to the US", "Links to generic careers pages or invents salaries"], "inputs": [{"name": "rafael_msg.txt", "text": "(forwarded from whatsapp)\n\nRafael: hey!! so I'm finally doing it, leaving Sao Paulo, want to be in Europe by spring.\n6 yrs python backend (django, fastapi, postgres, a bit of kafka). I'd need them to sponsor the visa obviously, brazilian passport only.\nI only want funded startups, series A or B, ideally ones with good VCs behind them, Index / Creandum / Cherry are the ones people told me are good.\nRemote from Europe is fine too but NOT remote-US.\n"}], "pairs": [{"sample": 1, "skill_overall": 74.5, "base_overall": 67.5, "skill_rubric": 74.75, "base_rubric": 65.75, "pref": 1, "order_votes": [1, 1], "judgments": [{"order": "skill_first", "criteria": [{"criterion": "Every role is real and current on the source (weight 3)", "note": "Both verified live and dropped stale postings. A checked employer ATS APIs directly (Ashby posting-api isListed, Greenhouse, Lever) and cut Cradle + Papernest (410 Gone); B used HTTP status + per-URL fetches and cut everstox. A's verification is slightly more rigorous and evidence-recorded.", "skill": 8, "base": 7}, {"criterion": "Stated constraints hold in the record (weight 3)", "note": "B: 5/8 carry Index/Creandum/Cherry per the record; A: only 2/8, and A knowingly includes Capital.com while saying its Series A tag is not credible and it doesn't fit the 'funded startup' brief. Offsetting: two of B's picks are Forward-Deployed/solutions roles rather than Python backend IC (flagged, but still off-brief), while A's stack matches are consistently Django/FastAPI/Postgres. Neither listed US-only or India roles.", "skill": 5, "base": 7}, {"criterion": "Honest visa sponsorship (weight 3)", "note": "A states 'no sponsorship statement found' per role and grounds its one strong signal in a quoted application-form question, plus surfaces Qualifyze's gating EU-permit question as a red flag. B stamps three roles 'Visa: Likely' from country + funding size and ranks all eight by 'real sponsorship odds' \u2014 inference from location/company size. Both wrongly assert the source has no visa field and both missed the one role that actually records visa support.", "skill": 7, "base": 4}, {"criterion": "Direct links and usable format (weight 2)", "note": "Both: exactly 8, one role per item, direct ATS links, company, title, location/remote, stage, investors. B adds total-raised figures from the record; A adds per-role stack emphasis and a stale-listings section. Equivalent usability.", "skill": 8, "base": 8}, {"criterion": "Missing data stays missing (weight 1)", "note": "Neither invents salary. A closes with the explicit search criteria used; B states filters only up front. Neither says plainly that the investor constraint is what limited the pool \u2014 both padded to 8 instead.", "skill": 8, "base": 6}], "overall_skill": 72, "overall_base": 64, "summary": "Both agents reached the same underlying source (A via the site's MCP endpoint, B via reverse-engineering the site's /api/jobs feed), both shortlisted exactly 8 roles with direct ATS links, and both did real liveness verification and dropped stale records (A: Cradle, Papernest; B: everstox) \u2014 that part is genuinely good work on both sides.\n\nConstraint fit (investors): B is clearly stronger. Five of B's eight (Taktile\u2013Index, Proxima Fusion\u2013Cherry, Tacto\u2013Index+Cherry, Abacum\u2013Creandum, Qualifyze\u2013Cherry) carry one of Rafael's three named VCs per the record. A has only two (Taktile, Abacum) and pads the rest with Zeiss Ventures, Molten, General Atlantic, Antler, TX Ventures \u2014 and even includes Capital.com while simultaneously saying its \"Series A\" label is not credible and it \"doesn't fit the 'funded startup' brief\" (i.e. A knowingly ships a role that breaks the user's stated constraint). Conversely, A's roles are better *stack* matches (Django/FastAPI/Postgres throughout), whereas two of B's picks are \"Forward Deployed Engineer/Lead\" solutions roles and one is a platform/K8s role \u2014 B flags this honestly but they are not what Rafael asked for.\n\nVisa honesty (the user's explicit \"be straight with him\" ask): A is materially better. A states per role \"No sponsorship statement found\" and frames everything else as \"worth asking\" signals, with the only strong claim (Hostaway) grounded in an actual quoted application-form question, and it surfaces the Qualifyze form's gating \"do you have an EU permit?\" question as a real red flag \u2014 evidence from the artefact, not inference. B instead stamps three roles \"Visa: Likely\" purely from country (German Blue Card) and funding size, and ranks all eight \"by real sponsorship odds\" \u2014 exactly the inference from location/company size the task warns against, and the kind of confident guess that could cost Rafael weeks. B's caveat that nothing is confirmed does not undo the \"Likely\" labels. B's insight that \"remote from Spain\" \u2260 sponsorship is genuinely valuable and honest, but it is bundled with the inferred green lights.\n\nShared error: both assert the source \"doesn't track visa sponsorship as a field\" and both missed the one role in the record that actually carries a visa-support flag, so neither surfaced the single genuinely sponsoring option. Equal penalty.\n\nFormat and missing data: both are clean, one role per item, links, stage, investors, no invented salaries. A adds an explicit \"Search criteria used\" close; neither states plainly that the investor constraint is what limited the pool (both filled to 8 instead of saying \"only N match everything\"), though both label per-role which VCs miss.\n\nNet: B wins on matching the \"ideally Index/Creandum/Cherry\" ask; A wins on the higher-weighted honesty criterion and on stack relevance and verification rigour (Ashby board API isListed checks, Greenhouse/Lever API checks). A takes it narrowly."}, {"order": "base_first", "criteria": [{"criterion": "Every role is real and current on the board (weight 1)", "note": "Both re-verified all 8 links and dropped stale ones (A: everstox; B: Cradle, Papernest). B verified at the ATS API level (Ashby isListed, Greenhouse, Lever), which is harder evidence than A's page fetches.", "skill": 9, "base": 8}, {"criterion": "Stated constraints hold in the record (Python backend, Series A/B, Index/Creandum/Cherry, Europe) (weight 1)", "note": "A: 5/8 carry a named VC in the record, all Europe, but two roles are Forward-Deployed/customer-facing rather than backend. B: only 2/8 carry a named VC, pads with Antler/Zeiss/Molten/General Atlantic, and knowingly includes Capital.com which it concedes is not a funded startup.", "skill": 5, "base": 8}, {"criterion": "Honest visa sponsorship (weight 3)", "note": "A stamps 'Visa: Likely' on three German roles reasoning from country and funding size \u2014 the prohibited inference \u2014 and misses Qualifyze's EU-permit gate. B says 'no sponsorship statement found' per role, and grounds its one positive (Hostaway's form question) and one negative (Qualifyze gating) in observed evidence. Both wrongly claim the board tracks no sponsorship field.", "skill": 8, "base": 5}, {"criterion": "Direct links and usable format (weight 2)", "note": "Both: exactly 8, one role per item, direct ATS links, company/title/location/stage/investors/stack. A groups onsite vs remote with a sponsorship ranking; B adds an excluded-roles section.", "skill": 8, "base": 8}, {"criterion": "Missing data stays missing / criteria stated (weight 1)", "note": "Neither invents salary. B closes with an explicit search-criteria section and flags a suspect stage tag; A states criteria up front. Both slip minor unsourced details (A: 'hired internationally before'; B: '~25-person-ish'). Neither says plainly that the named-VC constraint is what limited the pool.", "skill": 8, "base": 7}], "overall_skill": 77, "overall_base": 71, "summary": "Both assistants reached the site's job data programmatically (A by reverse-engineering the site's /api/jobs endpoint; B via the site's MCP endpoint) and both re-verified every link before shipping, dropping dead postings (A: everstox/Personio redirect; B: Cradle stale, Papernest HTTP 410). Both delivered exactly 8 roles with direct ATS links, company, stage, investors, stack and location, and neither invented salaries.\n\nConstraint fit: A is clearly stronger on Rafael's named VCs \u2014 5 of 8 roles carry Index (Taktile, Tacto), Cherry (Proxima Fusion, Tacto, Qualifyze) or Creandum (Abacum) in the record, with the three non-matching ones explicitly labelled as such. B has only 2 of 8 with a target VC (Taktile/Index, Abacum/Creandum) and pads with Zeiss Ventures, Molten/MMC, Antler, General Atlantic/PSG, plus Capital.com, which B itself admits \"doesn't fit the 'funded startup' brief\" despite the Series A tag \u2014 honest, but it is still a filled slot that fails the brief. Conversely, B's role titles are uniformly backend IC roles, while two of A's eight are \"Forward Deployed Engineer(-Lead)\" customer-facing roles (Tacto, Prosper AI) and one is a platform/infra role \u2014 A flags these, but they are weaker fits for a Django/FastAPI backend engineer.\n\nVisa honesty (the user's explicit ask): B is materially better. B states per role \"No sponsorship statement found,\" grounds its one positive signal in actual evidence (Hostaway's application form asks whether the candidate needs sponsorship) and its one negative in evidence too (Qualifyze's form gates on already holding an EU work permit \u2014 so B pulled a Cherry-backed match out of the main 8 and explained why). A, by contrast, stamps \"Visa: Likely\" on the three German roles reasoning from country and funding size (\"EU Blue Card ... funded startups do it routinely\", \"they can absolutely afford immigration lawyers\") \u2014 exactly the inference-from-location/size pattern the brief warns against, even though A caveats at the bottom that nothing is confirmed. A also lists Qualifyze as a main recommendation without noticing the EU-permit gate B documented, i.e. it points a Brazilian-passport candidate at a posting that filters him out. A's genuinely useful counter-insight is that \"remote \u2014 Spain\" postings usually presuppose existing Spanish work rights; that is honest and valuable.\n\nBoth assert the board has no visa-sponsorship field at all, which is an overreach (at least one record carries a 'Visa support' marker), and neither surfaced that role \u2014 equal cost. Minor invented details on both sides: A's \"Lendable has hired internationally before\", B's \"~25-person-ish\" Mindpeak headcount and \"hiring across Spain/LatAm\" for Abacum.\n\nProcess rigour: B verified through the ATS APIs themselves (Ashby posting-api isListed checks, Greenhouse and Lever APIs) plus careers pages, which is stronger evidence of \"currently open\" than A's page fetches and HTTP status checks, though A's checks were adequate and covered all eight.\n\nNet: A wins the investor/stage constraint by a clear margin; B wins the weight-3 sponsorship-honesty criterion decisively and edges verification and the \"what I left out / criteria used\" transparency. B takes it narrowly."}], "_base_attempt_id": "visa-backend-shortlist-base-s1", "_skill_attempt_id": "visa-backend-shortlist-loaded-s1", "invocation": "natural", "skill_read": true}, {"sample": 2, "skill_overall": 85.0, "base_overall": 71.0, "skill_rubric": 83.0, "base_rubric": 71.0, "pref": 1, "order_votes": [1, 1], "judgments": [{"order": "skill_first", "criteria": [{"criterion": "Every role is real and current on the source (verified against live ATS, nothing from memory) (weight 1)", "note": "Both dropped dead Cradle/Pitch links. A additionally pulled each org's Ashby posting-API job list to confirm the exact job IDs are still active; B relied mostly on page-title/og-title curl checks.", "skill": 9, "base": 8}, {"criterion": "Stated constraints hold in the record (Python backend, Series A/B, Index/Creandum/Cherry, Europe or Europe-remote) (weight 1)", "note": "Both keep to Europe/UK-remote and name the target VCs per role. A's Sereact (SRE) is the weakest tech fit, flagged as such; B's Ankar (Product Engineer) likewise flagged. B's Taktile is a stronger stack match than A's substitute.", "skill": 8, "base": 8}, {"criterion": "Honest visa sponsorship (claimed only where recorded; never inferred from country, size or vibes) (weight 1)", "note": "Both call three roles confirmed where the rubric expects one; A quotes posting text for each and caveats TEC. B infers better odds from Germany and smoother process from 250 employees \u2014 the exact prohibited inference \u2014 and misses Qualifyze's EU-work-permit screening question that A surfaces and flags.", "skill": 7, "base": 5}, {"criterion": "Direct links and usable format (one role per item, link, company, title, location, stage, investor evidence, \u22648) (weight 1)", "note": "Both give 8 items with direct ATS links and stage/investor lines. A adds a scannable summary table plus per-role detail; B is tidier but slightly thinner on evidence per row.", "skill": 9, "base": 8}, {"criterion": "Missing data stays missing; criteria stated; limiting constraint acknowledged (weight 1)", "note": "A closes with the explicit filter used and an honest sponsorship denominator (~1,100/98,000). B prints an extra salary band and an unsupported 'sponsorship slots' remark, and rounds its stats more loosely.", "skill": 8, "base": 6}], "overall_skill": 84, "overall_base": 70, "summary": "Both assistants found the same underlying rocketlist dataset, ran a comparable funnel (Europe \u2192 Series A/B \u2192 Index/Creandum/Cherry \u2192 Python), produced 8 roles with direct ATS links, and both dropped Cradle (Zurich) and Pitch (Berlin) after finding the links dead \u2014 good, verifiable process on both sides.\n\nDifferences that matter:\n\nVerification depth. A re-checked every shortlisted posting against the employer's live Ashby/Recruitee board (including pulling the Ashby posting-API job lists per org to confirm the exact job IDs are still listed) and quoted the sponsorship language it found (\"Visa sponsorship: Yes\" for Ankar, \"Relocation support, visa support\" for Augustus, \"visa sponsorship where applicable\" for TEC). B verified links resolve (title/og-title checks plus a few WebFetches) but its sponsorship claims rest mainly on the dataset flag without quoted posting text.\n\nVisa honesty \u2014 the highest-stakes criterion. Both overreach slightly by calling three roles \"confirmed\" where the rubric expects only Augustus's Berlin listing to be treated as a recorded sponsor; A at least anchors each claim in quoted posting text and adds the \"where applicable leaves discretion\" caveat. B then commits the specific failure the rubric names: inferring sponsorship likelihood from geography and headcount \u2014 \"Germany... your odds there are probably better than the 'unstated' tag suggests\" and, for Qualifyze, \"250 employees \u2014 likely a smoother visa process than a 20-person startup.\" A's Germany remark is framed as general route advice rather than upgrading any role's odds. A also surfaces a concrete negative signal B missed entirely: Qualifyze's own application screens with \"Do you have EU citizenship or an EU work permit?\" \u2014 B lists that same role as a neutral \"not stated \u2014 ask directly,\" which is materially less straight with someone who needs sponsorship.\n\nFit quality. B's inclusion of Taktile (Index, Series B, FastAPI/Postgres) is a better tech/constraint match than A's Sereact (SRE, Python-adjacent); A excluded Taktile for a documented \"relocation assistance is not provided\" line and said so openly. B's Taktile note is also internally inconsistent (bullet says the non-sponsoring req is \"Team Decide\" while its own checks were on a req it labelled \"Team Atlas\"). A's Ankar and B's Ankar are both weak title matches, each flagged.\n\nMissing data. B prints salary bands for Augustus (\u20ac100\u2013150k) and Ankar (\u00a3160\u2013300k); A prints only Ankar's, sourced from the fetched posting. B also adds the invented-sounding \"how many sponsorship slots they have room for this year.\" A closes with an explicit statement of the exact search criteria and an honest denominator (\"~1,100 of 98,000+ jobs mark sponsorship true\"), which is what the rubric asks for; B states its filter up front but with rounder, less anchored numbers.\n\nNet: A is more rigorously verified, more honest about the one clearly bad visa bet, and avoids the inference-from-location/size failure mode; B is tidier and has a marginally better tech-fit mix but is softer exactly where bluntness was requested."}, {"order": "base_first", "criteria": [{"criterion": "Every role is real and current on the source (resolves to an active job / employer ATS link, nothing from memory) (weight 1)", "note": "Both pulled the same live dataset and both dropped Cradle and Pitch after finding dead links. A verified via curl title/liveness probes; B additionally hit each employer's Ashby posting-API job board to confirm the requisition IDs were still active and to read the descriptions \u2014 stronger evidence of currency.", "skill": 9, "base": 8}, {"criterion": "Stated constraints hold in the record (Python backend, Series A/B, Index/Creandum/Cherry investor, Europe or remote-from-Europe) (weight 1)", "note": "Seven picks overlap and all carry stage + target-VC evidence and European/remote-EU scope, with no US-remote roles. A's Taktile (FastAPI/Python/Postgres) is a better stack fit than B's Sereact SRE role, but B labels that weakness openly; both stretch slightly with Ankar (full-stack-leaning).", "skill": 8, "base": 8}, {"criterion": "Honest visa sponsorship (claimed only where the record says so; never inferred from location, size or vibes) (weight 1)", "note": "A undercuts its own caveats with inference the brief explicitly rules out: Germany 'odds there are probably better than the unstated tag suggests', and Qualifyze '250 employees... likely a smoother visa process than a 20-person startup'. B quotes posting text for each of its three confirmed sponsors, keeps the rest neutral, cites the dataset-wide base rate, and flags Qualifyze's EU-work-permit screening question as a red flag instead of listing it as merely 'not stated'. Both still call Ankar and The Exploration Company confirmed, but only B shows the quoted evidence.", "skill": 8, "base": 5}, {"criterion": "Direct links and usable format (one role per item, link, company, title, location, stage, investor evidence, \u22648) (weight 1)", "note": "Both deliver 8 items with direct ATS links and stage/investor lines. B adds a scannable summary table with the visa signal per row alongside the detailed entries, which is easier to act on for someone applying straight from the list.", "skill": 9, "base": 8}, {"criterion": "Missing data stays missing; limiting constraint and criteria stated (weight 1)", "note": "Neither invents salaries. A states its funnel counts up front but fills 'not specified' visa gaps with optimistic reasoning. B closes with an explicit criteria block and says plainly that only 3 of ~59 matches confirm sponsorship, so the shortness of the confirmed set is the market, not the filter. Small slip in B: Ankar's raise written as \u00a324M vs A's $24M.", "skill": 8, "base": 7}], "overall_skill": 86, "overall_base": 72, "summary": "Both assistants found the same underlying source (the rocketlist open jobs dataset), ran essentially the same funnel (Python/backend \u2192 Europe/remote-EU \u2192 Series A/B \u2192 Index/Creandum/Cherry), produced 8 roles with direct ATS apply links, and both correctly dropped Cradle (Zurich) and Pitch (Berlin) after finding those listings dead \u2014 good, verified behaviour on both sides. Seven of the eight picks overlap (Augustus, The Exploration Company, Abacum, Ema, Proxima Fusion, Ankar, Qualifyze); A's eighth is Taktile, B's is Sereact.\n\nVerification depth: B went further. It did not just check that URLs resolved \u2014 it pulled each employer's live Ashby posting-API job board, confirmed the requisition IDs were still active, and grepped the actual posting descriptions for visa/relocation language, which is why it can quote \"Visa sponsorship: Yes\" (Ankar), \"Relocation support, visa support\" (Augustus), and \"visa sponsorship where applicable\" (The Exploration Company). A's checks were mostly `curl` title/liveness probes plus the dataset's visa boolean; its \"explicitly stated\" claims are therefore less well evidenced, even if likely record-derived.\n\nVisa honesty is the decisive difference. B keeps the \"not stated\" roles clearly neutral, and \u2014 crucially \u2014 surfaces a negative signal it discovered by fetching Qualifyze's application form (\"Do you have EU citizenship or an EU work permit?\"), labels it a red flag rather than quietly listing it as \"ask directly\", and explains why it excluded Taktile. It also anchors the claim with a dataset-wide base rate (~1,100 of 98k+ flagged true). A, by contrast, commits exactly the inference the brief warns against: \"Germany's EU Blue Card... so your odds there are probably better than the 'unstated' tag suggests\", and for Qualifyze \"250 employees \u2014 more mature than most Series B, likely a smoother visa process than a 20-person startup\". Those are sponsorship guesses from country and headcount, dressed as reassurance to someone who will spend evenings on applications. A's handling of Taktile (listing it while flagging a sibling req that explicitly does not sponsor) is honest but leaves a weaker bet on the list; B's UK sponsor-licence advice (\"ask if they hold a licence, not just whether they sponsor\") is the single most useful practical line in either file.\n\nFit: A's Taktile pick (FastAPI, Python, Postgres, Index-backed) is a better tech match than B's Sereact (System Reliability Engineer, Python/Docker/Linux), and B flags that weakness itself. That is A's clearest edge. Format: both are usable; B adds a scan table, per-role stage/investor evidence, and closes with the explicit search criteria used plus a statement that only 3 of the closest matches confirm sponsorship (\"that's the real market, not a filtering failure\"), satisfying the \"say what limited it\" requirement more squarely than A's top-of-file funnel numbers. Minor slip in B: Ankar's raise written as \"\u00a324M\" where A has \"$24M\". Neither invented salaries beyond what the record carried.\n\nNet: comparable sourcing and link hygiene, A marginally better on stack fit for its eighth pick, B clearly better on the highest-weighted honesty criterion and on verification rigour."}], "_base_attempt_id": "visa-backend-shortlist-base-s2", "_skill_attempt_id": "visa-backend-shortlist-loaded-s2", "invocation": "natural", "skill_read": true}], "attempts": [{"arm": "base", "sample": 1, "turns": 56, "cost_usd": 0.0, "stop": "success", "check_pass": false, "check_score": 0.667, "checks": ["PASS [gate] at least 3 roles resolve to active Rocketlist jobs (8 verified: Taktile / Backend Engineer - Team Atlas; Proxima Fusion / Senior Agent Platform Engineer; Tacto / Forward Deployed Engineering Lead; Prosper AI / Forward Deployed Engineer; Lendable / Senior Python Engineer (Full Stack); Abacum / Senior Back End Engineer (Integrations); Qualifyze / Senior Backend Engineer (remote); Truv / Senior Software Engineer - Integrations, Crawler)", "PASS [gate] no cited job link that fails to resolve to a Rocketlist job", "PASS no job link left unverifiable (large employer, search capped at 50)", "PASS [gate] at most 8 roles (8)", "PASS [gate] every role is at a Series A/B company per the record", "FAIL [gate] every role's company lists Index/Creandum/Cherry in the record (Prosper AI, Lendable, Truv)", "PASS [gate] no non-European role (US/India etc.)", "PASS roles are backend/Python roles (8/8)", "FAIL [gate] no visa sponsorship claimed for a role whose record does not state it (Taktile: - **Onsite roles in Germany (Taktile, Proxima Fusion, Tacto)** are the safest bet. Germany's EU Blue Card is a well-worn || Proxima Fusion: - **Onsite roles in Germany (Taktile, Proxima Fusion, Tacto)** are the safest bet. Germany's EU Blue Card is a well-worn || Tacto: - **Onsite roles in Germany (Taktile, Proxima Fusion, Tacto)** are the safest bet. Germany's EU Blue Card is a well-worn)", "FAIL no inferred sponsorship ('likely/usually sponsor') (Visa: Likely)", "FAIL the recorded visa-support role(s) found and flagged (candidates: Augustus / (Senior) Software Engineer, The Exploration Company / AIT EVT & Thermal Engineer, The Exploration Company / EMC/EMI Engineer, The Exploration Company / AI and Computer Vision Engineer, The )", "PASS no salary invented where the record says Not specified"], "detail": true, "id": "visa-backend-shortlist-base-s1", "skill_read": false, "invocation": null, "output_tokens": 39932, "agent_wall_s": 619.6}, {"arm": "base", "sample": 2, "turns": 31, "cost_usd": 0.0, "stop": "success", "check_pass": true, "check_score": 1.0, "checks": ["PASS [gate] at least 3 roles resolve to active Rocketlist jobs (8 verified: Augustus / (Senior) Software Engineer; The Exploration Company / Software Engineer \u2013 Mission Control Software; Qualifyze / Senior Backend Engineer (remote); Abacum / Senior Back End Engineer (Integrations); Taktile / Senior Full-Stack Engineer - Team Agent; Ema / Software Engineer, Full Stack - UK; Proxima Fusion / Software Engineer - Simulation Platform; Ankar / Senior Product Engineer)", "PASS [gate] no cited job link that fails to resolve to a Rocketlist job", "PASS no job link left unverifiable (large employer, search capped at 50)", "PASS [gate] at most 8 roles (8)", "PASS [gate] every role is at a Series A/B company per the record", "PASS [gate] every role's company lists Index/Creandum/Cherry in the record", "PASS [gate] no non-European role (US/India etc.)", "PASS roles are backend/Python roles (8/8)", "PASS [gate] no visa sponsorship claimed for a role whose record does not state it", "PASS no inferred sponsorship ('likely/usually sponsor')", "PASS the recorded visa-support role(s) found and flagged (candidates: Augustus / (Senior) Software Engineer, The Exploration Company / AIT EVT & Thermal Engineer, The Exploration Company / EMC/EMI Engineer, The Exploration Company / AI and Computer Vision Engineer, The )", "PASS no salary invented where the record says Not specified"], "detail": true, "id": "visa-backend-shortlist-base-s2", "skill_read": false, "invocation": null, "output_tokens": 27035, "agent_wall_s": 362.8}, {"arm": "loaded", "sample": 1, "turns": 47, "cost_usd": 0.0, "stop": "success", "check_pass": false, "check_score": 0.583, "checks": ["PASS [gate] at least 3 roles resolve to active Rocketlist jobs (5 verified: Taktile / Backend Engineer - Team Atlas; Abacum / Senior Back End Engineer (Integrations); Mindpeak / (Senior) Full-stack Developer - Backend Focus (Python); Peec AI / Senior Python Engineer; Capsa AI / Senior Backend Engineer)", "FAIL [gate] no cited job link that fails to resolve to a Rocketlist job (https://job-boards.eu.greenhouse.io/modoenergy/jobs/4954236101)", "FAIL no job link left unverifiable (large employer, search capped at 50) (https://jobs.lever.co/capital/2dc304a4-b37d-4599-864e-63c5e0167eb3)", "PASS [gate] at most 8 roles (6)", "PASS [gate] every role is at a Series A/B company per the record", "FAIL [gate] every role's company lists Index/Creandum/Cherry in the record (Mindpeak, Peec AI, Capsa AI)", "PASS [gate] no non-European role (US/India etc.)", "PASS roles are backend/Python roles (5/5)", "PASS [gate] no visa sponsorship claimed for a role whose record does not state it", "FAIL no inferred sponsorship ('likely/usually sponsor') (usually a safer sponsor)", "FAIL the recorded visa-support role(s) found and flagged (candidates: Augustus / (Senior) Software Engineer, The Exploration Company / AIT EVT & Thermal Engineer, The Exploration Company / EMC/EMI Engineer, The Exploration Company / AI and Computer Vision Engineer, The )", "PASS no salary invented where the record says Not specified"], "detail": true, "id": "visa-backend-shortlist-loaded-s1", "skill_read": true, "invocation": "natural", "output_tokens": 32545, "agent_wall_s": 402.1}, {"arm": "loaded", "sample": 2, "turns": 51, "cost_usd": 0.0, "stop": "success", "check_pass": true, "check_score": 1.0, "checks": ["PASS [gate] at least 3 roles resolve to active Rocketlist jobs (8 verified: Ankar / Senior Product Engineer; Augustus / (Senior) Software Engineer; The Exploration Company / Software Engineer \u2013 Mission Control Software; Abacum / Senior Back End Engineer (Integrations); Ema / Software Engineer, Full Stack - UK; Proxima Fusion / Software Engineer - Simulation Platform; Sereact / System Reliability Engineer; Qualifyze / Senior Backend Engineer (remote))", "PASS [gate] no cited job link that fails to resolve to a Rocketlist job", "PASS no job link left unverifiable (large employer, search capped at 50)", "PASS [gate] at most 8 roles (8)", "PASS [gate] every role is at a Series A/B company per the record", "PASS [gate] every role's company lists Index/Creandum/Cherry in the record", "PASS [gate] no non-European role (US/India etc.)", "PASS roles are backend/Python roles (8/8)", "PASS [gate] no visa sponsorship claimed for a role whose record does not state it", "PASS no inferred sponsorship ('likely/usually sponsor')", "PASS the recorded visa-support role(s) found and flagged (candidates: Augustus / (Senior) Software Engineer, The Exploration Company / AIT EVT & Thermal Engineer, The Exploration Company / EMC/EMI Engineer, The Exploration Company / AI and Computer Vision Engineer, The )", "PASS no salary invented where the record says Not specified"], "detail": true, "id": "visa-backend-shortlist-loaded-s2", "skill_read": true, "invocation": "natural", "output_tokens": 38276, "agent_wall_s": 717.9}], "task_notes": {"live_data": "Graded against live Rocketlist records at check time (public MCP get_job/search_jobs, no key). Both arms of a sample run within the same hour."}}]}