US college web census

US college web census ยท fetched 2026-09-30

Datasets

4,961 colleges, one row per domain, fetched 2026-09-30. Free to reuse under CC BY 4.0; cite "Agent Exchange, US college web census 2026-09-30, https://colleges.agentexchange.work/".

FileContentsFormatSize
colleges-web-census-2026-09.csvOne row per domain (4,961 rows), readable columnsCSV1.3 MB
colleges-web-census-2026-09.ndjsonOne JSON record per line, same fields as the per-domain APINDJSON4.3 MB
findings.jsonAggregates: overall, by control and level, by state, named listsJSON313 KB

Per-domain API

GET https://colleges.agentexchange.work/college/<domain>.json returns one record with readable field names (CORS open), including recheck once the domain has been re-checked live. Example: 4cd.edu.json. Unknown domains answer 404 with {"error": ...}. Per-domain Atom feed: https://colleges.agentexchange.work/college/<domain>.xml; badge: .svg. Agents: agent card, llms.txt.

Gap lists

8 filtered lists of the colleges whose record lacks one control (colleges without an accessibility statement on the homepage, colleges without a privacy policy link on the homepage, colleges without a security.txt, colleges with an expired security.txt, colleges without hsts, colleges blocking both chatgpt-user and claude-user, colleges answering only over plain http, college homepages that redirect to another domain): the first 50 rows and counts by state are free as /leads/<list>.top.json, the full CSV or JSON is 1.00 USDC through x402. Index: /leads. Every row is also in the dumps above.

Live data

changes.xml: Atom feed of every difference found by the 6-hourly re-checks (last 100; HTML at /changes). live.json: re-check totals. trending.json: the 100 most fetched pages of the last 7 days by crawler family (sampled 1 in 4, scaled).

Columns

domain, name, control and level, city, state, ipeds_sector (IPEDS sector), reachable, http_only, https_status, final_host, hsts, security_txt (none / complete / incomplete / expired), security_txt_contact, security_txt_expires, security_txt_expired, security_txt_path, privacy_link, accessibility_link, robots_txt, robots_all_bots, robots_gptbot, robots_claudebot, robots_chatgpt_user, robots_claude_user, robots_perplexity_user, robots_googlebot, ai_bots_named, content_signal, llms_txt, agent_card, title. Robots verdicts are allowed / partly blocked / blocked.