US college web census

Census fetched 2026-09-30 · population: NCES IPEDS 2024-25 institutional characteristics header file HD2024 (active Title IV institutions with a web address) · datasets (CC BY 4.0) · live changes

The US college web census, September 2026: what 4,961 colleges and universities publish for students, security researchers and AI agents

Every active institution in the IPEDS 2024-25 institutional characteristics file with a website address (4,961 colleges, universities and other Title IV institutions, after merging campuses that share one domain) was fetched once over HTTPS on 2026-09-30, with robots.txt, security.txt (both locations), llms.txt and the A2A agent card. 4,763 answered with a website.

Reachable over HTTPS

96.0%

4,763 of 4,961 colleges

security.txt

3.1%

146 of reachable sites; 111 complete and current

Accessibility statement linked

33.2%

1,583 homepages

Privacy policy linked

46.4%

2,212 homepages

Block GPTBot

114

68 block ChatGPT-User; 98 ClaudeBot

llms.txt

604

0 publish an A2A agent card

What we found

Live re-checks

Since 2026-10-01 this host has re-checked 6,759 colleges (a rotating slice every 6 hours, each domain at most once per 14 days; last run 2026-10-25 18:11 UTC) and recorded 1,045 changes against the census records. What changed · Atom feed · live.json.

By control and level

Control and levelIn listReachablesecurity.txtvalidprivacy linkaccessibility linkHSTSblocks GPTBotblocks ChatGPT-Userllms.txtagent card
private nonprofit 4-year1,5281,485 (97.2%)20 (1.3%)1350.0%31.8%45.3%53341910
private for-profit less-than-2-year900836 (92.9%)3 (0.4%)133.6%9.2%31.9%322000
public 4-year871852 (97.8%)41 (4.8%)2354.0%55.8%52.0%2011470
public 2-year830790 (95.2%)24 (3.0%)1849.5%53.7%49.7%147340
private for-profit 2-year254243 (95.7%)2 (0.8%)152.3%9.9%28.4%11530
public less-than-2-year224215 (96.0%)53 (24.7%)5318.6%26.5%75.3%129120
private for-profit 4-year197189 (95.9%)0 (0.0%)054.0%17.5%41.3%51500
private nonprofit 2-year115114 (99.1%)3 (2.6%)244.7%15.8%35.1%42120
private nonprofit less-than-2-year4239 (92.9%)0 (0.0%)046.2%7.7%43.6%2150

One page per control and level with its domains: https://colleges.agentexchange.work/by-control-and-level.

By state

States with at least 20 colleges in the list; percentages are of reachable sites.

StateIn listReachablesecurity.txtprivacy linkaccessibility linkHSTSblocks GPTBot
California542522 (96.3%)446.7%39.1%46.7%6
New York372351 (94.4%)1148.4%37.0%43.6%26
Texas286274 (95.8%)1541.2%33.6%41.2%2
Pennsylvania259255 (98.5%)745.5%30.2%51.8%2
Florida250234 (93.6%)844.4%31.6%38.9%4
Ohio213208 (97.7%)1344.2%26.9%51.0%11
Illinois185181 (97.8%)251.9%29.3%50.8%3
North Carolina139136 (97.8%)343.4%36.8%44.9%3
Massachusetts131128 (97.7%)148.4%40.6%53.9%3
Michigan127126 (99.2%)143.7%28.6%42.9%2
Tennessee124119 (96.0%)731.1%13.4%49.6%4
Missouri123121 (98.4%)743.0%31.4%43.8%1
New Jersey116113 (97.4%)740.7%24.8%35.4%9
Virginia114109 (95.6%)247.7%33.0%47.7%2
Georgia113111 (98.2%)060.4%40.5%41.4%1
Puerto Rico9883 (84.7%)66.0%4.8%32.5%2
Oklahoma9289 (96.7%)1231.5%20.2%44.9%0
Louisiana8976 (85.4%)047.4%28.9%40.8%0
Minnesota8682 (95.3%)154.9%45.1%58.5%1
Washington8279 (96.3%)055.7%41.8%40.5%2
Indiana8079 (98.8%)154.4%24.1%38.0%1
Arizona7772 (93.5%)031.9%18.1%45.8%1
Arkansas7671 (93.4%)550.7%25.4%54.9%2
Kentucky7372 (98.6%)169.4%43.1%29.2%2
Wisconsin7170 (98.6%)157.1%48.6%50.0%1
Colorado7068 (97.1%)057.4%44.1%45.6%1
Kansas6865 (95.6%)044.6%32.3%38.5%0
Alabama6761 (91.0%)149.2%42.6%34.4%0
Iowa6764 (95.5%)260.9%28.1%45.3%0
South Carolina6766 (98.5%)037.9%37.9%37.9%2
Maryland6461 (95.3%)067.2%42.6%62.3%6
West Virginia6157 (93.4%)1135.1%21.1%52.6%2
Oregon5857 (98.3%)142.1%36.8%45.6%1
Utah5146 (90.2%)052.2%30.4%41.3%1
Mississippi4847 (97.9%)042.6%34.0%31.9%0
Connecticut4442 (95.5%)066.7%35.7%40.5%0
Nebraska4040 (100.0%)552.5%40.0%65.0%1
New Mexico3835 (92.1%)240.0%45.7%28.6%0
Maine3332 (97.0%)150.0%28.1%31.3%0
New Hampshire2928 (96.6%)053.6%53.6%42.9%1
Montana2726 (96.3%)438.5%34.6%61.5%1
Idaho2525 (100.0%)060.0%40.0%40.0%0
South Dakota2525 (100.0%)060.0%40.0%48.0%1
Hawaii2321 (91.3%)142.9%33.3%33.3%0
North Dakota2323 (100.0%)056.5%47.8%43.5%1
District of Columbia2121 (100.0%)042.9%28.6%47.6%0
Nevada2121 (100.0%)047.6%23.8%42.9%2

Ranked table and one page per state with its domains: https://colleges.agentexchange.work/by-state.

Named lists

Colleges without a security.txt (sample of 300) (first 60 of 300)

Sites with an expired security.txt (14)

Sites publishing an llms.txt (first 60 of 604)

Sites telling ChatGPT-User and Claude-User not to open pages for users (48)

Sites blocking GPTBot (OpenAI's training crawler) (first 60 of 114)

Homepages that redirect to another domain (first 60 of 165)

Sites with a Content-Signal line in robots.txt (59)

Colleges without an accessibility link on the homepage (sample of 300) (first 60 of 300)

Gap lists for vendors: who is missing which control

Each list is the colleges whose 2026-09-30 record lacks one control, with counts by state, the first 50 rows free, an Atom feed of who enters or leaves it, and the full CSV or JSON for 1.00 USDC (x402). All gap lists.

ListCollegesOf the listWho buys this
Colleges without an accessibility statement on the homepage3,18064.1%Accessibility vendors and agencies, EdTech and campus web platform vendors with WCAG 2.1 AA offerings (Title II for public, Title III for private institutions).
Colleges without a privacy policy link on the homepage2,55151.4%Privacy-compliance vendors, student-data-privacy tooling, agencies that write notices.
Colleges without a security.txt4,61793.1%Security teams, vulnerability-disclosure and attack-surface vendors, MSPs serving higher education.
Colleges with an expired security.txt140.3%Security teams and vulnerability-disclosure vendors: a lapsed file is a specific, verifiable conversation opener.
Colleges without HSTS2,62252.9%Security teams, MSPs and hosting vendors serving higher education.
Colleges blocking both ChatGPT-User and Claude-User481.0%AI-visibility vendors and digital agencies: these sites are closed to ChatGPT and Claude users.
Colleges answering only over plain HTTP871.8%MSPs, hosting and web agencies: campus sites still without working HTTPS.
College homepages that redirect to another domain1653.3%Web agencies and domain-consolidation vendors; security teams checking stale redirects.

What colleges are expected to publish

Look up any college

Questions

How many college websites publish a security.txt?

146 of the 4,763 reachable college websites (3.1%) publish a security.txt; 111 are complete and current and 14 have expired.

How many colleges block AI crawlers?

114 college websites block GPTBot, 98 block ClaudeBot and 68 block ChatGPT-User in robots.txt; 48 block both ChatGPT-User and Claude-User; 448 name at least one AI crawler explicitly; 604 publish an llms.txt.

Do college websites link an accessibility statement?

33.2% of reachable college homepages link an accessibility statement and 46.4% link a privacy policy. 45.0% send an HSTS header.

Which institutions are included?

Active institutions in the IPEDS 2024-25 institutional characteristics file (HD2024) that list a website address: 4,961 domains on 2026-09-30, after merging campuses and branches that share one domain. The table by control and level separates public, private nonprofit and private for-profit institutions at each level.

Can I reuse the data?

Yes. The findings, the CSV and every per-college JSON record are published under CC BY 4.0; cite Agent Exchange and link the report.

Method and limits

Population: NCES IPEDS 2024-25 institutional characteristics header file HD2024 (active Title IV institutions with a web address) with each institution's control (public, private nonprofit, private for-profit), level (4-year, 2-year, less-than-2-year), city, state and IPEDS sector.. Fetch: one HTTPS request to the apex (then www.), following redirects, 8-second timeout, user agent agentexchange-census/1.0; then robots.txt, /.well-known/security.txt, /security.txt, /llms.txt and /.well-known/agent-card.json on the final host. "Reachable" means any HTTP response over HTTPS, including 403s from bot protection. Homepage links are detected from anchor targets and visible text, so a policy that exists but is not linked from the homepage is not counted. A robots.txt "blocked" verdict means the agent's own group (or the * group) disallows the whole site; "partly blocked" means some paths are disallowed. One snapshot per domain; slow sites and sites that block automated requests are counted as unreachable or reachable-with-error as the server answered. Live re-checks: every 6 hours this host re-fetches the homepage, robots.txt, security.txt and llms.txt of a rotating slice of up to 120 domains (each domain at most once per 14 days, same rules and user agent) and records any difference from the census record on the domain page and in the changes list; a 403, 429 or 503 to our request is recorded as inconclusive, not as a change. Per-domain records: lookup; downloads: CSV, NDJSON and JSON (CC BY 4.0).

Run one of these sites?

Instant AI visibility report for your college: $9 · $49 full report or $99 monthly monitoring · Hosted accessibility feedback and privacy request desk

Other censuses

US government web census (.gov) · World government web census · Top 100,000 websites census · US bank web census · US credit union web census · US school district web census · World university web census · Every Agent Exchange property (network map) · Agent-readiness grade and fix guides · Compliance Desk: hosted request processes · MCP registry integrity report · Agent Economy Index (daily x402 and MCP census) · Agent Exchange directory (x402 store)