Census fetched 2026-09-30 · population: NCES IPEDS 2024-25 institutional characteristics header file HD2024 (active Title IV institutions with a web address) · datasets (CC BY 4.0)
The US college web census, September 2026: what 4,961 colleges and universities publish for students, security researchers and AI agents
Every active institution in the IPEDS 2024-25 institutional characteristics file with a website address (4,961 colleges, universities and other Title IV institutions, after merging campuses that share one domain) was fetched once over HTTPS on 2026-09-30, with robots.txt, security.txt (both locations), llms.txt and the A2A agent card. 4,763 answered with a website.
Reachable over HTTPS
96.0%
4,763 of 4,961 colleges
security.txt
3.1%
146 of reachable sites; 111 complete and current
Accessibility statement linked
33.2%
1,583 homepages
Privacy policy linked
46.4%
2,212 homepages
Block GPTBot
114
68 block ChatGPT-User; 98 ClaudeBot
llms.txt
604
0 publish an A2A agent card
What we found
- security.txt: 146 of 4,763 reachable sites (3.1%) publish one; 111 are complete and current (a Contact line and an unexpired Expires line, RFC 9116); 14 have expired; 16 serve it only at /security.txt instead of /.well-known/security.txt.
- Accessibility: 33.2% of reachable sites link an accessibility statement from the homepage.
- Privacy: 46.4% link a privacy policy from the homepage.
- HTTPS: 4,763 of 4,961 domains (96.0%) answered over HTTPS; 87 answer only over plain HTTP; 45.0% of reachable sites send HSTS; 165 homepages redirect to a different domain.
- AI crawlers and assistants: 4,106 sites serve a robots.txt; 114 block GPTBot, 98 ClaudeBot and 68 ChatGPT-User; 48 disallow every bot; 48 block both ChatGPT-User and Claude-User; 60 block GPTBot while leaving OAI-SearchBot allowed; 448 name an AI crawler explicitly; 59 carry a Content-Signal line.
- Machine-readable surfaces: 604 publish an llms.txt and 0 an A2A agent card; 2,266 homepages carry schema.org structured data.
By control and level
| Control and level | In list | Reachable | security.txt | valid | privacy link | accessibility link | HSTS | blocks GPTBot | blocks ChatGPT-User | llms.txt | agent card |
|---|---|---|---|---|---|---|---|---|---|---|---|
| private nonprofit 4-year | 1,528 | 1,485 (97.2%) | 20 (1.3%) | 13 | 50.0% | 31.8% | 45.3% | 53 | 34 | 191 | 0 |
| private for-profit less-than-2-year | 900 | 836 (92.9%) | 3 (0.4%) | 1 | 33.6% | 9.2% | 31.9% | 3 | 2 | 200 | 0 |
| public 4-year | 871 | 852 (97.8%) | 41 (4.8%) | 23 | 54.0% | 55.8% | 52.0% | 20 | 11 | 47 | 0 |
| public 2-year | 830 | 790 (95.2%) | 24 (3.0%) | 18 | 49.5% | 53.7% | 49.7% | 14 | 7 | 34 | 0 |
| private for-profit 2-year | 254 | 243 (95.7%) | 2 (0.8%) | 1 | 52.3% | 9.9% | 28.4% | 1 | 1 | 53 | 0 |
| public less-than-2-year | 224 | 215 (96.0%) | 53 (24.7%) | 53 | 18.6% | 26.5% | 75.3% | 12 | 9 | 12 | 0 |
| private for-profit 4-year | 197 | 189 (95.9%) | 0 (0.0%) | 0 | 54.0% | 17.5% | 41.3% | 5 | 1 | 50 | 0 |
| private nonprofit 2-year | 115 | 114 (99.1%) | 3 (2.6%) | 2 | 44.7% | 15.8% | 35.1% | 4 | 2 | 12 | 0 |
| private nonprofit less-than-2-year | 42 | 39 (92.9%) | 0 (0.0%) | 0 | 46.2% | 7.7% | 43.6% | 2 | 1 | 5 | 0 |
By state
States with at least 20 colleges in the list; percentages are of reachable sites.
| State | In list | Reachable | security.txt | privacy link | accessibility link | HSTS | blocks GPTBot |
|---|---|---|---|---|---|---|---|
| CA | 542 | 522 (96.3%) | 4 | 46.7% | 39.1% | 46.7% | 6 |
| NY | 372 | 351 (94.4%) | 11 | 48.4% | 37.0% | 43.6% | 26 |
| TX | 286 | 274 (95.8%) | 15 | 41.2% | 33.6% | 41.2% | 2 |
| PA | 259 | 255 (98.5%) | 7 | 45.5% | 30.2% | 51.8% | 2 |
| FL | 250 | 234 (93.6%) | 8 | 44.4% | 31.6% | 38.9% | 4 |
| OH | 213 | 208 (97.7%) | 13 | 44.2% | 26.9% | 51.0% | 11 |
| IL | 185 | 181 (97.8%) | 2 | 51.9% | 29.3% | 50.8% | 3 |
| NC | 139 | 136 (97.8%) | 3 | 43.4% | 36.8% | 44.9% | 3 |
| MA | 131 | 128 (97.7%) | 1 | 48.4% | 40.6% | 53.9% | 3 |
| MI | 127 | 126 (99.2%) | 1 | 43.7% | 28.6% | 42.9% | 2 |
| TN | 124 | 119 (96.0%) | 7 | 31.1% | 13.4% | 49.6% | 4 |
| MO | 123 | 121 (98.4%) | 7 | 43.0% | 31.4% | 43.8% | 1 |
| NJ | 116 | 113 (97.4%) | 7 | 40.7% | 24.8% | 35.4% | 9 |
| VA | 114 | 109 (95.6%) | 2 | 47.7% | 33.0% | 47.7% | 2 |
| GA | 113 | 111 (98.2%) | 0 | 60.4% | 40.5% | 41.4% | 1 |
| PR | 98 | 83 (84.7%) | 6 | 6.0% | 4.8% | 32.5% | 2 |
| OK | 92 | 89 (96.7%) | 12 | 31.5% | 20.2% | 44.9% | 0 |
| LA | 89 | 76 (85.4%) | 0 | 47.4% | 28.9% | 40.8% | 0 |
| MN | 86 | 82 (95.3%) | 1 | 54.9% | 45.1% | 58.5% | 1 |
| WA | 82 | 79 (96.3%) | 0 | 55.7% | 41.8% | 40.5% | 2 |
| IN | 80 | 79 (98.8%) | 1 | 54.4% | 24.1% | 38.0% | 1 |
| AZ | 77 | 72 (93.5%) | 0 | 31.9% | 18.1% | 45.8% | 1 |
| AR | 76 | 71 (93.4%) | 5 | 50.7% | 25.4% | 54.9% | 2 |
| KY | 73 | 72 (98.6%) | 1 | 69.4% | 43.1% | 29.2% | 2 |
| WI | 71 | 70 (98.6%) | 1 | 57.1% | 48.6% | 50.0% | 1 |
| CO | 70 | 68 (97.1%) | 0 | 57.4% | 44.1% | 45.6% | 1 |
| KS | 68 | 65 (95.6%) | 0 | 44.6% | 32.3% | 38.5% | 0 |
| AL | 67 | 61 (91.0%) | 1 | 49.2% | 42.6% | 34.4% | 0 |
| IA | 67 | 64 (95.5%) | 2 | 60.9% | 28.1% | 45.3% | 0 |
| SC | 67 | 66 (98.5%) | 0 | 37.9% | 37.9% | 37.9% | 2 |
| MD | 64 | 61 (95.3%) | 0 | 67.2% | 42.6% | 62.3% | 6 |
| WV | 61 | 57 (93.4%) | 11 | 35.1% | 21.1% | 52.6% | 2 |
| OR | 58 | 57 (98.3%) | 1 | 42.1% | 36.8% | 45.6% | 1 |
| UT | 51 | 46 (90.2%) | 0 | 52.2% | 30.4% | 41.3% | 1 |
| MS | 48 | 47 (97.9%) | 0 | 42.6% | 34.0% | 31.9% | 0 |
| CT | 44 | 42 (95.5%) | 0 | 66.7% | 35.7% | 40.5% | 0 |
| NE | 40 | 40 (100.0%) | 5 | 52.5% | 40.0% | 65.0% | 1 |
| NM | 38 | 35 (92.1%) | 2 | 40.0% | 45.7% | 28.6% | 0 |
| ME | 33 | 32 (97.0%) | 1 | 50.0% | 28.1% | 31.3% | 0 |
| NH | 29 | 28 (96.6%) | 0 | 53.6% | 53.6% | 42.9% | 1 |
| MT | 27 | 26 (96.3%) | 4 | 38.5% | 34.6% | 61.5% | 1 |
| ID | 25 | 25 (100.0%) | 0 | 60.0% | 40.0% | 40.0% | 0 |
| SD | 25 | 25 (100.0%) | 0 | 60.0% | 40.0% | 48.0% | 1 |
| HI | 23 | 21 (91.3%) | 1 | 42.9% | 33.3% | 33.3% | 0 |
| ND | 23 | 23 (100.0%) | 0 | 56.5% | 47.8% | 43.5% | 1 |
| DC | 21 | 21 (100.0%) | 0 | 42.9% | 28.6% | 47.6% | 0 |
| NV | 21 | 21 (100.0%) | 0 | 47.6% | 23.8% | 42.9% | 2 |
Named lists
Colleges without a security.txt (sample of 300) (first 60 of 300)
akbible.edu · tuskegee.edu · selmau.edu · umobile.edu · ussa.edu · mesacc.edu · pmi.edu · asu.edu · riosalado.edu · nau.edu · hcu.edu · herzing.edu · prescott.edu · gccaz.edu · southuniversity.edu · auburn.edu · phoenixcollege.edu · chartercollege.edu · stillman.edu · miles.edu · faulkner.edu · huntingdon.edu · brooklinecollege.edu · cochise.edu · aum.edu · athens.edu · uwa.edu · arizona.edu · gcu.edu · alaskapacific.edu · scottsdalecc.edu · alasu.edu · aamu.edu · eac.edu · jsu.edu · uas.alaska.edu · una.edu · prescott.erau.edu · www2.oakwood.edu · uamont.edu · ua.edu · hsu.edu · tsoa.edu · arkansasbaptist.edu · obu.edu · gatewaycc.edu · southmountaincc.edu · troy.edu · npc.edu · samford.edu · atu.edu · uab.edu · shc.edu · montevallo.edu · uaa.alaska.edu · ualr.edu · astate.edu · bst.edu · southalabama.edu · uapb.edu
Sites with an expired security.txt (14)
tsc.fl.edu · hilo.hawaii.edu · vinu.edu · unomaha.edu · unk.edu · princeton.edu · unl.edu · villanova.edu · tamu.edu · pvamu.edu · tamus.edu · westliberty.edu · uog.edu · lccc.wy.edu
Sites publishing an llms.txt (first 60 of 604)
selmau.edu · ussa.edu · pmi.edu · herzing.edu · jeffersonstate.edu · rsi.edu · aum.edu · athens.edu · carrington.edu · uti.edu · bst.edu · sautech.edu · bellusacademy.edu · uams.edu · calstate.edu · americancareercollege.edu · anc.edu · arizonachristian.edu · ozarks.edu · uark.edu · cbc.edu · academyart.edu · calbeautycollege.edu · caofb.com · empire.edu · ucsf.edu · sofia.edu · fresno.edu · idi.edu · fivebranches.edu · lcad.edu · fuller.edu · westernu.edu · labi.edu · haven.edu · nhi.edu · otis.edu · newschoolarch.edu · concorde.edu · epic.edu · pacificoaks.edu · simpsonu.edu · uei.edu · collegesoflaw.edu · sjvc.edu · yoec.edu · riohondo.edu · saybrook.edu · ibt.edu · rmcad.edu · coloradotech.edu · morgancc.edu · naropa.edu · schillingdouglas.edu · theconservatory.org · fairfield.edu · scf.edu · dawncareerinstitute.edu · fullsail.edu · lssc.edu
Sites telling ChatGPT-User and Claude-User not to open pages for users (48)
csudh.edu · msmu.edu · baisbinyominacademy.com · peru.edu · talmudicalacademynj.com · mesivtaofeasternparkway.com · rabbinicalcollegebethshraga.com · rabbinicalacademyrabbichaimberlin.com · torahtemimahtalmudicalseminary.com · kc.edu · osu.edu · gannon.edu · coastal.edu · brazosport.edu · yvcc.edu · abc.edu · baismedroshelyon.com · rabbijacobjoseph.com · yeshivasnovominsk.com · atenasuniversity.edu · digipen.edu · yeshivaofmachzikaihadas.com · vcfa.edu · bethmedrashmeoryitzchok.com · betmedrashgadolaterettorah.com · yeshivazichronaryeh.com · yeshivasholomshachna.com · centralyeshivabethjoseph.com · yeshivabaisaharon.com · mechonlhoyroa.com · yeshivaohrnaftoli.com · baismedrashmayanhatorah.com · yeshivagedolahkerenhatorah.com · yeshivasemekhatorah.com · yeshivachemdashatorah.com · alaskacareercollege.edu · pgcc.edu · bergen.org · bcc.cuny.edu · charzannebeautycollege.com · pstcc.edu · mercercountyschools.sites.thrillshare.com · cmtarroyo.com · bnoszionbobov.com · seminarybnoschaim.com · evvayloisschoolofbeauty.com · dtcc.edu · www2.cuny.edu
Sites blocking GPTBot (OpenAI's training crawler) (first 60 of 114)
lyon.edu · csudh.edu · msmu.edu · sksm.edu · baisbinyominacademy.com · fullsail.edu · mdc.edu · svdp.edu · highlands.edu · elmhurst.edu · ambs.edu · nku.edu · sullivan.edu · goucher.edu · umd.edu · ndm.edu · muih.edu · maritime.edu · usna.edu · hillsdale.edu · merrimack.edu · stkate.edu · csl.edu · tmcc.edu · anselm.edu · peru.edu · talmudicalacademynj.com · crcds.edu · mesivtaofeasternparkway.com · juilliard.edu · rabbinicalcollegebethshraga.com · rabbinicalacademyrabbichaimberlin.com · torahtemimahtalmudicalseminary.com · barton.edu · dickinsonstate.edu · villa.edu · macuniversity.edu · udayton.edu · kc.edu · osu.edu · ncstatecollege.edu · gannon.edu · sdstate.edu · sco.edu · coastal.edu · fisk.edu · brazosport.edu · ensign.edu · yvcc.edu · abc.edu · uwsp.edu · baismedroshelyon.com · rabbijacobjoseph.com · bmcc.edu · fnu.edu · yeshivasnovominsk.com · atenasuniversity.edu · digipen.edu · ecollege.edu · yeshivaofmachzikaihadas.com
Homepages that redirect to another domain (first 60 of 165)
ussa.edu → ussu.edu · prescott.erau.edu → erau.edu · www2.oakwood.edu → oakwood.edu · rstc.edu → www.reidstate.edu · pcec.edu → uofac.edu · arizonachristian.edu → azcu.edu · williamsbaptistuniversity.com → williamsbu.edu · eacc.edu → www.uaeacc.edu · csum.edu → maritime.calpoly.edu · fidm.edu → asufidm.asu.edu · pardeerand.edu → www.rand.edu · welcome.solano.edu → www.solano.edu · moler.org → moler.edu · ucdenver.edu → www.cudenver.edu · waynescollegeofbeauty.com → waynescollegeofbeautyinc.com · daytonabeach.erau.edu → erau.edu · hccfl.edu → www.hcfl.edu · lwtc.edu → lwtc.collierschools.com · georgestonecollege.edu → gstc.escambiaschools.org · palmbeachstate.edu → www.pbsc.edu · tfc.edu → utf.edu · truett.edu → tmu.edu · tbc.edu → tcj.edu · sacn.edu → www.osfhealthcare.org · sfmccon.edu → www.osfhealthcare.org · ega.edu → www.georgiasouthern.edu · vincennesbeautycollege.com → www.vincennesbeautycollege.org · westmichigancollegeofbarberingandbeauty.com → keluaransdy.biz · mcc.iavalley.edu → www.iavalley.edu · ecc.iavalley.edu → www.iavalley.edu · mchs.edu → mercyhealthu.edu · averyjamesschool.com → www.deltacollege.com · umf.maine.edu → farmington.edu · wwwcp.umes.edu → www.umes.edu · muih.edu → ndm.edu · middlesex.mass.edu → www.middlesex.edu · necc.mass.edu → necc.edu · bfit.edu → franklincummings.edu · mghihp.edu → www.mgbu.edu · arbor.edu → springarbor.edu · dbidownriver.edu → www.detroitbusinessinstitute.edu · otc.edu → ozarkstech.edu · midwestinstitute.com → www.midwestinstitute.edu · poplarbluffschools.net → www.pbmules.com · slchc.edu → stepfulcollege.edu · cccb.edu → centralbible.edu · eastwickcollege.edu → eastwick.edu · njcu.edu → www.kean.edu · bloomfield.edu → www.montclair.edu · jfkmuhlenbergschools.org → www.hackensackmeridianhealth.org · berktradeschool.com → electricalandplumbingschool.com · funeraleducation.org → aami.edu · www2.cuny.edu → www.cuny.edu · cvph.org → www.uvmhealth.org · nycpm.edu → nycpm.touro.edu · new.unca.edu → www.unca.edu · auburncc.org → www.auburncareercenter.org · collegeatsoutheastern.com → judsoncollege.com · firelands.bgsu.edu → www.bgsu.edu · acwhcc.org → www.heartlandtech.org
Sites with a Content-Signal line in robots.txt (59)
sautech.edu · epic.edu · fullsail.edu · svdp.edu · boisebible.edu · eureka.edu · ravenscroftcollege.com · bethanylb.edu · spatech.edu · hairacademymd.com · montgomerybeautyschool.com · awardbeautyschool.com · unionky.edu · hillsdale.edu · michigancollegeofbeauty.com · bmc.edu · juilliard.edu · tc.columbia.edu · carolina.edu · heritagebiblecollege.edu · psu.edu · austinbeautycollege.com · cruinstitute.com · texascollegecosmetology.com · universalcollegeofbeauty.com · academyofcosmetology.edu · huntingtonschoolofbeauty.com · bmcc.edu · estescosmetology.com · bennettcareerinstitute.org · omegainstitutes.com · hairacademy2.com · chestercareercollege.edu · intercoast.edu · escoffier.edu · lci.edu · ace.edu · mtnstmassage.com · marketticosmetology.com · littlerockbeautyschool.com · mandalynacademy.com · academyla.com · capilo.com · cintaaveda.edu · catholiciu.edu · cosmetologyacademyoftexarkana.com · jba.edu · electricaltrainingcenter.edu · americancollegeofbarbering.com · hvac-tech.com · caanacademy.org · elim.edu · entouragebeautyne.com · arkansasweldingacademy.edu · vskicosmetology.com · medicalcareercollege.edu · ctkhealthcare.com · baltimorebeautybarberschool2.com · bcc.edu
Colleges without an accessibility link on the homepage (sample of 300) (first 60 of 300)
akbible.edu · tuskegee.edu · selmau.edu · umobile.edu · mesacc.edu · pmi.edu · riosalado.edu · nau.edu · hcu.edu · herzing.edu · gccaz.edu · southuniversity.edu · auburn.edu · phoenixcollege.edu · chartercollege.edu · miles.edu · huntingdon.edu · brooklinecollege.edu · aum.edu · uwa.edu · gcu.edu · alaskapacific.edu · scottsdalecc.edu · eac.edu · una.edu · www2.oakwood.edu · uamont.edu · hsu.edu · tsoa.edu · arkansasbaptist.edu · obu.edu · gatewaycc.edu · southmountaincc.edu · samford.edu · uab.edu · uaa.alaska.edu · ualr.edu · bst.edu · uapb.edu · cca.edu · dinecollege.edu · uams.edu · uci.edu · uafs.edu · arizonachristian.edu · hendrix.edu · alliant.edu · ucsb.edu · cbc.edu · ucdavis.edu · afi.com · williamsbaptistuniversity.com · caltech.edu · artcenter.edu · saumag.edu · academyart.edu · bryanuniversity.edu · acchs.edu · csustan.edu · buc.edu
What colleges are expected to publish
- ADA Title II web rule for public institutions (WCAG 2.1 Level AA): Public colleges and universities are state and local government entities and must make their web content and mobile apps conform to WCAG 2.1 Level AA: entities serving 50,000 or more people from April 24, 2026, smaller ones from April 26, 2027 (28 CFR Part 35, subpart H). The census checks only whether the homepage links an accessibility statement. Source
- ADA Title III for private institutions: Private colleges are public accommodations under Title III of the Americans with Disabilities Act; the Department of Justice's position is that their websites must be accessible to people with disabilities. Source
- Section 504 of the Rehabilitation Act (34 CFR Part 104): Every institution receiving federal financial assistance, public or private, may not exclude people with disabilities from its programs and activities, which the Department of Education applies to websites and online courses. Source
- Higher Education Act consumer information (34 CFR 668.41-668.43): Institutions in the federal student aid programs must make consumer information such as costs, accreditation, refund policies, completion rates and campus safety readily available to enrolled and prospective students, and most publish it on their websites. Source
- Clery Act annual security report (34 CFR 668.46): By October 1 each year, institutions must distribute an annual security report with three years of campus crime statistics to current students and employees; posting it on the website with a notice is an accepted method. Source
- FERPA annual notification (34 CFR 99.7): Institutions must notify students annually of their rights under the Family Educational Rights and Privacy Act; many publish the notice online. The census checks only for a privacy link on the homepage. Source
- Title IX coordinator contact (34 CFR 106.8): Institutions must notify students and employees of the Title IX coordinator's name, office, email and telephone number and display that information prominently on their websites. Source
- RFC 9116 security.txt: The security.txt standard defines the /.well-known/security.txt location and requires Contact and Expires fields so researchers can report vulnerabilities; the census marks a file complete when both are present and unexpired. Source
Look up any college
Questions
How many college websites publish a security.txt?
146 of the 4,763 reachable college websites (3.1%) publish a security.txt; 111 are complete and current and 14 have expired.
How many colleges block AI crawlers?
114 college websites block GPTBot, 98 block ClaudeBot and 68 block ChatGPT-User in robots.txt; 48 block both ChatGPT-User and Claude-User; 448 name at least one AI crawler explicitly; 604 publish an llms.txt.
Do college websites link an accessibility statement?
33.2% of reachable college homepages link an accessibility statement and 46.4% link a privacy policy. 45.0% send an HSTS header.
Which institutions are included?
Active institutions in the IPEDS 2024-25 institutional characteristics file (HD2024) that list a website address: 4,961 domains on 2026-09-30, after merging campuses and branches that share one domain. The table by control and level separates public, private nonprofit and private for-profit institutions at each level.
Can I reuse the data?
Yes. The findings, the CSV and every per-college JSON record are published under CC BY 4.0; cite Agent Exchange and link the report.
Method and limits
Population: NCES IPEDS 2024-25 institutional characteristics header file HD2024 (active Title IV institutions with a web address) with each institution's control (public, private nonprofit, private for-profit), level (4-year, 2-year, less-than-2-year), city, state and IPEDS sector.. Fetch: one HTTPS request to the apex (then www.), following redirects, 8-second timeout, user agent agentexchange-census/1.0; then robots.txt, /.well-known/security.txt, /security.txt, /llms.txt and /.well-known/agent-card.json on the final host. "Reachable" means any HTTP response over HTTPS, including 403s from bot protection. Homepage links are detected from anchor targets and visible text, so a policy that exists but is not linked from the homepage is not counted. A robots.txt "blocked" verdict means the agent's own group (or the * group) disallows the whole site; "partly blocked" means some paths are disallowed. One snapshot per domain; slow sites and sites that block automated requests are counted as unreachable or reachable-with-error as the server answered. Per-domain records: lookup; downloads: CSV, NDJSON and JSON (CC BY 4.0).
Run one of these sites?
Instant AI visibility report for your college: $9 · $49 full report or $99 monthly monitoring · Hosted accessibility feedback and privacy request desk
Other censuses
Government web census (.gov) · US bank web census · US school district web census · US public library web census · Top 100,000 websites census · AI crawler census (top 10,000 sites)