The Site Scanning program builds on a series of earlier projects to offer real-time, automatic data about all federal websites. It begins by taking the .gov registry export and joins several additional data sources to aggregate a list of subdomains within those domains. The result is approximately 25,000 websites on 1,200 domains.
The system then runs a series of scans against the complete list of websites daily, making the results available as open data. While many users want the data in a spreadsheet or machine-readable file, we’ve also created an API that allows interactive access to the data.