The Freshness Problem
When you aggregate data from 50+ directories, the data is never perfectly synchronized. Directory A might update hourly. Directory B might update weekly. Directory C might be maintained by a volunteer who updates it when they have time. The result is a dataset where some entries were last verified minutes ago and others weeks ago.
Users expect the data they see to be current. If they're checking an MCP server's security grade, they want the grade to reflect the tool's current dependency state, not its state from last month. If they're looking at download counts, they want recent numbers, not stale ones.
Crawling Strategies
Different sources warrant different crawling frequencies. Package registries (npm, PyPI) that update continuously are crawled frequently. Community-maintained awesome lists that change weekly are crawled less often. GitHub repository metrics are fetched periodically since they change relatively slowly.
Priority-based crawling helps allocate resources. Popular tools (higher download counts, more stars, more directory presence) are refreshed more frequently than less-used ones. This ensures that the tools most people are looking at have the freshest data.
Event-driven crawling supplements scheduled crawling. When a tool publishes a new version on npm, that event triggers a data refresh for that specific tool rather than waiting for the next scheduled crawl. This approach keeps actively-developed tools current without increasing the overall crawl load.
Staleness Detection
Each data point has a timestamp indicating when it was last verified. Staleness thresholds define how old data can be before it needs refreshing. A download count from two days ago is probably fine. A security score from six months ago might be misleading if dependencies have changed.
Different data types have different staleness tolerances. Static metadata (tool name, author, description) rarely changes and can be cached longer. Dynamic metrics (download counts, star counts, security scores) change more frequently and need more frequent updates. Volatility-based caching adjusts refresh rates to match how quickly each data type actually changes.
Score Recalculation
Security scores and quality metrics need periodic recalculation as underlying data changes. A tool's security grade might change when a dependency publishes a security patch. A tool's quality score might change when its maintenance activity increases or decreases.
Full score recalculation for 100,000+ tools is computationally expensive. Incremental recalculation, triggered by changes in underlying data, is more efficient. When a tool's dependency tree changes, only that tool's security score needs recalculation. When a tool gets added to a new directory, only its directory presence score updates.
Communicating Freshness
Transparency about data freshness helps users calibrate their trust in the information they see. Showing "last updated" timestamps, indicating when scores were last recalculated, and flagging data that might be stale all help users make informed decisions.
When data can't be verified (because a source is temporarily unavailable, for example), the appropriate response is to show the last known data with a staleness indicator rather than removing the tool from results. Users can then decide whether the last-known data is recent enough for their purposes.
Related Reading
- How Skillful.sh Aggregates Data from 50+ Directories
- How Trending Detection Works for AI Tools
- How Search Works Across 100,000+ AI Tools
Search 137,000+ AI tools on Skillful.sh. View ecosystem statistics.