Build Scalable API Analytics Without Fetching Every Record

If your dashboard needs a trend, fetching every underlying record is often the wrong bargain: more work, more data to handle, and detail your users may not need.

GitHub reports rebuilding its repository star-history API from 2,500 requests to 24. Its summary says the API provides trends without returning individual identities or requiring unbounded database work. That is a useful design prompt for any small team: can your API answer the question with an aggregate instead of a pile of records?

Start with the question, not the records

Suppose a product page needs to show how sign-ups changed week by week. If the interface only displays a weekly trend, retrieving every sign-up record and counting them in the client is unnecessary. The API can return counts grouped by week instead.

This is not a rule to aggregate everything. Individual records remain necessary for tasks such as investigating a specific account or exporting data. The decision is about matching the response to the job: use detailed records when people need detail; use a summary when they need a pattern.

Migrate from record fan-out in four steps

1. Inventory the question. Identify the chart or report, who uses it, and the exact decisions it supports. Write down the dimensions it needs, such as time period or product category.
2. Define a bounded response. Specify the grouping, time range, and filters the endpoint accepts, plus the shape of the aggregate it returns. Avoid an endpoint that silently retrieves an unlimited history.
3. Implement the summary at the data source. Calculate grouped results close to the stored data rather than asking the client to fetch and combine every underlying item. Keep the query limits and response size explicit.
4. Compare before switching. Run the new response alongside the current report for representative periods. Check that totals and grouping match, then move the consumer over and retire the record-by-record path if it has no other users.

Choose what to retain—and what to leave out

Aggregates can reduce unnecessary exposure of individual-level information, but they also remove detail. A weekly count cannot answer which specific account contributed to a change. Keep access to underlying records only where a real operational need justifies it, and protect that path separately.

Act now if your dashboard or integration makes repeated requests for individual records only to calculate totals or trends. Wait if users genuinely need the underlying records, or if you have not confirmed the report’s required filters and time resolution. Do not replace a useful drill-down with an aggregate that cannot support the work.

Make the response fit the decision

GitHub’s reported redesign is an example, not proof that every API should return summaries. The practical lesson is to measure whether your consumers need identities, records, or just a trend—and design the smallest response that serves them.

Takeaway: Before scaling a data-heavy endpoint, ask what decision its consumer is making. Your next action: trace one report back to the records it fetches and see whether a bounded aggregate would answer the same question.

Source: GitHub’s account of rebuilding star history without the stargazers.