Data refresh frequency is how often a dataset is rebuilt from its sources. It sounds like a single number, and it is quoted like one, but a buyer who acts on that single number will regularly find the data on screen older than the cadence they were sold. There are four clocks involved, and only the last of them affects a decision.
Four clocks, not one#
| Clock | What it measures | Who usually quotes it |
|---|---|---|
| Collection cadence | How often the vendor gathers records from source | The vendor, in marketing material |
| Processing lag | Time from collection to a record being cleaned, classified and reconciled | Rarely quoted; ask for it |
| Publication cadence | How often processed data is released into the product | Sometimes quoted, often confused with collection |
| Effective freshness | Age of the newest usable record when the buyer opens the dashboard | The number that matters |
Effective freshness is roughly the sum of the other three, and the difference can be large. A dataset collected daily, processed in a weekly batch and published monthly has a daily collection cadence and an effective freshness measured in weeks. Neither party is being dishonest — they are quoting different clocks. The buyer's job is to ask for the fourth one explicitly: on an average day, how old is the newest record I can see?
Match cadence to the decision cycle#
The correct cadence is a property of the decision, not of the dataset. Buying a faster refresh than the decision requires costs money and adds revision noise; buying a slower one means acting late.
| Decision | Cadence that supports it |
|---|---|
| Responding to a competitor's price move | Daily to weekly |
| Adjusting promotional plans in-season | Weekly to monthly |
| Competitive positioning review | Monthly |
| Category sizing and annual planning | Monthly to quarterly |
| Multi-year trend and structural research | Quarterly, with a consistent history |
Two practical consequences follow. First, most market intelligence questions inside a commercial organisation sit in the middle rows, and the middle rows are well served by a monthly cadence. Second, buying for the fastest row when your actual decisions live in the middle is a common and expensive procurement error — the speed is paid for and never used, while the dimensions that would have mattered, such as coverage depth and history length, went unexamined.
Why faster is not automatically better#
Freshness trades against stability, and the trade is structural rather than a sign of poor engineering.
Early periods are incomplete by construction. Returns, cancellations, disputed transactions and late-settling orders resolve over days or weeks. A figure published before they settle is a first estimate, and first estimates move. The fresher the figure, the more likely it is to be revised.
Short windows amplify noise. A category's week-to-week movement is dominated by promotional calendars, platform events and seasonal effects. Reading a strategic signal off a single fresh week usually means reading a promotion. The signal a buyer wants generally requires a window long enough for those effects to average out.
Frequent publication without a revision policy is worse than a slower cadence. If figures change quietly after release, then any analysis circulated internally has an unknown shelf life. A vendor that publishes monthly and states a revision window is more useful than one that publishes weekly and never says whether the numbers move.
The buyer's rule that follows: choose the slowest cadence that still lets the decision be made on time, and spend the difference on coverage and history instead.
Red flags in a cadence claim#
- A single site-wide figure. Cadence varies by source and by platform; an aggregate claim conceals the variation. Ask per data type.
- A vendor claiming
"real-time"without naming a subject. A vendor using the phrase may mean collection, processing or availability — three different claims, and the phrase alone distinguishes none of them. The follow-up question is simply: for which data, measured from what event to what event? - Collection cadence quoted as though it were availability. The most common substitution, and usually not deliberate.
- No stated processing lag. Every pipeline has one. A vendor that has never measured it does not know its own effective freshness.
- No revision or restatement policy. See Data Provenance and Due Diligence — this belongs to provenance as much as to cadence.
- Cadence claims that do not match the sample. The cheapest test of all: open the sample dataset supplied during evaluation and look at the date of the newest record. Compare it to what was said in the meeting.
Questions to ask a vendor#
- For each data type we intend to use, what is the collection cadence, the processing lag and the publication cadence — as three separate numbers?
- On a typical day, what is the age of the newest record available to a user?
- Does effective freshness vary by platform or by category within the dataset? Where is it slowest?
- Are published figures revised after release? Within what window, and are buyers notified?
- When methodology changes, is history restated, and is the change dated and documented?
- What happens to freshness when a source has a collection failure — is the gap filled, flagged, or left empty?
Question six is worth asking even though it sounds operational. The answer reveals whether the vendor treats a gap as something to disclose or something to smooth over, and that disposition tends to generalise to everything else in the dataset.
Cadence in the wider comparison#
Refresh frequency is one dimension among several, and it is the one most likely to be over-weighted, because it is the easiest to demonstrate in a sales meeting. A dashboard that updates while the buyer watches is memorable; a documented restatement policy is not. The comparison framework in Market Intelligence Platform Comparison Criteria exists partly to correct for that — cadence should be scored alongside Data Coverage and Completeness and source quality rather than ahead of them.
Where to look next#
For what determines the achievable cadence in the first place, see Market Intelligence Data Sources. For the audit trail behind a published figure, see Data Provenance and Due Diligence. For the full procurement process, see How to Evaluate Market Intelligence Providers.
Common questions#
What is the difference between collection cadence and effective freshness?#
Collection cadence is how often the vendor gathers data from source. Effective freshness is the age of the newest usable data point at the moment a buyer looks at the dashboard, and it is the only one of the two that affects a decision. The gap between them is processing lag plus publication cadence. A dataset collected daily but processed in a weekly batch and published on the first of the month has a daily collection cadence and an effective freshness that can exceed a month. Vendors quote the first number because it is the most impressive one. Buyers should ask for the second, phrased as: on an average day, how old is the newest record I can see?
Is a faster refresh always better?#
No, and assuming so leads to a predictable mistake. Faster cadence buys responsiveness and costs stability. Early figures for a period are incomplete by construction — returns, cancellations and late-settling transactions have not yet resolved — so a fresher number is more likely to be revised. For a pricing response that is an acceptable trade, because acting a week earlier is worth more than a small revision. For annual planning or a board-level baseline it is a poor trade, because the plan gets built on a figure that later moves. The right cadence is the slowest one that still lets the decision be made on time.
Why does refresh frequency vary by platform within the same dataset?#
Because the constraint is rarely the vendor's own systems. It is what each source makes available and how often, what volume has to be processed and reconciled, and what commercial or technical terms govern access. A social platform publishing a continuous public stream of short posts can support a daily rebuild. A marketplace whose sales figures must be derived from demand signals and then reconciled across sellers and variants generally cannot. This is why a single site-wide cadence claim is close to meaningless, and why the useful question is per data type and per platform rather than in aggregate.
What should a buyer ask about restatement?#
Three questions. First, are published figures ever revised after their initial release, and within what window? Second, when a revision occurs, is the buyer notified, or does the number simply change under an already-circulated analysis? Third, when the underlying methodology changes, is the history restated to match, and is the change dated and documented? A vendor with no revision policy at all is usually not more accurate — it is more likely publishing first estimates and leaving them in place. A documented revision window is a sign of operational maturity, not a weakness, and it is what lets a buyer know whether a figure quoted in a board pack will still be that figure next quarter.