Daily Refresh vs Change-Triggered Refresh: How to Choose for Each Source
Ask a team that buys an external data feed what comes up at renewal, once the bill and the freshness stop lining up: should we refresh daily, or only when something changes? It sounds like one data refresh frequency setting. It is a decision per source.
There is rarely one right model for a whole feed. Refresh on a schedule where values change constantly, where a missed change cannot be recovered, or where you need a dated record every day. Refresh on change where most visits would find nothing and the change is visible to a light check. Most feeds end up mixed, set source by source and often field by field.
One honest caveat: on most websites, "change-triggered" still means checking on a schedule. What changes is what happens after the check.
By the end, you will settle each source on four properties, know the failure each model hides, and hold nine scope lines for your next provider conversation.
- What "change-triggered" means: on most websites, a frequent cheap check plus a comparison, not a signal from the website.
- How to decide: per source, often per field, on four properties: change rate, cost of a late or missed change, change signal, and history.
- What each model hides: daily is late by design; a detector broken by a redesign reports the same silence as an unchanged page.
- What to write down: the change definition, the full re-baseline and the billing unit belong in the scope.
Daily vs change-triggered refresh at a glance
Run your source list against this table; the four questions below settle close calls.
| If the source or field... | Use | Example |
|---|---|---|
| Changes on most visits | Scheduled refresh, at the lag you can tolerate | A competitor's price page: a trigger would fire on most visits anyway |
| Changes rarely, and a light check can see the change | Change-triggered refresh, plus a periodic full pass | Policy, terms or documentation pages |
| Publishes a reliable change feed or file | Triggered from the feed, plus scheduled reconciliation | The weekly National Provider Identifier (NPI) change files |
| Changes silently, with no public event | Scheduled refresh, by field | A provider's practice affiliation |
| Must be shown as it stood on every date | Scheduled snapshots, or a change log with as-observed timestamps | A point-in-time history for audit or backtest |
| Has a freshness clock set by someone else | Mixed: a calendar for the clock, a trigger for reported changes | Health-plan provider directories |
One source can match two rows, which is why most lists end up mixed.
What does each refresh model actually do?
Start with what the two names hide: both models run on a schedule. Here "daily" stands for any fixed schedule, hourly or weekly included.
Scheduled full refresh re-collects every page on every run. Its data is up to one interval behind the source, about half on average, plus processing (arithmetic, not a measurement). Warehouse teams call the same split full refresh vs incremental refresh.
Change-triggered refresh, on most websites, is a scheduled check plus a comparison, with full extraction only when the comparison finds a change. It does not mean the website tells you it changed.
Push exists only where the publisher built it, usually for its own participants. W3C's WebSub standard lets a publisher announce updates to subscribers. Amazon's Selling Partner API sends offer-change notifications only to sellers with an active offer on that item. The Centers for Medicare & Medicaid Services (CMS) publishes weekly files from its NPI registry, NPPES, covering "newly assigned NPIs, updates to NPI data, and newly deactivated NPIs covering that week."
Even the cheapest check is a visit. In HTTP, a page's Last-Modified date is the time the server "believes" it last changed, and the standard does not require servers to send it. A conditional request lets a collector ask whether a page has changed and skip downloading it if not, so it saves the transfer, not the visit. Where a website prices crawler requests, as Cloudflare's pay-per-crawl does for AI crawlers (closed beta as of October 2026), a check that finds nothing still costs money.
So the cost splits three ways:
- Visits: both models pay them.
- Extraction and delivery: daily pays on every run; change-triggered pays only on a change.
- Downstream processing: the same split.
Whatever teams typically watch, the saving is never in the visits.
How do you choose a refresh model per source?
Ask four questions, in this order, for each source and often for each field.
1. How often does the part you use change?
Count fields, not pages. A page that "changes daily" because of an ad or a timestamp has not changed for you.
The published measurements of how often pages change are old, with data collected between 1999 and 2007, but four large studies agree on a shape. A large share of pages did not change for months, a smaller share changed on almost every visit, and much of what registered as change was cosmetic: in a 2002–2003 weekly crawl of 150 million pages, the authors found that web pages that change "usually either change only in their markup or in trivial ways". A 2004 study also found that how often a page changes does not predict how much it changes, which is why a scope has to name the fields that matter.
We found no newer method-disclosed measurement as of October 2026. The lean: scheduled when your fields move on most visits; change-triggered when they sit quiet.
2. What does a late or missed change cost?
Ask about late and missed separately, and whether either is recoverable.
For a team keeping a retrieval corpus current, a late change costs more than it looks. In an ACL 2025 benchmark built from Wikipedia edits, adding a single outdated passage cut a 70B model's overall score by more than 24%, while six merely distracting passages moved its accuracy by less than 1%. It also changes what a change-triggered refresh has to do: the detected change must replace or retire the old version, because refreshing more often does nothing if the index still holds both.
The lean: scheduled when you need every state, including ones that vanish between visits, or when a day late is fine; change-triggered when only the current state matters and a day late is too late.
3. Does the source tell you when it changed?
Yes (a published feed), partly (dates that may be wrong) or no, which covers most pages.
| Signal | What it tells you | How it fails | What to ask |
|---|---|---|---|
| Published change feed or delta file | Which records changed | Covers only the publisher's own fields (NPPES weekly files cover registry data, not practice moves) | Which fields does the feed cover, and what reconciles it? |
| Server dates and validators (Last-Modified, ETag) | What the server "believes" changed (RFC 9110) | Optional; Google reported in December 2024 that only 0.017% of its crawler's fetches could be answered from its cache | Has this signal been checked against observed changes on this source? |
| Sitemap lastmod | The date the page last changed, "not when the sitemap is generated" (sitemaps.org) | Google uses it only if it proves "consistently and verifiably" accurate; Bing found about one in five sitemaps that set it set it wrongly (2023) | Same |
| None (most pages) | Nothing | A trigger is a frequent light check plus a comparison | What is checked, how often, and what counts as a change? |
A date is a claim, not proof. The lean: change-triggered only on a feed you have verified; otherwise scheduled, or a trigger on your own light check.
4. Do you need proof of the value on every date?
Audit, backtesting and point-in-time reporting need the value as it stood on each date. Scheduled snapshots give you that by construction. A change log can stand in, but only when it records that the value was checked and found unchanged, with detection time, and only if detection was complete. A log with gaps cannot rebuild a past state, and a refresh that overwrites old values destroys it. Snapshots cost more; change logs cost completeness.
Three sources through the four questions
The verdicts in the last row are our reasoning, not a published rule.
| Competitor price page | Provider record (NPI plus affiliation) | Document library for a retrieval index | |
|---|---|---|---|
| 1. Field change | Price, availability | Registry rarely; affiliation silently | Mostly quiet; new ones unannounced |
| 2. Late or missed | A missed price usually leaves no trace | Plan's legal clocks: 90 days, and 2 business days of receiving a provider's change | One outdated passage cut a 70B model's score by more than 24% |
| 3. Signal | None | Registry yes; affiliation no | Partly, once verified |
| 4. History | Often | Verification records | Current state; retire old versions |
| Fits | Scheduled, on the price field | Mixed: weekly files, monthly reload, calendar for affiliation | Triggered on a verified signal, plus a full pass |
When the answers split, split the source: put the fields that need a calendar on one schedule and let the rest trigger.
The hidden failure in each model
Whichever model a source gets, it fails in a way no run report shows.
Daily refresh
Daily refresh is late by design, even when every run succeeds. It is also blind to anything that appears and vanishes between visits. In a 2023 study of one moderated Q&A platform, using the operator's own records of when answers were posted and removed, collecting ten minutes after posting would have missed 2.1% of answers and collecting eight hours after would have missed 5.4%, because they were already gone. A run that succeeded is not a value that is right.
Change-triggered refresh
Change-triggered refresh misses what its check cannot see: content behind logins or loaded by scripts, and pages that look different to different viewers. It also flags noise: in an hourly crawl (2007 data, published 2009) the authors found "many instances of randomly changing advertisements", plus page-served times and visitor counters.
Competitor pricing is where both failure modes bite at once. A retail price at a past moment usually leaves no independent record, so a change that lands and reverts between two daily visits is gone for good. Change alerts are noisy in the other direction: when a product disappears from a page, one comparison-site operator listed the possible causes as a scraper error, a block, a site change or the site being down for maintenance, any of which looks like a real removal. The common fix, requiring several matching observations before alerting, cuts false alarms by delaying the real ones.
The headline failure is quieter.
No alert is not proof of no change. After a redesign, a broken detector reports exactly what an unchanged page reports: nothing.
Silence is ambiguous unless the scope demands a separate "could not check" state, and a periodic full re-baseline is the only control.
Visiting quiet pages less often saves visits and costs a completeness guarantee, as one large research crawler describes.
"Instead, our crawler forms an idea of how fast particular pages change, notices pages that change very slowly or never, and crawls those pages less and less often. This is good because it allows the crawler to focus on newly discovered pages, and pages that frequently change. But it's bad in that it makes it hard to understand how complete the dataset is."
Greg Lindahl, Chief Technology Officer, Common Crawl Foundation, "Measuring Crawled Coverage of a Website in Common Crawl", 20 July 2026.
Why do most feeds end up mixed?
Because each model fails differently, most source lists land on a written mix of three parts:
- A cheap check on a schedule: a conditional request, a delta file or a single-field read.
- Full extraction on a confirmed change: under a written confirmation rule.
- A periodic full re-baseline: catches what the checks missed, including a silently broken detector, and finds new pages a known-URL check never visits.
The re-baseline recurs, and what it catches feeds back into the change definition. Even sitemaps.org notes that "Crawlers may periodically crawl pages marked 'never' so that they can handle unexpected changes to those pages."
An event trigger only works for changes that leave a public trace; fields that change silently need a calendar. One provider record can need both models at once. The federal NPI registry publishes weekly change files and a monthly deactivation file alongside a monthly full file, and CMS tells users to reload the full file each month to stay current: the registry's own fields get a change feed plus a re-baseline. Nothing comparable exists when a provider changes practice: the 30-day update duty covers only some providers and only registry fields, so affiliation has to be re-checked on a calendar. As of October 2026, US law for health plans sets both clocks: verify every directory listing at least every 90 days, and update within 2 business days of receiving a provider's change. The regulator did not pick one model. It wrote both into the same rule.
Outside healthcare the lesson holds: what counts as a change is a scope decision, not a tool setting.
"As for what you consider a change that requires clients to refresh their caches, that's up to you. Our recommendation is that you require a cache refresh on significant changes to your content; if you only updated the copyright date at the bottom of your page, that's probably not significant."
Gary Illyes, Google Search Central, "Crawling December: HTTP caching", 9 December 2024.
Forage AI runs website change monitoring with real-time alerts as a fully managed service, built to each client's own fields, so the client decides what is watched and what counts as a change, and the data stays the client's.
Data freshness SLA lines to copy into your scope
A written mix needs a scope, and the scope has to say which clock it measures. Data-quality practice already separates the two clocks: DAMA Netherlands' 2020 dimensions paper measures timeliness from when the real-world value was created to when the data is available, and treats on-time delivery against a schedule as a different property, punctuality. Ask for the first.
Self-serve monitoring tools usually meter the check itself, while managed-extraction quotes are usually built on sources, records and frequency. Before you compare quotes, ask which unit you are billed in (line 8 below). Write the target down and you can alert against it.
These nine lines are our recommendation, not an industry standard; the brackets are yours.
1. Freshness target: [field or source] is no more than [X hours/days] behind the source, measured from when the source changed, not from when the job ran.
2. Change detection: for each source, state the signal used (published feed, API filter, page check plus comparison), how often it is checked, and the fallback when the signal is missing or wrong.
3. What counts as a change: only changes to these fields count: [list], with these thresholds: [list]. Ignore ads, timestamps, layout and markup.
4. Confirmation rule: state how a change is confirmed before it is delivered, and the delay that rule adds.
5. Removed vs missing: a record is reported as removed only after [N] checks or a confirmed removal at source. Blocks, errors and empty pages are reported separately, by type.
6. Re-baseline: a full collection runs every [period] to catch changes the checks missed, and the number it caught is reported each time.
7. Misses and silence: "no change" must be distinguishable from "could not check". Every change found late or by the re-baseline is logged with when it happened at source and when it was delivered.
8. Billing unit (ask): are we billed per source, per check, per extraction or per delivered change?
9. History: keep [every observation / every change with before and after values and an as-observed timestamp] for [period]. Deletions and superseded versions are part of the delivery.
Start with your own source list
The choice is per source, a mix is normal, and it holds only while the scope names the change, the re-baseline and the billing unit. Take your own list through the four questions first.
Forage AI builds and runs managed extraction and change monitoring to your fields, with real-time alerts delivered by API or webhook into your own systems. To see how the split would fall on your sources, start with a change audit of your actual target pages.
FAQ
How often should a data provider refresh the data?
Per source, and often per field, decided on four properties: how often the fields change, what a late or missed change costs, whether the source signals a change, and whether you need history. Then ask for freshness lag at delivery, not run frequency: a data refresh rate says how often a job runs, not how far behind the source the value is.
Is change-triggered refresh cheaper than daily refresh?
On extraction, delivery and processing, often; on visits, no. A conditional check saves the transfer, not the visit, and websites that price crawler requests charge for checks too. Whether you save depends on the billing unit.
Can a change-triggered setup miss changes?
Yes, in three ways: it misses what its check cannot see, a check of known URLs never finds new pages, and a detector broken by a redesign reports the same silence as an unchanged page. A periodic full re-baseline is the control for all three.
Related articles
- Website Change Monitoring: Your Competitive Advantage: what teams watch with change monitoring, and why.
- Data Quality Dimensions: how to measure timeliness and the lags behind it.
- Top Competitor Price Tracking Tools: how pricing teams track competitors, source by source.
- US Healthcare Provider Data Sources: the public provider data sources and how each one refreshes.
- RAG Data Ingestion Best Practices: keeping a retrieval index current, including deletions.
Sai is a data infrastructure enthusiast who has spent the past two to three years following the AI space closely, from the infrastructure layer to the fast-growing world of data for AI. He is genuinely curious about how modern data pipelines get built and where the data industry is heading, and he writes insightful pieces on the core topics that shape this niche.