public class AdaptiveFetchSchedule extends AbstractFetchSchedule
delta = fetchTime - modifiedTimefetchTime + fetchInterval - delta * SYNC_DELTA_RATEfetchInterval = delta.
NOTE: values of DEC_FACTOR and INC_FACTOR higher than 0.4f may destabilize
the algorithm, so that the fetch interval either increases or decreases
infinitely, with little relevance to the page changes. Please use
main(String[]) method to test the values before applying them in a
production system.
| Modifier and Type | Field and Description |
|---|---|
protected float |
DEC_RATE |
protected float |
INC_RATE |
defaultInterval, maxIntervalSECONDS_PER_DAY, STATUS_MODIFIED, STATUS_NOTMODIFIED, STATUS_UNKNOWN| Constructor and Description |
|---|
AdaptiveFetchSchedule() |
| Modifier and Type | Method and Description |
|---|---|
static void |
main(String[] args) |
void |
setConf(Configuration conf) |
CrawlDatum |
setFetchSchedule(Text url,
CrawlDatum datum,
long prevFetchTime,
long prevModifiedTime,
long fetchTime,
long modifiedTime,
int state)
Sets the
fetchInterval and fetchTime on a
successfully fetched page. |
calculateLastFetchTime, forceRefetch, initializeSchedule, setPageGoneSchedule, setPageRetrySchedule, shouldFetchgetConfclone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, waitgetConfpublic void setConf(Configuration conf)
setConf in interface ConfigurablesetConf in class AbstractFetchSchedulepublic CrawlDatum setFetchSchedule(Text url, CrawlDatum datum, long prevFetchTime, long prevModifiedTime, long fetchTime, long modifiedTime, int state)
AbstractFetchSchedulefetchInterval and fetchTime on a
successfully fetched page. NOTE: this implementation resets the retry
counter - extending classes should call super.setFetchSchedule() to
preserve this behavior.setFetchSchedule in interface FetchSchedulesetFetchSchedule in class AbstractFetchScheduleurl - url of the pagedatum - page description to be adjusted. NOTE: this instance, passed by
reference, may be modified inside the method.prevFetchTime - previous value of fetch time, or 0 if not available.prevModifiedTime - previous value of modifiedTime, or 0 if not available.fetchTime - the latest time, when the page was recently re-fetched. Most
FetchSchedule implementations should update the value in @see
CrawlDatum to something greater than this value.modifiedTime - last time the content was modified. This information comes from
the protocol implementations, or is set to < 0 if not available.
Most FetchSchedule implementations should update the value in @see
CrawlDatum to this value.state - if FetchSchedule.STATUS_MODIFIED, then the content is considered to be
"changed" before the fetchTime, if
FetchSchedule.STATUS_NOTMODIFIED then the content is known to be
unchanged. This information may be obtained by comparing page
signatures before and after fetching. If this is set to
FetchSchedule.STATUS_UNKNOWN, then it is unknown whether the page was
changed; implementations are free to follow a sensible default
behavior.Copyright © 2021 The Apache Software Foundation