Module fetching 

Asynchronous retrieval of missing data.

FetchingDataSource combines a local storage implementation with a remote data availability provider to create a data sources which caches data locally, but which is capable of fetching missing data from a remote source, either proactively or on demand.

This implementation supports three kinds of data fetching.

Proactive Fetching

Proactive fetching means actively scanning the local database for missing objects and proactively retrieving them from a remote provider, even if those objects have yet to be requested by a client. Doing this increases the chance of success and decreases latency when a client does eventually ask for those objects. This is also the mechanism by which a query service joining a network late, or having been offline for some time, is able to catch up with the events on the network that it missed.

Proactive fetching is implemented by a background task which performs periodic scans of the database, identifying and retrieving missing objects. This task is generally low priority, since missing objects are rare, and it will take care not to monopolize resources that could be used to serve requests.

Active Fetching

Active fetching means reaching out to a remote data availability provider to retrieve a missing resource, upon receiving a request for that resource from a client. Not every request for a missing resource triggers an active fetch. To avoid spamming peers with requests for missing data, we only actively fetch resources that are known to exist somewhere. This means we can actively fetch leaves and headers when we are requested a leaf or header by height, whose height is less than the current chain height. We can fetch a block when the corresponding header exists (corresponding based on height, hash, or payload hash) or can be actively fetched.

Passive Fetching

For requests that cannot be actively fetched (for example, a block requested by hash, where we do not have a header proving that a block with that hash exists), we use passive fetching. This essentially means waiting passively until the query service receives an object that satisfies the request. This object may be received because it was actively fetched in responsive to a different request for the same object, one that permitted an active fetch. Or it may have been fetched proactively.

Modules

block 🔒

Fetchable implementation for BlockQueryData and PayloadQueryData.

header 🔒

Header fetching.

leaf 🔒

Fetchable implementation for LeafQueryData.

transaction 🔒

Transaction fetching.

vid 🔒

Fetchable implementation for VidCommonQueryData.

Structs

AggregatorMetrics 🔒

Builder

Builder for FetchingDataSource with configuration.

Fetcher 🔒

Asynchronous retrieval and storage of Fetchable resources.

FetchingDataSource

The most basic kind of data source.

Heights 🔒

Notifiers 🔒

Pruner

ScannerMetrics 🔒

Traits

AvailabilityProvider

A provider which can be used as a fetcher by the availability service.

FetchRequest 🔒

Fetchable 🔒

Objects which can be fetched from a remote DA provider and cached in local storage.

RangedFetchable 🔒

ResultExt 🔒

Storable 🔒

An object which can be stored in the database.

Functions

passive 🔒

Turn a fallible passive fetch future into an infallible “fetch”.

range_chunks 🔒

Break a range into fixed-size chunks.

range_chunks_rev 🔒

Break a range into fixed-size chunks, starting from the end and moving towards the start.

select_some 🔒

Get the result of the first future to return Some, if either do.

Type Aliases

PassiveFetch 🔒