Struct Builder

pub struct Builder<Types, S, P> {

    storage: S,
    provider: P,
    backoff: ExponentialBackoffBuilder,
    rate_limit: usize,
    range_chunk_size: usize,
    minor_scan_interval: Duration,
    major_scan_interval: usize,
    major_scan_offset: usize,
    proactive_range_chunk_size: Option<usize>,
    active_fetch_delay: Duration,
    chunk_fetch_delay: Duration,
    proactive_fetching: bool,
    aggregator: bool,
    aggregator_chunk_size: Option<usize>,
    types_migration_batch_size: u64,
    leaf_only: bool,
    _types: PhantomData<Types>,
}

Builder for FetchingDataSource with configuration.

Fields

storage: S

provider: P

backoff: ExponentialBackoffBuilder

rate_limit: usize

range_chunk_size: usize

minor_scan_interval: Duration

major_scan_interval: usize

major_scan_offset: usize

proactive_range_chunk_size: Option<usize>

active_fetch_delay: Duration

chunk_fetch_delay: Duration

proactive_fetching: bool

aggregator: bool

aggregator_chunk_size: Option<usize>

types_migration_batch_size: u64

leaf_only: bool

_types: PhantomData<Types>

Implementations

impl<Types, S, P> Builder<Types, S, P>

pub fn new(storage: S, provider: P) -> Self

Construct a new builder with the given storage and fetcher and the default options.

pub fn leaf_only(self) -> Self

pub fn with_min_retry_interval(self, interval: Duration) -> Self

Set the minimum delay between retries of failed operations.

pub fn with_max_retry_interval(self, interval: Duration) -> Self

Set the maximum delay between retries of failed operations.

pub fn with_retry_multiplier(self, multiplier: f64) -> Self

Set the multiplier for exponential backoff when retrying failed requests.

pub fn with_retry_randomization_factor(self, factor: f64) -> Self

Set the randomization factor for randomized backoff when retrying failed requests.

pub fn with_retry_timeout(self, timeout: Duration) -> Self

Set the maximum time to retry failed operations before giving up.

pub fn with_rate_limit(self, with_rate_limit: usize) -> Self

Set the maximum number of simultaneous fetches.

pub fn with_range_chunk_size(self, range_chunk_size: usize) -> Self

Set the number of items to process at a time when loading a range or stream.

This determines:

• The number of objects to load from storage in a single request
• The number of objects to buffer in memory per request/stream
• The number of concurrent notification subscriptions per request/stream

pub fn with_minor_scan_interval(self, interval: Duration) -> Self

Set the time interval between minor proactive fetching scans.

See proactive fetching.

pub fn with_major_scan_interval(self, interval: usize) -> Self

Set the interval (denominated in minor scans) between major proactive fetching scans.

See proactive fetching.

pub fn with_major_scan_offset(self, offset: usize) -> Self

Set the offset (denominated in minor scans) before the first major proactive fetching scan.

This is useful when starting multiple nodes at the same time: major proactive scans can have a measurable impact on the performance of the node for a brief time while the scan is running, so it may be desirable to prevent a group of nodes from all doing major scans at the same time. This can be achieved by giving each node a different major_scan_offset.

See also proactive fetching.

pub fn with_proactive_range_chunk_size(self, range_chunk_size: usize) -> Self

Set the number of items to process at a time when scanning for proactive fetching.

This is similar to Self::with_range_chunk_size, but only affects the chunk size for proactive fetching scans, not for normal subscription streams. This can be useful to tune the proactive scanner to be more or less greedy with persistent storage resources.

By default (i.e. if this method is not called) the proactive range chunk size will be set to whatever the normal range chunk size is.

pub fn with_active_fetch_delay(self, active_fetch_delay: Duration) -> Self

Add a delay between active fetches in proactive scans.

This can be used to limit the rate at which this query service makes requests to other query services during proactive scans. This is useful if the query service has a lot of blocks to catch up on, as without a delay, scanning can be extremely burdensome on the peer.

pub fn with_chunk_fetch_delay(self, chunk_fetch_delay: Duration) -> Self

Adds a delay between chunk fetches during proactive scans.

In a proactive scan, we retrieve a range of objects from a provider or local storage (e.g., a database). Without a delay between fetching these chunks, the process can become very CPU-intensive, especially when chunks are retrieved from local storage. While there is already a delay for active fetches (active_fetch_delay), situations may arise when subscribed to an old stream that fetches most of the data from local storage.

This additional delay helps to limit constant maximum CPU usage and ensures that local storage remains accessible to all processes, not just the proactive scanner.

pub fn disable_proactive_fetching(self) -> Self

Run without proactive fetching.

This can reduce load on the CPU and the database, but increases the probability that requests will fail due to missing resources. If resources are constrained, it is recommended to run with rare proactive fetching (see with_major_scan_interval, with_minor_scan_interval), rather than disabling it entirely.

pub fn disable_aggregator(self) -> Self

Run without an aggregator.

This can reduce load on the CPU and the database, but it will cause aggregate statistics (such as transaction counts) not to update.

pub fn with_aggregator_chunk_size(self, chunk_size: usize) -> Self

Set the number of items to process at a time when computing aggregate statistics.

This is similar to Self::with_range_chunk_size, but only affects the chunk size for the aggregator task, not for normal subscription streams. This can be useful to tune the aggregator to be more or less greedy with persistent storage resources.

By default (i.e. if this method is not called) the proactive range chunk size will be set to whatever the normal range chunk size is.

pub fn with_types_migration_batch_size(self, batch: u64) -> Self

Sets the batch size for the types migration. Determines how many (leaf, vid) rows are selected from the old types table and migrated at once.

pub fn is_leaf_only(&self) -> bool

impl<Types, S, P> Builder<Types, S, P>

where Types: NodeType, Payload<Types>: QueryablePayload<Types>, Header<Types>: QueryableHeader<Types>, S: PruneStorage + VersionedDataSource + HasMetrics + MigrateTypes<Types> + 'static, for<'a> S::ReadOnly<'a>: AvailabilityStorage<Types> + PrunedHeightStorage + NodeStorage<Types> + AggregatesStorage<Types>, for<'a> S::Transaction<'a>: UpdateAvailabilityStorage<Types> + UpdateAggregatesStorage<Types>, P: AvailabilityProvider<Types>,

pub async fn build(self) -> Result<FetchingDataSource<Types, S, P>>

Build a FetchingDataSource with these options.