Struct InMemoryRateLimiter

pub struct InMemoryRateLimiter { /* private fields */ }

An in-memory rate limiter based on a token bucket algorithm.

This is an in-memory rate limiter, so it cannot rate limit across different processes.

The rate limiter only allows time-based rate limiting and does not take into account any information about the input or the output, so it cannot be used to rate limit based on the size of the request.

It is thread safe and can be used in either a sync or async context.

The in-memory rate limiter is based on a token bucket. The bucket is filled with tokens at a given rate. Each request consumes a token. If there are not enough tokens in the bucket, the request is blocked until there are enough tokens.

These tokens have NOTHING to do with LLM tokens. They are just a way to keep track of how many requests can be made at a given time.

Current limitations:

• The rate limiter is not designed to work across different processes. It is an in-memory rate limiter, but it is thread safe.
• The rate limiter only supports time-based rate limiting. It does not take into account the size of the request or any other factors.

Example

use ferriclink_core::rate_limiters::InMemoryRateLimiter;
use std::time::Duration;

let rate_limiter = InMemoryRateLimiter::new(
    0.1,  // Can only make a request once every 10 seconds
    0.1,  // Wake up every 100 ms to check whether allowed to make a request
    10.0, // Controls the maximum burst size
);

// Use with a language model
// let model = ChatAnthropic::new()
//     .with_rate_limiter(rate_limiter);

Implementations

impl InMemoryRateLimiter

pub fn new( requests_per_second: f64, check_every_n_seconds: f64, max_bucket_size: f64, ) -> Self

Create a new in-memory rate limiter based on a token bucket.

These tokens have NOTHING to do with LLM tokens. They are just a way to keep track of how many requests can be made at a given time.

This rate limiter is designed to work in a threaded environment.

It works by filling up a bucket with tokens at a given rate. Each request consumes a given number of tokens. If there are not enough tokens in the bucket, the request is blocked until there are enough tokens.

Arguments

• requests_per_second - The number of tokens to add per second to the bucket. The tokens represent “credit” that can be used to make requests.
• check_every_n_seconds - Check whether the tokens are available every this many seconds. Can be a float to represent fractions of a second.
• max_bucket_size - The maximum number of tokens that can be in the bucket. Must be at least 1. Used to prevent bursts of requests.

Panics

Panics if max_bucket_size is less than 1.0.

pub async fn available_tokens(&self) -> f64

Get the current number of available tokens

pub fn max_bucket_size(&self) -> f64

Get the maximum bucket size

pub fn requests_per_second(&self) -> f64

Get the requests per second rate

pub fn check_every_n_seconds(&self) -> f64

Get the check interval in seconds

pub fn to_config(&self) -> InMemoryRateLimiterConfig

Convert to a serializable configuration

pub fn from_config(config: InMemoryRateLimiterConfig) -> Self

Create from a serializable configuration

Trait Implementations

impl BaseRateLimiter for InMemoryRateLimiter

fn acquire(&self, blocking: bool) -> Result<bool>

Attempt to acquire the necessary tokens for the rate limiter.

fn aacquire<'life0, 'async_trait>( &'life0 self, blocking: bool, ) -> Pin<Box<dyn Future<Output = Result<bool>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Attempt to acquire the necessary tokens for the rate limiter. Async version.

impl Clone for InMemoryRateLimiter

fn clone(&self) -> InMemoryRateLimiter

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for InMemoryRateLimiter

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.