Struct InMemoryCache

pub struct InMemoryCache { /* private fields */ }

Cache that stores things in memory.

This is a simple in-memory cache implementation that stores cached values in a HashMap. It supports optional size limits and LRU eviction.

Implementations

impl InMemoryCache

pub fn new() -> Self

Create a new in-memory cache with no size limit.

pub fn with_max_size(max_size: Option<usize>) -> Self

Create a new in-memory cache with a maximum size.

Arguments

• max_size - The maximum number of items to store in the cache. If None, the cache has no maximum size. If the cache exceeds the maximum size, the oldest items are removed.

Panics

Panics if max_size is Some(0).

pub async fn stats(&self) -> CacheStats

Get cache statistics.

pub async fn size(&self) -> usize

Get the current cache size.

pub async fn is_empty(&self) -> bool

Check if the cache is empty.

pub fn max_size(&self) -> Option<usize>

Get the maximum cache size.

Trait Implementations

impl BaseCache for InMemoryCache

fn lookup( &self, prompt: &str, llm_string: &str, ) -> Result<Option<CachedGenerations>>

Look up based on prompt and llm_string.

fn update( &self, prompt: &str, llm_string: &str, return_val: CachedGenerations, ) -> Result<()>

Update cache based on prompt and llm_string.

fn clear(&self) -> Result<()>

Clear cache that can take additional keyword arguments.

fn alookup<'life0, 'life1, 'life2, 'async_trait>( &'life0 self, prompt: &'life1 str, llm_string: &'life2 str, ) -> Pin<Box<dyn Future<Output = Result<Option<CachedGenerations>>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait,

Async look up based on prompt and llm_string.

fn aupdate<'life0, 'life1, 'life2, 'async_trait>( &'life0 self, prompt: &'life1 str, llm_string: &'life2 str, return_val: CachedGenerations, ) -> Pin<Box<dyn Future<Output = Result<()>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait,

Async update cache based on prompt and llm_string.

fn aclear<'life0, 'async_trait>( &'life0 self, ) -> Pin<Box<dyn Future<Output = Result<()>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Async clear cache that can take additional keyword arguments.

impl Debug for InMemoryCache

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for InMemoryCache

fn default() -> Self

Returns the “default value” for a type.