OpenAnima 0.2.0
Middleware for conversational NPCs that interact with the player, the world and each other
Loading...
Searching...
No Matches
Classes | Typedefs
OpenAnima::Provider Namespace Reference

Inference backends and their settings. More...

Classes

struct  Call
 
class  IProvider
 
class  RestProvider
 
struct  RestProviderConfig
 

Typedefs

using DeltaSink = std::function< void(std::string_view text)>
 

Detailed Description

Inference backends and their settings.

One interface to implement, one configuration type to fill, and the REST implementation for the chat-completions contract.

Typedef Documentation

◆ DeltaSink

using OpenAnima::Provider::DeltaSink = typedef std::function<void(std::string_view text)>

Receives the answer as a provider produces it.

An empty sink means the caller wants the whole answer at once, and a provider is then free to ask for it in a single response. That is not only an optimisation: some servers report the token usage only when the answer is not streamed.