xinference.client.handlers.ChatModelHandle.generate

ChatModelHandle.generate(prompt: str, generate_config: PytorchGenerateConfig | None = None) → Completion | Iterator[CompletionChunk]

Creates a completion for the provided prompt and parameters via RESTful APIs.

Parametri:

• prompt (str) – The user’s message or user’s input.

• generate_config (Optional["PytorchGenerateConfig"]) – Additional configuration for the chat generation. «PytorchGenerateConfig» -> Configuration for pytorch model

Ritorna:

Stream is a parameter in generate_config. When stream is set to True, the function will return Iterator[«CompletionChunk»]. When stream is set to False, the function will return «Completion».

Tipo di ritorno:

Union[«Completion», Iterator[«CompletionChunk»]]

Solleva:

RuntimeError – Fail to generate the completion from the server. Detailed information provided in error message.