Diving Deeper with Structured Outputs | by Armin Catovic | Sep, 2024

Serving to improve your understanding and optimum utilization of structured outputs and LLMs

Within the previous article, we have been launched to structured outputs utilizing OpenAI. For the reason that common availability launch in ChatCompletions API (v1.40.0), structured outputs have been utilized throughout dozens of use instances, and spawned quite a few threads on OpenAI forums.

On this article, our purpose is to give you an excellent deeper understanding, dispel some misconceptions, and offer you some recommendations on the way to apply them in essentially the most optimum method attainable, throughout totally different eventualities.

Structured outputs are a approach of implementing the output of an LLM to observe a pre-defined schema — normally a JSON schema. This works by remodeling the schema right into a context free grammar (CFG), which throughout the token sampling step, is used along with the beforehand generated tokens, to tell which subsequent tokens are legitimate. It’s useful to think about it as making a regex for token technology.

OpenAI API implementation truly tracks a restricted subset of JSON schema options. With extra common structured output options, corresponding to Outlines, it’s attainable to make use of a considerably bigger subset of the JSON schema, and even outline fully customized non-JSON schemas — so long as one has entry to an open weight mannequin. For the aim of this text, we’ll assume the OpenAI API implementation.

In response to JSON Schema Core Specification, “JSON Schema asserts what a JSON doc should appear to be, methods to extract data from it, and the way to work together with it”. JSON schema defines six primitive sorts — null, boolean, object, array, quantity and string. It additionally defines sure key phrases, annotations, and particular behaviours. For instance, we are able to specify in our schema that we count on an array and add an annotation that minItems shall be 5 .

Pydantic is a Python library that implements the JSON schema specification. We use Pydantic to construct strong and maintainable software program in Python. Since Python is a dynamically typed language, knowledge scientists don’t essentially suppose when it comes to variable sorts — these are sometimes implied of their code. For instance, a fruit can be specified as:

fruit = dict(
title="apple",
coloration="crimson",
weight=4.2
)

…whereas a operate declaration that returns “fruit” from some piece of information would usually be specified as:

def extract_fruit(s):
...
return fruit

Pydantic alternatively permits us to generate a JSON-schema compliant class, with correctly annotated variables and kind hints, making our code extra readable/maintainable and typically extra strong, i.e.

class Fruit(BaseModel):
title: str
coloration: Literal['red', 'green']
weight: Annotated[float, Gt(0)]
def extract_fruit(s: str) -> Fruit:
...
return fruit

OpenAI truly strongly recommends using Pydantic for specifying schemas, versus specifying the “uncooked” JSON schema instantly. There are a number of causes for this. Firstly, Pydantic is assured to stick to the JSON schema specification, so it saves you further pre-validation steps. Secondly, for bigger schemas, it’s much less verbose, permitting you to jot down cleaner code, sooner. Lastly, the openai Python package deal truly does some housekeeping, like setting additionalProperties to False for you, whereas when defining your schema “by-hand” utilizing JSON, you would want to set these manually, for each object in your schema (failing to take action ends in a fairly annoying API error).

As we alluded to beforehand, the ChatCompletions API gives a restricted subset of the complete JSON schema specification. There are quite a few keywords that are not yet supported, corresponding to minimal and most for numbers, and minItems and maxItems for arrays — annotations that may be in any other case very helpful in decreasing hallucinations, or constraining the output measurement.

Sure formatting options are additionally unavailable. For instance, the next Pydantic schema would end in API error when handed to response_format in ChatCompletions:

class NewsArticle(BaseModel):
headline: str
subheading: str
authors: Listing[str]
date_published: datetime = Area(None, description="Date when article was revealed. Use ISO 8601 date format.")

It could fail as a result of openai package deal has no format dealing with for datetime , so as a substitute you would want to set date_published as a str and carry out format validation (e.g. ISO 8601 compliance) post-hoc.

Different key limitations embrace the next:

• Hallucinations are nonetheless attainable — for instance, when extracting product IDs, you’d outline in your response schema the next: product_ids: Listing[str] ; whereas the output is assured to supply an inventory of strings (product IDs), the strings themselves could also be hallucinated, so on this use case, it’s possible you’ll need to validate the output towards some pre-defined set of product IDs.
• The output is capped at 4096 tokens, or the lesser quantity you set inside the max_tokens parameter — so despite the fact that the schema shall be adopted exactly, if the output is simply too giant, it will likely be truncated and produce an invalid JSON — particularly annoying on very giant Batch API jobs!
• Deeply nested schemas with many object properties could yield API errors — there’s a limitation on the depth and breadth of your schema, however typically it’s best to stay to flat and easy constructions— not simply to keep away from API errors but in addition to squeeze out as a lot efficiency from the LLMs as attainable (LLMs typically have hassle attending to deeply nested constructions).
• Extremely dynamic or arbitrary schemas should not attainable — despite the fact that recursion is supported, it isn’t attainable to create a extremely dynamic schema of let’s say, an inventory of arbitrary key-value objects, i.e. [{"key1": "val1"}, {"key2": "val2"}, ..., {"keyN": "valN"}] , for the reason that “keys” on this case should be pre-defined; in such a situation, the best choice is to not use structured outputs in any respect, however as a substitute go for normal JSON mode, and supply the directions on the output construction inside the system immediate.

With all this in thoughts, we are able to now undergo a few use instances with ideas and methods on the way to improve the efficiency when utilizing structured outputs.

Creating flexibility utilizing non-compulsory parameters

Let’s say we’re constructing an online scraping utility the place our objective is to gather particular parts from the online pages. For every internet web page, we provide the uncooked HTML within the person immediate, give particular scraping directions within the system immediate, and outline the next Pydantic mannequin:

class Webpage(BaseModel):
title: str
paragraphs: Non-compulsory[List[str]] = Area(None, description="Textual content contents enclosed inside <p></p> tags.")
hyperlinks: Non-compulsory[List[str]] = Area(None, description="URLs specified by `href` area inside <a></a> tags.")
pictures: Non-compulsory[List[str]] = Area(None, description="URLs specified by the `src` area inside the <img></img> tags.")

We’d then name the API as follows…

response = shopper.beta.chat.completions.parse(
mannequin="gpt-4o-2024-08-06",
messages=[
{
"role": "system",
"content": "You are to parse HTML and return the parsed page components."
},
{
"role": "user",
"content": """
<html>
<title>Structured Outputs Demo</title>
<body>
<img src="test.gif"></image>
<p>Hello world!</p>
</body>
</html>
"""
}
],
response_format=Webpage
)

…with the next response:

{
'pictures': ['test.gif'],
'hyperlinks': None,
'paragraphs': ['Hello world!'],
'title': 'Structured Outputs Demo'
}

Response schema equipped to the API utilizing structured outputs should return all the desired fields. Nevertheless, we are able to “emulate” non-compulsory fields and add extra flexibility utilizing the Non-compulsory kind annotation. We may truly additionally use Union[List[str], None] — they’re syntactically precisely the identical. In each instances, we get a conversion to anyOf key phrase as per the JSON schema spec. Within the instance above, since there are not any <a></a> tags current on the internet web page, the API nonetheless returns the hyperlinks area, however it’s set to None .

Decreasing hallucinations utilizing enums

We talked about beforehand that even when the LLM is assured to observe the supplied response schema, it might nonetheless hallucinate the precise values. Including to this, a recent paper discovered that implementing a set schema on the outputs, truly causes the LLM to hallucinate, or degrade when it comes to its reasoning capabilities.

One technique to overcome these limitations, is to try to make the most of enums as a lot as attainable. Enums constrain the output to a really particular set of tokens, inserting a chance of zero on the whole lot else. For instance, let’s assume you are attempting to re-rank product similarity outcomes between a goal product that incorporates a description and a singular product_id , and top-5 merchandise that have been obtained utilizing some vector similarity search (e.g. utilizing a cosine distance metric). Every a kind of top-5 merchandise additionally include the corresponding textual description and a singular ID. In your response you merely want to receive the re-ranking 1–5 as an inventory (e.g. [1, 4, 3, 5, 2] ), as a substitute of getting an inventory of re-ranked product ID strings, which can be hallucinated or invalid. We setup our Pydantic mannequin as follows…

class Rank(IntEnum):
RANK_1 = 1
RANK_2 = 2
RANK_3 = 3
RANK_4 = 4
RANK_5 = 5
class RerankingResult(BaseModel):
ordered_ranking: Listing[Rank] = Area(description="Offers ordered rating 1-5.")

…and run the API like so:

response = shopper.beta.chat.completions.parse(
mannequin="gpt-4o-2024-08-06",
messages=[
{
"role": "system",
"content": """
You are to rank the similarity of the candidate products against the target product.
Ranking should be orderly, from the most similar, to the least similar.
"""
},
{
"role": "user",
"content": """
## Target Product
Product ID: X56HHGHH
Product Description: 80" Samsung LED TV
## Candidate Products
Product ID: 125GHJJJGH
Product Description: NVIDIA RTX 4060 GPU
Product ID: 76876876GHJ
Product Description: Sony Walkman
Product ID: 433FGHHGG
Product Description: Sony LED TV 56"
Product ID: 777888887888
Product Description: Blueray Sony Player
Product ID: JGHHJGJ56
Product Description: BenQ PC Monitor 37" 4K UHD
"""
}
],
response_format=RerankingResult
)

The ultimate result’s merely:

{'ordered_ranking': [3, 5, 1, 4, 2]}

So the LLM ranked the Sony LED TV (i.e. merchandise quantity “3” within the record), and the BenQ PC Monitor (i.e. merchandise quantity “5”), as the 2 most comparable product candidates, i.e. the primary two parts of the ordered_ranking record!

On this article we gave an intensive deep-dive into structured outputs. We launched the JSON schema and Pydantic fashions, and related these to OpenAI’s ChatCompletions API. We walked by means of quite a lot of examples and showcased some optimum methods of resolving these utilizing structured outputs. To summarize some key takeaways:

• Structured outputs as supported by OpenAI API, and different third celebration frameworks, implement solely a subset of the JSON schema specification — getting higher knowledgeable when it comes to its options and limitations will provide help to make the precise design choices.
• Utilizing Pydantic or comparable frameworks that observe JSON schema specification faithfully, is very really useful, because it lets you create legitimate and cleaner code.
• While hallucinations are nonetheless anticipated, there are alternative ways of mitigating these, just by a alternative of response schema design; for instance, by using enums the place applicable.

In regards to the Creator

Armin Catovic is a Secretary of the Board at Stockholm AI, and a Vice President and a Senior ML/AI Engineer on the EQT Group, with 18 years of engineering expertise throughout Australia, South-East Asia, Europe and the US, and quite a lot of patents and top-tier peer-reviewed AI publications.