Analyzing Consistency in LLM Outputs Leveraging Colourful Queries

First Contact

LLMs have made an impressive evolution especially in the last few years. The rise of neural networks and the introduction of transformer architecture have significantly advanced language generation capabilities. A genuine breakthrough.

Initial introductions via an overflow of news articles, posts, webcasts, and opinions everywhere lead to interesting conversations.

Exploring (First Steps)

There are multiple ways to leverage the power of LLMs, playing around with chatGPT in my humble beginnings lead to very mixed results.

So Many Questions…

A lot of discussions, more posts, webcasts and opinions just lead to more questions.

A Colourful Idea

Exploring colours using ChatGPT with regards to hex-codes, meaning, symbolism etc…

Formulating a set up to analyse LLMs responses using colours.

Why Colours?

R & OpenAI API

# connecting R with opanAI api

library(openai)
# ensure API key is available in global environment (.Renv) 

#list of prompts (with role specified)
message_list <- list(
  list("role" = "system",
       "content" = "You're a colour expert"),
  list("role" = "user",
       "content" = "Provide an idiom or proverb for the colour green")
) 

create_chat_completion(
  model = "gpt-3.5-turbo", # state which model to use
  # different models can be more suitable for different tasks
  messages = message_list, # list with prompts
  temperature = colour_temp # temperature (creativity) parameter
  ) 

Introducing Randomness

Example:

Rain? Meatballs?

The Ingredients

The prompt, the colour categories, the temperatures

# colour-categorizing prompt
message_list <-
    list(
      list("role" = "system",
           "content" = "You're a colour expert and always use" 
                       "black, white, red, blue, green, yellow and purple" 
                       "as main colour categories"),
      list("role" = "user",
           "content" = "please provide the main colour category" 
                       "for the following hex code: ", hex_colour)
    ) 
  
# temperature settings
temperature <- c(0, 0.5, 1, 1.25, 1.5, 1.75, 1)

colour_chat <- create_chat_completion(
  model = "gpt-3.5-turbo",
  messages = message_list, # list with prompts
  temperature = temperature[t]
  ) 
  

Colour Categories and Set-Up

• 405 colours
  • R named colours and their hex codes
• 9 different temperature settings (between 0 and 2)
• run N times

OpenAI API usage and limits:

• limited number of requests per day | 10,000 RPD
• limited number of requests per minute | 500 RPM
• limited number of tokens per minute | 60,000 TPM

Categorizing Colours

Categorizing Colour Categories

Categorizing Colours Surprises

Categorizing Colours in Context

Categorizing Colours in Context

An important aspect are the instructions and the model’s understanding and application when generating a response. Only with higher temperature settings the returned response included a reasoning for the stated colour category.

Categorizing Colours in Context

On the other hand low temperature settings, even if claiming to having read the instructions more often fail to adhere to them

Categorizing Colours in Context

Unix-code errors and text-overflow, incoherent sentences, symbols/emojis or slight gibberish occurred more often for higher temperature settings.

Colourful Proverbs

Creating or quoting idioms that involve colours.

Does the symbolism change?

chat_colours <- c(
  "blue", "red", "green", "yellow", "purple", "black", "white", "gray", "orange"
  , "brown", "violet", "pink", "lavender", "cornflowerblue", "azure", 
  "aubergine", "sage", "gold", "khaki", "cobalt")
  
message_list <-
    list(
      list("role" = "system",
           "content" = "you're a colour expert"),
      list("role" = "user",
           "content" = "please create an idiom using the colour blue")
    )

Colourful Proverbs Symbolism

This requires retrieving the first response and including it in the message list.

message_list <- 
      list(
        list("role" = "system",
             "content" = "you're a colour expert"),
        list("role" = "user",
             "content" = paste0("please create an idiom using the colour blue")),
        list("role" = idiom[[blue]]$response$choice$message.role,
             "content" = llm_text[[blue]]$response$choice$message.content),
        list("role" = "user",
             "content" = "categorize the semtiment of the idiom")
      )

Create vs Quote

Paint the Town Something

The most common phrase…

Paint the town purple but not blue

25% of all generated idioms.

Idioms

Most fun examples were generated with higher temperature settings.

in conlcusion?