DSPy Tutorial

DSPy

DSPy is a new framework for developing applications using large language models (LLMs) that aims to solve the fragility and complexity issues of traditional LLM-based pipelines. It aims to make building complex, multi-step LLM applications more straightforward and robust by providing a declarative programming model and automated optimization capabilities.

• DSPy provides a declarative, composable, and Pythonic syntax for building LLM-based programs, replacing the need for manual prompt engineering
• DSPy includes an “automatic compiler” that can optimize the prompts, instructions, and LM weights for each component of an LLM program, improving the overall quality and efficiency
• DSPy allows chaining together multiple LLM calls into a program, where the output of one LLM call becomes the input to the next
• DSPy introduces abstractions like “Signatures” to cleanly represent the input/output specifications of each LLM component
• DSPy is positioned as a more structured and programmable alternative to other LLM integration frameworks like LangChain

Language Models

In DSPy, a Language Model (LM) is a key component used to perform various tasks such as answering questions, generating text, and making predictions. Some key points about LMs in DSPy:

• DSPy supports using both remote LMs managed as services (e.g. OpenAI, Cohere, Anthropic) and local LMs that you host yourself (e.g. HuggingFace models)
• To use an LM in DSPy, you first create an instance of the appropriate LM class (e.g. dspy.OpenAI for OpenAI models) and configure it as the default LM with dspy.configure
• You can then call the LM directly with a raw prompt string, but the recommended way is to use DSPy modules and signatures which provide a more structured way to interact with the LM
• DSPy’s key innovation is its ability to algorithmically optimize the prompts and weights of LMs, especially when used in multi-step pipelines This allows DSPy to significantly improve the reliability and performance of LMs on specific tasks compared to manual prompt engineering

import dspy
from groq_lm import LLAMA3_MODEL, GroqLM

gpt3_turbo = dspy.OpenAI(model="gpt-3.5-turbo-1106", max_tokens=300)
gpt4_turbo = dspy.OpenAI(model="gpt-4-turbo", max_tokens=300)
llama3 = dspy.OllamaLocal(model="llama3", max_tokens=300)
groq_llama3 = GroqLM(model=LLAMA3_MODEL, max_tokens=300)

Groq model used today: llama3-70b-8192

gpt3_turbo("hello! this is a raw prompt to GPT-3.5")

['Hello! How can I assist you today?']

groq_llama3("hello! this is a raw prompt to GroqLLAMA3")

["Hello! I'm GroqLLaMA3, nice to meet you! What's on your mind? Do you have a question, topic, or task you'd like to discuss or explore? I'm all ears (or rather, all text)!"]

dspy.configure(lm=gpt3_turbo)

qa = dspy.ChainOfThought("question -> answer")
response = qa(question="How many floors are in the castle David Gregory inherited?")
print(response)

Prediction(
    rationale='produce the answer. We know that the castle has a total of 7 floors, including the basement and the attic.',
    answer='The castle David Gregory inherited has 7 floors.'
)

with dspy.context(lm=gpt4_turbo):
    for idx in range(5):
        response = qa(
            question="How many floors are in the castle David Gregory inherited?",
            config=dict(temperature=0.7 + 0.0001 * idx),
        )
        print(f"{idx+1}.", response)

1. Prediction(
    rationale="determine the number of floors in the castle inherited by David Gregory. We need to analyze the provided information about the castle or typical castle structures if specific details are not given. Since the question does not specify details about the castle's size or style, we generally assume a typical castle might have at least 2 to 5 floors, including the main floors and possibly a basement or attic. However, without specific information about this particular castle, a precise answer cannot be given.",
    answer='The number of floors in the castle David Gregory inherited is not specified in the available information.'
)
2. Prediction(
    rationale='determine the number of floors in the castle David Gregory inherited. We need to check the provided information or context surrounding David Gregory and his castle. Since no specific information about the castle is provided in the question, we cannot ascertain the number of floors based on the given data alone. Typically, a castle might have several floors ranging from two to potentially five or more in grander structures, but without specific details or references, the exact number cannot be confirmed.',
    answer='The number of floors in the castle David Gregory inherited is not specified in the provided information.'
)
3. Prediction(
    rationale='determine the number of floors in the castle inherited by David Gregory. First, we would need information about the castle itself, which is not provided in the question. Without specific details about the castle or a source of information that mentions the number of floors in the castle, we cannot definitively answer the question.',
    answer='The number of floors in the castle David Gregory inherited is not specified in the provided information.'
)
4. Prediction(
    rationale="determine the number of floors in the castle inherited by David Gregory. Since the information about the number of floors in the castle is not provided in the given text or context, we can't directly know the answer. Usually, castles can vary greatly in size and number of floors, often having between two and five floors, including towers and possible underground levels. Without specific details about this particular castle, stating an exact number would be speculative.",
    answer='The number of floors in the castle David Gregory inherited is not specified in the provided information.'
)
5. Prediction(
    rationale="determine the number of floors in the castle inherited by David Gregory. First, if the number of floors in the castle was mentioned in a document, story, or data provided prior, we would refer to that. Since we don't have that document or story at hand, and there is no widely known information regarding a castle owned by David Gregory, we would need more specific information or context to accurately provide an answer.",
    answer='The number of floors in the castle David Gregory inherited cannot be determined with the information provided.'
)

with dspy.context(lm=gpt4_turbo):
    qa = dspy.ChainOfThought("question -> answer")
    response = qa(question="What did David Gregory inherit?")
    print(response)

Prediction(
    rationale="Question: What did David Gregory inherit?\nReasoning: Let's think step by step in order to determine what David Gregory inherited. We need to consider the context or source of information about David Gregory, as there could be multiple individuals with that name. If we are referring to a specific story, book, or historical account, details from that source would guide us. Without specific context, we can't provide a precise answer, but we can hypothesize based on common scenarios involving inheritance.",
    answer='It is unclear what David Gregory inherited without additional context or information about which David Gregory is being referred to.'
)

Signatures

Inline Signatures

sentence = (
    "it's a charming and often affecting journey."  # example from the SST-2 dataset.
)

classify = dspy.Predict("sentence -> sentiment")
classify(sentence=sentence)

Prediction(
    sentiment='Positive'
)

document = """The 21-year-old made seven appearances for the Hammers and netted his only goal for them in a Europa League qualification round match against Andorran side FC Lustrains last season. Lee had two loan spells in League One last term, with Blackpool and then Colchester United. He scored twice for the U's but was unable to save them from relegation. The length of Lee's contract with the promoted Tykes has not been revealed. Find all the latest football transfers on our dedicated page."""

summarize = dspy.ChainOfThought("document -> summary")
summarize(document=document)

Prediction(
    rationale="produce the summary. We need to highlight the player's performance for West Ham, his loan spells in League One, and his new contract with Barnsley.",
    summary='The 21-year-old player made seven appearances and scored one goal for West Ham in the Europa League. He had loan spells in League One with Blackpool and Colchester United, scoring twice for the latter. He has now signed a contract with Barnsley, but the length of the contract has not been disclosed.'
)

Class-based Signatures

class Emotion(dspy.Signature):
    """Classify emotion among sadness, joy, love, anger, fear, surprise."""

    sentence = dspy.InputField()
    sentiment = dspy.OutputField()


sentence = "i started feeling a little vulnerable when the giant spotlight started blinding me"  # from dair-ai/emotion

classify = dspy.Predict(Emotion)
classify(sentence=sentence)

Prediction(
    sentiment='Fear'
)

class CheckCitationFaithfulness(dspy.Signature):
    """Verify that the text is based on the provided context."""

    context = dspy.InputField(desc="facts here are assumed to be true")
    text = dspy.InputField()
    faithfulness = dspy.OutputField(
        desc="True/False indicating if text is faithful to context"
    )


context = "The 21-year-old made seven appearances for the Hammers and netted his only goal for them in a Europa League qualification round match against Andorran side FC Lustrains last season. Lee had two loan spells in League One last term, with Blackpool and then Colchester United. He scored twice for the U's but was unable to save them from relegation. The length of Lee's contract with the promoted Tykes has not been revealed. Find all the latest football transfers on our dedicated page."

text = "Lee scored 3 goals for Colchester United."

faithfulness = dspy.ChainOfThought(CheckCitationFaithfulness)
faithfulness(context=context, text=text)

Prediction(
    rationale="produce the faithfulness. We know that Lee had two loan spells in League One last term, with Blackpool and then Colchester United. He scored twice for the U's but was unable to save them from relegation. The text states that Lee scored 3 goals for Colchester United, which is not accurate based on the context.",
    faithfulness='False'
)

Modules

Multiple Completions

question = "What's something great about the ColBERT retrieval model?"

# 1) Declare with a signature, and pass some config.
classify = dspy.ChainOfThought("question -> answer", n=5)

# 2) Call with input argument.
response = classify(question=question)

# 3) Access the outputs.
for i in range(5):
    print(f"{i+1}. {response.completions[i].answer}")
    print(f"   Rationale: {response.completions[i].rationale}")

1. One great thing about the ColBERT retrieval model is its ability to efficiently retrieve relevant documents, improving search accuracy and speed.
   Rationale: produce the answer. We can consider the fact that the ColBERT retrieval model is designed to efficiently retrieve relevant documents for a given query, improving search accuracy and speed.
2. One great thing about the ColBERT retrieval model is its use of BERT and document-level embeddings, as well as its innovative pre-training objective that improves retrieval accuracy.
   Rationale: produce the answer. We can start by considering its use of BERT embeddings and document-level embeddings to improve retrieval accuracy. The ColBERT model also utilizes a novel pre-training objective that encourages learning from hard negatives, further enhancing its performance.
3. Its efficiency in retrieving and ranking passages from large-scale document collections.
   Rationale: produce the answer. We can start by considering the fact that ColBERT is designed to efficiently retrieve and rank passages from large-scale document collections. This means that it can quickly find relevant information, making it a powerful tool for information retrieval tasks.
4. One great aspect of the ColBERT retrieval model is its efficient handling of large-scale datasets and complex query structures.
   Rationale: produce the answer. We can consider its ability to handle large-scale datasets efficiently and effectively, its support for document relevance scoring, and its ability to handle complex query structures.
5. One great aspect of the ColBERT retrieval model is its improved efficiency and accuracy in information retrieval tasks.
   Rationale: produce the answer. We can consider the fact that the ColBERT retrieval model has been designed to improve efficiency and accuracy in information retrieval tasks. It uses efficient nearest neighbor search algorithms and has been shown to outperform other state-of-the-art models in retrieval tasks.

Chain of Thought with Hint

# Define a simple signature for basic question answering
class BasicQA(dspy.Signature):
    """Answer questions with short factoid answers."""

    question = dspy.InputField()
    answer = dspy.OutputField(desc="often between 1 and 5 words")


# Pass signature to ChainOfThought module
generate_answer = dspy.ChainOfThoughtWithHint(BasicQA)

# Call the predictor on a particular input alongside a hint.
question = "What is the color of the sky?"
hint = "It's what you often see during a sunny day."
pred = generate_answer(question=question, hint=hint)

print(f"Question: {question}")
print(f"Predicted Answer: {pred.answer}")

Question: What is the color of the sky?
Predicted Answer: Blue

Multi-Chain Comparison

class BasicQA(dspy.Signature):
    """Answer questions with short factoid answers."""

    question = dspy.InputField()
    answer = dspy.OutputField(desc="often between 1 and 5 words")


# Example completions generated by a model for reference
completions = [
    dspy.Prediction(
        rationale="I recall that during clear days, the sky often appears this color.",
        answer="blue",
    ),
    dspy.Prediction(
        rationale="Based on common knowledge, I believe the sky is typically seen as this color.",
        answer="green",
    ),
    dspy.Prediction(
        rationale="From images and depictions in media, the sky is frequently represented with this hue.",
        answer="blue",
    ),
]

# Pass signature to MultiChainComparison module
compare_answers = dspy.MultiChainComparison(BasicQA)

# Call the MultiChainComparison on the completions
question = "What is the color of the sky?"
final_pred = compare_answers(completions, question=question)

print(f"Question: {question}")
print(f"Final Predicted Answer (after comparison): {final_pred.answer}")
print(f"Final Rationale: {final_pred.rationale}")

Question: What is the color of the sky?
Final Predicted Answer (after comparison): Blue
Final Rationale: say that the color of the sky is blue.

Program of Thought

# Define a simple signature for basic question answering
class GenerateAnswer(dspy.Signature):
    """Answer questions with short factoid answers."""

    question = dspy.InputField()
    answer = dspy.OutputField(desc="often between 1 and 5 words")


# Pass signature to ProgramOfThought Module
pot = dspy.ProgramOfThought(GenerateAnswer)

# Call the ProgramOfThought module on a particular input
question = "Sarah has 5 apples. She buys 7 more apples from the store. How many apples does Sarah have now?"
result = pot(question=question)

print(f"Question: {question}")
print(f"Final Predicted Answer (after ProgramOfThought process): {result.answer}")

Question: Sarah has 5 apples. She buys 7 more apples from the store. How many apples does Sarah have now?
Final Predicted Answer (after ProgramOfThought process): 12

ReAct

# Define a simple signature for basic question answering
class BasicQA(dspy.Signature):
    """Answer questions with short factoid answers."""

    question = dspy.InputField()
    answer = dspy.OutputField(desc="often between 1 and 5 words")


# Pass signature to ReAct module
react_module = dspy.ReAct(BasicQA)

# Call the ReAct module on a particular input
question = "What is the color of the sky?"
result = react_module(question=question)

print(f"Question: {question}")
print(f"Final Predicted Answer (after ReAct process): {result.answer}")

Question: What is the color of the sky?
Final Predicted Answer (after ReAct process): The color of the sky is typically blue during the day and black at night.

Retrieve

import dspy

colbertv2_wiki17_abstracts = dspy.ColBERTv2(
    url="http://20.102.90.50:2017/wiki17_abstracts"
)

retrieval_response = colbertv2_wiki17_abstracts(
    "When was the first FIFA World Cup held?", k=5
)

for result in retrieval_response:
    print("Text:", result["text"], "\n")

Text: History of the FIFA World Cup | The FIFA World Cup was first held in 1930, when FIFA president Jules Rimet decided to stage an international football tournament. The inaugural edition, held in 1930, was contested as a final tournament of only thirteen teams invited by the organization. Since then, the World Cup has experienced successive expansions and format remodeling to its current 32-team final tournament preceded by a two-year qualifying process, involving over 200 teams from around the world. 

Text: 1950 FIFA World Cup | The 1950 FIFA World Cup, held in Brazil from 24 June to 16 July 1950, was the fourth FIFA World Cup. It was the first World Cup since 1938, the planned 1942 and 1946 competitions having been cancelled owing to World War II. It was won by Uruguay, who had won the inaugural competition in 1930, clinching the cup by beating the hosts Brazil 2–1 in the deciding match of the four-team final group (this was the only tournament not decided by a one-match final). It was also the first tournament where the trophy was referred to as the Jules Rimet Cup, to mark the 25th anniversary of Jules Rimet's presidency of FIFA. 

Text: 1970 FIFA World Cup | The 1970 FIFA World Cup was the ninth FIFA World Cup, the quadrennial international football championship for men's national teams. Held from 31 May to 21 June in Mexico, it was the first World Cup tournament staged in North America, and the first held outside Europe and South America. Teams representing 75 nations from all six populated continents entered the competition, and its qualification rounds began in May 1968. Fourteen teams qualified from this process to join host nation Mexico and defending champions England in the sixteen-team final tournament. El Salvador, Israel, and Morocco made their first appearances at the final stage, and Peru their first since 1930. 

Text: 1974 FIFA World Cup | The 1974 FIFA World Cup, the tenth staging of the World Cup, was held in West Germany (including West Berlin) from 13 June to 7 July. The tournament marked the first time that the current trophy, the FIFA World Cup Trophy, created by the Italian sculptor Silvio Gazzaniga, was awarded. The previous trophy, the Jules Rimet Trophy, had been won for the third time by Brazil in 1970 and awarded permanently to the Brazilians. The host nation won the title beating the Netherlands in the final, 2–1. The victory was the second for West Germany, who had also won in 1954. Australia, East Germany, Haiti and Zaire made their first appearances at the final stage, and the Netherlands and Poland their first since 1938. 

Text: 1954 FIFA World Cup | The 1954 FIFA World Cup, the fifth staging of the FIFA World Cup, was held in Switzerland from 16 June to 4 July. Switzerland was chosen as hosts in July 1946. The tournament set a number of all-time records for goal-scoring, including the highest average goals scored per game. The tournament was won by West Germany, who defeated Hungary 3–2 in the final, giving them their first title. 

dspy.settings.configure(rm=colbertv2_wiki17_abstracts)

query = "When was the first FIFA World Cup held?"

# Call the retriever on a particular query.
retrieve = dspy.Retrieve(k=3)
topK_passages = retrieve(query).passages

print(f"Top {retrieve.k} passages for question: {query} \n", "-" * 30, "\n")

for idx, passage in enumerate(topK_passages):
    print(f"{idx+1}]", passage, "\n")

Top 3 passages for question: When was the first FIFA World Cup held? 
 ------------------------------ 

1] History of the FIFA World Cup | The FIFA World Cup was first held in 1930, when FIFA president Jules Rimet decided to stage an international football tournament. The inaugural edition, held in 1930, was contested as a final tournament of only thirteen teams invited by the organization. Since then, the World Cup has experienced successive expansions and format remodeling to its current 32-team final tournament preceded by a two-year qualifying process, involving over 200 teams from around the world. 

2] 1950 FIFA World Cup | The 1950 FIFA World Cup, held in Brazil from 24 June to 16 July 1950, was the fourth FIFA World Cup. It was the first World Cup since 1938, the planned 1942 and 1946 competitions having been cancelled owing to World War II. It was won by Uruguay, who had won the inaugural competition in 1930, clinching the cup by beating the hosts Brazil 2–1 in the deciding match of the four-team final group (this was the only tournament not decided by a one-match final). It was also the first tournament where the trophy was referred to as the Jules Rimet Cup, to mark the 25th anniversary of Jules Rimet's presidency of FIFA. 

3] 1970 FIFA World Cup | The 1970 FIFA World Cup was the ninth FIFA World Cup, the quadrennial international football championship for men's national teams. Held from 31 May to 21 June in Mexico, it was the first World Cup tournament staged in North America, and the first held outside Europe and South America. Teams representing 75 nations from all six populated continents entered the competition, and its qualification rounds began in May 1968. Fourteen teams qualified from this process to join host nation Mexico and defending champions England in the sixteen-team final tournament. El Salvador, Israel, and Morocco made their first appearances at the final stage, and Peru their first since 1930. 

Data

Example Objects

qa_pair = dspy.Example(question="This is a question?", answer="This is an answer.")

print(qa_pair)
print(qa_pair.question)
print(qa_pair.answer)

Example({'question': 'This is a question?', 'answer': 'This is an answer.'}) (input_keys=None)
This is a question?
This is an answer.

Input Keys

In a typical machine learning dataset (comprising training, testing, and validation sets), there are distinct “inputs” and “labels”. In DSPy, the Example objects offer a with_inputs() method, allowing certain fields to be designated as inputs while the remaining fields serve as metadata or labels.

# Single Input.
qa_pair = qa_pair.with_inputs("question")
print(qa_pair.inputs())
print(qa_pair.labels())

Example({'question': 'This is a question?'}) (input_keys=None)
Example({'answer': 'This is an answer.'}) (input_keys=None)

Metrics

DSPy is a machine learning framework that requires thoughtful consideration of automatic metrics for evaluating and optimizing programs. A metric is a function that assesses the quality of program output. Simple tasks might use metrics like accuracy or F1 score, while complex tasks may require more detailed metrics, possibly involving AI feedback from language models. It’s important to start with a simple metric and iterate to improve it.

Simple Metrics

A DSPy metric is a Python function that compares example and pred outputs, returning a numerical score. It can also accept a trace argument for optimization purposes.

def validate_answer(example, pred, trace=None):
    return example.answer.lower() == pred.answer.lower()


validate_answer(qa_pair, qa_pair)

True

from dspy import evaluate

evaluate.metrics.answer_exact_match(qa_pair, qa_pair)

True

qa_ctx = dspy.Example(
    question="This is a question?",
    answer="This is an answer.",
    context=["This is an answer."],
)

evaluate.metrics.answer_passage_match(qa_pair, qa_ctx)

True

RAG Tutorial

Configure LM and RM

import dspy

turbo = dspy.OpenAI(model="gpt-3.5-turbo")
colbertv2_wiki17_abstracts = dspy.ColBERTv2(
    url="http://20.102.90.50:2017/wiki17_abstracts"
)

dspy.settings.configure(lm=turbo, rm=colbertv2_wiki17_abstracts)

Load the Dataset

from dspy.datasets import HotPotQA

# Load the dataset.
dataset = HotPotQA(
    train_seed=1, train_size=20, eval_seed=2023, dev_size=50, test_size=0
)

# Tell DSPy that the 'question' field is the input. Any other fields are labels and/or metadata.
trainset = [x.with_inputs("question") for x in dataset.train]
devset = [x.with_inputs("question") for x in dataset.dev]

len(trainset), len(devset)

/Users/dothomps/.pyenv/versions/3.11.9/envs/blog/lib/python3.11/site-packages/datasets/table.py:1421: FutureWarning: promote has been superseded by promote_options='default'.
  table = cls._concat_blocks(blocks, axis=0)

(20, 50)

Build the Signature

class GenerateAnswer(dspy.Signature):
    """Answer questions with short factoid answers."""

    context = dspy.InputField(desc="may contain relevant facts")
    question = dspy.InputField()
    answer = dspy.OutputField(desc="often between 1 and 5 words")

Build the Pipeline

class RAG(dspy.Module):
    def __init__(self, num_passages=3):
        super().__init__()

        self.retrieve = dspy.Retrieve(k=num_passages)
        self.generate_answer = dspy.ChainOfThought(GenerateAnswer)

    def forward(self, question):
        context = self.retrieve(question).passages
        prediction = self.generate_answer(context=context, question=question)
        return dspy.Prediction(context=context, answer=prediction.answer)

Optimize the Pipeline

from dspy.teleprompt import BootstrapFewShot


# Validation logic: check that the predicted answer is correct.
# Also check that the retrieved context does actually contain that answer.
def validate_context_and_answer(example, pred, trace=None):
    answer_EM = dspy.evaluate.answer_exact_match(example, pred)
    answer_PM = dspy.evaluate.answer_passage_match(example, pred)
    return answer_EM and answer_PM


# Set up a basic teleprompter, which will compile our RAG program.
teleprompter = BootstrapFewShot(metric=validate_context_and_answer)

# Compile!
compiled_rag = teleprompter.compile(RAG(), trainset=trainset)

 55%|█████▌    | 11/20 [00:00<00:00, 62.21it/s]

Execute the Pipeline

# Ask any question you like to this simple RAG program.
my_question = "What castle did David Gregory inherit?"

# Get the prediction. This contains `pred.context` and `pred.answer`.
pred = compiled_rag(my_question)

# Print the contexts and the answer.
print(f"Question: {my_question}")
print(f"Predicted Answer: {pred.answer}")
print(f"Retrieved Contexts (truncated): {[c[:200] + '...' for c in pred.context]}")

Question: What castle did David Gregory inherit?
Predicted Answer: Kinnairdy Castle
Retrieved Contexts (truncated): ['David Gregory (physician) | David Gregory (20 December 1625 – 1720) was a Scottish physician and inventor. His surname is sometimes spelt as Gregorie, the original Scottish spelling. He inherited Kinn...', 'Gregory Tarchaneiotes | Gregory Tarchaneiotes (Greek: Γρηγόριος Ταρχανειώτης , Italian: "Gregorio Tracanioto" or "Tracamoto" ) was a "protospatharius" and the long-reigning catepan of Italy from 998 t...', 'David Gregory (mathematician) | David Gregory (originally spelt Gregorie) FRS (? 1659 – 10 October 1708) was a Scottish mathematician and astronomer. He was professor of mathematics at the University ...']

turbo.inspect_history(n=1)

Answer questions with short factoid answers.

---

Question: At My Window was released by which American singer-songwriter?
Answer: John Townes Van Zandt

Question: "Everything Has Changed" is a song from an album released under which record label ?
Answer: Big Machine Records

Question: The Victorians - Their Story In Pictures is a documentary series written by an author born in what year?
Answer: 1950

Question: Which Pakistani cricket umpire who won 3 consecutive ICC umpire of the year awards in 2009, 2010, and 2011 will be in the ICC World Twenty20?
Answer: Aleem Sarwar Dar

Question: Having the combination of excellent foot speed and bat speed helped Eric Davis, create what kind of outfield for the Los Angeles Dodgers?
Answer: "Outfield of Dreams"

Question: Who is older, Aleksandr Danilovich Aleksandrov or Anatoly Fomenko?
Answer: Aleksandr Danilovich Aleksandrov

Question: The Organisation that allows a community to influence their operation or use and to enjoy the benefits arisingwas founded in what year?
Answer: 2010

Question: Tombstone stared an actor born May 17, 1955 known as who?
Answer: Bill Paxton

Question: In what year was the club founded that played Manchester City in the 1972 FA Charity Shield
Answer: 1874

Question: which American actor was Candace Kita guest starred with
Answer: Bill Murray

Question: Which is taller, the Empire State Building or the Bank of America Tower?
Answer: The Empire State Building

Question: Which company distributed this 1977 American animated film produced by Walt Disney Productions for which Sherman Brothers wrote songs?
Answer: Buena Vista Distribution

---

Follow the following format.

Context: may contain relevant facts

Question: ${question}

Reasoning: Let's think step by step in order to ${produce the answer}. We ...

Answer: often between 1 and 5 words

---

Context:
[1] «Tae Kwon Do Times | Tae Kwon Do Times is a magazine devoted to the martial art of taekwondo, and is published in the United States of America. While the title suggests that it focuses on taekwondo exclusively, the magazine also covers other Korean martial arts. "Tae Kwon Do Times" has published articles by a wide range of authors, including He-Young Kimm, Thomas Kurz, Scott Shaw, and Mark Van Schuyver.»
[2] «Kwon Tae-man | Kwon Tae-man (born 1941) was an early Korean hapkido practitioner and a pioneer of the art, first in Korea and then in the United States. He formed one of the earliest dojang's for hapkido in the United States in Torrance, California, and has been featured in many magazine articles promoting the art.»
[3] «Hee Il Cho | Cho Hee Il (born October 13, 1940) is a prominent Korean-American master of taekwondo, holding the rank of 9th "dan" in the martial art. He has written 11 martial art books, produced 70 martial art training videos, and has appeared on more than 70 martial arts magazine covers. Cho won several national and international competitions as a taekwondo competitor, and has appeared in several films, including "Fight to Win", "Best of the Best", "Bloodsport II", and "Bloodsport III". He founded the Action International Martial Arts Association (AIMAA) in 1980, and is its President. Cho is a member of both "Black Belt" magazine's Hall of Fame and "Tae Kwon Do Times" magazine's Hall of Fame.»

Question: Which magazine has published articles by Scott Shaw, Tae Kwon Do Times or Southwest Art?

Reasoning: Let's think step by step in order to produce the answer. We know that "Tae Kwon Do Times" has published articles by Scott Shaw, as mentioned in the context.

Answer: Tae Kwon Do Times

---

Context:
[1] «Rosario Dawson | Rosario Isabel Dawson (born May 9, 1979) is an American actress, producer, singer, comic book writer, and political activist. She made her film debut in the 1995 teen drama "Kids". Her subsequent film roles include "He Got Game", "Men in Black II", "25th Hour", "Rent", "Sin City", "Death Proof", "Seven Pounds", "", and "Top Five". Dawson has also provided voice-over work for Disney and DC.»
[2] «Sarai Gonzalez | Sarai Isaura Gonzalez (born 2005) is an American Latina child actress who made her professional debut at the age of 11 on the Spanish-language ""Soy Yo"" ("That's Me") music video by Bomba Estéreo. Cast as a "nerdy" tween with a "sassy" and "confident" attitude, her performance turned her into a "Latina icon" for "female empowerment, identity and self-worth". She subsequently appeared in two get out the vote videos for Latinos in advance of the 2016 United States elections.»
[3] «Gabriela (2001 film) | Gabriela is a 2001 American romance film, starring Seidy Lopez in the title role alongside Jaime Gomez as her admirer Mike. The film has been cited as an inspiration behind the Premiere Weekend Club, which supports Latino film-making.»

Question: Which American actress who made their film debut in the 1995 teen drama "Kids" was the co-founder of Voto Latino?

Reasoning: Let's think step by step in order to produce the answer. We know that the actress who made her film debut in "Kids" is Rosario Dawson, and she is also known for her political activism and involvement in Voto Latino.

Answer: Rosario Dawson

---

Context:
[1] «Battle of Kursk | The Battle of Kursk was a Second World War engagement between German and Soviet forces on the Eastern Front near Kursk (450 km south-west of Moscow) in the Soviet Union during July and August 1943. The battle began with the launch of the German offensive, Operation Citadel (German: "Unternehmen Zitadelle" ), on 5 July, which had the objective of pinching off the Kursk salient with attacks on the base of the salient from north and south simultaneously. After the German offensive stalled on the northern side of the salient, on 12 July the Soviets commenced their Kursk Strategic Offensive Operation with the launch of Operation Kutuzov (Russian: Кутузов ) against the rear of the German forces in the northern side. On the southern side, the Soviets also launched powerful counterattacks the same day, one of which led to a large armoured clash, the Battle of Prokhorovka. On 3 August, the Soviets began the second phase of the Kursk Strategic Offensive Operation with the launch of Operation Polkovodets Rumyantsev (Russian: Полководец Румянцев ) against the German forces in the southern side of the Kursk salient.»
[2] «Operation Mars | Operation Mars, also known as the Second Rzhev-Sychevka Offensive Operation (Russian: Вторая Ржевско-Сычёвская наступательная операция), was the codename for an offensive launched by Soviet forces against German forces during World War II. It took place between 25 November and 20 December 1942 around the Rzhev salient in the vicinity of Moscow.»
[3] «Kholm Pocket | The Kholm Pocket (German: "Kessel von Cholm" ; Russian: Холмский котёл ) was the name given for the encirclement of German troops by the Red Army around Kholm south of Leningrad, during World War II on the Eastern Front, from 23 January 1942 until 5 May 1942. A much larger pocket was simultaneously surrounded in Demyansk, about 100 km to the northeast. These were the results of German retreat following their defeat during the Battle of Moscow.»

Question: What is the code name for the German offensive that started this Second World War engagement on the Eastern Front (a few hundred kilometers from Moscow) between Soviet and German forces, which included 102nd Infantry Division?

Reasoning: Let's think step by step in order to produce the answer. We know that the German offensive that started the Battle of Kursk was called Operation Citadel.

Answer: Operation Citadel

---

Context:
[1] «Kerry Condon | Kerry Condon (born 4 January 1983) is an Irish television and film actress, best known for her role as Octavia of the Julii in the HBO/BBC series "Rome," as Stacey Ehrmantraut in AMC's "Better Call Saul" and as the voice of F.R.I.D.A.Y. in various films in the Marvel Cinematic Universe. She is also the youngest actress ever to play Ophelia in a Royal Shakespeare Company production of "Hamlet."»
[2] «Corona Riccardo | Corona Riccardo (c. 1878October 15, 1917) was an Italian born American actress who had a brief Broadway stage career before leaving to become a wife and mother. Born in Naples she came to acting in 1894 playing a Mexican girl in a play at the Empire Theatre. Wilson Barrett engaged her for a role in his play "The Sign of the Cross" which he took on tour of the United States. Riccardo played the role of Ancaria and later played Berenice in the same play. Robert B. Mantell in 1898 who struck by her beauty also cast her in two Shakespeare plays, "Romeo and Juliet" and "Othello". Author Lewis Strang writing in 1899 said Riccardo was the most promising actress in America at the time. Towards the end of 1898 Mantell chose her for another Shakespeare part, Ophelia im Hamlet. Afterwards she was due to join Augustin Daly's Theatre Company but Daly died in 1899. In 1899 she gained her biggest fame by playing Iras in the first stage production of Ben-Hur.»
[3] «Judi Dench | Dame Judith Olivia "Judi" Dench, {'1': ", '2': ", '3': ", '4': "} (born 9 December 1934) is an English actress and author. Dench made her professional debut in 1957 with the Old Vic Company. Over the following few years, she performed in several of Shakespeare's plays in such roles as Ophelia in "Hamlet", Juliet in "Romeo and Juliet", and Lady Macbeth in "Macbeth". Although most of her work during this period was in theatre, she also branched into film work and won a BAFTA Award as Most Promising Newcomer. She drew strong reviews for her leading role in the musical "Cabaret" in 1968.»

Question: Who acted in the shot film The Shore and is also the youngest actress ever to play Ophelia in a Royal Shakespeare Company production of "Hamlet." ?

Reasoning: Let's think step by step in order to produce the answer. We know that the actress who played Ophelia in a Royal Shakespeare Company production of "Hamlet" is Kerry Condon.

Answer: Kerry Condon

---

Context:
[1] «David Gregory (physician) | David Gregory (20 December 1625 – 1720) was a Scottish physician and inventor. His surname is sometimes spelt as Gregorie, the original Scottish spelling. He inherited Kinnairdy Castle in 1664. Three of his twenty-nine children became mathematics professors. He is credited with inventing a military cannon that Isaac Newton described as "being destructive to the human species". Copies and details of the model no longer exist. Gregory's use of a barometer to predict farming-related weather conditions led him to be accused of witchcraft by Presbyterian ministers from Aberdeen, although he was never convicted.»
[2] «Gregory Tarchaneiotes | Gregory Tarchaneiotes (Greek: Γρηγόριος Ταρχανειώτης , Italian: "Gregorio Tracanioto" or "Tracamoto" ) was a "protospatharius" and the long-reigning catepan of Italy from 998 to 1006. In December 999, and again on February 2, 1002, he reinstituted and confirmed the possessions of the abbey and monks of Monte Cassino in Ascoli. In 1004, he fortified and expanded the castle of Dragonara on the Fortore. He gave it three circular towers and one square one. He also strengthened Lucera.»
[3] «David Gregory (mathematician) | David Gregory (originally spelt Gregorie) FRS (? 1659 – 10 October 1708) was a Scottish mathematician and astronomer. He was professor of mathematics at the University of Edinburgh, Savilian Professor of Astronomy at the University of Oxford, and a commentator on Isaac Newton's "Principia".»

Question: What castle did David Gregory inherit?

Reasoning: Let's think step by step in order to produce the answer. We know that David Gregory inherited Kinnairdy Castle.

Answer: Kinnairdy Castle

'\n\n\nAnswer questions with short factoid answers.\n\n---\n\nQuestion: At My Window was released by which American singer-songwriter?\nAnswer: John Townes Van Zandt\n\nQuestion: "Everything Has Changed" is a song from an album released under which record label ?\nAnswer: Big Machine Records\n\nQuestion: The Victorians - Their Story In Pictures is a documentary series written by an author born in what year?\nAnswer: 1950\n\nQuestion: Which Pakistani cricket umpire who won 3 consecutive ICC umpire of the year awards in 2009, 2010, and 2011 will be in the ICC World Twenty20?\nAnswer: Aleem Sarwar Dar\n\nQuestion: Having the combination of excellent foot speed and bat speed helped Eric Davis, create what kind of outfield for the Los Angeles Dodgers?\nAnswer: "Outfield of Dreams"\n\nQuestion: Who is older, Aleksandr Danilovich Aleksandrov or Anatoly Fomenko?\nAnswer: Aleksandr Danilovich Aleksandrov\n\nQuestion: The Organisation that allows a community to influence their operation or use and to enjoy the benefits arisingwas founded in what year?\nAnswer: 2010\n\nQuestion: Tombstone stared an actor born May 17, 1955 known as who?\nAnswer: Bill Paxton\n\nQuestion: In what year was the club founded that played Manchester City in the 1972 FA Charity Shield\nAnswer: 1874\n\nQuestion: which American actor was Candace Kita guest starred with\nAnswer: Bill Murray\n\nQuestion: Which is taller, the Empire State Building or the Bank of America Tower?\nAnswer: The Empire State Building\n\nQuestion: Which company distributed this 1977 American animated film produced by Walt Disney Productions for which Sherman Brothers wrote songs?\nAnswer: Buena Vista Distribution\n\n---\n\nFollow the following format.\n\nContext: may contain relevant facts\n\nQuestion: ${question}\n\nReasoning: Let\'s think step by step in order to ${produce the answer}. We ...\n\nAnswer: often between 1 and 5 words\n\n---\n\nContext:\n[1] «Tae Kwon Do Times | Tae Kwon Do Times is a magazine devoted to the martial art of taekwondo, and is published in the United States of America. While the title suggests that it focuses on taekwondo exclusively, the magazine also covers other Korean martial arts. "Tae Kwon Do Times" has published articles by a wide range of authors, including He-Young Kimm, Thomas Kurz, Scott Shaw, and Mark Van Schuyver.»\n[2] «Kwon Tae-man | Kwon Tae-man (born 1941) was an early Korean hapkido practitioner and a pioneer of the art, first in Korea and then in the United States. He formed one of the earliest dojang\'s for hapkido in the United States in Torrance, California, and has been featured in many magazine articles promoting the art.»\n[3] «Hee Il Cho | Cho Hee Il (born October 13, 1940) is a prominent Korean-American master of taekwondo, holding the rank of 9th "dan" in the martial art. He has written 11 martial art books, produced 70 martial art training videos, and has appeared on more than 70 martial arts magazine covers. Cho won several national and international competitions as a taekwondo competitor, and has appeared in several films, including "Fight to Win", "Best of the Best", "Bloodsport II", and "Bloodsport III". He founded the Action International Martial Arts Association (AIMAA) in 1980, and is its President. Cho is a member of both "Black Belt" magazine\'s Hall of Fame and "Tae Kwon Do Times" magazine\'s Hall of Fame.»\n\nQuestion: Which magazine has published articles by Scott Shaw, Tae Kwon Do Times or Southwest Art?\n\nReasoning: Let\'s think step by step in order to produce the answer. We know that "Tae Kwon Do Times" has published articles by Scott Shaw, as mentioned in the context.\n\nAnswer: Tae Kwon Do Times\n\n---\n\nContext:\n[1] «Rosario Dawson | Rosario Isabel Dawson (born May 9, 1979) is an American actress, producer, singer, comic book writer, and political activist. She made her film debut in the 1995 teen drama "Kids". Her subsequent film roles include "He Got Game", "Men in Black II", "25th Hour", "Rent", "Sin City", "Death Proof", "Seven Pounds", "", and "Top Five". Dawson has also provided voice-over work for Disney and DC.»\n[2] «Sarai Gonzalez | Sarai Isaura Gonzalez (born 2005) is an American Latina child actress who made her professional debut at the age of 11 on the Spanish-language ""Soy Yo"" ("That\'s Me") music video by Bomba Estéreo. Cast as a "nerdy" tween with a "sassy" and "confident" attitude, her performance turned her into a "Latina icon" for "female empowerment, identity and self-worth". She subsequently appeared in two get out the vote videos for Latinos in advance of the 2016 United States elections.»\n[3] «Gabriela (2001 film) | Gabriela is a 2001 American romance film, starring Seidy Lopez in the title role alongside Jaime Gomez as her admirer Mike. The film has been cited as an inspiration behind the Premiere Weekend Club, which supports Latino film-making.»\n\nQuestion: Which American actress who made their film debut in the 1995 teen drama "Kids" was the co-founder of Voto Latino?\n\nReasoning: Let\'s think step by step in order to produce the answer. We know that the actress who made her film debut in "Kids" is Rosario Dawson, and she is also known for her political activism and involvement in Voto Latino.\n\nAnswer: Rosario Dawson\n\n---\n\nContext:\n[1] «Battle of Kursk | The Battle of Kursk was a Second World War engagement between German and Soviet forces on the Eastern Front near Kursk (450 km south-west of Moscow) in the Soviet Union during July and August 1943. The battle began with the launch of the German offensive, Operation Citadel (German: "Unternehmen Zitadelle" ), on 5 July, which had the objective of pinching off the Kursk salient with attacks on the base of the salient from north and south simultaneously. After the German offensive stalled on the northern side of the salient, on 12 July the Soviets commenced their Kursk Strategic Offensive Operation with the launch of Operation Kutuzov (Russian: Кутузов ) against the rear of the German forces in the northern side. On the southern side, the Soviets also launched powerful counterattacks the same day, one of which led to a large armoured clash, the Battle of Prokhorovka. On 3 August, the Soviets began the second phase of the Kursk Strategic Offensive Operation with the launch of Operation Polkovodets Rumyantsev (Russian: Полководец Румянцев ) against the German forces in the southern side of the Kursk salient.»\n[2] «Operation Mars | Operation Mars, also known as the Second Rzhev-Sychevka Offensive Operation (Russian: Вторая Ржевско-Сычёвская наступательная операция), was the codename for an offensive launched by Soviet forces against German forces during World War II. It took place between 25 November and 20 December 1942 around the Rzhev salient in the vicinity of Moscow.»\n[3] «Kholm Pocket | The Kholm Pocket (German: "Kessel von Cholm" ; Russian: Холмский котёл ) was the name given for the encirclement of German troops by the Red Army around Kholm south of Leningrad, during World War II on the Eastern Front, from 23 January 1942 until 5 May 1942. A much larger pocket was simultaneously surrounded in Demyansk, about 100 km to the northeast. These were the results of German retreat following their defeat during the Battle of Moscow.»\n\nQuestion: What is the code name for the German offensive that started this Second World War engagement on the Eastern Front (a few hundred kilometers from Moscow) between Soviet and German forces, which included 102nd Infantry Division?\n\nReasoning: Let\'s think step by step in order to produce the answer. We know that the German offensive that started the Battle of Kursk was called Operation Citadel.\n\nAnswer: Operation Citadel\n\n---\n\nContext:\n[1] «Kerry Condon | Kerry Condon (born 4 January 1983) is an Irish television and film actress, best known for her role as Octavia of the Julii in the HBO/BBC series "Rome," as Stacey Ehrmantraut in AMC\'s "Better Call Saul" and as the voice of F.R.I.D.A.Y. in various films in the Marvel Cinematic Universe. She is also the youngest actress ever to play Ophelia in a Royal Shakespeare Company production of "Hamlet."»\n[2] «Corona Riccardo | Corona Riccardo (c. 1878October 15, 1917) was an Italian born American actress who had a brief Broadway stage career before leaving to become a wife and mother. Born in Naples she came to acting in 1894 playing a Mexican girl in a play at the Empire Theatre. Wilson Barrett engaged her for a role in his play "The Sign of the Cross" which he took on tour of the United States. Riccardo played the role of Ancaria and later played Berenice in the same play. Robert B. Mantell in 1898 who struck by her beauty also cast her in two Shakespeare plays, "Romeo and Juliet" and "Othello". Author Lewis Strang writing in 1899 said Riccardo was the most promising actress in America at the time. Towards the end of 1898 Mantell chose her for another Shakespeare part, Ophelia im Hamlet. Afterwards she was due to join Augustin Daly\'s Theatre Company but Daly died in 1899. In 1899 she gained her biggest fame by playing Iras in the first stage production of Ben-Hur.»\n[3] «Judi Dench | Dame Judith Olivia "Judi" Dench, {\'1\': ", \'2\': ", \'3\': ", \'4\': "} (born 9 December 1934) is an English actress and author. Dench made her professional debut in 1957 with the Old Vic Company. Over the following few years, she performed in several of Shakespeare\'s plays in such roles as Ophelia in "Hamlet", Juliet in "Romeo and Juliet", and Lady Macbeth in "Macbeth". Although most of her work during this period was in theatre, she also branched into film work and won a BAFTA Award as Most Promising Newcomer. She drew strong reviews for her leading role in the musical "Cabaret" in 1968.»\n\nQuestion: Who acted in the shot film The Shore and is also the youngest actress ever to play Ophelia in a Royal Shakespeare Company production of "Hamlet." ?\n\nReasoning: Let\'s think step by step in order to produce the answer. We know that the actress who played Ophelia in a Royal Shakespeare Company production of "Hamlet" is Kerry Condon.\n\nAnswer: Kerry Condon\n\n---\n\nContext:\n[1] «David Gregory (physician) | David Gregory (20 December 1625 – 1720) was a Scottish physician and inventor. His surname is sometimes spelt as Gregorie, the original Scottish spelling. He inherited Kinnairdy Castle in 1664. Three of his twenty-nine children became mathematics professors. He is credited with inventing a military cannon that Isaac Newton described as "being destructive to the human species". Copies and details of the model no longer exist. Gregory\'s use of a barometer to predict farming-related weather conditions led him to be accused of witchcraft by Presbyterian ministers from Aberdeen, although he was never convicted.»\n[2] «Gregory Tarchaneiotes | Gregory Tarchaneiotes (Greek: Γρηγόριος Ταρχανειώτης , Italian: "Gregorio Tracanioto" or "Tracamoto" ) was a "protospatharius" and the long-reigning catepan of Italy from 998 to 1006. In December 999, and again on February 2, 1002, he reinstituted and confirmed the possessions of the abbey and monks of Monte Cassino in Ascoli. In 1004, he fortified and expanded the castle of Dragonara on the Fortore. He gave it three circular towers and one square one. He also strengthened Lucera.»\n[3] «David Gregory (mathematician) | David Gregory (originally spelt Gregorie) FRS (? 1659 – 10 October 1708) was a Scottish mathematician and astronomer. He was professor of mathematics at the University of Edinburgh, Savilian Professor of Astronomy at the University of Oxford, and a commentator on Isaac Newton\'s "Principia".»\n\nQuestion: What castle did David Gregory inherit?\n\nReasoning: Let\'s think step by step in order to\x1b[32m produce the answer. We know that David Gregory inherited Kinnairdy Castle.\n\nAnswer: Kinnairdy Castle\x1b[0m\n\n\n'

for name, parameter in compiled_rag.named_predictors():
    print(name)
    print(parameter.demos[0])
    print()

generate_answer
Example({'augmented': True, 'context': ['Tae Kwon Do Times | Tae Kwon Do Times is a magazine devoted to the martial art of taekwondo, and is published in the United States of America. While the title suggests that it focuses on taekwondo exclusively, the magazine also covers other Korean martial arts. "Tae Kwon Do Times" has published articles by a wide range of authors, including He-Young Kimm, Thomas Kurz, Scott Shaw, and Mark Van Schuyver.', "Kwon Tae-man | Kwon Tae-man (born 1941) was an early Korean hapkido practitioner and a pioneer of the art, first in Korea and then in the United States. He formed one of the earliest dojang's for hapkido in the United States in Torrance, California, and has been featured in many magazine articles promoting the art.", 'Hee Il Cho | Cho Hee Il (born October 13, 1940) is a prominent Korean-American master of taekwondo, holding the rank of 9th "dan" in the martial art. He has written 11 martial art books, produced 70 martial art training videos, and has appeared on more than 70 martial arts magazine covers. Cho won several national and international competitions as a taekwondo competitor, and has appeared in several films, including "Fight to Win", "Best of the Best", "Bloodsport II", and "Bloodsport III". He founded the Action International Martial Arts Association (AIMAA) in 1980, and is its President. Cho is a member of both "Black Belt" magazine\'s Hall of Fame and "Tae Kwon Do Times" magazine\'s Hall of Fame.'], 'question': 'Which magazine has published articles by Scott Shaw, Tae Kwon Do Times or Southwest Art?', 'rationale': 'produce the answer. We know that "Tae Kwon Do Times" has published articles by Scott Shaw, as mentioned in the context.', 'answer': 'Tae Kwon Do Times'}) (input_keys=None)

Evaluate the Pipeline

from dspy.evaluate.evaluate import Evaluate

# Set up the `evaluate_on_hotpotqa` function. We'll use this many times below.
evaluate_on_hotpotqa = Evaluate(
    devset=devset, num_threads=1, display_progress=False, display_table=5
)

# Evaluate the `compiled_rag` program with the `answer_exact_match` metric.
metric = dspy.evaluate.answer_exact_match
evaluate_on_hotpotqa(compiled_rag, metric=metric)

	question	example_answer	gold_titles	context	pred_answer	answer_exact_match
0	Are both Cangzhou and Qionghai in the Hebei province of China?	no	{'Qionghai', 'Cangzhou'}	['Cangzhou | Cangzhou () is a prefecture-level city in eastern Hebei province, People\'s Republic of China. At the 2010 census, Cangzhou\'s built-up ("or metro") area...	Yes	False
1	Who conducts the draft in which Marc-Andre Fleury was drafted to the Vegas Golden Knights for the 2017-18 season?	National Hockey League	{'2017 NHL Expansion Draft', '2017–18 Pittsburgh Penguins season'}	['2017–18 Pittsburgh Penguins season | The 2017–18 Pittsburgh Penguins season will be the 51st season for the National Hockey League ice hockey team that was...	National Hockey League	✔️ [True]
2	The Wings entered a new era, following the retirement of which Canadian retired professional ice hockey player and current general manager of the Tampa Bay...	Steve Yzerman	{'Steve Yzerman', '2006–07 Detroit Red Wings season'}	['Steve Yzerman | Stephen Gregory "Steve" Yzerman ( ; born May 9, 1965) is a Canadian retired professional ice hockey player and current general manager...	Steve Yzerman	✔️ [True]
3	What river is near the Crichton Collegiate Church?	the River Tyne	{'Crichton Collegiate Church', 'Crichton Castle'}	["Crichton Collegiate Church | Crichton Collegiate Church is situated about 0.6 mi south west of the hamlet of Crichton in Midlothian, Scotland. Crichton itself is...	River Tyne	✔️ [True]
4	In the 10th Century A.D. Ealhswith had a son called Æthelweard by which English king?	King Alfred the Great	{'Æthelweard (son of Alfred)', 'Ealhswith'}	["Æthelweard of East Anglia | Æthelweard (died 854) was a 9th-century king of East Anglia, the long-lived Anglo-Saxon kingdom which today includes the English counties...	Alfred the Great	False

... 45 more rows not displayed ...

48.0

Evaluate Retrieval

def gold_passages_retrieved(example, pred, trace=None):
    gold_titles = set(map(dspy.evaluate.normalize_text, example["gold_titles"]))
    found_titles = set(
        map(dspy.evaluate.normalize_text, [c.split(" | ")[0] for c in pred.context])
    )

    return gold_titles.issubset(found_titles)


compiled_rag_retrieval_score = evaluate_on_hotpotqa(
    compiled_rag, metric=gold_passages_retrieved
)

	question	example_answer	gold_titles	context	pred_answer	gold_passages_retrieved
0	Are both Cangzhou and Qionghai in the Hebei province of China?	no	{'Qionghai', 'Cangzhou'}	['Cangzhou | Cangzhou () is a prefecture-level city in eastern Hebei province, People\'s Republic of China. At the 2010 census, Cangzhou\'s built-up ("or metro") area...	Yes	False
1	Who conducts the draft in which Marc-Andre Fleury was drafted to the Vegas Golden Knights for the 2017-18 season?	National Hockey League	{'2017 NHL Expansion Draft', '2017–18 Pittsburgh Penguins season'}	['2017–18 Pittsburgh Penguins season | The 2017–18 Pittsburgh Penguins season will be the 51st season for the National Hockey League ice hockey team that was...	National Hockey League	✔️ [True]
2	The Wings entered a new era, following the retirement of which Canadian retired professional ice hockey player and current general manager of the Tampa Bay...	Steve Yzerman	{'Steve Yzerman', '2006–07 Detroit Red Wings season'}	['Steve Yzerman | Stephen Gregory "Steve" Yzerman ( ; born May 9, 1965) is a Canadian retired professional ice hockey player and current general manager...	Steve Yzerman	✔️ [True]
3	What river is near the Crichton Collegiate Church?	the River Tyne	{'Crichton Collegiate Church', 'Crichton Castle'}	["Crichton Collegiate Church | Crichton Collegiate Church is situated about 0.6 mi south west of the hamlet of Crichton in Midlothian, Scotland. Crichton itself is...	River Tyne	✔️ [True]
4	In the 10th Century A.D. Ealhswith had a son called Æthelweard by which English king?	King Alfred the Great	{'Æthelweard (son of Alfred)', 'Ealhswith'}	["Æthelweard of East Anglia | Æthelweard (died 854) was a 9th-century king of East Anglia, the long-lived Anglo-Saxon kingdom which today includes the English counties...	Alfred the Great	False

... 45 more rows not displayed ...

Multi-Hop Question-Answering

import dspy

turbo = dspy.OpenAI(model="gpt-3.5-turbo")
colbertv2_wiki17_abstracts = dspy.ColBERTv2(
    url="http://20.102.90.50:2017/wiki17_abstracts"
)

dspy.settings.configure(lm=turbo, rm=colbertv2_wiki17_abstracts)

from dspy.datasets import HotPotQA

# Load the dataset.
dataset = HotPotQA(
    train_seed=1, train_size=20, eval_seed=2023, dev_size=50, test_size=0
)

# Tell DSPy that the 'question' field is the input. Any other fields are labels and/or metadata.
trainset = [x.with_inputs("question") for x in dataset.train]
devset = [x.with_inputs("question") for x in dataset.dev]

len(trainset), len(devset)

/Users/dothomps/.pyenv/versions/3.11.9/envs/blog/lib/python3.11/site-packages/datasets/table.py:1421: FutureWarning: promote has been superseded by promote_options='default'.
  table = cls._concat_blocks(blocks, axis=0)

(20, 50)

class GenerateAnswer(dspy.Signature):
    """Answer questions with short factoid answers."""

    context = dspy.InputField(desc="may contain relevant facts")
    question = dspy.InputField()
    answer = dspy.OutputField(desc="often between 1 and 5 words")

class GenerateSearchQuery(dspy.Signature):
    """Write a simple search query that will help answer a complex question."""

    context = dspy.InputField(desc="may contain relevant facts")
    question = dspy.InputField()
    query = dspy.OutputField()

from dsp.utils import deduplicate


class SimplifiedBaleen(dspy.Module):
    def __init__(self, passages_per_hop=3, max_hops=2):
        super().__init__()

        self.generate_query = [
            dspy.ChainOfThought(GenerateSearchQuery) for _ in range(max_hops)
        ]
        self.retrieve = dspy.Retrieve(k=passages_per_hop)
        self.generate_answer = dspy.ChainOfThought(GenerateAnswer)
        self.max_hops = max_hops

    def forward(self, question):
        context = []

        for hop in range(self.max_hops):
            query = self.generate_query[hop](context=context, question=question).query
            passages = self.retrieve(query).passages
            context = deduplicate(context + passages)

        pred = self.generate_answer(context=context, question=question)
        return dspy.Prediction(context=context, answer=pred.answer)

# Ask any question you like to this simple RAG program.
my_question = "How many storeys are in the castle that David Gregory inherited?"

# Get the prediction. This contains `pred.context` and `pred.answer`.
uncompiled_baleen = SimplifiedBaleen()  # uncompiled (i.e., zero-shot) program
pred = uncompiled_baleen(my_question)

# Print the contexts and the answer.
print(f"Question: {my_question}")
print(f"Predicted Answer: {pred.answer}")
print(f"Retrieved Contexts (truncated): {[c[:200] + '...' for c in pred.context]}")

Question: How many storeys are in the castle that David Gregory inherited?
Predicted Answer: five
Retrieved Contexts (truncated): ['David Gregory (physician) | David Gregory (20 December 1625 – 1720) was a Scottish physician and inventor. His surname is sometimes spelt as Gregorie, the original Scottish spelling. He inherited Kinn...', 'Cantacuzino Castle | Cantacuzino Castle is situated on Zamora Street in Bușteni, Romania. The building, whose construction was completed in 1911, was conducted by the architect Gregory Cerchez at the ...', 'Gregory Tarchaneiotes | Gregory Tarchaneiotes (Greek: Γρηγόριος Ταρχανειώτης , Italian: "Gregorio Tracanioto" or "Tracamoto" ) was a "protospatharius" and the long-reigning catepan of Italy from 998 t...', 'Kinnairdy Castle | Kinnairdy Castle is a tower house, having five storeys and a garret, two miles south of Aberchirder, Aberdeenshire, Scotland. The alternative name is Old Kinnairdy....', 'Kinnordy House | Kinnordy House (alternative spellings: Kynnordy, Kinardy, Kinnordie and Kinorde) is an estate house near Kirriemuir in Angus, Scotland. The first house was built in the 1680s, when In...']

def validate_context_and_answer_and_hops(example, pred, trace=None):
    if not dspy.evaluate.answer_exact_match(example, pred):
        return False
    if not dspy.evaluate.answer_passage_match(example, pred):
        return False

    hops = [example.question] + [
        outputs.query for *_, outputs in trace if "query" in outputs
    ]

    if max([len(h) for h in hops]) > 100:
        return False
    if any(
        dspy.evaluate.answer_exact_match_str(hops[idx], hops[:idx], frac=0.8)
        for idx in range(2, len(hops))
    ):
        return False

    return True

from dspy.teleprompt import BootstrapFewShot

teleprompter = BootstrapFewShot(metric=validate_context_and_answer_and_hops)
compiled_baleen = teleprompter.compile(
    SimplifiedBaleen(), teacher=SimplifiedBaleen(passages_per_hop=2), trainset=trainset
)

 90%|█████████ | 18/20 [00:00<00:00, 37.83it/s]

from dspy.evaluate.evaluate import Evaluate


# Define metric to check if we retrieved the correct documents
def gold_passages_retrieved(example, pred, trace=None):
    gold_titles = set(map(dspy.evaluate.normalize_text, example["gold_titles"]))
    found_titles = set(
        map(dspy.evaluate.normalize_text, [c.split(" | ")[0] for c in pred.context])
    )
    return gold_titles.issubset(found_titles)


# Set up the `evaluate_on_hotpotqa` function. We'll use this many times below.
evaluate_on_hotpotqa = Evaluate(
    devset=devset, num_threads=1, display_progress=True, display_table=5
)

uncompiled_baleen_retrieval_score = evaluate_on_hotpotqa(
    uncompiled_baleen, metric=gold_passages_retrieved,
)

compiled_baleen_retrieval_score = evaluate_on_hotpotqa(
    compiled_baleen, metric=gold_passages_retrieved
)

print(f"## Retrieval Score for uncompiled Baleen: {uncompiled_baleen_retrieval_score}")
print(f"## Retrieval Score for compiled Baleen: {compiled_baleen_retrieval_score}")

Average Metric: 23 / 50  (46.0): 100%|██████████| 50/50 [03:37<00:00,  4.35s/it]
Average Metric: 32 / 50  (64.0): 100%|██████████| 50/50 [03:40<00:00,  4.42s/it]
## Retrieval Score for uncompiled Baleen: 46.0
## Retrieval Score for compiled Baleen: 64.0

	question	example_answer	gold_titles	context	pred_answer	gold_passages_retrieved
0	Are both Cangzhou and Qionghai in the Hebei province of China?	no	{'Qionghai', 'Cangzhou'}	['Qionghai | Qionghai () is one of the seven county-level cities of Hainan province, China. Although called a "city", Qionghai refers to a large land...	No	False
1	Who conducts the draft in which Marc-Andre Fleury was drafted to the Vegas Golden Knights for the 2017-18 season?	National Hockey League	{'2017 NHL Expansion Draft', '2017–18 Pittsburgh Penguins season'}	["2017 NHL Expansion Draft | The 2017 NHL Expansion Draft was an expansion draft conducted by the National Hockey League on June 18–20, 2017 to...	National Hockey League	✔️ [True]
2	The Wings entered a new era, following the retirement of which Canadian retired professional ice hockey player and current general manager of the Tampa Bay...	Steve Yzerman	{'Steve Yzerman', '2006–07 Detroit Red Wings season'}	['Steve Yzerman | Stephen Gregory "Steve" Yzerman ( ; born May 9, 1965) is a Canadian retired professional ice hockey player and current general manager...	Steve Yzerman	False
3	What river is near the Crichton Collegiate Church?	the River Tyne	{'Crichton Collegiate Church', 'Crichton Castle'}	['River Esk, Dumfries and Galloway | The River Esk (Scottish Gaelic: "Easg" ), also called the Border Esk, is a river in Dumfries and Galloway,...	River Nith	False
4	In the 10th Century A.D. Ealhswith had a son called Æthelweard by which English king?	King Alfred the Great	{'Æthelweard (son of Alfred)', 'Ealhswith'}	['10th century in England | Events from the 10th century in the Kingdom of England.', 'Beorhtnoð æthling of Kent | Beorhtnoð æthling of Kent was...	King Alfred the Great	✔️ [True]

... 45 more rows not displayed ...

	question	example_answer	gold_titles	context	pred_answer	gold_passages_retrieved
0	Are both Cangzhou and Qionghai in the Hebei province of China?	no	{'Qionghai', 'Cangzhou'}	['Cangzhou | Cangzhou () is a prefecture-level city in eastern Hebei province, People\'s Republic of China. At the 2010 census, Cangzhou\'s built-up ("or metro") area...	No	✔️ [True]
1	Who conducts the draft in which Marc-Andre Fleury was drafted to the Vegas Golden Knights for the 2017-18 season?	National Hockey League	{'2017 NHL Expansion Draft', '2017–18 Pittsburgh Penguins season'}	['2017 NHL Entry Draft | The 2017 NHL Entry Draft was the 55th NHL Entry Draft. The draft was held from June 23–24, 2017, at...	2017 NHL Entry Draft	False
2	The Wings entered a new era, following the retirement of which Canadian retired professional ice hockey player and current general manager of the Tampa Bay...	Steve Yzerman	{'Steve Yzerman', '2006–07 Detroit Red Wings season'}	['Steve Yzerman | Stephen Gregory "Steve" Yzerman ( ; born May 9, 1965) is a Canadian retired professional ice hockey player and current general manager...	Steve Yzerman	False
3	What river is near the Crichton Collegiate Church?	the River Tyne	{'Crichton Collegiate Church', 'Crichton Castle'}	["Crichton Collegiate Church | Crichton Collegiate Church is situated about 0.6 mi south west of the hamlet of Crichton in Midlothian, Scotland. Crichton itself is...	River Tyne	✔️ [True]
4	In the 10th Century A.D. Ealhswith had a son called Æthelweard by which English king?	King Alfred the Great	{'Æthelweard (son of Alfred)', 'Ealhswith'}	['Æthelstan | Æthelstan or Athelstan (Old English: "Æþelstan" , "Æðelstān" , meaning "noble stone"; 89427 October 939) was King of the Anglo-Saxons from 924 to...	King Alfred the Great	✔️ [True]

... 45 more rows not displayed ...