Prompt Engineering is Dead; Build LLM Applications with DSPy Framework

Databricks
23 Jul 202442:17

TLDRIn a talk at Data n Summit, the speaker declares 'Prompt Engineering is Dead' and introduces the DSPy framework for building Large Language Model (LLM) applications. They argue for a shift from traditional prompt engineering to a more strategic approach, emphasizing the importance of defining clear tasks, collecting data, and setting up evaluation metrics. The speaker highlights the potential of agents in interacting with the world and generating intellectual property, contrasting with simple reliance on third-party LLMs. The DSPy framework is praised for its ability to automate and optimize prompt engineering, allowing for more effective LLM application development.

Takeaways

  • 🔍 The presenter humorously admits their 'clickbait' title but justifies it due to the audience's presence, hinting at the engaging nature of the topic.
  • 📉 The speaker has undergone a 'mindset shift' regarding prompt engineering, indicating a significant evolution in their approach to working with language models.
  • 🤖 There's an emphasis on the importance of agents and their increasing prevalence, suggesting a move towards more interactive AI systems.
  • 🛠 The last 20% improvement on top of large language models (LLMs) is identified as challenging, implying that while initial progress is fast, refining AI systems requires deeper work.
  • 🚀 The potential of future AI models is acknowledged, but the presenter encourages leveraging current tools to build effective LLM applications.
  • 🏛 The presenter, standing on 'the shoulders of giants', credits researchers for their foundational work, highlighting the importance of building on existing research.
  • 💼 A real-world perspective is offered, focusing on practical applications in business and consulting, rather than purely theoretical discussions.
  • 🛍️ The retail industry is specifically mentioned as a field of application for AI, indicating a sector where the speaker sees potential for growth.
  • 🔗 The DSP (DeepSpeed) framework is introduced as a tool for building LLM applications, emphasizing its role in enhancing development efficiency.
  • 🔧 The presenter suggests a shift from manual 'prompt engineering' to a more programmatic approach, streamlining the process of refining AI interactions.

Q & A

  • What is the main theme of the talk at the Data n Summit?

    -The main theme of the talk is a shift in mindset regarding prompt engineering and the introduction of the DSPy framework for building applications with large language models (LLMs).

  • Why did the speaker choose the title 'Prompt Engineering is Dead; Build LLM Applications with DSPy Framework'?

    -The speaker chose the title as a form of clickbait to attract attention, while acknowledging that prompt engineering still has relevance, but suggesting that the DSPy framework offers a more structured and programmatic approach to building LLM applications.

  • What is the significance of the speaker's mention of 'agents' in the context of LLMs?

    -The speaker refers to 'agents' as a way to extend the capabilities of LLMs by enabling them to interact with the world and other systems, which is seen as a more valuable approach than simply using raw LLMs provided by third parties.

  • What does the speaker suggest as an issue with relying solely on raw LLMs provided by third parties?

    -The speaker suggests that relying solely on raw LLMs does not allow for the creation of intellectual property for the company, and there is a lack of optimization and customization that can be achieved with a more tailored approach.

  • What is the DSP framework and how does it relate to LLM applications?

    -The DSP framework is a structure that allows for the programmatic interaction with LLMs, enabling the optimization of prompts and the building of more sophisticated applications that can interact with other systems and the world.

  • Why is the speaker hiring an ML engineering leader with a passion for retail?

    -The speaker is hiring an ML engineering leader passionate about retail to help build and optimize LLM applications within the retail industry, leveraging the DSP framework to create valuable business solutions.

  • What are some of the prompting strategies discussed in the talk?

    -The prompting strategies discussed include zero-shot, few-shot, being polite, Chain of Thought, chain of density, react thinking, and prompt injection.

  • How does the speaker suggest evaluating the quality of prompts in LLM applications?

    -The speaker suggests that the first step in evaluating prompt quality is to gather data and test cases, followed by iterative refinement of the prompt based on the evaluation metrics and test outcomes.

  • What role does optimization play in the DSP framework?

    -Optimization in the DSP framework is crucial for fine-tuning the interaction with LLMs. It involves using examples to programmatically find the best prompts and parameters that yield the desired outcomes.

  • How does the speaker demonstrate the practical application of the DSP framework?

    -The speaker demonstrates the practical application of the DSP framework by walking through a coding example where they use the framework to build a sentiment analysis pipeline with a language model hosted in Databricks.

Outlines

00:00

📖 Introduction to Prompt Engineering and AI Agents

The speaker begins by acknowledging the audience at Data n Summit and expresses gratitude for attending the late session. They admit to using a 'clickbait' title but justify it by the attendance. The speaker shares their personal shift in understanding prompt engineering and its applications. They position themselves as a practitioner applying research in business and consulting, particularly in retail. They also give credit to the researchers doing the foundational work. The speaker announces a job opening for an ML engineering leader passionate about retail. The session's agenda is outlined to cover agents, prompting strategies, evaluation strategies, and the DSP framework, with a promise to include coding examples for those interested.

05:01

🤖 The Evolution and Importance of AI Agents

The paragraph delves into the concept of AI agents and their increasing prevalence, referencing the morning's keynote. The speaker discusses the potential of AI agents to interact with the world beyond text inputs, emphasizing the value of such interactions for productivity gains. They differentiate between various types of language models (LLMs) and how they fit into the agent approach. The speaker advocates for building systems that interact with the world around them to create intellectual property and reduce reliance on third-party tools. They mention optimization techniques and the importance of staying current with research, highlighting papers and frameworks that contribute to the field.

10:05

🔍 Deep Dive into Prompting Strategies and Evaluation

This section focuses on prompting strategies when dealing with LLMs, starting with early ideas like knowledge fusion and evolving to more advanced techniques like Chain of Thought. The speaker discusses the emergence of prompt engineering as a job role and the various strategies being researched and utilized. They emphasize the importance of data collection and evaluation in building language model systems, suggesting a scientific approach over random guessing. The speaker also touches on the idea of automating prompt optimization and the need for a strategic evaluation process.

15:06

📊 Discussing Frameworks and Strategies for Optimization

The speaker introduces the DSP framework, expressing enthusiasm for its structured approach to working with LLMs. They outline the general workflow, comparing it to the data science process, and emphasize the importance of defining tasks, collecting data, setting up configurations, and evaluating results. The speaker also discusses the concept of optimization in the context of prompt engineering, suggesting that the current methods are rudimentary and have room for growth. They propose the idea of optimizing at the mathematical layer rather than just the linguistic layer.

20:06

🛠️ Practical Implementation of the DSP Framework

The speaker provides a practical demonstration of implementing the DSP framework, starting with the importance of focusing on data. They discuss the simplicity of configuring the framework within Databricks and the ability to use various LMs and tools within the framework. The speaker also highlights the community support for DSP and the availability of connectors for Databricks. They explain the concept of signatures, modules, and optimizers within the framework, giving examples of how to define and use them to build a pipeline for interacting with LMs.

25:09

💻 Coding Examples and Optimizing Prompts

The speaker presents coding examples to illustrate how to use the DSP framework with a dataset of Reddit comments. They demonstrate the process of data cleaning, defining metrics, setting up signatures, and creating modules for optimization. The speaker shows how to interact with LMs using the framework and optimize prompts programmatically. They also caution about the potential costs of extensive LM calls during the optimization process and provide a brief overview of different optimization techniques available within the framework.

30:11

📈 Evaluating and Iterating on AI Agent Performance

In this section, the speaker discusses the process of evaluating AI agent performance using the DSP framework. They detail the creation of an evaluator, the use of metrics, and the importance of using both train and test data for accurate assessments. The speaker demonstrates how to improve accuracy through various optimization techniques, including few-shot learning and instruction optimization. They conclude with a live example of how these optimizations can significantly enhance the performance of an AI agent.

35:12

🎉 Conclusion and Final Thoughts

The speaker concludes the presentation by summarizing the key points and thanking the audience for their time. They reflect on the effectiveness of using the DSP framework for prompt engineering and AI agent development. The speaker is proud to have demonstrated the practical applications of the framework and expresses optimism about its potential for future advancements in AI. The session ends with applause from the audience.

Mindmap

Keywords

💡Prompt Engineering

Prompt engineering refers to the process of carefully crafting input prompts to elicit specific, desired responses from language models. In the context of the video, the speaker discusses a shift in mindset regarding prompt engineering, suggesting that instead of relying on raw large language models (LLMs), businesses should build more meaningful applications by integrating LLMs with other systems and tools. The video emphasizes the importance of going beyond simple prompt tuning to create systems that can interact with the world and generate intellectual property.

💡DSP Framework

The DSP (Differentiable Sequential Programming) Framework is a tool mentioned in the video for building and optimizing applications with LLMs. It provides a structured way to define tasks, collect data, set up configurations, and evaluate outcomes. The framework allows for the automation of testing and optimization, which the speaker believes is crucial for developing efficient and effective LLM applications. It stands out for its optimizer component, which can programmatically improve the interaction with language models based on given data examples.

💡Chain of Thought

The 'Chain of Thought' is a prompting strategy discussed in the video where the language model is asked to explain its reasoning step by step. This method has been shown to improve the quality of responses from LLMs by making them think through the problem-solving process. The video suggests that this technique is an important part of prompt engineering but also indicates that the DSP framework can automate and optimize such strategies.

💡Few-shot Learning

Few-shot learning is a concept in machine learning where an algorithm is provided with a few examples and must generalize from those to make predictions on new, unseen data. In the video, the speaker mentions using few-shot optimization, where the framework injects examples into the prompt to help the LLM produce better outcomes. This is showcased as a method to improve the accuracy of sentiment analysis on Reddit comments.

💡Language Models (LMs)

Language models, or LMs, are AI models that understand and generate human-like text based on the input they receive. The video discusses the evolution of interaction with LMs, starting from simple queries to more complex strategies that involve optimizing prompts and integrating LMs with other tools and systems. The speaker argues for a shift from basic prompt engineering to building comprehensive applications that leverage the capabilities of LMs.

💡Data Science Process

The data science process mentioned in the video refers to the systematic steps followed in data science projects, from defining the problem and collecting data to evaluating results and iterating on the model. The speaker emphasizes that prompt engineering and building LLM applications should follow a similar scientific process, starting with data collection and moving towards optimization and evaluation.

💡DevOps

DevOps is a set of practices that combines software development (Dev) and IT operations (Ops) to shorten the systems development life cycle and provide continuous delivery of high-quality software. The video draws a parallel between DevOps and the need for automation in testing and deployment of LLM applications, suggesting that the principles of DevOps can be applied to streamline the development and optimization of AI systems.

💡Retrieval-Augmented Generation (RAG)

Retrieval-Augmented Generation, or RAG, is a technique that combines retrieval of relevant information with generation of new text. The video mentions RAG as part of the agent approach, where LLMs interact with external systems to retrieve information and then generate responses. This is seen as a way to create more dynamic and useful applications that can provide实质性的 actions and not just text outputs.

💡Optimization

In the context of the video, optimization refers to the process of improving the performance of LLM applications through various techniques such as few-shot learning, instruction tuning, and fine-tuning. The speaker discusses the use of the DSP framework's optimizers to automate this process, allowing the system to empirically determine the best configurations and prompts for a given task.

💡Artificial General Intelligence (AGI)

Artificial General Intelligence, or AGI, is the idea of creating AI systems that possess the ability to understand or learn any intellectual task that a human being can do. The video speaker mentions AGI as a potential future state where a 'master model' could handle all types of tasks, making some current approaches to building LLM applications potentially obsolete.

Highlights

The presenter challenges the current state of prompt engineering and introduces the DSPy framework as a new approach.

The talk emphasizes a shift in mindset regarding prompt engineering and its applications.

The presenter shares their perspective as a practitioner in business and consulting, focusing on applying ideas to industry.

The importance of building on top of large language models (LLMs) for creating meaningful applications is discussed.

The potential for future AI models to replace the need for agents is mentioned, but the presenter suggests we are years away from such a reality.

The concept of agents and their ability to interact with the world beyond text is explored as a source of real value and productivity gains.

The presenter advocates for building systems that interact with the world, rather than relying solely on tools provided by third parties.

The DSP framework is introduced as a phenomenal tool for building LLM applications.

The talk covers four main areas: agents, prompting strategies, evaluation strategies, and the DSP framework.

The presenter discusses the evolution of prompt engineering, starting from simple prompts to more complex strategies like Chain of Thought.

The importance of data in the prompt engineering process is highlighted, emphasizing the need to collect data before crafting prompts.

The presenter suggests automating the testing and optimization process in language model applications, similar to DevOps practices.

The potential for language models to score and optimize their own prompts is introduced as a cutting-edge area of research.

The DSP framework's workflow is outlined, including defining tasks, collecting data, setting up configurations, and optimizing pipelines.

The presenter demonstrates how to use the DSP framework with code examples, showcasing its ease of use and flexibility.

The talk concludes with a live coding session where the presenter builds a sentiment analysis model using the DSP framework.

The effectiveness of different prompting strategies, such as zero-shot and few-shot learning, is demonstrated through the coding examples.

The presenter emphasizes the importance of defining clear evaluation metrics when working with language models.