Prompt Engineering is Dead; Build LLM Applications with DSPy Framework

The speaker begins by acknowledging the audience at Data n Summit and expresses gratitude for attending the late session. They admit to using a 'clickbait' title but justify it by the attendance. The speaker shares their personal shift in understanding prompt engineering and its applications. They position themselves as a practitioner applying research in business and consulting, particularly in retail. They also give credit to the researchers doing the foundational work. The speaker announces a job opening for an ML engineering leader passionate about retail. The session's agenda is outlined to cover agents, prompting strategies, evaluation strategies, and the DSP framework, with a promise to include coding examples for those interested.

The paragraph delves into the concept of AI agents and their increasing prevalence, referencing the morning's keynote. The speaker discusses the potential of AI agents to interact with the world beyond text inputs, emphasizing the value of such interactions for productivity gains. They differentiate between various types of language models (LLMs) and how they fit into the agent approach. The speaker advocates for building systems that interact with the world around them to create intellectual property and reduce reliance on third-party tools. They mention optimization techniques and the importance of staying current with research, highlighting papers and frameworks that contribute to the field.

This section focuses on prompting strategies when dealing with LLMs, starting with early ideas like knowledge fusion and evolving to more advanced techniques like Chain of Thought. The speaker discusses the emergence of prompt engineering as a job role and the various strategies being researched and utilized. They emphasize the importance of data collection and evaluation in building language model systems, suggesting a scientific approach over random guessing. The speaker also touches on the idea of automating prompt optimization and the need for a strategic evaluation process.

The speaker introduces the DSP framework, expressing enthusiasm for its structured approach to working with LLMs. They outline the general workflow, comparing it to the data science process, and emphasize the importance of defining tasks, collecting data, setting up configurations, and evaluating results. The speaker also discusses the concept of optimization in the context of prompt engineering, suggesting that the current methods are rudimentary and have room for growth. They propose the idea of optimizing at the mathematical layer rather than just the linguistic layer.

The speaker provides a practical demonstration of implementing the DSP framework, starting with the importance of focusing on data. They discuss the simplicity of configuring the framework within Databricks and the ability to use various LMs and tools within the framework. The speaker also highlights the community support for DSP and the availability of connectors for Databricks. They explain the concept of signatures, modules, and optimizers within the framework, giving examples of how to define and use them to build a pipeline for interacting with LMs.

The speaker presents coding examples to illustrate how to use the DSP framework with a dataset of Reddit comments. They demonstrate the process of data cleaning, defining metrics, setting up signatures, and creating modules for optimization. The speaker shows how to interact with LMs using the framework and optimize prompts programmatically. They also caution about the potential costs of extensive LM calls during the optimization process and provide a brief overview of different optimization techniques available within the framework.

In this section, the speaker discusses the process of evaluating AI agent performance using the DSP framework. They detail the creation of an evaluator, the use of metrics, and the importance of using both train and test data for accurate assessments. The speaker demonstrates how to improve accuracy through various optimization techniques, including few-shot learning and instruction optimization. They conclude with a live example of how these optimizations can significantly enhance the performance of an AI agent.

The speaker concludes the presentation by summarizing the key points and thanking the audience for their time. They reflect on the effectiveness of using the DSP framework for prompt engineering and AI agent development. The speaker is proud to have demonstrated the practical applications of the framework and expresses optimism about its potential for future advancements in AI. The session ends with applause from the audience.