Implicit In-Context Learning: Supercharge LLMs with DSPy

When communicating, we often convey implicit meanings or subtext that are not directly stated but hinted at through language, behavior, or context. This is known as implicit in-context in pragmatics. For example, in a workplace setting, a boss asking their subordinate “What do you think of this plan?” may seem like seeking an opinion, but the implicit context could indicate the boss wants to hear specific improvement suggestions rather than a simple good or bad evaluation. The subordinate needs to interpret the true intent based on their understanding of the boss and work background.

Expressing such subtext is a sophisticated language art in human communication. However, it can be challenging for large language models (LLMs) to understand the implicit context intended by humans. Yet, scenarios requiring LLMs to grasp “implicit in-context” are quite common, such as sentiment analysis, intent recognition, relation extraction, text classification, and more. Constructing prompts for implicit in-context learning in LLMs is a difficult task.

Limitations of Traditional In-Context Learning

We are familiar with the “in-context learning” (ICL) capabilities of LLMs, where the core idea is to provide a few demonstration examples before the test query to guide the model’s inference. This “few-shot learning” ability has shown LLMs’ remarkable adaptability and flexibility. However, traditional ICL also has limitations:

1. Demonstration examples are directly concatenated before the query in token form, leading to a dramatic increase in computation and memory overhead.
2. The selection and ordering of demonstration examples are very sensitive, and low-quality examples can seriously impact performance.

Introducing Implicit In-Context Learning (I2CL)

To overcome these limitations, researchers from Rutgers University proposed a revolutionary paradigm called Implicit In-Context Learning (I2CL).

I2CL ingeniously incorporates demonstration information in vector form into the LLM’s activation space, achieving performance close to few-shot learning while maintaining zero-shot inference speed. Let’s take a closer look at I2CL.

Context Vectorization: Extracting Refined Information

The first step of I2CL is to independently vectorize each demonstration example. Specifically, using a pre-trained tokenizer and LLM, researchers extract the output activation vectors of the multi-head attention (MHA) and multi-layer perceptron (MLP) modules from the last residual stream of each example as demonstration vectors. These vectors are then permutation-invariantly aggregated into a unified context vector.

The essence of this step lies in the last residual stream’s ability to effectively summarize the overall information of the sequence, while the linear representation hypothesis provides a theoretical basis for linear transformations between demonstration vectors at different abstraction levels. Through this clever vectorization approach, I2CL condenses the originally lengthy token sequences into a refined context representation.

Context Injection: Implicitly Fusing Information

The second step of I2CL is to fuse the context vector with the query’s activation vectors. Unlike attention-based fusion, the researchers adopted a simple yet effective linear combination: linearly weighting the context vector with the query’s MHA and MLP outputs, then applying it to the query’s residual stream.

By introducing learnable scalar weights, I2CL can adaptively control the injection strength of contextual information at different layers. More importantly, the entire process only involves a small number of scalar multiplications and element-wise additions, resulting in extremely low computational overhead. This allows I2CL to match the speed of zero-shot learning during inference.

To further enhance robustness, the researchers introduced random Gaussian noise perturbations to the residual stream during the calibration phase. Through this “noise self-calibration” mechanism, the learnable scalar weights can better generalize to potential downstream variations.

During inference, I2CL injects the context vector into the forward computation of each transformer layer in the form of residuals. Specifically, the MHA part of the context vector is first linearly combined with the query’s MHA output and added to the residual stream. The intermediate result then undergoes a nonlinear transformation. Finally, the MLP part of the context vector is linearly combined with the transformed result and added to the residual stream again to obtain the final output of the current layer. It is worth mentioning that the linear combination coefficients of the MHA and MLP parts are learnable parameters, allowing the model to adaptively adjust the injection strength of contextual information at different layers. These parameters are learned through a “self-calibration” process on the demonstration examples while introducing slight Gaussian noise to improve robustness.

Experimental Results and Significance

Experiments show that I2CL has achieved remarkable results on standard text classification tasks. In the classic few-shot settings (e.g., 1-shot, 5-shot), I2CL significantly outperforms traditional ICL methods and zero-shot baselines. Impressively, I2CL’s performance in extremely low-resource scenarios (e.g., 10-shot) already surpasses the level of ICL. This powerfully demonstrates I2CL’s enormous potential in few-shot learning. Through context vectorization, I2CL can maximize the extraction of rich semantic information contained in very few samples. The soft fusion mechanism of context injection allows this information to be flexibly embedded into the inference process, achieving stronger generalization and task adaptation capabilities.

The significance of I2CL goes beyond achieving new performance levels in few-shot learning tasks. More importantly, it opens up new possibilities for the application of pre-trained language models. In practice, we often face the challenge of scarce labeled data, especially for customized tasks in vertical domains. Thanks to I2CL’s outstanding few-shot learning capabilities, we only need to prepare a small number of high-quality demonstration examples to quickly adapt pre-trained models and build powerful natural language processing systems without large-scale manual annotation and model fine-tuning. This will greatly reduce development costs and shorten application cycles.

Practical Application: Intent Recognition in Customer Service

Let’s take intent recognition in an intelligent customer service system as an example to illustrate in detail how to use I2CL to construct a high-quality prompt.

Suppose our customer service system needs to handle various user inquiries about an e-commerce platform, including order queries, logistics tracking, returns and exchanges, etc. Traditional intent recognition methods require a large amount of labeled data covering various ways of asking for each intent. However, collecting and annotating so much training data is very time-consuming and labor-intensive.

Now, using I2CL, we only need a few examples to build a powerful intent recognition prompt. The specific steps are as follows:

1. Define intent types and examples

First, we define several main intent types and prepare 1-3 typical ways of asking for each intent as demonstrations. For example:

Order query intent: “When will the order I placed yesterday be shipped?” “How do I view my order details?”

Logistics tracking intent: “Where is my package? Why haven’t I received it yet?” “Can you tell me the tracking number? I want to check the delivery progress.”

Return and exchange intent: “I received a product with quality issues, how do I apply for a return?” “Can I still exchange after seven days?”

Account issue intent: “I forgot my login password, how do I reset it?” “How do I change the phone number bound to my account?”

2. Generate context vector

We input these intent examples into a pre-trained language model (such as GPT-3), extract the activation state at the last position of the model for each example, and then aggregate them to obtain a unified context vector. This vector contains the representation information of different intents in the semantic space.

3. Inject context vector

On a small number of examples, we fine-tune a set of scalar weights used to control the injection strength of the context vector at different layers of the model. The fine-tuning process appropriately adds noise to improve generalization. Now, we have obtained an intent recognition prompt that contains rich contextual information.

4. Zero-shot intent inference

For new user inquiries, we input them into the above prompt. The pre-trained model will directly predict the intent label of the inquiry under context enhancement, without requiring additional training data.

Here’s an Actual Inference Example. Suppose the user inputs a new inquiry: “The computer I bought last week is broken, can I get a new one?”

The intent recognition prompt will append contextual information, as shown in the image:

Putting the user’s question into the prompt, the language model will comprehensively consider the example context and output “Return and exchange intent” as the prediction result.

We can see that I2CL has built a powerful intent recognizer using only a few examples. The key points are:

1. The examples cover the main intent types and contain rich semantic information;
2. The context vector refines and fuses this information;
3. The scalar weights control the injection strength of contextual information at different layers of the model;
4. Injecting noise improves the generalization ability of the prompt.

When a new question arrives, the pre-trained model can directly make an accurate judgment of its intent under context enhancement, without retraining. Compared with traditional supervised learning, I2CL greatly reduces the dependence of intent recognition tasks on labeled data, providing a flexible and efficient new approach for quickly building intelligent customer service systems.

Implementing I2CL with DSPy

You can also implement the above intent recognition using DSPy:

# Define the DSPy implementation of the I2CL model

class GenerateIntentLabel(dspy.Signature):
    """Generate intent label from a given query."""
    context = dspy.InputField(desc="Demonstrations of intents and examples.")
    query = dspy.InputField(desc="The user query to classify.")
    intent = dspy.OutputField(desc="The predicted intent label.")

class I2CL(dspy.Module):
    def __init__(self):
        super().__init__()
        self.generate_intent_label = dspy.ChainOfThought(GenerateIntentLabel)

    def forward(self, query):
        context = self.retrieve_demonstrations()
        prediction = self.generate_intent_label(context=context, query=query)
        return dspy.Prediction(context=context, intent=prediction.intent)

    def retrieve_demonstrations(self):
        return demonstrations

    def generate_intent_label(self, context, query):
        prompt = f"{context}nUser query: {query}nIntent category: "
        response = openai.ChatCompletion.create(model="deepseek-chat")
        return response.choices[0].message.content

Conclusion

Implicit In-Context Learning (I2CL) is a groundbreaking paradigm that enables large language models to understand and utilize implicit contextual information with very few examples. By vectorizing demonstration examples and injecting the context vector into the model’s forward computation, I2CL achieves performance close to few-shot learning while maintaining the speed of zero-shot inference. This opens up new possibilities for quickly adapting pre-trained models to various natural language processing tasks with minimal labeled data.

The practical application of I2CL in intent recognition for customer service systems demonstrates its potential to greatly reduce development costs and application cycles. As research in this area continues to advance, we can expect I2CL and related techniques to play an increasingly important role in making large language models more efficient, flexible, and accessible for real-world applications.