For a model to succeed in the context task, it must execute these steps:
- Extract the answer from model’s prior knowledge
- Extract the answer from the context
- Decide whether to answer according to the context or prior knowledge
To follow either context or prior knowledge, they fine-tuned the model + few shot.
How about the truthfulness direction? Can we combine it here? Maybe if the truthfulness direction is high, we go for the prior knowledge?