Fine Tuning Language Models with Self-Generated Instructions

Why do Language Models need fine tuning?

Large scale Language Models like GPT-3 have shown to perform few-shot learning remarkably well, but are less successful at zero-shot learning for a reason that, it is harder for models to perform well on prompts that are not similar to that of the training data. Hence, Language Models are further fine-tuned on databases that contain natural language instructions.

Large “instruction-tuned” language models (fine-tuned to respond to instructions) have demonstrated a remarkable ability to generalize zero-shot to new tasks.

For example, Open-AI’s Instruct GPT uses a similar method to improve the performance of the GPT-3 Language Model, making it better at understanding human requests.

Why are Self-Generated Instructions preferred over natural language instructions to fine-tune Language Models?

Fine tuning language models with natural language instructions work really well but has a major flaw – they require large amounts of human instructions to train. Such datasets are limited in quantity and narrow in scope, with most containing similar instructions. Hence, SELF-INSTRUCT, a framework for improving the instruction-following capabilities of Pre-trained Language Models, is a way to generate instructions, input, and output samples using a Language Model, then prune them, before using them to fine-tune the original model, to apply for down-stream applications.

How to fine-tune Language Models with Self-Generated Instructions?

General process of fine-tuning a language model using self-instruction is as follows:

1. A small initial dataset, containing a few sample instructions, inputs and outputs, to be created.

   Example of the instructions:
   Instruction: “A colour description has been provided. Find the CSS code of the colour.”
   Input: “A light red colour with medium light shade of pink”
   Output: “#DB7093”

2. The language model to be prompted to generate a new instruction. Based on the new instruction, the model generates appropriate inputs and outputs. To ensure diversity of instructions, the SELF-INSTRUCT pipeline removes new instructions that are similar to existing instructions. This process to be run multiple times till the desired number of instructions and input/output samples are generated.
3. Finally, the language model should be finetuned using these self-generated instructions to suit to down-stream applications.

Additionally, combining manual instructions with self-instructions could help improve the performance of language models even further.

Applications of Fine-tuned Language Models with self-generated instructions

• Fine-tuning language models is essential for any task that requires natural language understanding.
• Self-Instruction does not require a large number of instructions to train on, hence it can be useful in scenarios where sample instructions cannot be found or limited.
• Some significant applications of self-generation instructions are in natural language understanding of regional languages, chatbots in regional languages, language translations (especially between two relatively less spoken languages), text summarization for niche subjects etc.

References for further reading

1. Self-Instruct: Aligning Language Model with Self Generated Instructions
2. To implement click here.