Automated Red Teaming ; OpenAI’s Novel Methods for LLM Attack Simulation

Novel methods for automated red teaming of large language models (LLMs) is proposed by OpenAI recently. The approach is to factorize the red-teaming task into generating diverse attack goals and then training a reinforcement learning (RL) attacker to achieve those goals effectively and diversely. Pretty interesting, contributions include using automatically generated rule-based rewards and a multi-step RL process that encourages stylistic diversity in attacks.

The methods are applied to two tasks: indirect prompt injection and safety “jailbreaking,” that shows improved diversity and effectiveness compared to prior approaches.

Paper