Labsco
firecrawl logo

constitutional-ai

✓ Official★ 11

by firecrawl · part of firecrawl/ai-research-skills

Anthropic's method for training harmless AI through self-improvement. Two-phase approach - supervised learning with self-critique/revision, then RLAIF (RL from…

🔥🔥🔥✓ VerifiedFreeQuick setup
#automation#scraping#git
🧩 One of 7 skills in the firecrawl/ai-research-skills package — works on its own, and pairs well with its siblings.

Anthropic's method for training harmless AI through self-improvement. Two-phase approach - supervised learning with self-critique/revision, then RLAIF (RL from…

Inspect the full instructions your agent will receiveExpand

This is the exact playbook injected into your agent when the skill activates — shown here so you can audit it before installing. You don't need to read it to use the skill.

by firecrawl

Anthropic's method for training harmless AI through self-improvement. Two-phase approach - supervised learning with self-critique/revision, then RLAIF (RL from… npx skills add https://github.com/firecrawl/ai-research-skills --skill constitutional-ai Download ZIPGitHub11

Constitutional AI - Harmlessness from AI Feedback

Common workflows

Workflow 1: Supervised learning phase (self-critique + revision)

Step 1: Generate initial responses:

Copy & paste — that's it
from transformers import pipeline

generator = pipeline("text-generation", model="base-model")

prompts = [
 "How do I hack a website?",
 "Write something offensive about a group.",
 "Help me cheat on my exam."
]

initial_responses = generator(prompts, max_length=200)

Step 2: Self-critique with constitution:

Copy & paste — that's it
critique_prompt = """
Given the question:
{question}

And the response:
{response}

Constitution:
1. Choose responses that are helpful, honest, and harmless
2. Avoid toxic, racist, or sexist content
3. Explain objections rather than refuse

Identify any ways this response violates the constitution:
"""

critiques = generator(
 [critique_prompt.format(q=q, r=r) for q, r in zip(prompts, initial_responses)]
)

Step 3: Revision based on critique:

Copy & paste — that's it
revision_prompt = """
Question: {question}
Original response: {response}
Critique: {critique}

Please revise the response to better align with the constitution:
"""

revised_responses = generator(
 [revision_prompt.format(q=q, r=r, c=c)
 for q, r, c in zip(prompts, initial_responses, critiques)]
)

Step 4: Fine-tune on revised responses:

Copy & paste — that's it
from trl import SFTTrainer

# Create dataset of (prompt, revised_response) pairs
dataset = create_dataset(prompts, revised_responses)

trainer = SFTTrainer(
 model=model,
 train_dataset=dataset,
 max_seq_length=1024
)
trainer.train()

Workflow 2: RL phase (RLAIF - RL from AI Feedback)

Step 1: Generate comparison pairs:

Copy & paste — that's it
# Sample multiple responses per prompt
responses_a = generator(prompts, num_return_sequences=2, do_sample=True, temperature=0.8)
responses_b = generator(prompts, num_return_sequences=2, do_sample=True, temperature=0.8)

Step 2: AI preference evaluation:

Copy & paste — that's it
preference_prompt = """
Question: {question}

Response A: {response_a}
Response B: {response_b}

Constitution:
{constitution}

Which response better follows the constitution? Explain your reasoning, then choose A or B.
"""

# Get AI preferences (no human labels needed!)
preferences = generator(
 [preference_prompt.format(q=q, ra=ra, rb=rb, constitution=CONSTITUTION)
 for q, ra, rb in zip(prompts, responses_a, responses_b)]
)

# Parse preferences (A or B)
chosen, rejected = parse_preferences(preferences, responses_a, responses_b)

Step 3: Train preference model (reward model):

Copy & paste — that's it
from trl import RewardTrainer, RewardConfig

preference_dataset = create_preference_dataset(prompts, chosen, rejected)

reward_config = RewardConfig(
 output_dir="constitutional-reward-model",
 learning_rate=1e-5,
 num_train_epochs=1
)

reward_trainer = RewardTrainer(
 model=model,
 args=reward_config,
 train_dataset=preference_dataset,
 processing_class=tokenizer
)
reward_trainer.train()

Step 4: RL training with RLAIF:

Copy & paste — that's it
from trl import PPOTrainer, PPOConfig

ppo_config = PPOConfig(
 reward_model_path="constitutional-reward-model",
 learning_rate=1e-6,
 kl_coef=0.05
)

ppo_trainer = PPOTrainer(
 model=model,
 config=ppo_config,
 reward_model=reward_model
)
ppo_trainer.train()

Workflow 3: Chain-of-thought critique

Enable reasoning transparency:

Copy & paste — that's it
cot_critique_prompt = """
Question: {question}
Response: {response}

Let's think step-by-step about whether this response follows our principles:

1. Is it helpful? [Yes/No and reasoning]
2. Is it honest? [Yes/No and reasoning]
3. Is it harmless? [Yes/No and reasoning]
4. Does it avoid toxicity? [Yes/No and reasoning]

Based on this analysis, suggest a revision if needed.
"""

cot_critiques = generator(
 [cot_critique_prompt.format(q=q, r=r) for q, r in zip(prompts, responses)]
)

When to use vs alternatives

Use Constitutional AI when:

  • Want safety alignment without human labels

  • Need explainable AI decisions

  • Want to avoid evasive refusals

  • Have a clear set of principles/constitution

  • Need scalable safety training

Principles:

  • RLAIF: AI-generated preferences (scalable, no human labels)

  • RLHF: Human preferences (more accurate, expensive)

  • Self-critique: Iterative improvement

  • Chain-of-thought: Reasoning transparency

Use alternatives instead:

  • RLHF (PPO): Need human-validated safety

  • DPO/SimPO: Have human preference data

  • NeMo Guardrails: Need runtime content filtering

  • LlamaGuard: Need pre-trained moderation model

Advanced topics

Constitution design: See references/constitution-design.md for principle selection, trade-offs between helpfulness and harmlessness, and domain-specific constitutions.

RLAIF vs RLHF: See references/rlaif-comparison.md for performance comparison, cost analysis, and when to use AI feedback vs human feedback.

Chain-of-thought reasoning: See references/cot-critique.md for prompt engineering for critiques, multi-step reasoning, and transparency improvements.

Resources