Working with LLMs & Prompting
The quality of what you get out depends on what you put in.
An LLM is only as useful as the instructions you give it. Prompting is the skill of communicating clearly with a model so it returns accurate, useful, and safe results. For DevOps, that means turning it into a reliable assistant for logs, configs, and troubleshooting.
🎯 Learning Objectives
By the end of this lesson you will:
- Write clear, effective prompts
- Provide the right context to ground a model
- Request structured, parseable output
- Recognise and mitigate hallucination
The Anatomy of a Good Prompt
Strong prompts usually have four parts:
| Part | Purpose | Example |
|---|---|---|
| Role | Who the model should act as | “You are a senior SRE.” |
| Task | What to do | “Explain this error and suggest a fix.” |
| Context | The data to work from | The actual log lines |
| Format | How to respond | “Reply as: Cause / Fix / Command.” |
💡 Be specific, not polite
Models don’t need pleasantries — they need clarity. “Summarise the top 3 errors in these logs as a bulleted list” beats “Could you please take a look at these logs?” every time.
Context Is Everything
A model can’t know your systems unless you tell it. Compare:
# Weak — no context
"Why is my pod failing?"
# Strong — grounded in real data
"You are a Kubernetes expert. Given this output, identify the
cause and the fix:
$ kubectl describe pod web-xyz
State: Waiting
Reason: ImagePullBackOff
Message: manifest for myapp:1.4 not found"The second prompt gives the model exactly what it needs to reason about your problem — not a generic one.
Ask for Structured Output
When another program will consume the answer, ask for a structured format like JSON:
Prompt: "Classify this alert. Respond ONLY as JSON:
{ severity: 'low|medium|high', service: string, action: string }"
Response:
{ "severity": "high", "service": "checkout", "action": "restart pods" }💡 Structured output enables automation
When you constrain the model to a schema, its output becomes machine-readable — the bridge between an LLM and an automated pipeline or agent tool.
Few-Shot Examples
Showing a couple of examples (“few-shot”) steers the model far better than describing the task in words alone:
Convert log lines to severity labels.
"connection timeout after 30s" → WARN
"disk usage at 95%" → CRITICAL
"user logged in successfully" → INFO
"failed to bind to port 8080" → ???The examples teach the pattern; the model completes the last line consistently.
Hallucination — the Big Risk
LLMs can state wrong things confidently. They may invent a command flag, a config option, or an API that doesn’t exist.
⚠ Never run AI output blindly
Treat model output as a knowledgeable suggestion, not gospel. Always verify commands, config, and especially anything destructive before running it — in DevOps, a hallucinated --force flag can cause real damage. Ground the model in real docs and data, and keep a human in the loop for risky actions.
Ways to reduce hallucination:
- Provide context — give it the real docs/logs so it doesn’t guess.
- Ask it to cite — “quote the exact log line that supports your conclusion.”
- Request uncertainty — “say ‘unsure’ if the data doesn’t show it.”
- Verify — check commands against real documentation before executing.
🧪 Hands-on Lab
Turn a Vague Prompt into a Strong One
- Start with a weak prompt: “fix my deployment”
- Rewrite it with role, task, context (paste real error output), and output format
- Add a constraint that reduces hallucination
🧠 Knowledge Check
What is the most reliable way to get a model to reason about YOUR specific problem?
What is 'hallucination' in the context of LLMs?
💼 Interview Preparation
How would you safely use an LLM to help operate production systems?
Summary
You can now write grounded, structured prompts, use few-shot examples, and guard against hallucination. But prompting alone doesn’t let a model act — it needs tools. Next, you’ll learn how the Model Context Protocol connects AI to real systems.