Building an AI Chatbot with OpenAI API
A working path from a 20-line chat loop to a streaming, budget-aware chatbot endpoint: message history, token trimming, retries, moderation and a FastAPI route.

A colleague asked me for an FAQ bot for an internal tool, "just a quick one". My first version resent the entire conversation on every turn, never trimmed anything and waited for the full answer before showing a character. It was fine in a demo. Within a week someone pasted a long document into the chat, the request failed with a context-length error, and people complained that the bot "freezes" for several seconds before answering.
Those three problems (history growth, hard limits and perceived latency) are what separate a demo chatbot from one people can use. This post builds the practical version step by step with the OpenAI Python SDK. Memory beyond a single session, tool calling and retrieval each have their own deeper posts, linked below, so here I stay with the core loop.
Quick takeaways
- A chatbot is a loop that sends a list of messages and appends the reply. The model remembers nothing between calls; your code owns the history.
- Trim history to a token budget on every turn, keeping the system prompt and the most recent messages.
- Stream tokens to the client. Total time barely changes, but the answer starts appearing almost immediately.
- Let the SDK retry transient failures, set a timeout, and run user input through the moderation endpoint.
- Read the model name from configuration. Models are retired and replaced; hard-coded names turn into outages.
How do I build the smallest working chatbot?
Install the SDK and set your key as an environment variable (never commit it):
pip install openai tiktoken fastapi uvicorn
export OPENAI_API_KEY="..."
The core loop is short. It keeps a messages list, sends it, prints the streamed reply and stores the answer so the next turn has context:
import os
from openai import OpenAI
MODEL = os.environ.get("CHAT_MODEL", "gpt-4o-mini") # choose a current model for your account
client = OpenAI(max_retries=3, timeout=30.0)
SYSTEM = "You are a concise support assistant for our internal developer portal. If you are unsure, say so."
messages = [{"role": "system", "content": SYSTEM}]
while True:
user_input = input("you> ").strip()
if not user_input:
break
messages.append({"role": "user", "content": user_input})
stream = client.chat.completions.create(model=MODEL, messages=messages, stream=True)
reply = ""
for chunk in stream:
delta = chunk.choices[0].delta.content if chunk.choices else None
if delta:
print(delta, end="", flush=True)
reply += delta
print()
messages.append({"role": "assistant", "content": reply})
max_retries and timeout on the client cover the boring failures: connection resets, 429 rate limits and 5xx responses are retried with backoff. The system message sets the role and the "say you are unsure" rule, which does more for trust than any clever prompt. Prompts that need careful engineering are covered in Prompt Engineering Patterns That Improve LLM Output Quality.
How do I stop the conversation from exceeding the context window?
The history grows with every turn, and so does your bill. A simple, predictable policy is a sliding window: always keep the system message, then keep as many of the newest messages as fit into a token budget.
import tiktoken
ENCODING = tiktoken.get_encoding("o200k_base") # tokenizer used by recent OpenAI chat models
def count_tokens(message: dict) -> int:
# Approximate: content tokens plus a few tokens of per-message overhead.
return len(ENCODING.encode(message["content"])) + 4
def trim_history(messages: list[dict], budget: int = 6000) -> list[dict]:
system, rest = messages[0], messages[1:]
kept: list[dict] = []
used = count_tokens(system)
for message in reversed(rest):
cost = count_tokens(message)
if used + cost > budget:
break
kept.append(message)
used += cost
return [system, *reversed(kept)]
Call trim_history(messages) right before each request. Two details are easy to miss. Count the budget for the model's reply as well as the input, and reject or summarise a single user message that is larger than the whole budget instead of silently dropping it. When older context matters (preferences, earlier decisions), a sliding window is not enough; the options for summaries and long-term storage are in Building Conversational AI Memory.
How do I expose the chatbot over HTTP with streaming?
For a web UI, stream with server-sent events. FastAPI runs a synchronous generator in a thread pool, so the plain SDK client works without extra async code:
from fastapi import FastAPI, HTTPException
from fastapi.responses import StreamingResponse
from pydantic import BaseModel
app = FastAPI()
class ChatRequest(BaseModel):
messages: list[dict] # the client sends the history; validate roles in real code
@app.post("/chat")
def chat(request: ChatRequest):
last_user = request.messages[-1]["content"]
moderation = client.moderations.create(model="omni-moderation-latest", input=last_user)
if moderation.results[0].flagged:
raise HTTPException(status_code=400, detail="Message rejected by content policy")
history = trim_history([{"role": "system", "content": SYSTEM}, *request.messages])
def event_stream():
stream = client.chat.completions.create(model=MODEL, messages=history, stream=True)
for chunk in stream:
if chunk.choices and chunk.choices[0].delta.content:
yield f"data: {chunk.choices[0].delta.content}\n\n"
yield "data: [DONE]\n\n"
return StreamingResponse(event_stream(), media_type="text/event-stream")
Run it with uvicorn app:app --reload. In production, add authentication, per-user rate limits (cost control is a security feature), a cap on message length, and server-side validation of the roles in messages so clients cannot inject their own system prompt. Log conversations only as far as your privacy policy allows, with identifiers removed.
What should I add after the basics work?
Add things when a real user need appears, not before. The usual order in my projects:
- Your own knowledge. Answers that must come from your documents need retrieval; start with chunking strategies for RAG.
- Actions. If the bot must look up an order or open a ticket, use tool calling, with validation and retry limits, as described in Engineering Reliable AI Agents.
- Quality tracking. Keep a small set of real questions with expected answers and re-run it before every prompt or model change.
What should I do this week?
Run the loop above against your own use case and ask it ten questions you already know the answers to. Write down the failures, because they become your first test set. Then add the token-trimming function and the streaming endpoint, in that order, and ship it to five colleagues before you build anything fancier.
Uğur Kaval
AI/ML Engineer & Full Stack Developer specializing in building innovative solutions with modern technologies. Passionate about automation, machine learning, and web development.


