Building an AI Agent from Scratch on AWS: Key Learnings with a Dash of Real Talk

Let's be honest—building an AI agent isn’t all sunshine and rainbows. Sure, there’s a lot of hype around AI these days, but actually getting an agent to do something useful is a different story. My journey of creating an AI agent using AWS Lambda, DynamoDB, Bedrock, and S3 was full of surprising challenges and a few “Aha!” moments. Spoiler alert: It's not just about calling an LLM and waiting for magic. Here’s what I learned along the way.

1. Model How Your Agent Will Work, Like a “Real Agent”

One surefire way to sabotage your AI project? Try to make your agent do everything . Seriously, if you start by imagining an all-knowing, all-doing super-agent, you’re setting yourself up for disappointment. The biggest trap in agent-building is boiling the ocean—trying to tackle every task under the sun, only to end up with a confused, underwhelming mess. Instead, think of your AI as a “human agent” tasked with a specific mission. What would a real human do to achieve that goal in a professional and effective manner? When I set out to build a parental coach agent, I zeroed in on the one thing that mattered most: the communication model with the parent. Not just any communication, but one that was as effective and repetitive as possible.

Now, if you have a year (or a magical time-turner), you could make that communication model as sophisticated as you like. You could let your agent “think,” “reason,” and “decide” on the best interaction style for each conversation. Sounds impressive, right? It’s also the quickest way to get lost in complexity and waste precious time. Here’s the kicker: making the communication model deterministic—predictable and consistent—isn’t anti-agent. In fact, it’s the smartest move you can make. It frees up your resources to focus on what really matters: effectively coaching the parent. Just because the communication model is deterministic doesn’t mean the agent itself is static or boring. Quite the opposite.

I found it downright fascinating to watch my agent reason and analyze each situation. It decided which questions to include in the next cycle to gather the information it needed—strategically and thoughtfully. The “deterministic” model gave it just enough structure to be effective, without turning it into a predictable robot. It was proof that sometimes, the smartest systems are the ones that don’t try to do everything

2. Prompting Strategy Matters (More Than You Think)

I started off thinking I could just tell the LLM what I wanted, and voilà—perfect results would pop out. Yeah, no. Directly asking for results often led to responses that were either too vague or just plain wrong. It was like giving someone a map without directions. I soon realized the key was to break down the problem the way a human would tackle it. For example, instead of saying, “Hey LLM, generate a plan,” I had to nudge it through a thought process:

●      Analyze the Input: What’s the most important information here? ●      Reprioritize Goals: Based on that, should any goals change or new ones be added? ●      Propose an Action Plan: Now, how do we get from here to success? This layered approach turned out to be a game-changer. It gave the LLM enough context to provide thoughtful, structured answers. The moral of the story? LLMs aren’t mind-readers—they need a breadcrumb trail to follow.

3. LLM Inference Takes Time (And That’s a Problem)

If you think cloud infrastructure is all about speed and scalability, wait till you run a large language model on it. LLMs take their sweet time, especially when you’re expecting a nuanced response. And guess what? AWS API Gateway has a 30-second timeout. Not nearly enough. To get around this, I had to split my Lambda function into two parts using AWS EventBridge. The first Lambda just stored data in DynamoDB and fired off an event. The heavy lifting—inference and processing—was done asynchronously in the second Lambda. It’s like saying, “I’ll get back to you,” and actually doing it. Problem solved.

4. Segmentation of LLM Usage: One Model Doesn’t Fit All

You’d think one powerful LLM could handle everything. I thought so too—until I realized how inefficient that was. Even though I used Llama 3 on Bedrock, the tasks varied wildly: ●      Big LLM (50Bn+ parameters) for deep reasoning and planning. ●      Coding LLM for generating HTML templates. Pro tip: Start with highly effective model like ChatGPT or Clause to generate base templates. Once you have very good templates, you can provide them as directions or hints to lower performing LLMs and then they will give you excellent results.

●      Lite LLM for quick summaries and email / messaging generation. Using the right model for the right job saved on cost and improved performance. It’s like using a sports car for racing, a pickup truck for hauling, and a scooter for quick errands—each serves its purpose perfectly.

5. Goal Management is Crucial (Or Your Agent Will Wander Aimlessly)

Here’s a rookie mistake: I assumed the agent could handle goals implicitly, like some sort of magical to-do list in its memory. Wrong. Agents need explicit goal tracking to stay focused. This means: ●      Maintaining a dynamic list of goals. ●      Reprioritizing goals as progress is made. ●      Aligning short-term tasks with the agent’s ultimate mission. Once I implemented this, my agent went from wandering aimlessly to executing tasks with purpose. Who knew agents needed project management too?

6. Measuring Progress is Non-Negotiable

It’s not enough to give your agent a list of goals—you need to know how well it’s performing. This meant tracking:

●      Objectives completed vs. pending tasks. ●      Decision-making success rates. ●      Real-time adjustments to strategy. I incorporated logging, monitoring, and analytics to get a clear view of the agent’s performance. Think of it as a fitness tracker, but for AI productivity.

7. Modeling the AI Agent

This is where all the pieces come together. Designing the AI agent wasn’t just about getting it to work—it was about getting it to work intelligently. Here’s how I structured the agent, breaking down each core concept and its role in the overall system. 7.1. Overview of the Agent Architecture At the core of this AI agent were four interconnected components that worked in harmony to deliver intelligent, dynamic responses: LLMs, Memory & Goals, Triggers, and Tools. Here’s how each part played its role.

How I modeled my AI Agent 7.2. Triggers

The agent was event-driven, reacting to specific user actions

●      New Intake Form Submitted: This signaled the arrival of a new user, initiating a personalized journey.

●      New Questionnaire Form Submitted: This updated the agent on user progress, prompting a reassessment of goals and next steps. ●      New Plan Form Submitted: This indicated that the user had executed a plan, allowing the agent to measure success and adjust future strategies. 7.3. LLMs These were the brains of the operation, handling everything from high-level reasoning to detailed content generation. I used a mix of models tailored to specific tasks:

●      Big LLM for strategic thinking, planning, and complex decision-making. This was the powerhouse, reserved for high-stakes reasoning that required deep context understanding. ●      Coding LLM for generating HTML forms. It used predefined templates to maintain consistency and efficiency, allowing rapid iteration without starting from scratch each time. ●      Lite LLM for summaries and communication, including drafting emails and generating concise reports. Lightweight and fast, it was perfect for routine tasks.

In my case, I could segment my LLMs as above 7.4. Memory Management

The agent wasn’t just reacting; it was learning and evolving. This required a robust memory architecture:

●      Goal Memory kept track of objectives, ensuring the agent was always working towards the right targets. ●      History Memory stored past interactions and decisions, providing context for future actions and avoiding repetitive conversations. ●      Working Memory maintained session-specific data, like intake forms and ongoing conversations, allowing the agent to provide contextually relevant responses. Although this was all DynamoDB, I modeled my agent across these 3 types of memories

7.5. Tools and Actions

These were the agent’s way of interacting with the outside world

●      Send Email and Send Text Messages for communication.

●      Publish Form to S3 for dynamic form generation and storage. ●      Store and Load from Memory to maintain state and context across sessions. Agents need to act, interact, and communicate

8. Building AI Agents is Harder Than It Seems (And That’s Okay)

Let’s clear up a myth: AI agents are not just simple scripts calling LLM APIs. They require:

●      Sophisticated orchestration of multiple models. ●      State management that doesn’t get tangled up. ●      Goal prioritization and progress tracking. ●      Resource optimization for cost-effective performance. Creating a truly effective AI agent involves building a system that delivers real value—not just a chatbot on steroids. And that, my friends, is where the real magic happens. Final Thoughts Building an AI agent from scratch isn’t easy, but it’s incredibly rewarding. From cracking the code on prompt engineering to mastering state management and memory design, each lesson has shaped how I approach AI development. And while AI agents are still evolving, the effort is worth it when they deliver real-world value and efficiency.

I’ve shared my journey—the good, the bad, and the ugly. If you’re building an AI agent, I hope these insights save you some headaches and help you create something truly impactful.