Lessons from building GenAI based applications — Juan Peredo
Feb 22, 2025 · 33:13
Juan Peredo details the hidden complexities of building GenAI applications, from model hosting and cost control to output validation and observability. He compares local (Ollama) vs cloud hosting (Modal, SkyPilot) and warns that an agent processing 3,000 calls/day with OpenAI O1 costs nearly $300,000/month, while LLaMA 3.3 70B drops that to $50,000/month. He explains techniques to mitigate hallucinations—prompt engineering, guardrails, RAG, and fine-tuning—each with trade-offs like added latency or cost. Peredo advocates externalizing prompts via LangChain Hub for easy iteration and future-proofing, and illustrates agent design with parallel calls to reduce latency. Finally, he stresses observability using tools like LangSmith to debug probabilistic failures, such as an LLM failing on case sensitivity.
