LangSmith

Here’s a concise introduction you can use to set the stage for your article on LangSmith:

LangSmith is a unified platform for building production-grade large language model (LLM) applications. It brings together observability, evaluation, and prompt-engineering workflows so teams can ship AI agents with confidence—whether they’re using LangChain or any other LLM framework.

At its core, LangSmith offers three pillars of functionality:

• Observability: Capture and inspect detailed traces of every LLM call, and configure metrics, dashboards, and alerts to catch issues in real time.
• Evals: Run automated and human-in-the-loop evaluations on production traffic to score performance, track regressions, and gather qualitative feedback.
• Prompt Engineering: Version and collaborate on prompts directly in the platform, so you can iterate quickly and roll back changes safely

Designed to be framework-agnostic, LangSmith integrates seamlessly with LangChain (or LangGraph) via a single environment variable, but can also work with any custom tooling or direct API calls you already have in place LangSmith.

This means teams can adopt end-to-end observability and testing without ripping out existing code—accelerating the journey from prototype to reliable, production-ready AI.

Here’s a rundown of the main JS/TS modules, classes, and functions you’ll find in the langsmith SDK, along with what each one does:

• Client The core class for interacting with the LangSmith REST API. Through its methods you can create and update runs, manage datasets & examples, submit feedback, run & log Evals, handle annotation queues, version prompts, and more—all via a single, unified client instance.
• traceable A function decorator / higher-order wrapper that instruments your entire function or pipeline. Any nested LLM calls or custom spans become children under a single trace in LangSmith, giving you end-to-end observability with zero manual span management.
• wrapOpenAI A plug-and-play wrapper for the OpenAI JS/TS client that automatically traces every LLM call you make—no boilerplate tracing code required.
• runTrees Utilities for extracting, serializing, and working with hierarchical “run trees” (i.e., nested traces). Useful if you need to post-process or analyze complex multi-step pipelines programmatically.
• evaluation namespace A collection of functions to define, execute, and list automated and human-in-the-loop evaluations (e.g. createEvaluation, runEvaluation, listEvaluations). Hooks directly into LangSmith’s Evals engine so you can score model outputs, detect regressions, and collect qualitative feedback.
• schemas TypeScript definitions and runtime validators for core LangSmith entities (Run, Dataset, Example, EvaluationResult, etc.), ensuring strong typing and correct structure when sending or receiving data.
• vercel (e.g. AISDKExporter / VercelExporter) OpenTelemetry exporters and helpers tailored for Vercel/Next.js serverless environments. Automatically capture and forward edge-function traces into LangSmith.
• singletons A tiny module to manage a global Client instance so you don’t re-initialize the client across multiple files or hot-reloads.
• utils (including anonymizer) Low-level helpers for payload sanitization, metadata extraction, and other common transformations before sending data to LangSmith.
• wrappers Additional client wrappers for other LLM providers (beyond OpenAI), enabling transparent tracing integration for whichever SDK you use.
• jest / vitest plugins Testing integrations that let you define LangSmith datasets and Evals as Jest or Vitest test cases—so your CI suite can automatically generate evaluation reports and feedback.

import { ChatOpenAI } from "@langchain/openai";
import { AgentExecutor, createOpenAIFunctionsAgent } from "langchain/agents";
import { Tool } from "langchain/tools";
import { ChatPromptTemplate, MessagesPlaceholder } from "@langchain/core/prompts";
import { LangChainTracer } from "langchain/callbacks";
import dotenv from "dotenv";
import readline from "readline";

dotenv.config();

// === LLM Model ===
const model = new ChatOpenAI({
  temperature: 0,
  modelName: "gpt-3.5-turbo",
  apiKey: process.env.OPENAI_API_KEY,
});

// === Tools ===
class CalculatorTool extends Tool {
  name = "calculator";
  description = "Useful for when you need to answer questions about math";

  async _call(input: string) {
    try {
      return eval(input).toString();
    } catch (error) {
      console.error(error);
      return "Error: " + error;
    }
  }
}

class TimerTool extends Tool {
  name = "timer";
  description = "Useful for when you need to track time";

  async _call(input: string) {
    return new Date().toLocaleTimeString();
  }
}

class WeatherTool extends Tool {
  name = "weather";
  description = "Fetches the current weather for a given city. Provide the city name as input.";

  async _call(city: string) {
    if (!city) return "Error: Please provide a city name.";

    try {
      const response = await fetch(`${process.env.WEATHER_API_URL}?q=${city}&appid=${process.env.WEATHER_API_KEY}&units=metric`);
      const data = await response.json();

      if (data.cod !== 200) return `Error: ${data.message}`;

      return `The weather in ${data.name} is ${data.weather[0].description} with a temperature of ${data.main.temp}°C.`;
    } catch (error) {
      return "Error fetching weather data.";
    }
  }
}

// === Main Run Function ===
async function run() {
  const tools = [new CalculatorTool(), new TimerTool(), new WeatherTool()];
  let chat_history: { role: string, content: string }[] = [];

  const prompt = ChatPromptTemplate.fromMessages([
    ["system", "You are a helpful AI assistant with access to tools. Follow these steps: " +
      "1. Think about the user's question " +
      "2. If a tool is needed, decide which one to use" +
      "3. Call the tool and observe its result " +
      "4. Respond to the user in a structured format " +
      "Do not respond until you have observed the tool's result."
    ],
    new MessagesPlaceholder("chat_history"),
    ["human", "{input}"],
    new MessagesPlaceholder("agent_scratchpad"),
  ]);

  const agent = await createOpenAIFunctionsAgent({
    llm: model,
    tools,
    prompt,
  });

  const executor = new AgentExecutor({
    agent,
    tools,
    callbacks: [new LangChainTracer()],
  });

  const rl = readline.createInterface({
    input: process.stdin,
    output: process.stdout,
  });

  const askQuestion = async () => {
    rl.question("You: ", async (input) => {
      const result = await executor.invoke(
        { input, chat_history, agent_scratchpad: [] },
        {
          runName: "CLI Agent Run",
          tags: ["cli", "langsmith", "tools"],
        }
      );

      console.log("🤖 Agent:", result.output);

      chat_history.push({ role: 'user', content: input });
      chat_history.push({ role: 'assistant', content: result.output });

      askQuestion();
    });
  };

  askQuestion();
}

run().catch(console.error);