Aug 2, 2023

Langfuse Update โ€” July 2023

Building out the core platform and preparing infrastructure for advanced analytics: Async SDKs, automated token counts, nested Trace UI, langchain integration, public GET API

Hi all, a lot has happened since our launch two weeks ago and I'm excited to share our July update:


The details ๐Ÿ‘‡

๐Ÿฆœ๐Ÿ”— Langchain Integration

For teams building their LLM app with Langchain, adopting Langfuse just got that much easier. We added a CallbackHandler to the Langfuse Python SDK that automatically traces your complex Langchain chains and agents. Simply pass it as a callback.

pip install langfuse
# Initialize Langfuse handler
from langfuse.callback import CallbackHandler
langfuse_handler = CallbackHandler(
    secret_key="sk-lf-...",
    public_key="pk-lf-...",
    host="https://cloud.langfuse.com", # ๐Ÿ‡ช๐Ÿ‡บ EU region
  # host="https://us.cloud.langfuse.com", # ๐Ÿ‡บ๐Ÿ‡ธ US region
)
 
# Your Langchain code
 
# Add Langfuse handler as callback (classic and LCEL)
chain.invoke({"input": "<user_input>"}, config={"callbacks": [langfuse_handler]})

Also works for run and predict methods.

chain.run(input="<user_input>", callbacks=[langfuse_handler])
conversation.predict(input="<user_input>", callbacks=[langfuse_handler])

โ†’ Integration docs

โšก๏ธ Improved SDKs for Python and JS/TS

Most Langfuse users integrate using our Python or JS/TS SDKs. We made improvements to both DX and performance:

  1. Fully async, avoid adding latency or blocking your app
  2. Typed (Pydantic/Typescript)
  3. Improved DX for nesting traces; no need to manually manage IDs
  4. Support for more runtimes: JS/TS SDKs work in Node.js, Edge (Deno, Vercel, Cloudflare) or in the browser to directly report scores (e.g. user feedback) to Langfuse

โ†’ SDK docs

โ›“๏ธ New Trace UI

Most users of Langfuse run complex LLM-based applications. To help monitor and debug these applications and understand how the different steps lead to the final output, we added a new nested trace UI.

Examples from our Langchain integration:

Debug view in Langfuse

๐Ÿงฎ Automated Token Counts

Usage/cost reporting is a key use case of Langfuse.

Until now, token counts needed to be ingested when logging new LLM calls. For OpenAI and Anthropic models, Langfuse now automatically calculates token counts based on the ingested prompts and completions. This is helpful as some APIs do not provide token counts in their responses. It also reduces integration effort.

ModelTokenizerUsed package
gpt*cl100k_basetiktoken (opens in a new tab)
claude*claude@anthropic-ai/tokenizer (opens in a new tab)

โ†’ Learn more

๐ŸŒŸ Human-in-the-loop Evaluation

Scores in Langfuse are essential to monitor the quality of your LLM app. Until now, scores were created via the Web SDK based on user feedback (e.g. thumbs up/down, implicit user feedback) or via the API (e.g. when running model-based evals).

Many of you wanted to manually score generations in the UI as you or your team browse production logs. We've added this to the Langfuse UI:

Add manual score in UI

โ†’ Learn more

๐Ÿ‘ฅ User Reporting

Add a userId when ingesting data into Langfuse and use it to get usage/cost broken down by user. Explore what individual users are doing in your LLM app.

More user-based analytics and features are coming soon, e.g. APIs to use this data in your own dashboards and usage-based billing.

User reporting

โ†’ Learn more

๐Ÿ“ˆ Analytics (alpha)

Currently we are working with a small group of alpha users to test our analytics features. We've focused a lot on getting this right and are excited to share more soon. Reach out if you want to share your needs or want to be part of the alpha. We are onboarding new users every week.

โ†’ More details

๐Ÿ‘ฉโ€๐Ÿ’ป Public GET API

We added a public GET API to Langfuse to support users consuming Langfuse data in their own applications. Check out the API reference for more details on the available endpoints. Need other endpoints? Shoot me a message: marc@langfuse.com

Example use cases:

  • Integrate user feedback (by use case) into your own dashboards
  • Fine-tune on low-quality generations
GET /traces
GET /traces/:traceId
GET /scores
GET /observations/:observationId

โ†’ API reference

๐Ÿ“š Improved Docs

We've shipped a lot over the last month and the docs were lagging behind. We've now updated the docs to reflect the latest changes and added a bunch of new content. This should make it way easier to get started with Langfuse and incrementally adopt more advanced features. We'd love to get your feedback on the docs, use the new widget on the bottom to ๐Ÿ‘/๐Ÿ‘Ž, thank you ๐Ÿ™

We also added a Q&A chatbot as one user was surprised that we don't have one yet. It's still a bit rough around the edges but we'll improve it over time. Give feedback when using it and you can be sure that it is all tracked and analyzed in Langfuse.

New docs

๐Ÿšข What's Next?

There is more coming in August. Stay tuned! We'll focus on shipping analytics to all users and further improvements to the UI/DX of the core platform. Anything you'd like to see? Join us on Discord and share your thoughts.

Subscribe to get monthly updates via email:

Follow along on Twitter (@Langfuse (opens in a new tab), @marcklingen (opens in a new tab))

Was this page useful?

Questions? We're here to help

Subscribe to updates