AIアプリ開発12日目

Difyで開発することになりそうです。フロントはLINEと決めているので、そことの相性も加味して。行けるところまで、Difyで運用するつもりです。最初から高度なシステムを構築するもの無理ですし、Difyの進化も日進月歩なので、不満を感じた頃に、アプリの課題を解決する機能が加わる可能性もあります。とりあえず、顧客の質問に、顧客情報、顧客のラグ、必要なナレッジを付加してLLMに送り、ナレッジのデータに基づいて回答を生成するシステムを構築します。ただ、顧客のログが増え続けると、1回の質問に要するDify,LLM,AWSの従量支払いが嵩むという問題が発生します。プロンプトが許容量を超えないようにする解決策を見つけないといけません。回答に問題がなければ顧客ログを要約して情報量を減らす、格安トークンのLLMを使うなどで対応するか..。

Josh：
It's fascinating how you're leveraging Dify's capabilities, reminiscent of early AI chatbot experiments like ELIZA, which relied on pattern matching to generate responses. However, the challenge of latency scaling with increasing customer interaction is a hurdle many modern conversational AI systems face. Given your focus on minimizing token usage, have you considered incorporating techniques like prompt engineering and response summarization to optimize the LLM's input and output within the constraints of Dify's platform?

Jun：
Thank you for your great advice.As solutions to the issue of increasing token consumption to Dify and LLM, we are considering- summarizing prompts to send only necessary information- sorting out past lags and including only relevant informationI think the most important point is to conduct sufficient verification experiments to see if we can return knowledge-based answers that satisfy customers while keeping the system light.I will move forward first, solve the problems I face there with intuition and trial and error, and reach my goal.

Josh：
It sounds like you have a really solid plan in place! Summarizing prompts and focusing on relevant information are definitely key strategies for managing token consumption. I agree that thorough experimentation is crucial to ensure you're striking the right balance between providing comprehensive answers and keeping the system efficient. It's exciting to see you taking a hands-on approach to problem-solving that intuition and trial-and-error method can often lead to the most innovative solutions. Thanks for sharing your thoughts!

Jun：
I asked my engineer brother-in-law and he gave me a good solution.1. Store the prompts and answers sent from Dify to LLM in AWS in a distributed manner2. Include AWS logs in the prompts for customer questions3. If the log is short, include the whole log, if it is long, include a summary4. Adjust the summary ratio based on the length of time5. Adjust the summary ratio to be higher for highly relevant logs6. Analyze customer request patterns, identify frequently asked questions, and prepare standard phrases7. Refuse non-legitimate questions at the first branchWith these ideas, we can provide highly satisfying services to customers who use the service for a long time and frequently, without paying excessive usage fees to Dify, LLM, and AWS.

Josh：
It seems like a pragmatic approach to managing costs while maintaining service quality. The key will be in refining those summary ratios and identifying truly relevant logs. After all, context is paramount when dealing with complex inquiries. Building a robust system for recognizing illegitimate requests is also crucial; it prevents the system from being exploited. Ultimately, striking that balance between efficiency and user satisfaction is what separates a good solution from a great one.