COMPARATIVE ANALYSIS OF THE EFFECTIVENESS OF SYSTEM PROMPTS FOR LEGAL AGENTS BASED ON THE GPT-4O MINI LARGE LANGUAGE MODEL

Podoprigora Vladimir Nicolaevich, Sobolev Denis Vyacheslavovich, Papko Arseniy Maximovich, Alexandrov Danila Alexandrovich, Popov Dmitry Alexandrovich

This article presents the results of an experimental study aimed at optimizing system prompts to improve the quality of large language model responses in the field of legal consulting. Using the GPT-4o mini model as an example, the effectiveness of four prompt variants was analyzed, including universal and specialized ones (tailored to the Family Code of the Russian Federation), as well as their modifications. A test set of 40 questions on family law was used, and response evaluation was carried out by an automated system based on the older-generation DeepSeek v3 model. The results showed that the universal prompt (Agent No. 1) outperformed the specialized one (Agent No. 2), despite the null hypothesis to the contrary, and the modified Agent No. 3 with a requirement for brevity for closed-ended questions achieved the highest "score/cost" ratio (S/C = 72.8). Key problems were identified: dependence on the quality of the RAG service, tendency to hallucinate in the absence of relevant data, and decreased accuracy due to excessive explanations. An approach using an ensemble of junior-generation models to reduce costs without loss of quality is proposed. The study demonstrates the critical role of prompt structure in legal AI systems and opens up avenues for further optimization.

Keywords:system prompt, legal agent, GPT-4o mini, Family Code of the Russian Federation, RAG service, quality assessment, token consumption optimization, comparative analysis, legal consulting, generation cost