Evaluación de modelos LLMs de código abierto

Abstract

Open-source large language models have proven to be useful tools for code generation; however, there is a scarcity of evaluations comparing their performance on programming problems with different levels of complexity. This study evaluates four open-source LLMs: Mistral 7B, Phi-3 Mini, Qwen2.5 7B, and DeepSeek-Coder 1.3B in generating solutions for Python programming problems, using nine exercises organized into three difficulty levels: easy, intermediate, and advanced. The models were selected for their availability under open-source licenses that permit academic use and free modification. The evaluation is conducted through a dual approach that includes automated analysis with DeepSeek Chat and human evaluation by an independent evaluator. Standardized criteria for functionality, code clarity, and error handling are applied, providing quantitative scores. The results show significant variations in model performance depending on problem complexity and their specialization, which is relevant for model selection in educational and software development contexts