Research Information System
4th International Conference on Trends in Advanced Researc, Konya, Turkey, 4 - 05 July 2025, pp.126-135, (Full Text)
Recent advancements in Artificial Intelligence (AI), particularly the development of Large Language Models (LLMs), have significantly transformed the field. Among these, ChatGPT, developed by OpenAI, has drawn considerable attention for its capabilities in information retrieval, content generation, and complex problem solving, with promising applications across diverse disciplines. This study aims to rigorously assess the performance of ChatGPT, one of the leading LLMs, in answering undergraduate-level civil engineering exam questions, a task that demands domain-specific knowledge and technical expertise. In the study, construction materials, one of the core subfields of civil engineering, were selected as the focus area. Exam questions with varying levels of difficulty and formats were used to evaluate ChatGPT's performance. The model's responses were assessed by an expert academic based on criteria such as technical accuracy, correctness, and clarity. For responses classified as entirely incorrect, the evaluation was repeated using a predefined six-point scale, considering prompt engineering techniques known to play a critical role in model performance. Role assignment, self-consistency, and tree of thought techniques were applied for this purpose. The findings indicate that prompt engineering significantly enhances model performance. According to expert evaluations based on the six-point scale used in the study, the average score increased from 1.78 with the original prompt to 2.28 with role prompting, 2.95 with self-consistency, and 3.00 with the tree of thought technique. These results clearly demonstrate that the tree of thought approach is highly effective in improving the model’s reasoning ability and producing more consistent and accurate responses by promoting step-by-step problem solving in complex tasks. This study highlights both the potential benefits and current limitations of large language models such as ChatGPT in educational and professional assessment processes. The findings aim to inform future research and discussions on the integration of artificial intelligence into civil engineering education and practice.