Abstract
It is a long-standing hypothesis that the concise and customized notation of a DSL improves the performance of developers when compared with a GPL. For non-technical domains—e.g., science—, this hypothesis lacks empirical evidence. Given this lack of empirical evidence, we evaluate a DSL for ecological modeling designed and implemented by us with regard to performance improvements of developers as compared to a GPL. We conduct an online survey with embedded controlled experiments among ecologists to assess the correctness and time spent of the participants when using a DSL for ecosystem simulation specifications compared with a GPL-based solution. We observe that (1) solving tasks with the DSL, the participants’ correctness point score was —depending on the task— on average 61 % up to 63 % higher than with the GPL-based solution and their average time spent per task was reduced by 31 % up to 56 %; (2) the participants subjectively find it easier to work with the DSL, and (3) more than 90 % of the subjects are able to carry out basic maintenance tasks concerning the infrastructure of the DSL used in our treatment, which is based on another internal DSL embedded into Java. The tasks of our experiments are simplified and our web-based editor components do not offer full IDE-support. Our findings indicate that the development of further DSL for the specific needs of the ecological modeling community should be a worthwhile investment to increase its members’ productivity and to enhance the reliability of their scientific results.
Similar content being viewed by others
Notes
For this purpose, we use CodeMirror (http://codemirror.net), for which we implemented a Sprat Ecosystem DSL syntax coloring plug-in.
These tests are all formulated to test against normality (the alternative hypothesis is that the sample is not normal). This means that the test, strictly speaking, can never show that a sample is likely to be normal because, as Bortz and Döring (2006) put it, “a non-significant result says nothing.”
References
Almorsy M, Grundy J, Sadus R, van Straten W, Barnes DG, Kaluza O (2013) A suite of domain-specific visual languages for scientific software application modelling. In: Symposium on Visual Languages and Human-Centric Computing (VL/HCC), 2013. IEEE, pp 91–94
April A, Abran A (2012) Software Maintenance Management: Evaluation and Continuous Improvement. Wiley
Basili VR, Caldiera G, Rombach HD (1994) Goal question metric paradigm. In: Encyclopedia of Software Engineering. Wiley, pp 528–532
Basili VR, Cruzes D, Carver JC, Hochstein LM, Hollingsworth JK, Zelkowitz MV, Shull F (2008) Understanding the high-performance-computing community: a software engineer’s perspective. IEEE Softw 25(4):29–36
Bortz J, Döring N (2006) Forschungsmethoden und Evaluation für Human- und Sozialwissenschaftler, 4th edn. Springer
Bortz J, Schuster C (2010) Statistik für Human- und Sozialwissenschaftler, 7th edn. Springer
Consel C, Marlet R (1998) Architecture software using a methodology for language development. In: Principles of Declarative Programming, LNCS, vol 1490. Springer, pp 170–194
Faulk S, Loh E, Vanter MLVD, Squires S, Votta LG (2009) Scientific computing’s productivity gridlock: How software engineering can help. Comput Sci Eng 11:30–39
Fowler M (2010) Domain-Specific Languages. Addison-Wesley
Galesic M (2006) Dropouts on the web: Effects of interest and burden experienced during an online survey. J Off Stat 22(2):313–328
Hevner AR, Linger RC, Webb Collins R, Pleszkoch MG, Walton GH (2005) The impact of function extraction technology on next-generation software engineering. Tech. rep. Carnegie Mellon University
Hoisl B, Sobernig S, Strembeck M (2014) Comparing three notations for defining scenario-based model tests: a controlled experiment. In: 9th International Conference on the Quality of Information and Communications Technology (QUATIC), pp 95–104
Jedlitschka A, Ciolkowski M, Pfahl D (2008) Reporting experiments in software engineering. In: Shull f, Singer J, DI Sjøberg (eds) Guide to advanced empirical software engineering. Springer, pp 201–228
Johanson AN (2016) Data and scripts for the Sprat Ecosystem DSL survey. doi:http://dx.doi.org/10.5281/zenodo.61373
Johanson AN, Hasselbring W (2014a) Hierarchical combination of internal and external domain-specific languages for scientific computing. In: Proceedings of the 2014 European Conference on Software Architecture Workshops, ACM, ECSAW’14, pp 17:1–17:8
Johanson AN, Hasselbring W (2014b) Sprat: Hierarchies of domain-specific languages for marine ecosystem simulation engineering. In: Proceedings TMS SpringSim’14, SCS, pp 187–192
Johanson AN, Hasselbring W, Oschlies A, Worm B (2016) Evaluating hierarchical domain-specific languages for computational science: Applying the Sprat approach to a marine ecosystem model. In: Carver j, Hong NPC, Thiruvathukal GK (eds) Software Engineering for Science. Chapman and Hall
Jones B, Kenward M (2014) Design and Analysis of Cross-Over Trials. Taylor & Francis
Kieburtz R, Hook J (1995) Software design for reliability and reuse (sdrr) project phase I final scientific and technical report. Tech. rep. Pacific Software Research Center
Kieburtz RB, McKinney L, Bell JM, Hook J, Kotov A, Lewis J, Oliva DP, Sheard T, Smith I, Walton L (1996) A software engineering experiment in software component generation. In: Proceedings of the 18th international conference on software engineering (ICSE’96), pp 542–552
Knight J (2002) Safety critical systems: challenges and directions. In: Proceedings ICSE’02. IEEE, pp 547–550
Kolovos DS, Paige RF, Kelly T, Polack FA (2006) Requirements for domain-specific languages. In: Proceedings of ECOOP Workshop on Domain-Specific Program Development (DSPD)
Korman AK (1971) Industrial and Organizational Psychology. Prentice-Hall
Kosar T, Martı PE, Barrientos PA, Mernik M, et al (2008) A preliminary study on various implementation approaches of domain-specific language. Inf Softw Technol 50(5):390–405
Kosar T, Mernik M, Carver JC (2012) Program comprehension of domain-specific and general-purpose languages: Comparison using a family of experiments. Empir Softw Eng 17(3):276–304
Kosar T, Bohra S, Mernik M (2016) Domain-specific languages: a systematic mapping study. Inf Softw Technol 71:77–91
Likert R (1932) A technique for the measurement of attitudes. Arch Psychol 22(140):5–55
Meliá S, Cachero C, Hermida JM, Aparicio E (2015) Comparison of a textual versus a graphical notation for the maintainability of mde domain models: an empirical pilot study. Softw Qual J:1–27
Mernik M, Heering J, Sloane AM (2005) When and how to develop domain-specific languages. ACM Comput Surv (CSUR) 37(4):316–344
Palyart M, Lugato D, Ober I, Bruel J (2012) MDE4HPC: An approach for using model-driven engineering in high-performance computing. In: Proceedings SDL’11: Integrating System and Software Modeling, LNCS, vol 7083, pp 247–261
Prabhu P, Jablin TB, Raman A, Zhang Y, Huang J, Kim H, Johnson NP, Liu F, Ghosh S, Beardl S, Oh T, Zoufaly M, Walker D, August DI (2011) A survey of the practice of computational science. In: State of the Practice Reports, ACM, SC’11, pp 19:1–19:12
Ricca F, Scanniello G, Torchiano M, Reggio G, Astesiano E (2010) On the effectiveness of screen mockups in requirements engineering: results from an internal replication
Shapiro SS, Wilk MB, Chen HJ (1968) A comparative study of various tests for normality. J Am Stat Assoc 63(324):1343–1372
Siegmund J, Siegmund N, Apel S (2015) Views on internal and external validity in empirical software engineering. In: IEEE/ACM 37Th IEEE international conference on software engineering (ICSE 2015), pp 9–19
da Silva AR (2015) Model-driven engineering: a survey supported by the unified conceptual model. Comput Lang Syst Struct 43:139–155
van Solingen R (1999) Berghout, E. McGraw-Hill, The Goal/Question/Metric Method: A Practical Guide for Quality Improvement of Software Development
Stahl T, Völter M (2006) Model-Driven Software development: Technology, Engineering, Management. Wiley
Webb Collins R, Hevner AR, Walton GH, Linger RC (2008) The impacts of function extraction technology on program comprehension: a controlled experiment. Inf Softw Technol 50(11):1165–1179
Wilcoxon F (1945) Individual comparisons by ranking methods. Biom Bull 1(6):80–83
Wohlin C, Runeson P, Höst M, Ohlsson MC, Regnell B, Wesslén A (2012) Experimentation in software engineering. Springer
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by: Romain Robbes
Appendix
Appendix
1.1 Appendix A: Meta-Model of the Sprat Ecosystem DSL
The meta-model is shown in Fig. 9.
1.2 Appendix B: Academic Background Questionnaire
-
1.
Your name [free-form question; optional]
-
2.
Your e-mail address [free-form question; optional]
-
3.
What is your highest educational qualification? (e.g., Bachelor, Master, Ph.D. etc.) [free-form question]
-
4.
In which discipline did you acquire this qualification? [free-form question]
-
5.
What is your current occupation? (e.g., bachelor/master/Ph.D. student, post-doc, professor etc.) [free-form question]
1.3 Appendix C: Programming Experience Questionnaire
-
1.
Have you ever written a computer program using a programming language? (e.g., C, Matlab, Java, R etc.) [yes/no]
-
2.
Have you ever been required to write a computer program for work/for your studies? [yes/no]
-
3.
If applicable, which programming languages (including languages for special purposes like R or Matlab) have you used so far? [free-form question; optional]
-
4.
How would you rate the level of your programming skills? [non-programmer (never written a program, difficulty with reading), beginner (problems writing easy programs, can read), regular (can write basic programs, no problems reading), advanced (can write complex programs), expert (years of experience)]
-
5.
How experienced are you with the programming language C/C++? [non-programmer (never written a program, difficulty with reading), beginner (problems writing easy programs, can read), regular (can write basic programs, no problems reading), advanced (can write complex programs), expert (years of experience)]
-
6.
How interested are you in programming in general? [6-point Likert scale]
-
7.
If applicable, what was the primary source of information for learning how to program? (e.g., internet, book, a colleague/friend etc.) [free-form question; optional]
-
8.
If applicable: I am confident that the programs I have written are correct (i.e., that they actually compute what I intended them to). [6-point Likert scale]
1.4 Appendix D: Parametrization and Recording Exercise
The exercises are shown in Figs. 10 and 11 (parametrization exercise) as well as Figs. 12 and 13 (recording exercise). Note that in the screenshots of the survey, the Sprat Ecosystem DSL is always referred to as EcoDSL; this was the working title of the DSL when conducting the survey.
-
1.
Concepts of marine ecosystem simulations—such as fish species—can easily be specified with the Sprat Ecosystem DSL. [6-point Likert scale]
-
2.
Concepts of marine ecosystem simulations—such as fish species—can easily be specified with C++. [6-point Likert scale]
-
3.
The Sprat Ecosystem DSL seems simple to use. [6-point Likert scale]
-
4.
C++ seems simple to use. [6-point Likert scale]
-
5.
The Sprat Ecosystem DSL programs were easy to understand. [6-point Likert scale]
-
6.
The C++ programs were easy to understand. [6-point Likert scale]
-
7.
The Sprat Ecosystem DSL seems too technical to specify concepts of marine ecosystem simulations. [6-point Likert scale]
-
8.
C++ seems too technical to specify concepts of marine ecosystem simulations. [6-point Likert scale]
-
9.
It is easy to introduce changes to existing Sprat Ecosystem DSL programs. [6-point Likert scale]
-
10.
It is easy to introduce changes to existing C++ programs. [6-point Likert scale]
-
11.
I was able to understand the meaning of the Sprat Ecosystem DSL source code quickly. [6-point Likert scale]
-
12.
I was able to understand the meaning of C++ source code quickly. [6-point Likert scale]
-
13.
Which difficulties did you have while using the Sprat Ecosystem DSL? [free-form question; optional]
-
14.
Which suggestions do you have for improving the Sprat Ecosystem DSL programming language? [free-form question; optional]
1.5 Appendix E: Exercise in Modifying the Sprat Ecosystem DSL
The exercise is shown in Fig. 14.
-
1.
How would you rate your expertise regarding the Java programming language? [non-programmer (never written a program, difficulty with reading), beginner (problems writing easy programs, can read), regular (can write basic programs, no problems reading), advanced (can write complex programs), expert (years of experience)]
-
2.
The meaning of the Java code was easy to comprehend. [6-point Likert scale]
-
3.
It was easy to recognize the statements in the Java code that were relevant for completing the task. [6-point Likert scale]
Rights and permissions
About this article
Cite this article
Johanson, A.N., Hasselbring, W. Effectiveness and efficiency of a domain-specific language for high-performance marine ecosystem simulation: a controlled experiment. Empir Software Eng 22, 2206–2236 (2017). https://doi.org/10.1007/s10664-016-9483-z
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10664-016-9483-z