In recent years research in formal verification of hardware and software has reached important progresses in the development of methodologies and tools to meet the increasing complexity of systems. The explicit role of Formal Verification is to find errors and to improve the reliability on the accuracy of system design, which implies a challenge for software engineering of this century. The purpose of this research is to perform a systematic review of the literature to establish the state of the art of research in formal verification during the last 10 years and to identify the approaches, methods, techniques and methodologies used, as well as the intensity of those research activities. During the process it was found that research in this field has doubled since 2005, and that the mean value of researches conducted year after year remains the same and that prevail the application in control and interaction systems. Additionally it was found that, the case study is the most used method and that empirical research is the most applied type.
En años recientes, la investigación en verificación formal de hardware y software ha logrado importantes progresos en el desarrollo de metodologías y herramientas para hacer frente a la creciente complejidad de los sistemas. La función explícita de la verificación formal es encontrar errores y mejorar la confianza en la exactitud del diseño del sistema, lo que supone un reto para la ingeniería de software de este siglo. El objetivo de esta investigación fue realizar una revisión sistemática a la literatura para determinar el estado del arte de la investigación en verificación formal en los últimos 10 años e identificar los enfoques, métodos, técnicas y metodologías empleadas, lo mismo que la intensidad de esa investigación. En el proceso se encontró que la investigación en esta área se duplicó a partir del año 2005, que hasta el momento mantiene un número promedio de investigaciones año tras año y que predomina la aplicación en sistemas de control e interacción. Además, que el estudio de caso es el método más utilizado y que la investigación empírica es la más aplicada.
Functional verification has become the bottleneck for the design of complex systems. Simulating designs is money-demanding and time-demanding and performing a complete simulation is almost impossible. Currently, as a solution for these problems, designers have started using formal methods to perform formal verification on most of products. But there is still a wide gap for the verification of big designs, which can be built but cannot be verified completely because of the complexity of the problems they deal with (Sülflow et al., 2009). This has caused that in many countries, the academic world, industry and governments must face the challenge of reducing this technological gap and proposing new and ingenious solutions for specifying, designing, structuring and applying test cases by using formal verification.
Formal verification is a crucial element in the development of the current complex information systems. Moore’s Law is still applied to determine the growth rate of the complexity of software and hardware products, but the complexity of verification becomes more complicated. In fact, theoretically, it augments exponentially with product’s complexity and doubles in the same way with time. The community of computer sciences recognizes that functional verification is an important obstacle for a design methodology, and that it demands up to 70% of developing time and resources. But, despite the significant amount of efforts and resources applied in verification, functional faults continue as the cause of the significant number of errors of the final product. In extreme situations, the errors are artifacts of the simulation because they are not detected due to their non-exhaustive nature of the verification which is based in simulation. The real fact is that it does not matter how much time is applied in simulation or how exhaustive is the test plan, any attempt to validate a design by using simulation is by itself incomplete for any system.
Formal verification (FV) is a systematic process that uses mathematical reasoning to verify that design specification remains the same during implementation. With this verification is possible to overcome the challenges of simulation because all the possible input values can be explored algorithmically or exhaustively. In other words, to achieve a high degree of observation of the product it is not necessary to exaggerate the design or creating multiple scenarios.
One of the objectives of FV is to guarantee the complete coverage of the space of the states in the tested design, to achieve that it uses and applies techniques like model verification through the exploration of space of states and automated techniques to demonstrate the theorems. Currently, the most automated and most accepted FV technique is Symbolic Model Verifier or SMV and, despite its success as an important method for the formal verification of sequential commercial designs, is still limited in relation to the size of the verifiable designs (Copty et al., 2001). Formal verification requires that engineers think different. For instance, simulation is empirical, this means that using trial and error to test all of the possible combinations and try to discover errors can take significant time. For this reason, it does not fully achieve it. Besides, because engineers must define and create a high number of input scenarios, they focus their efforts on breaking the design but not on which design must do. Formal verification, on the contrary, is mathematical and exhaustive and allows engineers become focused exclusively in finding which one is the correct behavior of the design.
The aim of this research is to conduct a systematic review in the literature regarding research in formal verification during the last decade, to determine the approaches, methods, techniques and research methodologies used and the intensity of these research activities. To achieve that, the paradigm of evidence-based research was used. The possibility of using this paradigm is proposed in Kitchenham et al. (2004) and Dyba et al. (2005), and the goal is to identify a question that can be answered, which could provide information and which can lead to evidences for that answer and evaluate it (Brereton et al., 2007). Thus, a systematic review to the literature is the first stage to conduct evidence-based research. The guidelines to perform a systematic review to the literature are explained in detail in Brereton et al. (2007) and Kitchenham (2009).
In the next section, the methodology applied in this research is described; the third section shows and analyzes the results obtained; the following section shows possible threats and limitations for validation and the last section gives details on conclusions and future work proposals.
Methodological processPerforming a systematic review to literature can be divided in three main stages (Brereton et al., 2007): (1) planning, (2) execution and (3) documentation, which in turn divides in a combination of other simpler procedures, as shown in Table 1.
Stages of a systematic review (Kitchenham, 2009)
| Stages | Procedures |
| Planning | Specifying research questions |
| Developing the review protocol | |
| Validating the review protocol | |
| Execution | Identifing relevant research |
| Selecting primary studies | |
| Assesing the quality of studies | |
| Extracting required data | |
| Synthesizing data | |
| Documentation | Writing the review report |
| Validating the report |
According to Kitchenham (2009) and Kitchenham et al. (2009), planning a systematic review involves six definitions:
- 1.
The research questions
- 2.
The searching process
- 3.
The inclusion and exclusion criteria
- 4.
Quality assessment
- 5.
Data collection
- 6.
Data analysis.
The research questions applied during the development of this research were:
- Q1:
¿In which fields of formal verification is conducted research currently?
- Q2:
¿Which application methodology is the most researched?
- Q3:
¿In which formal verification technique is conducted research more frequently?
- Q4:
¿Which approach and research method is the most used?
- Q5:
¿Which is the intensity of research activities in formal verification?
In order to answer to Q1, Q2, Q3 and Q4, it has been associated each primary study with an approach or research method, with a technique and applied methodology and with a covered field. To establish the figures that will indicate the intensity of research activity, regarding to Q5, it was identified a research corpus of the number of publications by year. The slope of the line for the FV was compared to the slope corresponding to the line which represents the research activities in functional verification.
Research processA systematic review about a specific subject must identify and highlight the specific sources about of the object of study; however; in the field of formal verification were not found that sources, because the related studies can be published in journals and conferences related to both functional verification and formal methods. The purpose of the search was to identify the primary studies which could be included or excluded from the definitive set of studies of the review. The plan involved an automated search in the ACM Digital Library, IEEE Digital Library, Science Direct and Springer Link, based on the timeline between January 2000 and April 2011. The parameters of the automated searching and their location in the study were the following:
- •
Formal verification: in the title. For all of the research questions.
- •
Discret mathematical, declarative language, formal language, formal method, formal specification and formal verification: in the abstract or in the content. For Q1.
- •
Experimentation, case study, stochastic and heuristic: in the abstract or in the content. For Q2.
- •
Peer, animation, simulation, agil methods and XP: in the abstract or in the content. For Q3.
- •
The observation of the results for Q1, Q2 y Q3 permitted classifying the approach and the research method for Q4. For the empirical research it was performed a search of terms experiment, survey, case study, empirical research in the abstract and within the content.
- •
Formal verification AND research: in the title and combined with each year of the timeline. For Q5.
The total number of articles recovered in this search was 552. However, most of them were identified by marginal relation and as a result of combining some keywords. The exclusion of the irrelevant articles was carried out manually, based on the exclusion and inclusion criteria defined as follows.
Inclusion and exclusion criteriaThe works selected like primary studies had to be relevant for the research topic, therefore it was applied the filtering process proposed in (Dyba and Dingsoyr, 2008):
- 1.
Identifying the relevant studies. Only complete works published in journals, full conference-congress y workshop were considered and short papers, extended abstracts and posters were dismissed. A number of 131 studies were excluded.
- 2.
Excluding studies based on their title. The exclusion criterion applied was the AND filter in the advanced search option of each digital library. A number of 28 studies were excluded.
- 3.
Excluding studies based on their abstracts. A number of 49 studies were excluded.
- 4.
Among the resultant studies selecting the most relevant for the research topic based on the full text. It was decided including only the works deeply related with the subject of Formal Verification. According to this criterion 145 works were excluded, which resulted in a final set having 199 articles considered like primary studies for the research.
The purpose of this stage is to validate the fact that the primary studies selected exhibit solidity regarding methodology and results. Considering the high standards of the review process performed by the selected journals and databases, it was concluded, based on the evidences that the primary studies selected exhibit good quality.
Data collectionAfter finishing the inclusion or exclusion process, the set of data of the primary studies was structured. During this stage the following attributes were collected:
- 1.
Type of event: journal, conference-congress, workshop.
- 2.
Published in: journal, proceedings.
- 3.
Publishing house: ACM, IEEE, Springer, Elsevier.
- 4.
Year of publishing: 2000 to 2011 timeline.
- 5.
Country.
- 6.
Classification of the approach and method. According to Glass et al. (2002), the main research and scientific approaches are: descriptive, explanatory and empirical and, according to Wohlin et al. (2000) and Dyba and Dingsoyr (2008), there are three methods of research used to evaluate techniques, methods and tools: survey, case study and experiment.
- 7.
Classification of the field. The selected fields for the research were: mathematical models, formal languages, automated models, declarative languages, formal methods and formal specification.
- 8.
Classification of the methodology. The analyzed methodologies were: experimentation, case study, stochastic and heuristics.
- 9.
Classification of the technique. The selected primary studies were classified according to the treatment given in the employed technique: peers, animation, simulation and agile methods.
For answering Q5 three types of articles were included according the following classification:
- •
Technological and scientific research article. Document that presents in a detailed manner the original results of finished research projects. Their structure generally has four sections: introduction, methodology, results and conclusions.
- •
Reflection article. Document that presents the results of finished research from an analytic, critical or interpretative point of view about a specific topic and considering original sources.
- •
Review article. Document which analyzes systematizes and integrates the results of published or non-published research about a science or technology field, having as purpose disseminating the advances and trends of development. A characteristic feature is that they present a detailed bibliographic review of, at least, 50 references.
The primary studies were tabulated and analyzed statistically with the objective of finding:
- 1.
Number of works published by year: Q5.
- 2.
Number of works published in journals and proceedings: Q5.
- 3.
Number of studies by country: Q5.
- 4.
Main topics covered in formal verification: Q5.
- 5.
Approach and research method: Q4.
- 6.
Field of formal verification in which research is performed: Q1.
- 7.
Application methodology: Q2.
- 8.
Used technique: Q3.
In order to understand the categories assigned for each study, the features of the data set of the primary studies were tabulated. It is important to notice the difference between research activity and research article. Research activity comprise the set of relevant articles which were included based on the title, that is, research, reflection and verification articles, whereas the research articles are the final result of applying the inclusion and exclusion criteria. Table 2 shows the dynamism of the research activity by year and event type.
According to these results the research in formal verification doubled since 2005, remaining constant the number of publications until present time. Figure 1 shows the comparison between the research activities in software engineering, SE vs FV. The topics covered by research in SE are diverse, among them we found formal verification, but for this analysis it was taken a different concept because of the research interests. As shown in Table 5, the industrial and academic groups are the biggest boosting agents of research in formal verification, but the universities quadruple the work performed in industry. Additionally, in Table 2 it can be seen that there are specialized journals, workshops and conferences that debate about the theoretical and practical applications of this subject.
When examining the activities regarding formal verification by country it can be seen that the United States provides more than half of total publications with 55%. The United States is followed by thr United Kingdom, Japan, China and Germany. In other representative countries, it was found that interest on FV started a little bit later than in those mentioned above. Table 3 presents the comparison between works which report research activities and other type of reports related to FV topics.
Because the purpose of this review to the literature is to inquire about methods, techniques and methodologies that apply research in formal verification, from this point in the article the work will be focused in 105 papers that disseminate results of research. Table 4 shows the publications in which the articles were found about the researches related to formal verification and the amount of works published in the timeline covered by this investigation. Figure 2 shows the profile of the most active countries in research in FV.
Journals and published Works on Formal Verification
| Publication | |
| Electronic Notes in Theorical Computer Science (ENTCS) | 20 |
| Formal Methods in System Design | 12 |
| IEEE Transactions on Software Engineering | 8 |
| IEEE Transactions on Systems, Man, and Cybernetics | 7 |
| International Journal on Software Tools for Technology Transfer (STTT) | 6 |
| Theorical Computer Science | 4 |
| IEEE Design & Test | 6 |
| Journal of Automated Reasoning | 3 |
| Formal Aspects of Computing | 3 |
| Computers in industry | 2 |
| Computer Standards & Interfaces | 2 |
| Science of Computer Programming | 2 |
| Journal of Systems Architecture: the Euromicro Journal | 2 |
| Real-Time Systems | 2 |
| Computer | 1 |
| IEEE Transactions on Computers | 1 |
| IEEE Software | 1 |
| IBM Journal of Research and Development | 1 |
| Journal of Computing Science in Colleges | 1 |
| Journal of Systems and Software | 1 |
| Journal of the ACM (JACM) | 1 |
| Journal of Parallel and Distributed Computing Systems | 1 |
| Future Generation Computer Systems | 1 |
| Journal of Symbolic Computation | 1 |
| Automation and Remote Control | 1 |
| Advances in Engineering Software | 1 |
| Journal of Electronic Testing: Theory and Applications | 1 |
| Environmental Modelling & Software | 1 |
| Integration, the VLSI Journal | 1 |
| International Journal of Parallel Programming | 1 |
| Programming and Computing Software | 1 |
| Nordic Journal of Computing | 1 |
| Informatics | 1 |
| Journal of Visual Languages and Computing | 1 |
| IEEE Transactions on Dependable and Secure Computing | 1 |
| EURASIP Journal on Embedded Systems | 1 |
| Annals of Software Enginnering | 1 |
| IEEE Transactions on Information Forensics and Security | 1 |
| Software Testing, Verification & Reliability | 1 |
| International Journal of Agent-Oriented Software Engineering | 1 |
Table 5 presents a comparison between the number of universities and the number of companies which perform research in formal verification and the number of published works.
Tables 6 and 7 show the results of the analysis regarding the classification of approaches and methods found in research in formal verification.
The most applied method is case study; this fact reinforces the result that indicates that universities have the highest participation in the research in this field. The research articles in formal verification use an approach of empirical research, because of the need of confirmation of the method and the applied model. Table 8 shows the results regarding the field of research in formal verification, it must be considered that they are inclusive.
The fields in which most of the work is performed are formal specification, mathematical models and formal methods. The last one allows describing the properties of the system through rigorous mathematics, to achieve this, they apply a language of formal specification which makes possible specifying the functionality of a program; this is due to the way formal verification is built: first it focuses on the specification, then the test model is built and finally the verification of the case study is checked. This is a complex process that involves different tools, some manual other automated. Table 9 shows a comparison between the methodologies used for formal verification in the analyzed works.
Since the case study method is used to apply formal verification and to check the results manually, it also appears like the prevailing methodology to validate the results of the empirical approach. The experimental part becomes evident in the participation of the industrial research. Table 10 presents the techniques used for research in FV for the primary studies.
The current techniques of development are best suited to the new paradigms and there are commercial tools that support the quality improvement of software. As information systems increase their complexity, the losses caused by faults are increasingly higher. Around 98% of the research articles describe simulation techniques, this with the purpose of controlling the input variables and the responses or expected outputs in the test environments. It is important to notice the fact that only 2% used the technique of peer checking, that in the reviews of the end-of-century literature was the most used.
Threats and limitationsIn this review we performed a detailed research of the literature based on the finding of 199 authors and different works, including some secondary studies – where references in the primary study were used to find other studies. However, it can be seen that, considering the increasing trend of works in this field, it is not possible to ensure that all of the articles in this field were recovered, particularly for 2011, because the research ended in April.
The studies that did not have the words formal verification in their title were not included in the set of primary studies; hence, it is possible that during the search process a significant number of studies related to the research field were excluded. Furthermore, the inclusion of works presented in workshops could change the results because their nature is different than that of journals and conferences. The difficulties to discern the established parameters in the research for the sources which only allowed access to the abstract, could also influenced the results of the classification.
Formal verification in the different countries and time periods has been grouped in thematic fields with the intention of identifying the interest fields in each of them, which do not necessarily correspond with that established to answer the research questions of this work. However, derived from the literature review itself, the suggestion arises for associating different functions to different needs and motivation features. Grouping all this roles and functions part of the detail that was possible to include in the analysis could be lost. In this review the term formal verification covers a number of roles in software engineering, like the tasks carried out by all the professionals that participate directly in the production of software. This causes limitations for the study because rarely they are defined or differentiated individually according to the practice, but it is also true that competences, roles and practices in this field have changed during the timeline covered by the review; for instance, in the early 2000 the role of the programmer/analyst was common; on the contrary, at mid 2005 they were already called software engineers. Therefore, researches and publications related to formal verification could be also be biased by these trends.
Conclusions and future workThe objective of this work was to summarize the state of the art about the scientific research in the field of formal verification and in order to achieve that a systematic review of the literature was performed, considered like the first step in the research paradigm based in evidence. FV has lately become a practical mean for detecting the presence of unwanted behaviors in software products, a required feature for critical models. The models for checking quality in the software industry and those used by the testers of advanced theorems, make easier performing complex analysis of specifications in an automated or semi-automated way.
Because of the nature of formal verification, the most representative research approach is the empirical, due in part to the need of checking in a case study the model created through observation and result analysis.
The research articles included in this study cover a wide variety of topics related to FV, like Petri Nets for control devices, digital circuits and processors –in which they are used to perform exhaustive verification processes in order to optimize the design–; the temporal logic to verify formally the concurrence of access to the control algorithms and the security specifications of information systems to ensure their security; formal semantics for business specifications; the verification of system requirements; the analysis of hierarchical processor, which divides into a set of conditions for the achievement of a simpler verification for reasoning, allowing to perform the test in the different architecture levels; the heuristic ones to formally and automatically verify complex systems like the next generations of microprocessors. Software engineering is facing up a permanent challenge with formal verification, because its goal is to reduce the gap between high-complexity systems and the applicability of good practices in the whole design process.
Formal specification is a topic that can be found in all the research articles of this study. Some of them describe the need for establishing methods of specification presentation and writing with features like: accessibility for the user based on the functional logic representation of knowledge, possibilities of automated analysis of conversion and translation to other languages developed in formal methods, the formal unified format for the exchange between different development systems and the graphic representation of the logic of the sentences in the programming language. Other feature found in the primary studies is that formal verification becomes integrated in different fields trough frameworks, which allow the development of applications to formally verify the systems which remain independent from the underlying technique of testing and from the new verification techniques about the level of word, like the abstraction of predicates and the theory of the module of satisfaction.
The research questions proposed in the methodology were answered according to the results obtained during the review. These results can be used in industry and academic institutions to plan new researches and to plan works which lead to automated formal verification. This field is a priority for the community, because the complexity of the systems in the coming decades will continue to increase, and manual testing will not be enough.
The results of this review propose new questions which could be solved by future researches. For instance, because software engineers have established a new professional group regarding to those established at the end of the century in computational sciences, there are topics and aspects related to FV that remain unsolved, this causes the need for further studies. It could be useful to evaluate how actively the formal methods in the curricula of different undergraduate programs are included, with respect to computational sciences; this could offer future results to achieve that total automation of software tests could be a reality. Additionally, it is necessary to continue working to develop a mathematical model to formalize software engineering.
Edgar Serna-M. Theoretical computational scientist with over 10 years of industry experience as project leader in information systems and as a software architect and university professor and researcher with over 20 years of experience. His areas of research are logic, software engineering, computer science, and formal methods and computer mathematics, around which has published books and articles, and participated in conference with papers in national and international events.
David Morales-V. Systems engineer and specialist with a specialization in programming methodologies and databases. With seven years of industries experience, it is graduated to the University of San Buenaventura Medellín. His areas of interest include programming fundamentals, data structures, databases, software engineering, geographic information systems and computer education, about which he has published several.