Analyzing R&D projects on health products

Abstract

The resources invested in research do not guarantee an immediate practical application. Companies and government increasingly seek mechanisms for prioritizing R&D projects appropriately when resources are insufficient. This study's objective was to develop and present a methodology used to evaluate the portfolio of research and help choosing the best investments in research. This methodology was applied at the Butanta Institute, in Brazil, an organization responsible for researching and developing, among other medicines, vaccines and other chemicals, Onco BCG, a medication for the treatment of bladder cancer. In the article, we present and analyze the methodology used at Butanta Institute. Conclusions show that literature on R&D portfolio management advises the use of risk and return criteria, when choosing among projects. In the case study of Butanta, the methodology of choice was based more on the customer's perspective.

Introduction

Choosing the best portfolio of projects of research and development (R&D) is an important discussion when dealing with technological strategy. One reason for that is the substantial amount of investment on research and which not always can be reflected into results. Another issue is the cash flow dilemma: expenses in research and development today can bring results in a long term perspective. This paper presents a methodology for evaluation of the R&D portfolio at a government institute, in Brazil, which is responsible for development of many medications.

Decisions regarding investment in R&D generated a line of studies under technological strategy, named portfolio selection process (Blanes i Vidal & Möller, 2016). In this field, past studies have come up with methodologies that support decision makers in choosing how they will invest their financial resources, given the amplitude of R&D projects available. Many of these methods are built on complex methods such as the fuzzy logic (Bhattacharyya, 2015). However, these methods do not fit all types of companies or the problems they face; one of these cases where there is a call for more adaptive models is the case of nonprofit organizations (Jeng & Huang, 2015).

One of the nonprofit sectors that demand such studies is the Health Programs, as the results of the research directly impacts the population well-being. Avoiding primary research can prevent the discovery of cure for many diseases.

When we look for answers for the problem of choosing the best R&D portfolio of projects, part of the literature show methodologies of evaluation of R&D portfolio based on the trade-off between risk and return. Those studies consider the financial impacts as the most important criteria for decision. This fact is expected, considering that the most part of the studies focus on private companies. When it comes to health developments, the search of the cure of important diseases depends on projects that have high costs and also high risks. In the trade-off between risk and return, used as the only criteria for the selection, these projects could be among the least preferable ones. This is specifically true for the case of non-governmental organizations (Lacerda, 2015). The choice of less risky projects could make impossible the cure of important diseases, in the future.

This study objectives to develop and apply a methodology for choosing the best portfolio of R&D when it comes to the development of one of the products of Butanta Institute: the Onco BCG vaccine. This vaccine is one of the medications designed for the treatment of bladder cancer.

Butanta Institute is a governmental agency affiliated with the Sao Paulo State Health Department, in Brazil. It carries out many activities aimed at preventing and correcting health problems faced by the population of the state. Among these activities, there are scientific and technological research projects developed by state-owned research institutions. Such research is very frequently out of touch with the Department of Health's priority needs. This happens partly for lack of awareness of the Department's needs, which are not taken into consideration. There are evidently other reasons for the mismatch. Martin (1994) highlights the importance of investing in innovations geared to clearly identified needs so as to increase the chances of success. Although his observation relates to private-sector companies, it is equally applicable to the public sector. Another possible reason for the divergence between strategic research needs and the research that is really developed is that, in the Butanta Institute case, the research is primarily funded and developed by scientists with academic scholarships provided by the government; given that, the ones who decide what subject will be researched are the scientists themselves, according to their own research interests. The demand for research comes not necessarily from the Butanta Institute or its product engineers, but it comes from the researchers themselves.

The study described here entailed the development of a methodology for evaluating R&D project portfolios and partial application to test its effectiveness. The experiment demonstrated the method's considerable potential as an ancillary instrument for evaluating an R&D portfolio. To test the methodology we chose BCG immunotherapy, a medication used in the treatment of bladder cancer. If these needs are identified, part of the research effort can be directed toward supporting them so as to increase the general public's satisfaction with the health services provided by the public sector, at a lower cost to society.

Conceptual framework

Procedures to manage portfolios of technology projects have been developed and applied by private enterprise as part of a broader process known as Technology Management. The purpose of Technology Management is to ensure that technology is used to leverage competitiveness by creating new products and processes, enhancing existing ones, cutting costs, and developing patents that guarantee a temporary monopoly. Among the tools of Technology Management is Strategic Technology Planning, which comprises several stages including project portfolio management. These concepts can also be used by public-sector entities. In this case the pursuit of profit is replaced by the desire to create quality products and services for society at a low cost and on a timely basis. Once priority needs have been identified, the existing project portfolio can be evaluated and new projects identified. Existing project selection methods can then be applied.

Technology is an ancillary instrument for the implementation of such programs. It boosts their performance in terms of quality and cost savings, so that healthcare services can be extended to more people for the same cost. Hence the importance of establishing R&D priorities.

It must be stressed out, however, that part of the resources allocated to research should be kept available for projects geared to the advancement of science. This is especially valid for governmental research institutions, although some private-sector firms invest a small proportion of available funds in scientific research.

Context of project portfolio decisions

Project portfolio management should be aligned with the organization's technology strategy, which in turn needs to be integrated with corporate strategy. Much of the specialist literature on strategic management indicates that one of the main reasons for which firms develop a strategic plan is the need to identify the future environment in which the firm intends to operate. This is done by analyzing a series of variables in order to establish basic guidelines to be followed by all members with a minimum of stability. There are many strategy concepts and typologies, each one advocated by authors who follow one of the many schools of thought on the subject that have emerged so far. This paper focuses on the approaches considered most relevant to the object of study.

On the method of strategy formulation, Mintzberg (1998) classifies strategies into: emerging strategies, deliberate strategies, and deliberately emerging strategies. For him strategies can either take shape or be formulated. A given strategy may emerge as a response to an evolving situation or be introduced deliberately through a process of formulation followed by implementation (Oliveira, Salazar, Crêspo, Costa, & Kovacs, 2015). Thus strategies are termed deliberate when constant planning intentions have been fully satisfied. Emerging strategies, on the other hand, derive from actions undertaken by the firm but not explicitly planned as such.

Because emerging strategies are comparatively flexible, they stimulate learning but hamper control, since actions are taken one at a time as the need arises. This type of strategy seems most suited to situations of technological paradigm change, in which a “window of opportunity” must be used quickly to establish a competitive advantage (which also may or may not last). Deliberate strategies, on the other hand, are intended to establish the best form of control but cannot accurately foresee the future and keep all the factors involved within expecting situations so as to achieve their goals (Neis & Pereira, 2016). As a trade-off between the two types of strategies Mintzberg proposes a deliberately emerging strategy, also known as process strategy. In this case, “management controls the strategy formulation process – focusing on designing structure, staffing, procedures and so on – but leaves content proper to others”.

In the specific case of technological innovation, Roussel, Saad, and Erickson (1991) stress the importance of integrating technology needs identification with corporate strategy. Applying this concept to the government sector, the Department of Health should clearly define its strategic priorities in terms of what products and services to make available to what segments of the population. The relationship between strategy and technology management is discussed in depth by Brockhoff (1998).

Bignetti (2001) cites Ansoff (1965) and Andrews (1971) to argue that classic studies of innovation generally refer to organizations which interact with relatively stable environments and are characterized by relatively long product and technology lifecycles. This is the case, for example, for firms in traditional industries not significantly affected by technological revolutions or new market preferences. Environmental uncertainty is reduced because results and demand are reasonably foreseeable. In such circumstances the actors tend to behave in accordance with a more deterministic view of the environment. The strategy prescribed is one of adapting the organization to environmental threats.

Technological progress can often be seen as deriving from the competition among technology regimes, design configurations or different options for a design common to a large number of firms. Mastery of a technology may produce irreversibilities and constraints: as the technology evolves, future developments may be tied to a particular paradigm (Dosi, 1982). New technological discontinuities usually come from outside the paradigm, from a different industry, or from a new current of knowledge (Utterback, 1994).

According to Bignetti (2001), from the traditional perspective technology management has so far evolved through five generations, from traditional models of technology-push and market-pull in the 1950s and 1960s to the integrated systems of the 1990s (Rothwell, 1993, cited by Bignetti, 2001). While markets, customers and suppliers were gradually inserted into studies as these generations developed, they are still considered external sources of information for a fundamentally internal process of R&D.

Technological competition is described as the result of a decision-making and social interaction process from which a technology – not necessarily the best – emerges victorious (Arthur, 1996). Contrary to expectations based on survival of the fittest, the final solution may often be an instance of survival of the boldest. The innovation process is seen not as a sequential process, from basic research to market, but as a spiral in which the development of technology is inherently tied to the implementation stage.

In the sphere of public health it is vital to define a clear and consistent strategy for action geared to meeting priority health needs for the different regions and population strata. Technology is an ancillary instrument in implementing such strategies. There are various prerequisites for adequate technology use. One of them, which is the focus for this paper, is to identify priority technology needs so that it is possible to evaluate how far the existing R&D project portfolio meets those needs.

Methods of project portfolio evaluation

Project selection is one of the various aspects of R&D project portfolio management. In this matter, there are many different methods that could be used for deciding which is the best research portfolio. Many of these tools have been based on fuzzy sets (Pérez, Caballero, Carazo, Gómez, & Liern, 2015) or complex methods based heavily on data and analysis, using techniques such as the analytic hierarchy process or linear programming (Parvaneh & El-Sayegh, 2016). Recent studies have, however, challenged researchers to offer methods that adapt to different types of companies and the problems they face (de Oliveira Filho, Silveira, & Ana, 2014). Also, they point to the importance of considering other variables beyond the financial aspects, such as sustainability (Brook & Pagnanelli, 2014).

An analysis of the types of R&D project portfolio selection tools showed four main categories, that we describe as follows.

• 1)

  Risk evaluation

  This type of method is based on decision trees and other financial methodologies to evaluate the tradeoff between risk and return, considering the R&D portfolio. According to Brook and Brewster (1999), developing and using a Customer Needs Tree is a simple and effective way of solving the problem, with specific benefits in idea generation, technology assessment, portfolio management and internal communication.

• 2)

  Portfolio grid

  The portfolio grid distributes R&D projects according to their probability of technical success and potential commercial value given success. Matheson and Menke (1994) adopt this approach, and classify projects in quadrants: bread and butter (projects with low technical difficulty and low marketing potential), white elephant (projects with high technical difficulty and low marketing potential), Oister (projects with high technical difficulty and high marketing potential) and pearl (projects with high potential for commercialization and low technical difficulty). Jolly (2002) also developed a model comprising two basic variables: technological competitiveness and technological attractiveness. Each can be measured by 16 factors.

• 3)

  Mixed methods

  It is common to adopt mixed methods, e.g., the ones based on financial analysis, that also classify projects on the portfolio matrix. Tritle, Scriven, and Fusfeld (2000) adopt a periodical review of the existing projects, followed by project portfolio graphs, metrics and decision trees.

• 4)

  Value added systems

  Some authors use a more targeted approach to customer needs than the previously shown methods. In private institutions, this may be related to the potential market for the product. In government institutions, the system lists the best projects through its potential contribution to society. Linton, Walsh, and Morabito (2002) use a decision model with more than one criterion, and chooses the priority projects that create higher value for the consumer.

Table 1 synthesizes these types of methods for evaluation of R&D projects as explained before.

Table 1.

Authors and categories of methodologies they have used.

Categories	Authors
Value creation	Linton et al. (2002)
Portfolio matrix	Matheson and Menke (1994)
Mixed systems	Tritle et al. (2000); Lint and Pennings (2001)
Trade-off risk × return	Graves, Ringuest, and Case (2000); Rzasa et al. (1990); Morris et al. (1991); Brook and Brewster (1999); Ringuest and Graves (2005); Kolisch, Meyer, and Mohr (2005)

Source: Authors.

Following the last approach in Table 1– trade-off risk × return – Osawa and Murakami (2002) developed and applied a two-stage methodology for evaluating R&D projects in industry. The first stage comprises the development of the evaluation platform in three steps:

• a)

  Philosophy: Identifies what lies behind the decision, based on information from the firm's business plan, strategies, vision and mission. Two types of criteria are defined: qualitative criteria to evaluate the suitability of a project in terms of the extent to which it matches the firm's strategy; and quantitative criteria relating to financial aspects, among others.

• b)

  Outline of five criteria: (1) strategic importance and technological effect; (2) probability of realization; (3) sales; (4) profit and (5) R&D efficiency.

• c)

  Outline of evaluation platform, comprising an analysis of three factors: Input, calculation, and output.

The second stage deals with utilization of the evaluation platform, as summarized in Fig. 1.

Fig. 1.

New methodology using evaluation platform.

Source: Osawa and Murakami (2002).

Chien (2002) presents a framework for evaluating alternative portfolios of R&D projects. The basis for his study is the need to evaluate inter-relations among projects. A set of very good projects is not necessarily best for the firm because relations among projects entail additional synergies not considered by the traditional method.

Evaluation of a portfolio of R&D projects should take account of technology needs. Technology needs identification should take place simultaneously along two main avenues. The first involves evaluating the production processes used for a product or service, and detecting any potential for enhancements that could give rise to research projects (Vasconcellos, 1990). The second consists of evaluating the scientific and technological trends that could completely eliminate a disease or drastically alter production techniques, such as the advances derived from molecular biology and genetic engineering. S-shaped curves and the concepts of disruptive technologies and technological obsolescence are discussed in depth by Sanderson and Uzumeri (1997), Christensen (1997), and Sahal (1991). Although this paper focuses on the first avenue outlined above, any technology needs identification process should include both avenues.

Recommendations for the evaluation of R&D portfolios

Considering the recommendations and best practices to the evaluation of R&D portfolio, to Hodder and Riggs (1985), the choice of R&D projects requires consideration of the risk of three distinct project phases: research, product development and sales. Matheson and Menke (1994) complement this advice with the idea that the selection should not focus on evaluation of the individual projects; to maximize one's return, there is need to make quality decisions at the portfolio level mixing high risk – high potential R&D with lower-risk projects that produce near-term returns through incremental improvements to existing products and processes.

Morris, Teisberg, and Kolbe (1991) criticize the use of methodologies based on the analysis of risk and return only. For them, this type of methodology can lead the manager to avoid dealing with the riskier projects. However, they understand that if two projects have the same expected payoffs and the same costs, but different risks, and different ranges of possible outcomes, a conscientious R&D manager should choose the riskier one. The authors also point that this decision is better even for risk-averse managers. If a manager can choose several projects, than he can benefit from having a portfolio of risk projects. This happens because a project with more downsized risk has a higher expected value in the commercial phase and this higher value outweighs the lower chance of success in the research phase.

Cooper, Edgett, and Kleinschmidt (1998) present the results of a survey in which financial methods were rated as having more weaknesses than strengths. For them, a good project follows six criteria: portfolios are aligned with the business objectives, contains very-high-value projects, reflect the business strategy, are done on time – no gridlock, have good balance of projects and finally they have the right number of projects.

Tritle et al. (2000) consider that the R&D portfolio must be in balance with corporate goals and strategies – nevertheless, assessment of qualitative information is critical. Kirchhoff, Merges, and Morabito (2001) also consider, based on their experience on Lucent Technologies Lab, that the value creation is better than focusing only on financial aspects.

Finally, Rzasa, Faulkner, and Sousa (1990) suggest that it is important to internalize the methodology at the group level and then internalize it at the business unit level.

In this study, we present a framework for R&D project evaluation that was not based on fuzzy sets, but instead considers the specific demands for a non-governmental agency.

Methods

As seen before, this study's objective is to present and analyze the case of Butanta Institute and its methodology for choosing the best portfolio of R&D when it comes to the development of one of its products: the Onco BCG vaccine.

This study presents the results of a practitioner work developed in Butanta Institute. Therefore, this paper is the result of an action research methodology.

To achieve this goal, the practitioner project was developed in 6 steps, described as follows.

• 1)

  First block of interviews: First, those responsible for the Butanta Institute, in order to collect information about the strategic direction of the organization were interviewed. This first block lasted for interviewing 4 h.

• 2)

  Second block of interviews: In sequence, respondents were mainly responsible for the BCG Onco product. This second set of interviews lasted 30 h and was conducted over several sessions. The purpose of interviews was with much depth to determine what are the surveys that were proposed to the institute and that would be viable for achievement. Thus, the aim of delineating which initial portfolio of research which is being treated. It is important to remember that, unlike what happens in private institutions, which often need to make use of creativity techniques to raise ideas for research to be conducted in the case of Butanta Institute these ideas have been proposed by researchers, presenting research projects that may or may not be funded by government agencies. These 30 h of interviews were aimed at collecting information about requests of previous studies and identify any needs that have not been proposed by researchers to date.

• 3)

  Third block of interviews: In the third block, were made with the main panels that serve the users of BCG Onco product. This block was designed to assess the factors critical to the success of BCG Onco from the viewpoint of its users. It is important to note that doctors who prescribe the drug the best people to provide information about their improvements and critical success factors, although users of BCG Onco are patients who are treated with this medication are. So were collected from people dealing with doctors.

• 4)

  Development of methodology to be used by the institute. Based on the information collected, broke for the elaboration of the method for selecting the portfolio of research at the institute. This method is described as follows, between the results obtained.

• 5)

  Application of the methodology for selecting the portfolio of research and data analysis. In this fifth step, we applied this methodology to the case of BCG Onco and sought to identify the research that have high potential alignment with the strategy of the institute and improvements to BCG Onco product based on the critical success factors raised in the third step.

• 6)

  Assessment methodology: The end of the application of the methodology, were conducted more interviews with those responsible for the BCG Onco order to verify the applicability of the methodology used.

Butanta Institute

Butanta Institute was founded in February 1901 to combat an outbreak of bubonic fever in the port city of Santos, in Brazil, and it has since become a center for research and production of antivenom sera. The institute currently focuses on the development of high social impact biopharmaceuticals:

• •

  Erythropoietin: Erythropoietin cloning and purification are under development by the institute for use by more than 25,000 kidney patients who are awaiting transplants and currently cost the health system some 50 million US dollars.

• •

  Lung surfactant: An industrial plant is under development for production of 500,000 doses of lung surfactant per year, to be used primarily in premature babies with immature lung syndrome due to insufficient time in the womb for lung formation. Another application is in adult patients with lung problems.

• •

  Anti-CD3: Anti-CD3 is under production to meet international demand as well as for distribution in the domestic market. It is used to constrain thrombus progression and as a topical anticoagulant.

In recent years Butanta Institute has stood out as the leading research institution in Sao Paulo State based on the number of indexed publications, similar to that of the main institutions belonging to the University of Sao Paulo, with which it collaborates in graduate programs for master's and doctoral degrees in biology and biotechnology.

Application of portfolio evaluation method

The method proposed in this paper for evaluating a portfolio of R&D projects comprises five stages, as shown in Fig. 2.

Fig. 2.

Method for R&D project portfolio evaluation and improvement.

Source: Butanta Institute, research data.

As shown in Fig. 2, the method was applied to a single product in order to test the procedure. Thus only portfolio projects relating to the selected product were evaluated. The procedure should be applied to all products of Butanta Institute, generating a large number of projects for selection using existing methods. Project interdependency should be taken into consideration during this stage (Oullet & Martel, 1995; Ringuest, Graves, & Case, 1999).

The analysis of the project portfolio for Onco BCG comprised the steps set out as follows, in accordance with the stages of the proposed methodology.

Strategic guidelines

A meeting with the Technical Committee enabled the main strategic guidelines of Butanta Institute to be identified. We also identified its main areas of activity and the segments of the community served by the institute.

Product selection

The next step involved an initial meeting with the institute's coordinators followed by meetings with those responsible for the potential products to be used in the study, as identified in the previous step. It was then possible to decide which product would be studied, taking into consideration factors such as the relevance of the product to the overall strategy of the institute. The product chosen in this case was BCG immunotherapy, also known as Onco BCG, currently used in the treatment of bladder cancer.

Design of evaluation matrix

This stage comprised two parts: identification and weighting of performance factors, and study of the production process to identify production stages.

Identification and weighting of performance factors: This step consisted of meetings with those responsible for manufacturing the chosen product to determine the performance factors customers expect from the product and the relative weight of each factor. These performance factors can be understood as key features to ensure that the customer's problem is resolved with quality and at low cost. Customers are defined not only as patients who receive the products for treatment but also as physicians who prescribe it and professionals responsible for application, such as nurses and once again physicians.

Performance factors for Onco BCG are as follows:

• •

  Longer shelf life – The longer the shelf life, the longer the product can be stored, meaning less waste and better control by the nurses and physicians who use it. In addition, a product that can be stored longer is more easily distributed to distant regions or locations that use it on a smaller scale. The latter can receive annual shipments of the product, for example.

• •

  Lower cost – Lower cost allows for production of larger quantities and thus coverage of an even larger segment of the population in need of the product.

• •

  Less side effects – Side effects are undesirable consequences of treatment with medication. Less side effects mean better quality treatment and more patient satisfaction.

• •

  Immunogenicity – Immunogenicity is the patient's response to the medication. A high degree of immunogenicity indicates that the product is doing its job and that treatment is proceeding satisfactorily.

• •

  Ease of application – The product should be easy and convenient to apply. This means, for example, rapid reconstitution and no need for an arsenal of equipment for administering the product.

• •

  Resistance to temperature variation – The medication needs to be stored, transported and handled under differing conditions in terms of temperature and the general environment. It should be formulated and packaged in such a way as to withstand temperature variations and to suit the environment in which it will be used.

Study of production process and identification of production stages

Once the performance factors had been determined, the Onco BCG production process was mapped out. More meetings were held to discuss details of the production process, divide it into stages, and diagnose each stage separately. This step of the methodology included a visit to the production facility, providing an opportunity to see how Onco BCG is manufactured and in particular the rigorous care taken at every stage of the process. The columns in Chart 4-1 represent the various stages of production.

Completion of matrix

The matrix produced in the previous step was completed in two stages: performance factors vs. stages of production; and identification of technology needs covered by projects in progress.

Performance factors vs. stages of production: To facilitate the analysis and as part of the proposed methodology, a matrix was created (Table 2) correlating the information on performance factors and production stages. Each cell in which the two variables intersect is used to show the importance of each production stage in achieving the performance factors for the end-product.

Table 2.

Matrix correlating information on production stages and performance factors.

Performance factors	Production stages
	Priority	Inputs				Innoculum preparation
		Culture medium	Lyophilization stabilizers	Glass vials and synthetic rubbers	Dilutant	BCG growth	Transfer to Erlenmeyer	Culture selection	Bacterial film filtration	Minimization of cell aggregates	Deep fermentation
Longer shelf life	A		2	1
Lower cost	B	3	2								3
Less side effects	B	1	1
Immunogenicity	A	3				3	2	2	1	1	3
Ease of application	B			1	2
Resistance to temperature variation	A					3					3

Performance factors	Production stages
	Formulation of product	Filling	Lyophilization	Quality testing							Storage at institute	Transportation	Storage at application site
				pH measurement	Colony forming units (CFU)	Oxygen takeup	Heat stability	Residual moisture	Reconstitution of product	Other
Longer shelf life			3		1	1	1				2	2	2
Lower cost			2		3	3	3			1	2	3	1
Less side effects													1
Immunogenicity			1		3					2			1
Ease of application									1				1
Resistance to temperature variation			3				1					2	2

1 = low potential impact; 2 = medium potential impact; 3 = high potential impact.

Source: Research data.

Each cell is ranked on a scale from 1 to 3. Cells with the highest potential to have a positive influence on the desired performance are ranked 3. Cells with little or no importance in terms of potential satisfaction of the performance criteria are left blank.

As shown in Table 2, identification of technology needs covered by projects in progress: the matrix presented above was completed in order to analyze the research projects in progress at the institute and measure their relevance. Relevance is represented in terms of the contribution of each production stage to achievement of the performance factors defined for the project. Thus by correlating performance factors with the corresponding stages of production it is possible to relate projects in progress with the product's performance requirements. It is important to note that needs identification is necessary but not sufficient. Scientists tend to resist demands not directly related to their specialties. A study by Cohen, Duberley, and Mcauley (1999) involving seven government-funded research institutions analyzed their need for outsourced research and the impact on the scientists working there, showing the significance of motivation to ensure that part of the work done by research institutions is geared to meeting society's needs.

For each project a detailed chart similar to the example in Table 3 was prepared, showing the production stages involved in the project and the performance factors relating to each stage. The charts were used to guide completion of the matrix presented in item 4.7 as follows.

Table 3.

Performance factors and production stages for each project.

Performance factor	Production stage
• Longer shelf life	• Lyophilization stabilizers • Lyophilization • Colony-forming units (CFU) • Oxygen takeup • Heat stability
• Lower cost	• Lyophilization stabilizers • Lyophilization • Colony-forming units (CFU) • Oxygen takeup • Heat stability
• Resistance to temperature variation	• Lyophilization • Heat stability

Source: Research data.

Example: Project 1 (P1) – production of freeze-dried BCG vaccine by sealing under argon

Objective: Implement the use of argon 5.0 (99.999% purity) for lyophilizer vacuum breaks after bulk lyophilization of BCG vaccine, thus assuring maximum product stability and rigorous control of residual moisture.

Proposals for new projects

This step of the methodology has two phases – technology gaps and technology trends.

Technology gaps: This phase of the study is designed to analyze the matrix completed in the preceding step, identify technology needs not covered by the existing project portfolio, and generate new project proposals.

Table 4 which is based on the matrix presented in Table 3, shows the analysis performed to determine new technology needs. Projects already in progress at the time this methodology was introduced are represented by numbers (P1, P2, …, Pn). Inclusion of an asterisk (P*1, P*2, …, P*n) indicates projects created in accordance with needs identified using the methodology. Triangles highlight significant correlations not covered by existing research and development projects. If the methodology was applied in its complete form, databases on projects in progress at the various research institutions linked to Butanta Institute would also be queried.

Table 4.

Projects covering each correlation of production stages and performance factors.

Performance factors	Production stages
	Priority	Inputs				Innoculum preparation
		Culture medium	Lyophilization stabilizers	Glass vials and synthetic rubbers	Dilutant	BCG growth	Transfer to Erlenmeyer	Culture selection	Bacterial film filtration	Minimization of cell aggregates	Deep fermentation
Longer shelf life	A		2 P1 P*2 P*5	1 P*5
Lower cost	B	3 P*2 P*4 P*5	2 P1 P5 P*2 P*5 P*6								3 Δ P*2 P*4 P*5
Less side effects	B	1 P5 P*1 P*2 P*5	1 P*2 P*4 P*5 P*6
Immunogenicity	A	3 P5 P*1 P*2 P*4 P*5				3 P5 P*1 P*2 P*4	2 P5 P*1 P*2 P*4	2 P5 P*1 P*2 P*4	1 P5 P*1 P*2 P*4	1 Δ P*2 P*4 P*5	3 Δ P*2 P*4
Ease of application	B			1 P*5	2 P*5 P*6
Resistance to temperature variation	A					3 P5 P*2					3 P*2 P*5

Performance factors	Production stages
	Formulation of product	Filling	Lyophilization	Quality testing							Storage at institute	Transportation	Storage at application site
				pH measurement	Colony forming units (CFU)	Oxygen takeup	Heat stability	Residual moisture	Reconstitution of product	Other
Longer shelf life			3 P1 P*2 P*5		1 P1 P5 P*2 P*4 P*5	1 P1 P5 P*2 P*4 P*5	1 P1 P*2 P*5				2 Δ	2 Δ	2 Δ P*8 P*10
Lower cost			2 P1 P2 P*2 P*5 P*6		3 P1 P4 P5 P*2 P*3 P*4 P*5 P*6	3 P1 P3 P4 P5 P*2 P*3 P*4 P*5 P*6	3 P1 P*2 P*5 P*6			1 P*6	2 Δ	3 Δ P*7	2 Δ P*8 P*9 P*11
Less side effects													1 Δ
Immunogenicity			1 Δ P*2 P*5 P*6		3 P5 P*1 P*2 P*4					2 P6 P7			1 Δ P*10
Ease of application									1 P*5 P*6				1 Δ
Resistance to temperature variation			3 P1 P*2 P*5 P*6				1 P1 P*2 P*5 P*6					2 Δ P*7	2 Δ P*7 P*9

Δ = needs not covered by any project in progress; Pn = project in progress n; P*n = new project *n.

Technology trends: The analysis performed to date needs to be supplemented by a second avenue of approach focusing on the trends in technology and science that could make the current production process used in manufacturing Onco BCG totally obsolete or unnecessary, owing for example to the introduction of other treatments that completely cure the diseases treated by Onco BCG. Examples of such trends in the case of Onco BCG include:

• •

  Research on recombinant BCG, using molecular biology to express proteins from other microorganisms that induce an immune response and afford protection against various diseases.

• •

  Research on recombinant BCG with altered production of mycobacterial proteins, which could be tested in the treatment of bladder cancer.

• •

  Research on recombinant BCG producing proteins that modulate the immune system in cancer treatment.

• •

  Research on fermentation routes to substitute cultivation in glass vials.

Closing remarks

This study aimed to develop and present a methodology for selecting the portfolio of research and development (R&D). The methodology presented was developed within the Butanta Institute, one connected with the Health Department of the State of Sao Paulo organization.

First, we sought to analyze the literature on the selection of research projects and development. The literature review showed that the authors address this question from four major perspectives. The first is composed of items that leave the concept of creating value to determine what the best investments in research and development. A second category of articles is worth the portfolio matrix techniques, we classify queries that may be developed along two axes of the matrix. There is also, in a third category, mixed systems, which draw on the concept of value creation and portfolio matrices. However, it is a significant number of authors in the fourth category – methods that use the concepts of risk and return for selecting the portfolio of research to be developed.

The method shown in this paper has some peculiarities in regard to previous studies. Much of the literature on R&D evaluation is based on complex mathematical tools and focuses on financial aspects, characteristics that are not appropriate for non-profit organizations (Schaeffer & Cruz-Reyes, 2016) – which is the case of Instituto Butanta. To adapt to this kind of organization, the proposed method has two main characteristics.

First, it considers the opinion of the group who is responsible for taking decisions. This has been previously pointed by Lee and Kim (2000) as necessary when the projects evaluated have possible impacts for society. Additionally, as in the case of Instituto Butanta, when projects are managed by a group of individuals, it is necessary to consider the group members’ opinions when deciding which project to invest.

Second, it takes into account the opinions of experts when judging the potential value of the projects. According to Liu, Wang, Ma, and Sun (2016), it is more adequate than financial measures when the projects’ potential gains are not necessarily monetary, but instead subjective or parts of larger technological discoveries.

This analysis of the literature showed that most of the theories that apply to private organizations can be applied in public organizations, in particular as regards the issue of improvements in health. However, the concepts of risk and return, when treated from a financial perspective, become not ideal tools for this situation, since the investments in research in the area of health are often risky, and managers cannot judge the best investment for those who bring minor scratches. For example, riskier research can often those related to large and significant improvements, such as curing major diseases.

Based on this literature, we developed and applied a methodology for the selection of research at the Butanta Institute. It was based on a substantial number of hours of interviews to obtain data that provided the basis for the study.

The results showed, first, that the selection of the portfolio of research should first be based on the strategic direction of the institution. Another assumption that is made is based on customers’ perceptions of the product studied – the BCG Onco – about which requirements should be prioritized when it sought to improve the product – what are the critical success factors.

The methodology and its evaluation by the responsible in the Butanta Institute, showed that this method was easy to apply and use, may be used by the institution for the selection of research to be conducted.

Based on the results, we conclude that, although the techniques are based on risk and financial return are of higher value, researchers and R&D managers must consider the strategic and client perspectives when selecting the best portfolio of research.

Conflicts of interest

The authors declare no conflicts of interest.