Scientific publications can be used to identify examples of scientific collaboration, the length of a given collaborative effort, the duration of its effects, and the number of scientific societies that have been created as a result. Among other projects involving this method, in the FP7 Health evaluation project we used bibliometric analysis in combination with network analysis to study the networks of co-publications. By employing this method we assessed the collaborative dimension of health research projects and the effect of EU funding on the formation of the European Research Area.
This method concerns the categorization and analysis of any type of information by its content. PPMI has vast experience with the method – for example, we used content analysis in a report prepared in the framework of the network of experts on the social dimension of education and training (NESET). The information coded using NVIVO software informed a synthesized analysis of FP6 and FP7 projects dealing with different aspects of education inequalities and policy and practice-level recommendations implicated therein.
Theory of change is most often applied in order to design better interventions in light of desired outcomes. Among other projects, we used it in an ex-ante evaluation project in order to provide recommendations on how to improve the quality of EU structural fund investments in Lithuania. We reconstructed every possible outcome for every foreseen intervention showcasing the logical pathways between the existing problems, aims of the action, intervention activities, results, outcomes and impacts. Moreover, we identified specific conditions that need to be in place in order to achieve the desired results and impacts.
A systematic review is a specialist review technique informing evidence-based policy making. It implies searching exhaustively to find all relevant research, assessing the quality of the research and using rigorous techniques to synthesise its findings. In cooperation with Centre for Innovation in the Early Years (VBJK) and Institute of Education (IoE), PPMI used this technique for the first time in 2013 while implementing Assessing childcare in Europe project. Following the systematic review methodology elaborated by the EPPI-Centre at the IoE, research team screened over 19 000 publications and prepared a synthesis of available research evidence on the links between continuous professional development interventions, working conditions and outcomes for children. Some principles of systematic literature reviews were latter applied by PPMI while carrying out a pilot evaluation of ESF-funded ALMP measures in Lithuania and reviewing the research literature in English, French and German on how initial teacher education (ITE) prepares student teachers to deal with diversity in the classroom.
Econometric analysis refers to a broad range of techniques exploring statistical connections between various economic parameters and aiming to estimate their changes under different circumstances. For example, we used econometric modelling to forecast labour supply and demand in Lithuania 2013-2018 as well as the impact of EU support 2014-2020 on GDP, employment, investment and prices.
We use cost-benefit or cost-effectiveness analysis to systematically assess the costs of an intervention and compare them to potential benefits under various scenarios. For example, we used CBA to implement the 3rd interim evaluation of Education, Audiovisual and Culture Executive Agency, evaluation of the operation of European Research Council Executive Agency (2012-2015) and evaluation of the operation of Research Executive Agency (2012-2015).
Counterfactual impact evaluation (CIE) is a set of evaluation techniques allowing to generate robust evidence about the impacts of specific policy interventions. PPMI pioneered these techniques in national context in 2011 with the first CIE of public policies in Lithuania. The evaluation assessed the impact of labour market integration services on employment of persons with disabilities and ex-offenders. To promote further the use of CIEs, in 2013, we analysed the feasibility of and prepared methodological guidance on applying CIE techniques for evaluating impacts of EU structural assistance in Lithuania. In 2015, PPMI completed a CIE on ESF-funded support for the registered unemployed. We analysed administrative data of over 900,000 unemployed persons to estimate the impact of subsidised employment and vocational training on the labour market outcomes for the unemployed, in particular during the economic downturn.
Survey is one of our most important methods. It allows to make quantitative generalisations about opinions of different people, such as policy makers or public policy target groups. For instance, in 2015 PPMI launched a wide survey programme for analysing the perception of the EU and of EU's policies abroad. The surveys were carried out in countries, such as China, India, Mexico, Russia, the US, Brazil, Japan and others. TNS Global implemented the fieldwork. Our experts designed the methodology, developed the questionnaire and analysed the results.
In Vicious circle project, as in number of other research cases, our team applied media analysis for analyzing the images of return migrants in the major news portals of Lithuania.
At PPMI we have applied SNA in evaluation of institutions such as the European Institute for Gender Equality (EIGE) and the European Centre for the Development of Vocational Training (Cedefop) among others. The method has allowed us to explore to what extent European agencies succeed at creating information-sharing networks involving key stakeholders and whether a particular initiative achieves added-value above and beyond that provided by similar initiatives. Finally, by applying SNA in the evaluation of funding for FP7 Health and Marie Skłodowska-Curie actions (MSCA), we explored the density of the networks formed, overall network structure, centrality and other parameters of the European collaborative research networks. This in turn allowed to assess the contribution of European Framework Programmes in developing a single European Research area, as well as to identify the most important project participants and the regions with the highest network concentration.
One of the projects in which we used social media analysis to get systematic empirical insight into how communication about Europe is constructed, disseminated and received via Twitter was perceptions of the EU abroad. By employing this method, we traced global views and reactions to Europe and identified who is reacting and responding to the official EU communication on Twitter and how Europe is perceived and understood in relation to other global powers. With the help of project partners from the National Center for Research on Europe (NCRE) of the University of Canterbury and Statistical Cybermetrics Research Group of the University of Wolverhampton we coordinated the collection of data from international high-profile events using the Twitter Applications Programming Interface (API) through a set of relevant and inclusive keyword queries.
The case study method is among the ones most often applied by our experts. For instance, in a project that analysed and compared how new forms of employment are applied and regulated in the EU-28 and Norway, PPMI has coordinated and implemented 66 case studies in 24 countries that provided a significant part of the evidence presented in the study. The case study unit in this project was an employment or service relationship between employers and dependent employees, or between clients and self-employed individuals. The main sources of information were interviews with various stakeholders, supplemented by data gathered in desk research. Rich qualitative data allowed not only description but also evaluation of the new employment forms.
Interviews are among the most widespread forms of gathering primary qualitative data, which features in mostly every study at PPMI. We have experience with structured to free-form interviews in various languages with diverse stakeholders ranging from those affected by policies to those at their helm. An example of a project that extensively relied on interviews to inform findings was a study for Eurofound 'Sustainable work throughout the life course: national policies and strategies', in the course of which we coordinated 10 national case studies and synthesized findings to give an overview of the different policies and programmes that allow both people to work longer and allow more people to work. In another study on research careers in Europe for EC DG Education and Culture, the PPMI team interviewed various EU-level officials and national stakeholders analysing the perception of research careers in Europe, issues facing individuals with dual-careers, and career restart opportunities in order to inform policy recommendations on how to improve EU-wide research career management.
When implementing evaluations and research studies, we often organise focus group discussions. This is an attractive way to collect data, validate research findings and make policy recommendations. For example, we co-organised focus groups in London, Budapest and Dublin on success factors of awareness raising and fundraising for literacy (European Literacy Policy Network). We also organised around 20 focus groups to discuss potential results and impacts of early childhood education and care on children’s education throughout compulsory schooling (ECEC-ESL study). Another focus group took place in Brussels (external evaluation of ECVET initiative) to reflect on an overall governance structure of ECVET. In focus groups, we gather together a broad range of stakeholders: EU-level and national policy makers, practitioners, experts, researchers, volunteers, project developers, teachers etc.