Observation Wheel - Monitoring and Evaluation / SmartEnCity.eu

Monitoring and Evaluation

Monitoring and evaluation are indispensable parts of planning and implementing smart city interventions. The Cities4ZERO methodology recommends developing an indicator system of Key Performance Indicators (KPIs) at the project level tailored to fit the project objectives, together with an evaluation plan, a data collection plan and a monitoring program, defining a baseline for the interventions as well as the expected performance, thus being able to assess the results after processing all the data. This data will also be valuable if the city has or decides to develop a City Information Open Platform to analyze and visualize all the data for better decision-making.

Right after implementing the interventions, it’s important to run performance tests and start the monitoring period, which ensures data availability for assessment. Once the monitoring period has ended and data has been collected for a significant period of time, all data and performance results are gathered and analyzed – it’s now time to assess the project and its impacts according to the performance indicators fixed before. In the SmartEnCity project, for instance, seven categories are used to assess the project: energy, mobility, life cycle assessment (LCA), ICTs, social acceptance, citizen engagement and economic performance.

The process of assessing the implemented actions needs the involvement of various experts from different fields of evaluation for a correct reporting and comparison. The results then feed into an evaluation report, focusing on all the key aspects, comparing the KPIs values before and after the intervention. At the city level, we can also make a comparison between the current data and the indicators at the city level. The main challenge at this step is to have complete and good quality sets of data. And this means a lot of work in the previous stages.

After the project has come to an end and the results have been evaluated, it’s time to reflect and complete a project review, checking if the project objectives have been fulfilled. In SmartEnCity, the project objectives were split into four categories: technical, environmental, social and economic objectives. Through gathering feedback from the key stakeholders, the local partnership can then validate if the interventions were aligned with the Strategic Plan and Action plan at the city level, and to what extent they are successful, exploitable and replicable. What was successful? What might have been done better? Why did some indicators differ from what was expected? From this reflection, key barriers, success factors, regulatory input and potential exploitable results can be extracted for future projects in the city.

How to use & tips

Enable monitoring and tracking project deployment performance. Through indicators at district/city scales, you will be able to track to what extent your goals have been achieved.
It’s not the same to assess mobility actions, district level actions or building renovation actions. The set of KPIs defined for the intervention assessment must focus on the objectives set for this intervention as well as take into account the specificities of this particular intervention.
Check what you want to calculate and what you need for this. In some cases, the calculation of a set of KPIs means that monitoring devices need to be installed. These needs have to be identified in advance to include their installation and verification as one of the subtasks in the schedule, through a specific monitoring plan. In parallel, a plan for data management must also be generated.
It is important to stress that not all indicators are based on data gathered from equipment, but also from the citizens, tenants and service users. Data can be collected, for example, through surveys and needs to be analysed accordingly. Make measurable what cannot be measured!
It is useful that, for some calculations, the experts involved are certified on an evaluation methodology (as the International Performance Measurement and Verification Protocol IPMVP, for example) to assure the quality of the work and the precision of the results.
If you are overwhelmed by the number of technical deliverables on monitoring and evaluation that SmartEnCity has produced (see below), just start by following the general Cities4ZERO methodology (focus especially on Steps 11, 13, 14 and 15) and then see what kind of information and guidance you need for continuing your learning process.

Don'ts

Don’t be afraid of unexpected results from monitoring and evaluation. It’s clear that a positive assessment supports the interventions developed in case they have been correctly finished and good evaluation results obtained. But even if the results are not as expected, a correct assessment is still valuable as it is the way to ensure better results in future interventions. We can only improve what we know!

Links to more information

Monitoring and Evaluation

Twitter