Next Article in Journal / Special Issue
Artificial Intelligence and Robotics in Smart City Strategies and Planned Smart Development
Previous Article in Journal
Sustainable City: Energy Usage Prediction Method for Electrified Refuse Collection Vehicles
Previous Article in Special Issue
IoT-Enabled Smart Sustainable Cities: Challenges and Approaches
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Review

Review of Smart City Assessment Tools

Institute of Systems and Robotics, Department of Electrical and Computer Engineering, University of Coimbra, 3030-290 Coimbra, Portugal
*
Author to whom correspondence should be addressed.
Submission received: 26 August 2020 / Revised: 21 September 2020 / Accepted: 24 September 2020 / Published: 30 September 2020
(This article belongs to the Special Issue Feature Papers for Smart Cities)

Abstract

:
Today’s cities are estimated to generate 80% of global GDP, covering only about 3% of the land, but contributing to about 72% of all global greenhouse gas emissions. Cities face significant challenges, such as population growth, pollution, congestion, lack of physical and social infrastructures, while trying to simultaneously meet sustainable energy and environmental requirements. The Smart City concept intends to address these challenges by identifying new and intelligent ways to manage the complexity of urban living and implement solutions for multidisciplinary problems in cities. With the increasing number of Smart City projects being implemented around the world, it is important to evaluate their strengths and weaknesses for their future improvement and evolution track record. It is, therefore, crucial to characterize and improve the proper tools to adequately evaluate these implementations. Following the Smart City implementation growth, several Smart City Assessment tools with different indicator sets have been developed. This work presents a literature review on Smart City Assessment tools, discussing their main gaps in order to improve future methodologies and tools. Smart City Assessment can deliver important performance indicators monitoring for the evaluation of multiple benefits for different actors and stakeholders, such as city authorities, investors and funding agencies, researchers, and citizens.

Graphical Abstract

1. Introduction

Today’s cities face significant challenges, such as increasing population, pollution, congestion, resource usage, lack of adequate physical and social infrastructures, sustainable economic growth, and increasingly stricter energy and environmental requirements [1,2].
According to the United Nations [3], more people live in urban areas than in rural areas. In 1950, only 30% of the world’s population was living in an urban environment, but in 2018, this number increased to 55%. In 2030, it is expected to increase to about 60% [4] and by 2050, the projections point to about 68% of the world’s population being urban [3]. The United Nations [3] states that “the growth in the urban population is driven by overall population increase and by the upward shift in the percentage living in urban areas”. Together, these two factors can add about 2.5 billion to the world’s urban population by 2050, with almost 90% of this growth happening in Asia and Africa [3]. The urbanization process is occurring at different rates, being considerably faster in developing regions than in the developed ones. Africa is expected to be the fastest urbanizing region and according to Joint Research Centre [5], in the last 25 years, the urban population has more than doubled in almost all countries across sub-Saharan Africa, although meanwhile, in many parts of North America and Europe, the urban population has been declining. In 2016, there were, globally, about 512 urban centers with at least 1 million inhabitants and 31 megacities with over 10 million inhabitants [6]. By 2030, these numbers are projected to increase to 662 urban centers and 41 megacities, most of which in developing regions [6].
Nowadays, cities are estimated to generate 80% of all economic growth [7], covering only about 3% of the land, but producing about 72% of all global greenhouse gas emissions [8]. All the continuous population growth, and especially in urban areas, put a lot of challenges in the design of cities and exceptional sustainability challenges, both on infrastructures and the environment [9].
The development challenges of cities have attracted many interdisciplinary fields of study. Computer Science is one of the multiple fields addressing city development challenges. The interest of Computer Science began at the end of the 20th century when cities were idealized to become a network of computers and other electronic based devices [10]. Nowadays, cities are composed of even more complex systems in constant evolution, in which the combination of multiple research areas is needed in order to overcome the arising endless challenges [10]. In order to make cities and communities more inclusive, safe, resilient, and sustainable, as per the “Sustainable Development Goal 11” of the 2030 Agenda for Sustainable Development [11], it is crucial to understand the key trends in the future of urbanization [4], and to evaluate city growth development and implementation measures to fulfill the multi-dimension human and sustainable requirements.
The “Smart City” concept is intended to address the referred challenges by identifying new and smart ways to manage the complexity of urban living and implement solutions for multidisciplinary problems ranging from energy consumption, resource management, environmental protection, security, quality of life, the efficiency of urban operation, and the availability of a wide variety of services [9,12,13].
The estimated number of smart city implementations varies depending on the definition and extent of the elements included in the analysis. A Navigant Research global market report [14] estimated that in 2017, there were more than 250 smart city project implementations in 178 cities worldwide. Europe is the top-performing geographical area with 12 cities ranking among the top 25 smart cities ranked by the IESE Cities in Motion Index 2018 [15]. Europe is aiming to have about 300 smart cities by the end of 2020 and India plans to build about 100 smart cities by 2022, impacting a population of almost 1 billion people [16].
“Smart City” as a new research concept began in 1992 with the book entitled “The Technopolis Phenomenon: Smart Cities, Fast Systems, Global Networks” [10,17]. Since then, this research area has not only attracted different research areas, but also many stakeholders, from governmental organizations to industries and social society, supporting the growing interest of the “Smart City” concept, within the global sustainable development framework.
According to Mora [10] the “Smart City” research established itself as a new scientific field in the year 2009, but despite all the growing number of publications, the concept is far from having a clear and established definition. Many definitions for the concept have been developed, but so far none has been commonly accepted by the scientific community. It is common to see different interpretations according to the context of the publications [13,18]. The absence of such a definition has been pointed out in several publications, namely [1,10,19]. According to the bibliometric analysis of the literature published between 1992 and 2012 performed by Mora [10], there is a tendency among publications to differentiate themselves, being difficult to acquire a common agreement.
Some publications mostly see Smart City as a technological innovation whereas others see it as social innovation. There is the idealization of a Smart City being supported by a technological combination of different IoT devices forming a wireless sensor network (WSN), for numerous control and monitoring applications fields namely for utilities, security, asset tracking, and smart metering [20]. The bibliometric analysis of the literature developed by Mora [10] points out that “researchers seem to agree in picturing the smart city as an urban environment in which an ICT-driven approach to urban sustainability is activated”. The same analysis states that there are although different research paths being followed. The two most cited groups of publications describe the smart city as a more techno-centric approach and the other, one of the most cited and influential documents in the area published between 1992 and 2012, Giffinger [21], as a balanced combination of human, social, cultural, economic, environmental, and technological factors [10]. Technology should be seen as a means to achieve the outcomes of a Smart City implementation according to the needs of its location (environment, energy, people, business, governance, etc).
The Smart City concept is still evolving and there is not yet a commonly accepted worldwide definition, but it is quite clear that Smart Cities leverage information and communication technologies to enhance service levels, citizen well-being, sustainability, and economic development. Expanding the Smart City concept will, for instance, improve the existing infrastructures, increase the digitalization and integration of facilities and services, expand the use of collaboration tools and automated service management platforms, increase the safety of citizens and transportation, making cities more attractive for stakeholders, and living up to citizens’ expectations. According to an OECD recent report [22], when well connected with inclusive growth objectives, Smart City implementations can, besides many other advantages, offer powerful tools to support the shift from in-person to remote service delivery, mitigating the fallout of the crisis on urban residents and businesses, including the most vulnerable ones, and empower new forms of local governance. However, the same report states as a possible threat to society the “increased inequality among digitally marginalized groups” and “possible abuse of citizen data, privacy and safety”.
Different and distinct Smart City concepts and architectures have emerged during the past decades, but all of them visualize it as being composed of different inter-related components, such as data, economy, people, governance, mobility, environment, and living [13,23]. All of the components are related to different aspects of urban living, such as communication, industry, education, e-democracy, logistics and infrastructures, efficiency and sustainability, security, and quality [23]. All this diversity of components and related aspects has attracted multi-disciplinary approaches and science fields to the Smart City developments, which has contributed to the enrichment of scientific knowledge and increase and scale-up of innovative technology developments.
Despite all the different research paths and lack of commonly accepted definition in the trending Smart City concept, one point is clear; the impact evaluation of Smart City implementations is extremely valuable and strongly needed. Every impact evaluation has the principle of permitting the characterization of achieved levels of performance according to a certain target goal. This is crucial for the planning of the city further developments and upgrades, to assess the strengths and weaknesses, for comparison with other implementations and to inform the different interested stakeholders about the implementation status.
This paper is focused on the assessment tools for Smart City implementation evaluation, which can also be mentioned as “Smart City Assessment (SCA)”. In the literature, it is possible to find different names for such evaluation tools, such as “rankings”, “ratings”, “indicators”, “key performance indicators (KPI)”, “indexes”, “frameworks”, and “assessment tools”. In the scope of this publication, the mentioned “Smart City Assessment (SCA)” will refer to an assessment process or tool that can evaluate the performance of a certain indicator in the scope of a Smart City concept implementation.
The remainder of the paper is structured as follows: Section 2 presents the method used for the literature review. Section 3 and Section 4 presents the results of the literature review in terms of the smart city concept and the smart city assessment, respectively. Then, the existing research and gaps in SCA are assessed in Section 5. Finally, Section 6 discusses the findings and presents the main conclusions.

2. Literature Review

In the literature, it is possible to find “Smart City” analogous concepts referred to in relatively different ways. Several different wordings appear, with “Green City”, “Sustainable City”, and “Intelligent City” being the most common [10,18,24,25,26,27,28]. Using the “Web of Science” search engine with the string: AK = ((“smart” OR “green” OR “sustainable” OR “intelligent”) AND (“city” OR “cities”))], it was possible to find 8,518 publications with the referred keywords since 1992. The searches on Web of Science (webofknowledge.com) were made on the 31 July 2020, and revealed that the number of publications has been constantly increasing over the years reaching a total of 1599 in 2019. In 2020, about 730 publications had already been published.
The wording is also quite diverse when referring to tools or studies that can provide an evaluation of Smart City implementations. In order to find and refine the search of articles related to the subject of this article, the following string was inserted in the search engine: AK = ((“smart” OR “green” OR “sustainable” OR “intelligent”) AND (“city” OR “cities”) AND (“tool” OR “assessment” OR “indicator” OR “key performance indicators” OR “KPI” OR “framework” OR “toolkit” OR “index” OR “rating” OR “ranking”)). In the last 20 years, a total of 372 publications were found, from which 298 (80%) were published in the last five years (since 2015). The years with the greatest number of publications were 2017 and 2018, with respectively 63 and 62 yearly publications. In 2020, there are already 41 publications captured by the search string. The countries with most publications are the Popular Republic of China, with about 79 publications (21%), Spain with 34 (9%), and Italy with 31 (8%). A more refined search was done to this preliminary screening of results, based on the publication’s number of citations and impact factor. Afterward, to further refine the selection an abstract reading was conducted that was able to identify the most adequate publications in the scope of this paper. This methodology was believed to identify the results that are most satisfactory possible for the review. Other publications could be read and cited, but with potentially less added value to the final result. The conclusions regarding the existing tool gaps were collected from the analyzed publications, preferring the ones that clearly stated a satisfactory and reasonable evaluation methodology.
It is clear by the demonstrated increasing number of articles being published over the years, that there is a growing research interest for the subject. This paper intends to present an updated review on existing SCA tools and their main gaps, being a starting point for those initiating their research on the subject. With SCA being a nascent field with a growing potential to evolve [13], having a good review of the strengths and weaknesses of existing tools can be important for future developments.

3. Smart City Concept

As already mentioned, in the literature, different expressions can be found referring to similar concepts of “Smart City”. The use of combined wordings can also be found, suggesting or definition new concepts, sometimes for differentiation purposes, like for instance the “Smart Sustainable City” concept [29]. Although sometimes not exactly being in the same focus, the use of all these different expressions has been creating terminological confusion [26,30,31]. Today, “Smart City” is probably the most widespread and accepted wording among citizens, media, investors and companies, and public authorities. It is being used by public authorities and the business sector, as it is a buzz word understandable by most of the targeted audiences [29].
According to Grabys [32], references to Smart City similar concepts can be traced back from the 1960s, when the literature refers to “informational or cybernetically planned cities” and after that, in the 1980s, when references appear to the “networked cities” or “computable cities”. The Smart City concept was introduced in 1994 [33]. Later, by 2010, the European Union (EU) began to use the “smart” wording to refer to sustainable projects and actions related to the urban environment [33]. With the appearance of Smart City project implementations supported by the European Union (EU), the interest regarding the subject began to increased significantly [34].
The literature review performed in 2014 by Mosannenzadeh [35], showed that the concept of Smart City was being developed in three main areas: Academic, Industrial, and Governmental, creating different points-of-view by each domain area. At the time, it was pointed out that there was no common definition for the concept and that the disparity of wordings used in different definitions was a sign of controversy in the concept [35]. Nowadays, and despite all the growing number of publications since then, there is still not yet a common and acceptable definition for the Smart City concept, although researchers seem to agree in the significant contribution of the use of technologies for urban development [10,35].
However, there seems to be huge differences and confusion between the concept definitions and what actually is being done in real-world implementations and even in the scope of the main definitions for most reputed international institutions. The literature refers to “Sustainable City” as being the umbrella where other concepts like “Smart City” are under. “Sustainable City” is referred as including a harmonized integration of social, economic, environmental, and institutional aspects, whereas “Smart City” is pointed out as having the main goal of improving the sustainability of cities through the application of ICT, the collaboration of key stakeholders, and integration of domains [36]. On the other side, other publications refer that one of the many attributes or dimensions of Smart Cities is “Sustainability” [9,13,28,37], and that “a general goal of Smart Cities is to improve sustainability with the help of technologies” [34].
The vision of Smart Cities according to the European Commission (EC) is beyond the use of ICT. It is seen as interactivity between different infrastructures with the main goal of providing benefits in all sectors. According to the EC [38] Smart Cities are “Cities using technological solutions to improve the management and efficiency of the urban environment”. The EC [38] refers to Smart City as “a place where traditional networks and services are made more efficient with the use of digital and telecommunication technologies for the benefit of its inhabitants and business”. It is also stated that “a smart city goes beyond the use of information and communication technologies (ICT) for better resource use and fewer emissions. It means smarter urban transport networks, upgraded water supply and waste disposal facilities and more efficient ways to light and heat buildings. It also means a more interactive and responsive city administration, safer public spaces and meeting the needs of an aging population” [38].
IEEE also has this broader understanding of the Smart City concept where technology is seen as an enabler for a better quality of life and to decrease environmental impacts [34]. IEEE Smart Cities Initiative refers that “a Smart City brings together technology, government and society and includes but is not limited to the following elements: A smart economy, Smart energy, Smart mobility, A smart environment, Smart living, Smart governance” [39].
The United Nations, on their “Smart Cities and infrastructure report” [40] also refer that there is no standardized commonly accepted definition for Smart City. In the report [40], it states, as a reference, the definition provided by the International Telecommunication Union (ITU) after analysis in 2014 about 100 different definitions, and presented at the ITU-T Y. 4900 recommendations [41]: “an innovative city that uses information and communication technologies (ICTs) and other means to improve quality of life, the efficiency of urban operation and services, and competitiveness, while ensuring that it meets the needs of present and future generations with respect to economic, social, environmental as well as cultural aspects”. Although this definition provided by ITU [41] is actually for “Smart Sustainable City”, the United Nations use it as a reference when mentioning to the definition of Smart City, stating also that “Governments and stakeholders need to work together to develop a common understanding of what Smart City means in their specific national and city-level contexts” [40]. Nowadays, the United Nations refers to the wording “Sustainable Cities and Communities” as a means to “make cities and human settlements inclusive, safe, resilient and sustainable” [42].

4. Smart City Assessment

As referred by Sharifi [13] Smart City Assessment (SCA) “is a nascent field with much potential for future development”. There is limited research analyzing SCA tools that Sharifi [13] attributes to the fact that SCA is a relatively new field of research and practice. The evaluation of Smart City implementations should be seen as critical, in order to historically document their strengths and weaknesses, for the scope of future improvements and to inform interested stakeholders about the level achieved in different target goals. SCA tools can also be used to present city-rankings, revealing the best (and the worst) places for certain activities, which is pointed out by literature to be a central instrument for assessing the attractiveness of urban regions [21,43,44]. The SCA and resulting rankings can thus be used by cities to improve their position in the competition with other regions. A highly reputed city ranking helps to improve the international image of a city and can, therefore, play a central role in its marketing strategy [21]. Giffinger [21] mentioned a detailed analysis and comparison of 10 German rankings. The city rankings referred to in (Giffinger et al., 2007), which have been published by the magazines focus, capital, impulse, and Bizz between 1994 and 2003, point out the following benefits for city rankings:
  • City rankings draw public attention to major issues affecting the quality of life of the citizens and promote healthy competition;
  • City rankings stimulate a broad discussion on regional development strategies;
  • Regional actors are forced to make their decisions transparent and comprehensible;
  • Positive changes are also registered outside the region;
  • The results in detail may initiate the learning effects of local actors;
  • On the other hand, the same report identifies the handicaps of city rankings;
  • City rankings tend to neglect complex interrelations in regional development;
  • The discussion is mainly focused on the bare rank;
  • Long-term development strategies may be threatened;
  • Existing stereotypes may be strengthened.
Based on the literature review presented by Sharifi [13], the next Table 1 presents the benefits of SCA for the different stakeholder’s categorization.
Table 1. perfectly summarizes the existence of multiple benefits for the SCA activities and that they may offer different perspectives for each one of the actors and stakeholders involved.

5. SCA Research Review and Identified Gaps

The assessment of citie’s smartness has received much more attention in recent years, but very few studies have analyzed SCA tools and their strengths and weaknesses [13]. According to Sharifi [13] there are two major approaches to the study of SCA tools: One, focused on providing an overview of the tools, and another one involving more detailed analyses of the tools to better understand their thematic focus and the typology of their indicators.
Albino [28] provided a brief overview of frameworks, rating systems, and a number of indicators, for existing smart city assessment until 2015. The analysis is performed in a more quantitative overview of the indicators used with very few qualitative details, thus it is not possible to evaluate the potential gaps of the presented assessment tools. The tools and frameworks presented are, for instance, smart ranking systems developed by the University of Vienna, the Intelligent Community Forum’s Smart 21 communities, the Global Power City Index, the Smarter Cities Ranking, the World’s Smartest Cities, the IBM Smart City, and the McKinsey Global Institute rankings.
The main objective of the CITYkeys European project [55] that started in February 2015 and ended two years later (2017) was to develop and validate a framework for transparent performance evaluation, including the definition of key performance indicators (KPIs) and guidelines for data collection and a performance measurement system prototype. The CITYkeys project consortium performed a survey of existing KPI systems for project evaluation in order to create an inventory of existing assessment frameworks or tools. The inventory included relevant H2020 and FP7 projects, as well as relevant projects from the countries of the three research organizations involved: Finland, the Netherlands, and Austria. It also included the status of relevant initiatives that were under development at the time, like the smart city framework of CEN/CENELEC and the Smart City Information System [56]. A total of 43 indicator frameworks were scanned for indicators that could be related to the CITYkeys pre-selected subthemes and thus potentially be used for the CITYkeys framework. Based on this inventory analyzed on Neumann [57], it is reported that in general terms, the analyzed frameworks suggested that the availability of the KPI was saturated. Neumann [57] refers that numerous indicators that existed at the time and where being used across different sectors in cities and districts in Europe, as well as globally. It also reported the following gaps in terms of indicator availability: Multilevel governance, education, employment, scalability, and replicability. The report suggests that there was a significant variation in the coverage of the different subthemes, including for instance the ”Energy and Mitigation” and ”Environment” subthemes [57]. It should be noted that the gap analysis was carried out only by linking subthemes (and their associated indicators) of existing frameworks with the selected subthemes of the CITYkeys framework, thus not being an exhaustive analysis of the existing indicators individually.
There have been also several approaches to standardize the indicators from which the frameworks or rankings can provide an assessment for Smart City implementation. Recently, Huovila [58] provided a quite extensive comparative analysis of existing standardized indicators for Smart City Assessment. Such publication performs an evaluation of seven recent existing standardized frameworks with 413 indicators. Table 2 shows the standards analyzed and their main categories addressed.
The analysis provided by Huovila [58] indicated that there is a lack of balance between the different indicators, namely between the indicators related to sustainability and smartness. There is also a strong emphasis on the smartness indicators [13,59]. Huovila [58] affirms that ISO, ETSI, and Sustainable Development Goal 11 are well documented, but the ITU standards have a short definition of the indicators, which may lead to different their calculation methods, leading to different results in each assessment. It is even stated, when referring to the lack of documentation, that “without provision of sufficient metadata and guidance, they might have limited implementation in practice and lack reliable cross-city comparability” [58].
The assessment tools use different indicators. These indicators should be selected taking into consideration multiple complex factors, but the developers of SCA tools need to take into account that the methods should follow simple and applicable guiding principles in order to approach city managers, that are interested in the big picture and not on technical or methodological details [58]. Huovila [58] refers that “further developments of indicator frameworks need to be embedded in the analysis of cities needs”. This is an important note since, sometimes, the indicators used in the cities evaluation are not in line with the city goals. This can, for instance, contribute to a poor evaluation of future city developments, that are evaluated according to indicators that have nothing to do with what was idealized by the city managers or planers.
Ahvenniemi [34] performed an analysis of a set of 16 SCA tools. In the analysis, it is stated that “while environmental sustainability is an essential target of smart cities environmental indicators are clearly underrepresented in the smart city frameworks analyzed”. Ahvenniemi [34] recommends that Smart City assessment “should not only use output indicators that measure the efficiency of deployment of smart solutions but always also impact indicators that measure the contribution towards the ultimate goals such as environmental, economic or social sustainability”.
A more detailed critical review about SCA tools was recently developed by Sharifi [13] where 34 selected SCA tools were analyzed was presented. This publication presents a rather interesting literature review about the Smart City concept and existing research on SCA tools, where the author identifies two main gaps: The first is that the majority of the studies conducted so far have mainly analyzed a limited number of tools, and the second is that the existing research is mainly focused on thematic focus and indicator type, neglecting many other important issues related to assessment tools [13]. Table 3 presents the 34 selected SCA tools analyzed by Sharifi [13].
The analysis performed by Sharifi [13] is quite extensive and presents the results of his analysis of all the 34 SCA tools according to the following sub-sections:
  • Comprehensiveness:
    a.
    The extent of inclusion of indicators in the selected tools;
    b.
    Distribution pattern of indicators
  • Stakeholder engagement;
  • Context sensitivity;
  • Strategic needs;
  • Uncertainty management;
  • Interlinkages and interoperability;
  • Temporal changes;
  • Flexibility;
  • Feasibility;
  • Presentation and communication of the results;
  • Action plans.
Several strengths and weaknesses are pointed out to each one of the SCA tools according to the main purpose of the previous sub-sections. In terms of weaknesses, which can be considered a gap in the assessed SCA tools, it is worth mentioning at least the following [13]:
  • There is a lack of balanced distribution of indicators;
  • The most tools have not taken measures to engage stakeholders in their development and implementation processes;
  • The majority of the selected tools do not consider locally specific conditions;
  • Overall, the issue of feasibility has not been well addressed across the tools;
  • Only about 25% of the tools provide recommendations on linking results to action plans.
After evaluating the selected SCA tools, using the developed framework, Sharifi [13] states that in “overall, limited success has been achieved and several improvements are needed as the tools continue to evolve over the coming years”. The author also affirms that in order to anticipate future changes and to avoid being overwhelmed by constantly changing dynamics of urban systems, SCA tools should consider the use of separate and scenario-making techniques for dealing with future uncertainties [13].
According to Sharifi [13] extensive analysis “more research is needed to gain further details about the suitability of individual SCA tools for guiding cities towards smartness” and that “such research should be informed by empirical findings from case studies to better understand the complexities of smart city assessment”. This statement shows the importance of testing the SCA tools in real scenarios through the evaluation of real-world Smart City implementations.

6. Discussion and Conclusions

With an increasing world population and the shift towards urban living environment [3], cities face multidisciplinary challenges that require innovative and intelligent solutions. Smart City implementations can help to overcome the limitations of traditional urban development models, improving Citie’s sustainability and resilience.
Even if no consensual definition for the Smart City concept is available, all of the existing ones have at least the same common goal: Improve the quality of urban life [95]. There are limited concept definitions provided by international organizations, and researchers and SCA tool developers tend to follow different paths. An international agreement on such definitions should be provided by an agreement between international sustainable development organizations (e.g., United Smart Cities program from UNECE – United Nations) and/or international standardization bodies (e.g., ISO, IEC and ITU).
The Smart City concept can be seen as a balanced combination of human, social, cultural, economic, environmental, energy, and technological developments, commonly integrated for the same purpose, aiming to overcome the main challenges of urban living. Technology should be seen as a means to achieve the outcomes of a Smart City implementation according to the needs of its location (environment, energy, people, business, governance, etc). We see Smart Cities in a broader approach, with the use of technologies facilitating the interaction between different utilities, infrastructures, services, and systems, with the main purpose of supporting the sustainable use of resources and increasing city administration efficiency, benefiting inhabitants, governance, and economic growth.
Smart City Assessment tools are important for the proper evaluation of Smart City implementations, bringing benefits for all the different involved stakeholders [13]. Such benefits can be very diverse, namely: Improvement of the international city image [21,43,44,45,46], prioritization of funding allocation [45], development of new strategies for performance improvement [47], and motivation for engagement in development activities and to communicate desires and priorities. The tools should enable the evaluation of the implementations regarding their impacts on the local objectives, but also on global established goals [45,46,53].
The main purpose of the Smart City Assessment is to give feedback and guidance for decision-making, enabling the assessment of whether the implementations are proceeding towards the wanted direction [34]. Many of the existing SCA tools are nowadays mainly used for promotional purposes and very few for an evaluation of what actually should be done in order to increase the performance of future developments.
The diverse approaches followed by different SCA tools can be partly explained by the fact that there is no consensus on the actual definition of the Smart City Concept [18]. This fact may introduce confusion into the main stakeholders involved. Even the main international organizations and standard bodies sometimes use differentiated wordings when addressing similar concepts. Therefore, an international effort should be made so that a common definition for the different concepts can be agreed.
This paper carries out an updated literature review of the main different approaches of the Smart City concept and of the different SCA tools being developed in the last decades. The different perceptions of the Smart City concept are somehow generating distinct SCA tools that exacerbate different components and aspects, according to the perspectives theirs developers. The following gaps, regarding the available SCA tools, were identified:
  • There is a lack of balance distribution of indicators;
  • The great majority of the SCA tools are static assessments or snapshots;
  • They present limitations when comparing cities with different scales;
  • In general, they do not measure or evaluate the impacts on the specific local needs or goals of the implementation;
  • The engagement of stakeholders in Smart City developments and implementation processes is not evaluated;
  • They do not evaluate the contribution to the United Nations Sustainable Development Goals (SDG) or other worldwide targets/goals;
  • The feasibility of the implementations is not evaluated.
The development of a single tool, able to overcome all the mentioned gaps of the existing SCA tools, is considered to be an quite ambitious task. Efforts should be made to at least develop an assessment tool capable of performing an evaluation track record, with, for instance, automated acquisition of several important indicators, and the possibility to compare implementations with different scales (namely in terms of area and financial dimensions).
The lack of balance between the indicators can be solved by the combination of different indicators sets already defined by the literature, and the inclusion of several others of which it is possible to obtain. Sometimes, this unbalance is related to the SCA tool developers that would like to emphasize specific outcomes of the implementations.
The proliferation of wireless sensor networks (WSN) capable of acquiring data in an automated and synchronous way [20] could provide real-time or quasi-real-time data for several of the indicators that compose a SCA tool. This would contribute to a constant update on the assessment of certain domains (i.e., water and energy consumption) of a Smart City implementations, that could provide insights about their evolution track record, for better definition of future improvements and actual verification of their efficiency. In certain domains, this constant monitoring of data could also provide relevant data to better evaluate the feasibility of the implemented measures.
The literature review presented in this paper provided insights about the existing gaps in the available SCA tools. Future research needs to be carried out on the analyses of indicators sets that could be included in a SCA tool, capable of overcoming several of the existing identified gaps. Also, it will be useful for the evaluation that, as much as possible, several of these indicators could be automatically acquired using automated information systems like sensing and metering devices and other data collection platforms, namely smart energy meters, smart water meters, outdoor air quality stations, traffic monitors, and waste management platforms. The authors are researching the development of such a SCA tool that can create added value in relation to the existing ones, contributing to overcome some of the existing identified gaps. The SCA tool under development will not only provide an overall rating of Smart City implementations, with all the multiple benefits for different stakeholders, but also will be able to provide different size implementation comparisons, historical overview and evolution, and different ratings for the main involved stakeholders, using a multi-criteria/multidimensional analysis. The different ratings will not only establish an overall rating of the city, but also a rating reflecting the different stakeholder’s points of view (i.e., city authorities, investors, researchers, citizens). This framework is believed to contribute significantly to the knowledge, evaluation, and empowerment of the stakeholders, increasing their willingness to actively contribute and participate in future implementations. Such an SCA tool can also be used in validation methodologies of both large- and small-scale pilot implementations, since it will consider the scale of the implementation.

Author Contributions

C.P.: Conceptualization, Methodology, Dala collection and assessment, Writing—original draft, Writing—review & editing. P.M.: Conceptualization, Writing—review & editing, Supervision. A.T.d.A.: Writing—review & editing, Supervision. All authors have read and agreed to the published version of the manuscript.

Funding

Support for this research was provided by OE - national funds of FCT/MCTES (PIDDAC) under project UID/EEA/00048/2019.

Conflicts of Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

References

  1. Chourabi, H.; Nam, T.; Walker, S.; Gil-Garcia, J.R.; Mellouli, S.; Nahon, K.; Theresa, A.P.; Scholl, H.J. Understanding smart cities: An integrative framework. In Proceedings of the 45th Hawaii International Conference on System Sciences, Maui, HI, USA, 4–7 January 2012; pp. 2289–2297. [Google Scholar] [CrossRef]
  2. British Standards Institution (BSI). The Role of Standards in Smart Cities; Issue 2 (August 2014). Available online: https://www.bsigroup.com/LocalFiles/en-GB/smart-cities/resources/The-Role-of-Standards-in-Smart-Cities-Issue-2-August-2014.pdf (accessed on 26 June 2020).
  3. United Nations. Department of Economic and Social Affairs. In World Urbanization Prospects—The 2018 Revision (ST/ESA/SER.A/420); United Nations: New York, USA, 2019; Available online: https://population.un.org/wup/ (accessed on 2 June 2020).
  4. United Nations. The World ’s Cities in 2018. World’s Cities 2018—Data Booklet (ST/ESA/SER.A/417). 2018. Available online: https://www.un.org/en/events/citiesday/assets/pdf/the_worlds_cities_in _2018_data_booklet.pdf, (accessed on 8 April 2020).
  5. Joint Research Centre; Freire, C.; Corban, S.C.; Ehrlich, D.; Florczyk, A.; Kemper, T.; Maffenini, L.; Melchiorri, M.; Pesaresi, M.; Schiavina, M.; et al. Atlas of the Human Planet 2019, EUR 30010 EN; Publications Office of the European Union: Luxembourg, 2019; ISBN 978-92-76-17418-9. [Google Scholar] [CrossRef]
  6. United Nations. The World’s Cities in 2016. Available online: https://www.un.org/en/development/desa/population/publications/pdf/urbanization/the_worlds_cities_in_2016_data_booklet.pdf (accessed on 8 April 2020).
  7. The World Bank. Urban Development. Available online: https://www.worldbank.org/en/topic/urbandevelopment/overview. (accessed on 26 June 2020).
  8. European Commission. 100 Climate-Neutral Cities by 2030—By and for the Citizens, Interim Report of the Mission Board for Climate-Neutral and Smart Cities; EC: Brussels, Belgium, 2020. [Google Scholar]
  9. Akande, A.; Cabral, P.; Gomes, P.; Casteleyn, S. The Lisbon ranking for smart sustainable cities in Europe. Sustain. Cities Soc. 2019, 44, 475–487. [Google Scholar] [CrossRef]
  10. Mora, L.; Deakin, M. Untangling Smart Cities, Chapter 3; The First Two Decades of Research on Smart City Development; Elsevier: Amsterdam, The Netherlands, 2019; pp. 57–87. [Google Scholar] [CrossRef]
  11. United Nations. United Nations Sustainable Development Goals Knowledge Platform. 2020. Available online: https://sustainabledevelopment.un.org/sdgs (accessed on 2 April 2020).
  12. Bibri, S.E.; Krogstie, J. Smart sustainable cities of the future: An extensive interdisciplinary literature review. Sustain. Cities Soc. 2017, 31. [Google Scholar] [CrossRef]
  13. Sharifi, A. A critical review of selected smart city assessment tools and indicator sets. J. Clean. Prod. 2019, 233, 1269–1283. [Google Scholar] [CrossRef]
  14. Navigante Research. More than 250 Smart City Projects Exist in 178 Cities Worldwide. Available online: https://guidehouseinsights.com/news-and-views/more-than-250-smart-city-projects-exist-in-178-cities-worldwide (accessed on 26 June 2020).
  15. IESE Insight. New York, London and Paris Firmly Established as the Smartest Cities. Available online: https://www.ieseinsight.com/doc.aspx?id=2124&ar=&idi=2&idioma=2 (accessed on 26 June 2020).
  16. European Commission. Developments and Forecasts on Continuing Urbanisation. Available online: https://ec.europa.eu/knowledge4policy/foresight/topic/continuing-urbanisation/developments-and-forecasts-on-continuing-urbanisation_en (accessed on 26 June 2020).
  17. David, R.W.S.; Gibson, V.; Kozmetsky, G. The Technopolis Phenomenon: Smart Cities, Fast Systems, Global Networks; Rowman & Littlefield Publishers, Inc: Lanham, MD, USA, 1992. [Google Scholar]
  18. Sharifi, A. A typology of smart city assessment tools and indicator sets. Sustain. Cities Soc. 2020, 53, 101936. [Google Scholar] [CrossRef]
  19. Alkandari, A.; Alnasheet, M.; Alshekhly, I.F.T. Smart Cities: Survey. Adv. Comput. Sci. Technol. Res. 2012, 2, 79–90. [Google Scholar]
  20. Vieira, A.; Patrão, C.; Gonçalves, T.; Monteiro, P.; Marcelino, S.; Filipe, E.; Damasceno, J.; Pereira, H.; Sousa, C.; Oliveira, P.; et al. WaterAMI—Water Automated Metering Infrastructure Based on an Energy Aware Wireless Mesh Network Communication Protocol. In Sustainable Energy for Smart Cities. SESC 2019. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering; Springer: Cham, Switzerland, 2020; Volume 315, pp. 209–220. [Google Scholar]
  21. Giffinger, R.; Fertner, C.; Kramar, H.; Kalasek, R.; Pichler-Milanović, N.; Meijers, E. Smart cities—Ranking of European medium-sized cities. Centre of Regional Science, Vienna University of Technology. October 2007, pp. 1–28. Available online: http://www.smart-cities.eu/download/smart_cities_final_report.pdf (accessed on 10 June 2020).
  22. OECD. Smart Cities and Inclusive Growth. Building on the outcomes of the 1st OECD Roundtable on Smart Cities and Inclusive Growth; OCED: Paris, France, 2020; pp. 1–59. Available online: https://www.oecd.org/cfe/cities/OECD_Policy_Paper_Smart_Cities_and_Inclusive_Growth.pdf (accessed on 18 June 2020).
  23. Samih, H. Smart cities and internet of things. J. Inf. Technol. Case Appl. Res 2019, 21, 3–12. [Google Scholar] [CrossRef] [Green Version]
  24. Akande, A.; Cabral, P.; Casteleyn, S. Assessing the Gap between Technology and the Environmental Sustainability of European Cities. Inf. Syst. Front. 2019. [CrossRef] [Green Version]
  25. Nam, T.; Pardo, T.A. Conceptualizing smart city with dimensions of technology, people, and institutions. In Proceedings of the 12th Annual International Digital Government Research Conference: Digital Government Innovation in Challenging Times, College Park, MD, USA, 12–15 June 2011; pp. 282–391. [Google Scholar] [CrossRef]
  26. Ojo, A.; Dzhusupova, Z.; Curry, E. Exploring the Nature of the Smart Cities Research Landscape. In Public Administration and Information Technology; Springer: Berlin/Heidelberg, Germany, 2016; Volume 11, pp. 23–47. [Google Scholar]
  27. Hollands, R.G. Will the real smart city please stand up? Intelligent, progressive or entrepreneurial? City 2008, 12, 303–320. [Google Scholar] [CrossRef]
  28. Albino, V.; Berardi, U.; Dangelico, R.M. Smart cities: Definitions, dimensions, performance, and initiatives. J. Urban Technol. 2015, 22, 3–21. [Google Scholar] [CrossRef]
  29. Höjer, M.; Wangel, J. Smart Sustainable Cities: Definition and challenges. In ICT Innovations for Sustainability. Advances in Intelligent Systems and Computing; Hilty, L.M., Aebischer, B., Eds.; Springer: Cham, Switzerland, 2015; Volume 310, pp. 333–349. [Google Scholar]
  30. Silva, B.N.; Khan, M.; Han, K. Towards sustainable smart cities: A review of trends, architectures, components, and open challenges in smart cities. Sustain. Cities Soc. 2018, 38, 697–713. [Google Scholar] [CrossRef]
  31. O’Dwyer, E.; Pan, I.; Acha, S.; Shah, N. Smart energy systems for sustainable smart cities: Current developments, trends and future directions. Appl. Energy 2019, 237, 581–591. [Google Scholar] [CrossRef] [Green Version]
  32. Gabrys, J. Programming environments: Environmentality and citizen sensing in the smart city. Environ. Plan. D Soc. Sp. 2014, 32, 30–48. [Google Scholar] [CrossRef] [Green Version]
  33. Dameri, R.P.; Cocchia, A. Smart City and Digital City: Twenty Years of Terminology Evolution. In Proceedings of the X Conference of the Italian Chapter of AIS, ITAIS 2013, Milan, Italy, 14 December 2013; pp. 1–8. [Google Scholar]
  34. Ahvenniemi, H.; Huovila, A.; Pinto-Seppä, I.; Airaksinen, M. What are the differences between sustainable and smart cities? Cities 2017, 60, 234–245. [Google Scholar] [CrossRef]
  35. Mosannenzadeh, F.; Vettorato, D. Defining Smart City. A Conceptual Framework Based on Keyword Analysis. TeMA J. L. Use Mobil. Environ. 2014, 998. [Google Scholar] [CrossRef]
  36. Mosannenzadeh, F.; Bisello, A.; Vaccaro, R.; D’Alonzo, V.; Hunter, G.W.; Vettorato, D. Smart energy city development: A story told by urban planners. Cities 2017, 64, 54–65. [Google Scholar] [CrossRef]
  37. Mohanty, B.S.P.; Choppali, U.; Kougianos, E. Everything You Wanted to Know About Smart Cities. IEEE Consum. Electr. M. 2016, 5, 60–70. [Google Scholar] [CrossRef]
  38. European Commission. What are smart cities? Available online: https://ec.europa.eu/info/eu-regional-and-urban-development/topics/cities-and-urban-development/city-initiatives/smart-cities_en (accessed on 4 August 2020).
  39. IEEE. IEEE Smart Cities. Available online: https://smartcities.ieee.org/ (accessed on 5 August 2020).
  40. United Nations. Smart cities and infrastructure Report of the Secretary-General Economic and Social Council, 26 February 2016. Available online: https://unctad.org/meetings/en/SessionalDocuments/ecn162016d2_en.pdf (accessed on 22 April 2020).
  41. ITU. Y.4901/L.1601—Key performance indicators related to the use of information and communication technology in smart sustainable cities; June 2016. Available online: https://www.itu.int/rec/T-REC-L.1601-201606-I/en (accessed on 12 April 2020).
  42. United Nations. Sustainable Development Goal 11+—Make Cities and Human Settlements Inclusive, Safe, Resilient and Sustainable; UN Department of Economic and Social Affairs: New York, NY, USA, 2019. [Google Scholar]
  43. Giffinger, R. Smart cities ranking: An effective instrument for the positioning of the cities? ACE Archit. City Environ. 2010. [Google Scholar] [CrossRef] [Green Version]
  44. Giffinger, R.; Haindlmaier, G.; Kramar, H. The role of rankings in growing city competition. Urban Res. Pract. 2010, 3, 299–312. [Google Scholar] [CrossRef]
  45. Airaksinen, M. Smart Cities, Can the Performance Be Measured? VTT Impulse: Espoo, Finland, 2016. [Google Scholar]
  46. Mohan, A.; Dubey, G.; Ahmed, F.; Sidhu, A. Smart Cities Index: A Tool for Evaluating Cities; Indian School of Business: Hyderabad, India, 2017. [Google Scholar]
  47. Caird, S.; Hudson, L.; Kortuem, G. A Tale of Evaluation and Reporting in UK Smart Cities Other Sally Caird with Lorraine Hudson and Gerd Kortuem; The Open University: Milton Keynes, UK, 2016; ISBN 9781473021082. [Google Scholar]
  48. Garau, C.; Pavan, V.M. Evaluating Urban Quality: Indicators and Assessment Tools for Smart Sustainable Cities. Sustainability 2018, 10, 575. [Google Scholar] [CrossRef] [Green Version]
  49. Zygiaris, S. Smart City Reference Model: Assisting Planners to Conceptualize the Building of Smart City Innovation Ecosystems. J. Knowl. Econ. 2013, 4, 217–231. [Google Scholar] [CrossRef]
  50. Fernandez-Anez, V.; Fernández-Güell, J.M.; Giffinger, R. Smart City implementation and discourses: An integrated conceptual model. The case of Vienna. Cities 2018, 78. [Google Scholar] [CrossRef]
  51. The Scottish Government. Scottish Cities Alliance, and Urban Tide. Smart Cities Maturity Model and Self-Assessment Tool: Guidance Note for Completion of Self-Assessment Tool. Developed for and with The Scottish Government by Urban Tide. October 2014. Available online: https://www.scottishcities.org.uk/site/assets/files/1103/smart_cities_readiness_assessment_-_guidance_note.pdf (accessed on 2 August 2020).
  52. Sang, Z.; Ding, H.; Higashi, M.; Nakamura, J.; Hara, M.; Hashitani, T.; Sugiura, J.; Di Carlo, C.; Girdinio, P.; Bolla, R.; et al. Key Performance Indicators Definitions for Smart Sustainable Cities; ITU-T: Geneva, Switzerland, 2015. [Google Scholar]
  53. Debnath, A.K.; Chin, H.C.; Haque, M.M.; Yuen, B. A methodological framework for benchmarking smart transport cities. Cities 2014, 37, 47–56. [Google Scholar] [CrossRef] [Green Version]
  54. Barsi, B. Beyond indicators, new methods in Smart city assessment. Smart Cities Reg. Dev. J. 2018, 3, 87–99. [Google Scholar]
  55. VTT. CITYkeys. Available online: www.citykeys-project.eu (accessed on 15 April 2020).
  56. VITO, DNV-GL, EUROCITIES, Steinbeis, Tecnalia, Th!nk-E. Smart Cities Information System (SCIS). Available online: https://smartcities-infosystem.eu (accessed on 26 July 2020).
  57. Neumann, H.-M.; Jakutyte-Walangitang, D.; Vielguth, S.; Züger, J.; Airaksinen, M.; Huovila, A.; Bosch, P.; Rovers, V.; Jongeneel, S.; Pangerl, E. D1.2—Overview of the Current State of the Art; VTT Impulse: Espoo, Finland, 2015. [Google Scholar]
  58. Huovila, A.; Bosch, P.; Airaksinen, M. Comparative analysis of standardized indicators for Smart sustainable cities: What indicators and standards to use and when? Cities 2019, 89, 141–153. [Google Scholar] [CrossRef]
  59. ISO 37120:2018—Sustainable Cities and Communities—Indicators for City Services and Quality of Life; ISO - International Standardization Organization: Geneva, Switzerland, 2018.
  60. ISO 37122:2019—Sustainable Cities and Communities—Indicators for Smart Cities; ISO - International Standardization Organization: Geneva, Switzerland, 2019.
  61. European Telecommunications Standards Institute. Key Performance Indicators for Sustainable Digital Multiservice Cities; ETSI: Sophia Antipolis, France, 2017. [Google Scholar]
  62. ITU. Y.4902/L.1602—Key Performance Indicators Related to the Sustainability Impacts of Information and Communication Technology in Smart Sustainable Cities; ITU: Geneva, Switzerland, 2016. [Google Scholar]
  63. ITU. Y.4903/L.1603—Key Performance Indicators for Smart Sustainable Cities to Assess the Achievement of Sustainable Development Goals; ITU: Geneva, Switzerland, 2016. [Google Scholar]
  64. Li, X.; Fong, P.S.W.; Dai, S.; Li, Y. Towards sustainable smart cities: An empirical comparative assessment and development pattern optimization in China. J. Clean. Prod. 2019, 215, 730–743. [Google Scholar] [CrossRef]
  65. Berrone, P.; Ricart, J.E. IESE Cities in Motion Index 2018; IESE Business School: Barcelona, Spain, 2018. [Google Scholar]
  66. Institute for Urban Strategies—The Mori Memorial Foundation. Global Power City Index. 2018. Available online: http://mori-m-foundation.or.jp/english/ius2/gpci2/2018.shtml (accessed on 4 August 2020).
  67. Innovation Cities Index. Innovation CitiesTM Index 2018: Global. Available online: https://www.innovation-cities.com/innovation-cities-index-2018-global/13935 (accessed on 4 August 2020).
  68. EasyPark Group. EasyPark. Available online: https://www.easyparkgroup.com/smart-cities-index/ (accessed on 4 August 2020).
  69. National Institute of Standards and Technology. A Consensus Framework for Smart City Architectures IES-City Framework (Internet-of-Things-Enabled Smart City Framework); NIST: Gaithersburg, MD, USA, 2018. [Google Scholar]
  70. Smart Cities Council. Smart Cities Guide for Built Environment Consultants; Smart Cities Council: Brisbane, Australia, 2018. [Google Scholar]
  71. Smart Cities Council. Smart Cities Readiness Guide: The Planning Manual for Building Tomorrow’s Cities Today; Smart Cities Council: Brisbane, Australia, 2015. [Google Scholar]
  72. Bloomberg Philanthropies. What Works Cities Certification Assessment Guide 2018. Available online: https://whatworkscities.bloomberg.org/certification/ (accessed on 4 August 2020).
  73. Smart Cities Council Australian New Zealand and Green Building Council of Australia. Code for Smart Communities; Smart Cities Council: Brisbane, Australia, 2018. [Google Scholar]
  74. Shen, L.; Huang, Z.; Wong, S.W.; Liao, S.; Lou, Y. A holistic evaluation of smart city performance in the context of China. J. Clean. Prod. 2018, 200, 667–679. [Google Scholar] [CrossRef]
  75. ESI. Top 50 Smart City Governments—Eden Strategy Institute and ONG&ONG Pte Ltd. Available online: https://static1.squarespace.com/static/5b3c517fec4eb767a04e73ff/t/5b513c57aa4a99f62d168e60 /1532050650562/Eden-OXD_Top+50+Smart+City+Governments.pdf (accessed on 28 July 2020).
  76. ASCIMER. Assessing Smart City Initiatives for the Mediterranean Region. Available online: http://www.eiburs-ascimer.transyt-projects.com/ (accessed on 22 August 2020).
  77. Fernandez-Anez, V.; Velazquez, G.; Perez-Prada, F.; Monzón, A. Smart City Projects Assessment Matrix: Connecting Challenges and Actions in the Mediterranean Region. J. Urban Technol. 2018, 1–25. [Google Scholar] [CrossRef]
  78. Woods, E.; Labastida, R.R.; Chow, T.; Citron, R.; Leuschner, P. UK Smart Cities Index 2017 Assessment of Strategy and Execution for the UK’s Leading Smart Cities; ITU: Geneva, Switzerland, 2017. [Google Scholar]
  79. Ericsson. Networked Society City Index 2014. Available online: https://www.ericsson.com/en/news/2014/11/networked-society-city-index-2014-released (accessed on 10 August 2020).
  80. PwC. Cities of Opportunity 7. Available online: https://www.pwc.com/us/en/library/cities-of-opportunity.html (accessed on 18 August 2020).
  81. Hemment, D.; Woods, M.; Appadoo, V.; Bui, L. Community Key Performance Indicators (Community KPIs) for the IoT and Smart Cities; Future Everything: Manchester, UK, 2016. [Google Scholar]
  82. Woods, E.; Omara, H.; Ravens, S.; Citron, R. Gulf States Smart Cities Index: Assessment of Strategy and Execution for 10 Cities; Navigant Consulting, Inc.: Chicago, IL, USA, 2016. [Google Scholar]
  83. Bannerjee, S.; Bone, J.; Finger, Y.; Haley, C. European Digital City Index—Methodology Report. 2016. Available online: https://digitalcityindex.eu/uploads/2016%20EDCi%20Construction%20Methodology%20FINAL.pdf (accessed on 4 August 2020).
  84. Walter, C.; Woodling, R.; Lathouwer, D.; Woodling, R. Smart City Domain Strategic Growth Map; Horizon 2020 European co-funded project: ESPRESSO - systEmic Standardisation apPRoach to Empower Smart citieS and cOmmunities; 2016; Available online: http://espresso.espresso-project.eu/wp-content/uploads/2017/05/D5.1-Smart-City-Strategic-Growth-Map.pdf (accessed on 4 August 2020).
  85. Wu, Z.; Pan, Y.; Ye, Q.; Kong, L. The City Intelligence Quotient (City IQ) Evaluation System: Conception and Evaluation. Engineering 2016, 2, 196–211. [Google Scholar] [CrossRef] [Green Version]
  86. Achaerandio, R.; Curto, J.; Bigliani, R.; Gallotti, G. Smart Cities Analysis in Spain 2012—The Smart Journey; International Data Corporation (IDC): Madrid, Spain, 2012. [Google Scholar]
  87. ITU. ITU-T Y.4900/L.1600 Overview of key Performance Indicators in Smart Sustainable Cities; ITU: Geneva, Switzerland, 2016. [Google Scholar]
  88. Economic Commission for Europe. The UNECE–ITU Smart Sustainable Cities Indicators; UNECE: Geneva, Switzerland, 2015. [Google Scholar]
  89. Cohen, B. The Smartest Cities in the World 2015: Methodology. Available online: https://www.fastcompany.com/3038818/the-smartest-cities-in-the-world-2015-methodology (accessed on 2 August 2020).
  90. Manville, C.; Cochrane, G.; Cave, J.; Millard, J.; Pederson, J.K.; Thaarup, R.K.; Liebe, A.; Wissner, M.; Massink, R.; Kotterink, B. Mapping Smart Cities in the EU; European Parliament: Brussels, Belgium, 2014.
  91. Urban Tide. Overview of Smart Cities Maturity Model; Developed for and with The Scottish Government by Urban Tide. 2015. Available online: https://static1.squarespace.com/static/5527ba84e4b09a3d0e89e14d/t/55aebffce4b0f8960472ef49/1437515772651/UT_Smart_Model_FINAL.pdf (accessed on 4 August 2020).
  92. Thielen, P.; Hemis, H.; Storch, A.; Lutz, M. Gradual development of Austrian Smart City Profiles, Blue Globe Report, SmartCities #2/2013; Klima energie fonds, Wien, Austria. 10 March 2013. Available online: https://www.klimafonds.gv.at/wp-content/uploads/sites/6/BGR22013KR11SE2F00690SmartCityProfilesv1.0.pdf (accessed on 4 August 2020).
  93. The Committee of Digital and Knowledge-based Cities of UCLG. Smart Cities Study 2012, International Study on Situation and Development of ICT, Innovation and Knowledge in Cities; UCLG: Barcelona, Spain, 2013. [Google Scholar]
  94. China Academy of Telecommunication Research of MIIT. Smart Cities Benchmarking in China; CAICT: Beijing, China, 2012. [Google Scholar]
  95. Merino-Saum, A.; Halla, P.; Superti, V.; Boesch, A.; Binder, C.R. Indicators for urban sustainability: Key lessons from a systematic analysis of 67 measurement initiatives. Ecol. Indic. 2020, 119, 106879. [Google Scholar] [CrossRef]
Table 1. Potential multiple benefits of Smart City Assessment (SCA) for different stakeholders [13].
Table 1. Potential multiple benefits of Smart City Assessment (SCA) for different stakeholders [13].
StakeholderPotential Benefits
Cities and city authorities
  • Monitoring performance to improve the international image and competitive position of the city in the eyes of investors, as well as creative citizens and the public [21,43,44,45,46]
  • Justify the value of smart city investments and interventions [47]
  • Identify strengths and weaknesses and guide smart city planning [21,43,44,48,49,50]
  • Track progress in achieving pre-defined goals and targets and identify the position of the city in its efforts towards smartness [48,51]
  • Understand the socio-economic and environmental implications of smart city projects [52]
  • Understand the technical requirements of smart city projects [52]
  • Learn from the experiences of peers (when assessment involves benchmarking) [21,44,50,53]
  • Identify and showcase best-practice cases to learn lessons from [21]
  • Enhance governance transparency [21,44,50,53]
  • Stimulate discussions among various stakeholders that may result in improved mobilization of resources [21,44,54]
Investors and funding agencies
  • Evidence-based evaluation of the completed or ongoing projects [47]
  • Scientific means for prioritizing funding allocation [45]
  • Enhanced ability to make decisions regarding the best sites for future investment [44,46]
  • Ability to identify and capitalize on new business opportunities [46]
Researchers
  • Develop new strategies for improvement of smart city performance [47]
  • Simplify the complexities of the smart city concept [9,24]
Citizens
  • Enhanced awareness about the benefits of smart city projects [47]
  • Ability to make informed decisions when it comes to future investments [45]
  • Motivation to engage in smart city development activities and to communicate their desires and priorities to city authorities [45,46,53]
Table 2. Summary of international standardized indicators analyzed in [58].
Table 2. Summary of international standardized indicators analyzed in [58].
Name, Type and AbbreviationMain Categories Addressed
ISO 37120:2018 sustainable development of communities – indicators for city services and quality of life
International non mandatory standard
Abbreviation: ISO 37120 [59]
Economy, education, energy, environment and climate change, finance, governance, health, housing, population and social conditions, recreation, safety, solid waste, sport and culture, telecommunication, transportation, urban/local agriculture and food security, urban planning, wastewater, water
ISO/DIS 37122:2019 sustainable development in communities - indicators for Smart cities
DIS = draft international standard
Abbreviation: ISO 37122 [60]
ETSI TS 103 463 key performance indicators for sustainable digital multiservice cities
TS = technical specification
Abbreviation: ETSI indicators [61]
ITU-T Y.4901/L.1601 key performance indicators related to the use of information and communication technology in Smart sustainable cities
Type: Recommendation
Abbreviation: ITU 4901 [41]
ICT, environmental sustainability, productivity, quality of life, equity and social inclusion, physical infrastructure
ITU-T Y.4902/L.1602 key performance indicators related to the sustainability impacts of information and communication technology in Smart sustainable cities
Type: Recommendation
Abbreviation: ITU 4902 [62]
Environmental sustainability, productivity, quality of life, equity and social inclusion, physical infrastructure
ITU-T Y.4903/L.1603 key performance indicators for Smart sustainable cities to assess the achievement of sustainable development goals
Recommendation
Abbreviation: ITU 4903 [63]
Economy, environment, society and culture
Sustainable Development Goal 11+ monitoring framework
UN Inter-Agency Expert Group definition
Abbreviation: UN SDG 11+ indicators [42]
UN SDG targets 11.1, 11.2, 11.3, 11.4, 11.5, 11.6, 11.7, 11a, 11b, 11c, 1.4, 6.3
Table 3. Summary of SCA tools analyzed in [13].
Table 3. Summary of SCA tools analyzed in [13].
ToolYearPrimary Developer(s)
Lisbon ranking for smart sustainable cities2019[9]
Smart Sustainable Cities China2019[64]
Cities in Motion Index2018Center for Globalization and Strategy and IESE Business School’s Department of Strategy [65]
Global Power City Index2018The Mori Memorial Foundation’s Institute for Urban Strategies [66]
Innovation Cities™ Index2018China Academy of Telecommunication Research and China Communications Standards Association [67]
EasyPark2018EasyPark Group [68]
IoT-Enabled Smart city framework2018National Institute of Standards and Technology [69]
Smart Cities Council’s tools and frameworks2018Smart Cities Council, Australia and New Zealand [70,71]
What Works Cities2018Bloomberg Philanthropies [72]
Code for Smart Communities2018Smart Cities Council Australian New Zealand and the Green Building Council of Australia [73]
China Smart City Performance2018Shen [74]
Smart City Governments2018Eden Strategy Institute and ONG&ONG Pte Ltd. [75]
Assessing Smart City Initiatives for the Mediterranean Region2017Universidad Politécnica of Madrid (UPM) [76,77]
Smart Cities Index- India2017Indian School of Business [45]
Juniper Research smart city frameworks2017Juniper Research [77]
UK Smart Cities Index2017Navigant Research [78]
CITYkeys2016Netherlands Organization for Applied Scientific Research (TNO) [55]
Networked Society City Index2016Ericsson in collaboration with Sweco [79]
Cities of Opportunity2016PricewaterhouseCoopers (PwC) [80]
Community KPIs for the IoT and Smart Cities2016Future Everything [81]
Gulf States Smart Cities Index2016Navigant Research [82]
European Digital Cities Index2016Nesta [83]
Smart City Strategic Growth Map2016ESPRESSO, European Commission [84]
City IQ Evaluation System2015Wu [85]
International Data Corporation (IDC) Smart City Analysis2015IDC [86]
Telecommunication and Standardization Sector of International Telecommunication Union (ITU)2015ITU-T Focus Group on Smart Sustainable Cities [87]
United Nations Economic Commission for Europe-ITU Smart Sustainable Cities Indicators2015UNECE Committee on Housing and Land Management, Environment Agency Austria, and ITU [88]
Smart Cities Ranking of European Medium-sized Cities2014TU Vienna, in cooperation with the University of Ljubljana and the TU of Delft [21]
Boyd-Cohen Smart City Index2014Boyd-Cohen [89]
Mapping Smart Cities in the EU2014RAND Europe, European Union (EU) [90]
Smart City Maturity Model and Self-Assessment Tool2014The Scottish Government and Scottish Cities Alliance [51,91]
Smart City Profiles2013Austrian Climate and Energy Fund and Environment Agency Austria [92]
United Cities and Local Governments (UCLG) smart cities study2012City of Bilbao and Committee of Digital and Knowledge-based Cities of UCLG [93]
Smart Cities Benchmarking in China2012China Academy of Telecommunication Research and China Communications Standards Association [94]

Share and Cite

MDPI and ACS Style

Patrão, C.; Moura, P.; Almeida, A.T.d. Review of Smart City Assessment Tools. Smart Cities 2020, 3, 1117-1132. https://doi.org/10.3390/smartcities3040055

AMA Style

Patrão C, Moura P, Almeida ATd. Review of Smart City Assessment Tools. Smart Cities. 2020; 3(4):1117-1132. https://doi.org/10.3390/smartcities3040055

Chicago/Turabian Style

Patrão, Carlos, Pedro Moura, and Anibal T. de Almeida. 2020. "Review of Smart City Assessment Tools" Smart Cities 3, no. 4: 1117-1132. https://doi.org/10.3390/smartcities3040055

Article Metrics

Back to TopTop