The Tall Tower Dataset (Ramon and Lledó, 2019a) is a unique collection of data from 222 tall towers resulting from an exhaustive process of identification of existing masts and their later data retrieval. Figure 2 presents the global distribution of the sites, which is highly heterogeneous. Most of the masts are located in Asia (51 %), mainly clustered in Iran, resulting from a national campaign aimed to boost renewable energies at a country level. Then, tall towers appear more spatially distributed over North America (23 %) and Europe (16 %), mirroring the important deployment of wind power that is taking place in those regions. Africa (8 %), Oceania (1 %) and Antarctica (1 %) follow. Unfortunately, it has been hard to retrieve data from South America, so no records from this area can be found in the Tall Tower Dataset.

Figure 2Global distribution of the 222 tall tower locations within the Tall Tower Dataset. Colours indicate the top measuring level for each tower. Further information can be found in Sect. S1 of the Supplement.

The height above the surface where the top sensor is located for each tower is also depicted in Fig. 2. On the one hand, masts placed in historical observatories (i.e. often having more than 20 years of data) tend to be short, with heights ranging between 18 and 50 m above the ground and usually consisting of one measuring level at the top of the pole. Two examples are the American masts in Utqiaġvik (formerly known as Barrow) and Mauna Loa. On the other hand, modern towers often reach 100 to 200 m of altitude. Indeed, most of the masts in northern Europe have been installed during the last 15–20 years and are generally taller than 80 m, usually reaching 150 to 200 m. However, the tallest structures are located in the USA, reaching the exceptional height of 500 m, allowing the placement of sensors at that height. The top anemometer at Walnut Grove tall tower in California is at 488 m above ground level. The number of measuring levels in these masts is almost always higher than three, and up to eight in the case of the FINO met masts.

A list of the towers included in the Tall Tower Dataset, as well as their main characteristics such as the owner institution, country, geographic coordinates or specific recording periods, can be found in Sect. S1 of the Supplement. The record lengths and other structural features such as height or instrumentation are quite diverse as they depend on the purpose they were designed for. Most of the towers are typically installed to provide in situ observations for experimental field campaigns within the research or industry fields. In this case, the tall towers are commonly referred to as meteorological masts or met masts, and they represent up to the 87 % of all the tall towers in the dataset. However, other sensors are installed over marine platforms (11 %) or at the top of lighthouses (1 %) to monitor the coastal weather conditions. Finally, 1 % of the towers are instrumented communication transmitters that take meteorological measurements at several platforms along with the antenna. Concerning the location, almost 80 % of these tall towers are found inland while the other 20 % are placed offshore.

Information indicating the representative features mentioned above is included in the dataset within the corresponding site metadata, which have been standardized for all the sites. This material was sometimes confusing, sparse or even missing in the datasets distributed by the owner data centres, especially when it comes to the conventions in which the initial data were prepared. For example, if the time zone in which the time stamps were delivered was not specified, it could be challenging to discern whether they are provided in local time or Coordinated Universal Time (UTC). Another example concerns the data units, which were not explicitly stated in a few cases either. In both of these confusing situations, the data provider was contacted to confirm the original convention. Further information on the diverse standards in which the data were provided as well as the final conventions employed in the Tall Tower Dataset can be found in Ramon and Lledó (2019b).

The time span of the 222 time series is depicted in Fig. 3a. First, we split the series according to their time resolution, which varies from every 10 min to once per hour. Most of the series, i.e. a total of 172, provide 10 min averaged data, meeting the WMO standard (WMO, 2007) for estimating mean wind speeds. The other 50 masts report 15 min, 20 min, 30 min or hourly data. Information on how these averages have been taken is hardly ever available. The fact is that resulting aggregated values vary depending on whether averages are taken over the horizontal wind components or speed and direction modules independently. WMO (2007) does not prefer one option over the other, as it may depend on the application or available instrumentation. Even though the effects of this choice are rather small, especially for higher wind speeds, it represents an additional source of uncertainty for the values themselves.

The total coverage of the Tall Tower Dataset ranges from 1984 to 2017. While the 90 % of the series span less than 20 years, 3 % cover 30 or more years. The precise beginning and end of recording periods can be found in Sect. S1 of the Supplement for each tall tower. Nevertheless, several of these masts have been recently installed, and measurements are currently operational. Missing data periods – 12.1 % of the dataset – appear sometimes embedded within the series.

Figure 3(a) Time coverage of the 222 tall towers depending on whether they report 10 min data (top) or lower resolutions up to 1 h data (bottom) and (b) time evolution of the total number of observations within the Tall Tower Dataset for wind speed (green), wind direction (blue), temperature (red), relative humidity (orange) and pressure (purple).

Download

Concerning the data retrieval process, the initial efforts focused on collecting the largest number of wind observations possible. Those records have been complemented with temperature, relative humidity and surface pressure data also measured at the different platforms along the tower. The time evolution of the number of these five variables is plotted in Fig. 3b. Most of the data fall within the 21st century, with a significant increase at the beginning of the millennium. Up to 2.7 million wind speed records have been retrieved for one single month, i.e. December 2015, which constitutes the month with the maximum amount of wind speed data. In the case of wind direction, the month with the highest number of records is October 2012 (2.1 million measurements). A decrease in the number of observations has been noticed from 2017 onwards. Generally, some of the data providers prefer to keep the most recent data and release them once measurements are preliminarily checked for gross errors. Temperature, relative humidity and pressure are not always available. We note that the fewest records correspond to barometric pressure, which is usually measured only at surface level (i.e. 2 m above ground level).

To ensure the high quality of tall tower wind data and guarantee the accuracy of any result derived from these records, a QC procedure needs to be carried out. The scientific literature has devoted efforts to QC wind data taken at surface stations (e.g. Dunn et al., 2012; Lucio-Eceiza et al., 2018a, b). However, no QC software has been specifically designed to tackle the same problem with tall tower observations, whose features vary considerably when compared to surface wind data (e.g. measurements are taken at higher altitudes, the spatial density of stations is considerably lower). Unique measuring techniques, such as the parallel measurements at different platforms along the mast or sensor redundancy at a given height, can also be taken into account to complement and enhance the typical QC.

Figure 4Summary of the workflow of the QCSS4TT routines applied over wind data within the Tall Tower Dataset.

Download

After a review process of the existing QC routines, a set of 18 sequential QC tests (two preliminary and 16 main tests) has been selected and designed to be performed over wind measurements. The Quality Control Software Suite for Tall Towers (QCSS4TT) designed here is applied to all the wind speed and wind direction data within the Tall Tower Dataset, regardless of whether they were previously quality controlled or not by the providing institution. A general description of the QCSS4TT is presented below in this section. The software is fully described in Sect. S2 in the Supplement.

The QC tests within the QCSS4TT are all intra-station checks, as they do not compare series from nearby tall towers. QC routines ingest entire time series of winds at a specific heights, whose time frequencies vary between 10 min and 1 h. The recommended sequence for the application of the QC tests is presented in Fig. 4. Checks are grouped in five categories depending on the purpose they were designed for. The two preliminary checks are designed to detect gross manipulation errors. Then, the 16 main QC tests ensure the limits, spatio-temporal and internal consistency of the wind speed and wind direction time series. We note that the routines can be run independently, with the exception of the quartile occurrences and isolated pass tests which feed on the output of other tests within the QCSS4TT. After deciding the appropriate order, the tests have been applied over the Tall Tower Dataset according to the flux diagram in Fig. 4.

Table 1Main QC tests summary. The abbreviations ws, wd and tmp stand for wind speed (in metres per second), wind direction (in meteorological degrees) and temperature (in degrees Celsius), respectively. The reader is referred to Sect. S2 in the Supplement for detailed information on each of the tests.

Download Print Version | Download XLSX

The QCSS4TT starts with two preliminary tests. Firstly, the time stamp check is carried out during the data formatting process and ensures that all the time stamps are included in the dataset and equally sampled according to the tower reporting frequency. Those time stamps that are either duplicated or wrongly spaced have been discarded, and those missing have been included, setting the corresponding record to “not available” (NA). Secondly, the surroundings check is performed by detecting nearby elements that could potentially perturb the wind flow and then produce unreal records. To do so, detailed descriptions of the encircling area of the mast as well as its possible changes over time are required.

Then, the 16 main QC tests follow. A summary can be found in Table 1 and complete information can be found in Sect. S2 in the Supplement. Most of them are standard checks typically performed over wind and other Essential Climate Variables such as temperature or precipitation. However, we propose two new QC tests (the so-called tower shadow and vertical ratios checks, respectively) here to guarantee the spatial consistency of the data by considering the special characteristics of the tall tower measurements since classic inter-station comparisons appear challenging due to poor spatial density of sites.

Table 2Flag level definitions.

Download Print Version | Download XLSX

After running the QCSS4TT, a natural number (hereafter referred to as QC flag or flag; see Table 2) is attached to each observation according to its nature and/or level of confidence. To decide which flag should be assigned to each observation, different threshold values have been set for each of the QC routines. The threshold selection has been based on the World Meteorological Organization (WMO) standards (WMO, 2007; Aguilar et al., 2003), QC software manuals (IOSS, 2017) or state-of-the-art bibliography (e.g. Jiménez et al., 2010). Many of these standards, if not all, have been developed specifically for surface winds (i.e. 10 m winds), whose features vary importantly when compared with winds observed at higher altitudes, such as those from the present work. After a preliminary test of the thresholds over the wind data within the Tall Tower Dataset, it was noted that some tests considerably overestimated the amount of erroneous data (also known as Type I errors; see Hubbard et al., 2004). The WMO allows adjustment of some of the fixed-value limits proposed in the WMO (2007) to reflect singular climate conditions more accurately. As the QCSS4TT aims to clean data from towers located all over the world regardless of the prevailing climate conditions in the area, thresholds need to be adjusted manually to not deem wrong the general and particular climate features observed in a wide variety of world climates. It is also vital to take into account that this sensitive experiment should reduce the number of Type I errors without increasing the number of invalid data that have been accepted by the tests (also referred to as Type II errors).

Based on these thresholds and the nature of the individual wind records, six different categories have been defined (Table 2), and each datum is flagged appropriately. The quality of a record is inferred automatically by checking if it passes all the tests successfully (flagged as “1”), passes the tests but might need further checks such as a visual inspection (hereafter referred to as suspect and marked as “2”) or fails at least one of the tests (flagged as “4”). When an observation is not considered suspect or wrong by any of the QC tests, additional levels may indicate that the observation was not evaluated by three or more tests (indicated as “0”) or corresponds to a calm period (“5”). Finally, missing values are flagged uniformly (categorized as “9”).

Wind records flagged as “4” are deemed to be erroneous data and thus unreliable. They have been removed by changing the original record to NA. Suspect data as well as those observations that have not been evaluated by all the QC tests remain unaltered because they might be potentially correct and usable for some applications. But in case the user prefers to impose their own level of restriction, we also include the raw data jointly with the flag values resulting from the quality controlling. Therefore, the data user is able to filter the raw data based on the flag values. Still in those cases, we strongly discourage the usage of data marked as erroneous (“4”).

The QCSS4TT has been applied sequentially over the Tall Tower Dataset according to the flux diagram in Fig. 4. We present here the global results obtained from the quality controlling of the Tall Tower Dataset, as well as a summary of the performance of the main tests.

As stated in Sect. 3, the surroundings check needs detailed original metadata of the tower location. Unfortunately, this valuable information is not always available so the surroundings check cannot be carried out over all the tower sites. The unique case when this QC test confirms that a series of wind speeds were disturbed by the surrounding forest occurs at Wallaby Creek met mast. After running the main QC routines, long sequences of wind speeds measured at the lowermost level of this met mast – placed at 10 m above surface – have been flagged as wrong. A close look at the site metadata reveals that the canopy well exceeds the 10 m height during the whole recording period, considerably reducing the observed wind speeds. Hence, all the individual observations of the Wallaby Creek 10 m wind series have to be used with caution, even those that have not been considered problematic by other tests.

Then, the main QC routines have examined each of the 240 371 908 wind speed and wind direction values individually in the Tall Tower Dataset and flagged them accordingly. After this process, 228 780 679 values (95.2 % of the total data) passed successfully all the checks and can be considered reliable. Conversely, 6 827 880 observations (2.8 %) have been considered erroneous by at least one of the QC tests. They have been replaced by NA, increasing the total number of missing data from 12.1 % to 14.6 %. A total of 1.8 % of the dataset is flagged as suspect. Some of the QC tests, particularly those that compute period-aggregated statistics such as moving averages or variances, require a minimum amount of data. Due to this constraint, 0.2 % of the data have not been evaluated by three or more QC tests to avoid the computation of such statistics with reduced sample sizes. Records identified as calm (i.e. wind speeds under 0.5 m s⁻¹) have also been skipped on purpose by a small group of tests, i.e. those that compute quotients between pairs of simultaneous observations. However, calm wind records can be trusted as they successfully passed all the other QC checks. The percentage of calm wind values is highly dependent on the geographical location of the tall tower. Met masts located in Southeast Asia contain the largest percentage of calm wind records, reaching up to 24 % of the total data.

Figure 5Percentage of data flagged as “fail” (dark) and “suspect” (light) by 15 of the main QC tests within the QCSS4TT. Asterisk (*) indicates that the QC test only flags data as suspect. Double asterisk (**) denotes that the QC test only flags data as erroneous.

Download

The amount of data flagged by each test is considerably different, as can be noticed in Fig. 5, which depicts the percentage of data flagged as erroneous and suspect by the main QC tests. We note that both the flat line and quartile occurrences checks have flagged the largest amount of data (1.74 %). The former detected the most substantial amount of erroneous data (1.52 %), followed by the repeated sequences and quartile occurrences tests (1.29 % and 0.88 %, respectively). The vertical ratios check has detected very few erroneous or suspect records, and the difference of extreme values test has flagged no data. The occurrences of 0 and 360 values tests is not included in Fig. 5 since this test does not flag individual records but the entire time series according to their quality. Results for this QC show that no wrong or suspicious time series have been detected after the inspection of the frequency of appearance of null wind speeds and 0 and 360 wind direction values.

The performance of the QCSS4TT needs to be assessed. Here, a benchmark experiment has been specifically designed to test the ability of the QCSS4TT in detecting wrong values. In the following, the preparation of the experiment and its results are described.

Table 3Percentage of the detected errors (%) depending on the proportion of data that were set to missing. The percentages of missing data are approximately 20 %, 10 %, 5 % and 0 % (NA-free). The table differentiates between land and offshore locations.

Download Print Version | Download XLSX

The setup of the experiment consists of generating a set of presumably QC-free time series where a set of errors will be purposely introduced later on. The time series have been extracted at 50 randomly selected points from the ERA5 reanalysis (Copernicus Climate Change Service (C3S), 2017) global grid at hourly frequency, thus meeting the requirements of the QCSS4TT concerning the time resolution. These time series span the 10-year period from 2007 to the end of 2016, which constitutes the time range with the largest number of records within the Tall Tower Dataset (Fig. 3b). To better emulate the features of the tall tower data, we retrieve two parallel series at each of the 50 points. These wind speeds are those provided at 10 and 100 m, respectively.

The set of 50 series is replicated fourfold. Three of these four groups of series are firstly modified by introducing missing data at random, either by erasing data individually or removing sequences of records. The percentages of missing data in these series are approximately 5 %, 10 % and 20 %. The introduction of missing records emulates the frequently observed sporadic sensor failures and no data periods within the wind speed series. Finally, the remaining group of series is left with no datum set to missing.

Figure 6Percentage of detection of seeded errors as a function of the magnitude of the random number r.

Download

The error “seeding” process is carried out following the methodology in Hubbard et al. (2004), where the performance of a set of basic QC tests for temperature and precipitation data is assessed. In this publication, a subset of 2 % of the total data is selected to be modified by introducing an error of magnitude:

where σ_x is the standard deviation of the time series x and r_i is a randomly selected number generated using a uniform distribution ranging from −3.5 to +3.5 specifically for the ith observation. Once the errors are inserted, the QCSS4TT is executed.

Table 3 summarizes the differences in the error detection depending on whether missing data are introduced or not into the wind series for both continental and oceanic locations. It is worth noting that the QCSS4TT shows a slight sensitivity to missing data, reducing the percentage of detected errors when the percentage of missing records increases. This decrease might be attributed to the fact that some QC tests are deactivated when a period with very sparse data is encountered. No important differences are noted between onshore and offshore sites, as the percentages of identified errors are quite similar. Results that follow in this section are presented for the set of series containing approximately 10 % of missing data, which is close to the average of absent records within the Tall Tower Dataset (12.5 %).

The QCSS4TT has detected on average nearly 40 % of all the seeded errors (see Table 3). Indeed, this result is at the average of the percentage of detection observed for precipitation data in Hubbard et al. (2004), which was 30 %–40 % for complex terrain sites and 40 %–50 % for the other locations.

At this stage of the experiment, it is important to study the role of the random number r, and particularly its magnitude, which subsequently influences the size of the error E. Values of r close to zero will introduce smaller errors, which will be less likely to be caught by any of the QC checks. Figure 6 presents the percentage of detection as a function of the r values, which have been grouped in intervals of 0.5 units. We note that the QC tests detect most of the biggest errors. However, the percentage of detection decreases as the magnitude of r does, as we expected. Thus, the smallest errors are usually skipped by all the QC tests. Indeed, this result mirrors the conservative philosophy employed in the threshold selection of the checks.

Finally, it has been observed that Type I errors have been made in 8 % of the total data, corresponding mainly to suspect flagging.

Records from 181 out of the 222 tall towers within the Tall Tower Dataset (Ramon and Lledó, 2019a) are publicly accessible through the following EUDAT repository: https://doi.org/10.23728/b2share.136ecdeee31a45a7906a773095656ddb. Data from the other 41 tall towers are not directly provided for download within the Tall Tower Dataset but can be downloaded from specific tower websites (mostly after registration). Links to all the 222 tall tower original sources are provided in a data catalogue website at http://talltowers.bsc.es (last access: February 2020), which also facilitates browsing the Tall Tower Dataset and visualizing the main climatic features of all the data.

The QCSS4TT code as well as an explanatory vignette on how to use it is also available via GitLab at: https://earth.bsc.es/gitlab/jramon/INDECIS-QCSS4TT (Ramon and Lledó, 2019c).

The supplement related to this article is available online at: https://doi.org/10.5194/essd-12-429-2020-supplement.

JR retrieved and formatted the tall tower data, produced the QCSS4TT code and carried out the benchmark experiment. He also wrote the first draft of this manuscript. This work has been done under the supervision of LL, who conceived the research, gave advice on the data collection, and assisted JR in several IT issues and code debugging. NPZ came up with the design of the benchmark experiment as well as some of the graphics to visualize the results. AS and FJDR also supervised this work and facilitated the retrieval of data. All authors contributed to the analysis of the results and to the writing and editing of the paper.

The authors declare that they have no conflict of interest.

The Tall Tower Dataset is made available in good faith to be used for non-commercial purposes. In no event will the authors be liable to any user or third party for any damage or loss resulting from any use or misuse of these data.

This research has been supported by the H2020 ERA-net ERA4CS (grant no. 690462).

This paper was edited by Kirsten Elger and reviewed by two anonymous referees.