As a further update, I was recently pondering over the question of how the Met Office knows what the 1991-2020 (and even earlier) averages are for Edinburgh Gogarbank even though that particular station didn't even exist before 1998 which is a whole 7 years after the beginning of that 30 year time frame. Just recently though, the Met Office published a video on their main YouTube channel about something called the HadUK grid and it turns out that this is actually what is used to determine those averages.
I got that answer when I posed the same question to Google's AI bot which is accessible at google.com. With that, there is the 2km grid which is capable of tracking back over a period of time and give an accurate picture of what the observations in a particular area would be if there was an actual weather station there. That in turn allowed the Met Office to be able to determine what the conditions were like in the Gogarbank area before its weather station was even setup, thus allowing them to be able to come up with those averages.
It turns out in fact, that this can be utilised to give similar averages for any area of the UK and in my case, I was even able to build my own 1991-2020 averages for my actual location here in NW Edinburgh by creating a python script for that which downloads data from open-meteo.com and then uses that to build those average. It is possible to directly download HadUK grid data from the Met Office site but that is rather bulky and difficult to navigate. At the other site at open-meteo.com, I can access data from Copernicus's ERA-5 land dataset which is similar in nature (if not the same) as the Met Office's HadUK grid data and I have now created a python script for doing that.
Developing such a script used to be a very time consuming process which required you to understand quite a lot of detailed maths and physics, and a lot of in depth programming concepts. I am now approaching retirement ago and my brain doesn't quite work as well as it used to, so it's a lot more difficult now for me to be able to deal with those concepts than it used to be. However, AI is now changing that. In today's world, there are a lot of time pressures involved which mean that in some situations, there might not be the time to waste on such time consuming processes, especially given that a lot of projects might have to be up and running very quickly.
Because of that, even professional computer programmers now collaborate with each other and use AI to help them to develop their projects. That is what I have started to do and by doing so, I am now able to produce these python scripts in only a small fraction of the time that it would have taken had I just tried to work on that on my own. That in turn makes quite a huge difference.
As for my own 1991-2020 averages, I found them to be very interesting. Since I live in the same city as Gogarbank, my averages are similar but they are not exactly the same because of the fact that I live in a different area. I live in NW Edinburgh but Gogarbank is actually in SW Edinburgh which technically makes it a part of the south of Edinburgh rather than the north of Edinburgh. If you study the data from there more closely, what you actually find is that as a result, the conditions there are much more likely to be more like what they are at a station such as Swanston, that they are at the botanic gardens in Edinburgh which is closer to the coast. Indeed, that was shown really well in 2019 when the temperature there rose to over 30°C, but only reached 26°C at the botanic gardens due to some slight cooling from the North Sea.
However, the botanic gardens are also very close to the city centre and that North Sea effect actually grows even stronger as you move even closer to the coast from there, yet there are no actual weather stations in that vicinity which can pick that up. As a result, it is possible for snow to be lying at the botanic gardens in the winter, with my area being completely snow free at the same time. That is something which these localised averages pick up really well, and I'm glad I've been able to find them.
This is good not just for long term averages though because it turns out that I can also utilise that to give a fairly accurate picture of what my own data would be like in terms of the actual current conditions if I actually had my own weather station to be able to record that. This then actually allows me to report the conditions specifically in my area without any need for an actual weather station. This will help me even more when I travel down to the Borders because there are no nearby major Met Office stations and yet, this system would probably still be able to give an accurate picture of the actual conditions there.
Furthermore, this also means that I would no longer have the same issues with missing data as I would for recording data from any station as this system automatically fills in those gaps. Another example of that comes with rainfall because I have lost count of the number of times when I have had a lot of rain here, only for very little of that to actually be picked up at any of the nearby major stations. This system actually picks that up really well though and with that, I can get a more accurate picture of what those rainfall totals actually are in this particular area.
With all of that in mind, I am now in the process of giving my own python script a complete overhaul and replacing the system which I have been using up until now, with a system which give reports for my actual area where I live, rather than for a station such as Gogarbank. That was a system which required me to understand more complex concepts such as dealing with raw BUFR/SYNOP data and looking back in it, it turns out that my old system was a lot more complicated than it actually needed to be.
On the other hand, the system which I am now creating to replace that will be a lot simpler and easier to use. As an example of what this system provides, I noticed that according to SEPA, there been 11.6 mm of rain during this month so far at the botanic gardens but only 10.6 mm at Gogarbank. Meanwhile, my new system is telling me that where I live, there has been a total of 12.4 mm of rain during this month so far. As for the temperatures, It is currently 8.2°C at Gogarbank and 7.4°C at the botanic gardens. Meanwhile, my new system is telling me that it is currently 8.8°C. This nicely shows that it is a west to east variation which is overriding everything in that regard just now. From that, I am seeing that this is a system which I am very happy with and so, I'm looking forward to bringing that into use in my future reports on here on the relevant threads for that.
Originally Posted by: johncs2016