On the internet we're all interconnected by fiber. If you let light carry your messages, it will travel with light speed obviously. The only problem are all the devices in between, routing and switching: They delay it, sometimes by quite a bit.
In general, but only in general, the farther you and your destination are located, the more of these devices you might need to traverse. BUT, these devices are usually strong backbones who can handle a tremendous amount of data, so sometimes you might even have a better connection to a server a bit farther away, just like I experienced a few years ago in Germany connecting to Switzerland.
These router interconnections are happening on the basis of agreements between organizations or other entities who are in control of certain Autonomous Systems, or AS. Between those AS, there are two kinds of agreements: Transmit and Peering. While you pay a usually better connected AS owner than yourself to forward your traffic, peering is done mutually with lower cost, i.e., both agree to forward the other's traffic - and they publically make it known. Hence, they are publically viewable, for example on HE's BGP toolkit. While this is not the Holy Grail (it certainly has it's weaknesses), it gives a much better idea of how well you are connected to certain servers. So when I started making sure the servers I connect to have a direct BGP route to the AS of AirVPN's servers, throughput problems were rarely seen, even if it meant to connect to Switzerland instead of Frankfurt (which is nearer to my physical location).
Also what can happen with geographically closer servers is that an ISP in Germany for example might route traffic through a US east coast server because it's cheaper, less loaded or simply because the ISP doesn't "trust" the only big peering provider in Germany he has an agreement with (= ranks it down or so). The BGP toolkit can uncover this to some extent. Routing outside your ISP is quite a stage for politics.