[Alex D] - There is an electronic music company called cycling 74, who make MAX/MSP among other things; and it could just possibly be a reference to that.
[Mark S] - Many have suspected the title as being a reference to Albert Hofmann's trip on LSD known as Bicycle Day.
[Forfy] - The Peel Session version was used in season two, episode twelve of Six Feet Under.
Samples / Lyrics
The eerie choir heard throughout much of Happy Cycling is sampled from Norman Jewison's Jesus Christ Superstar, specifically "The Crucifixion". 
The backmasked vocals beginning roughly at 6:25 are thought to be of an interview with Jeff Lynne of the Electric Light Orchestra. The subject seems to be ELO's supposed use of backmasking on their song "Eldorado" from the album of the same name, and the controversy thereof. Though the vocals are occasionally difficult to understand when reversed, the following transcript is relatively complete:
(Interviewer's voice): "... to tell us about backwards masking ??? and the Electric Light Orchestra. A whole crowd of ... got all ... because they thought that if ... but, when it was hand-spun backwards on a turntable, and it said something like 'Christ is inferno'. Anyway, it was enough to ... cause some problems for ELO and Jeff Lynne in particular."
(Jeff Lynne's voice): "It is absolutely ...don't ask why I said that. It didn't say anything, anything of the sort. And uhm, that was total...in fact, it was a mistake, isn't it. Because anyone who can write a song played forwards and have it say something else backwards has got to be some kind of genius, and that I ain't! I was upset at first by the accusations, you know, but now, but now I would find them funny ... we're totally innocent of all those claims ... and we're all God-fearing chaps!"
[Sam Lambert] has pointed out yet another of those little details in the track that can be so easily overlooked. During the first part of the track, whenever the seagull sounds are heard, there is also a sound like a man singing, rather faint and tinny.
The bird calls are sampled from Vangelis - La Fête sauvage. Sample starts at 2:43. 
Listen to the backmasked vocals at the end of "Happy Cycling". The clip has been reversed, and hard panned to the right channel to boost the vocals.