My friend told me the following about FP's and a cast member confirmed it to my husband:

Fast Passes can be used anytime after the starting time. The ending time doesn't matter. So if you miss your specified hour to ride it doesn't matter. Just come anytime.

If this is true, and we did do it once, then this would mess up the system. I guess this is why the FP lines had 20 minute waits sometimes. The whole purpose is to only have "so" many people coming back at certain times.

I am sure this varies with cast members. I think they should enforce the hour time on your FP.