Hi ncrelae! I honestly don't know of a site on the Internet that sells replacement parts for the Exceed-RC Pitts.
However, attaching/re-attaching the rudder to the vertical stabilizer and installing the horizontal stabilizer is really easy.
For the rudder (which should have come preattached to the vertical stabilizer, so I'm assuming you need to re-attach it), I like using AC-style hinges like
these on hobbycity.com. Just make two small slits in the rudder (one each at about 1" from the top and bottom) and then line the rudder up to the horizontal stabilizer and make two corresponding slits in the vertical stabilizer. Then, squirt a little Gorilla, Sumo, or CA glue in each of the slits, insert the two AC hinges in the rudder, then slide them into the corresponding slits in the vert. stabilizer. Let it dry, and you should be good to go.
For attaching the horiontal stabilizer to the fuselage, you really should glue that. I recommend using a light, but very tough glue like Gorilla or Sumo. Alternately, you can use CA or 5-minute epoxy, which would also work fine. You could try taping it on or even going with some kind of breakaway velcro solution, but I wouldn't want to risk losing the plane due to it working its way lose. Glue that sucker back on and be done with it. :)
Good luck!