I'd go with fraps, people complain about the huge file size, but really once you render everything out, it shrinks down. It is AVI but since you hopefully are planning on using an editing software for your video. It doesn't matter for example a leauge of legends game will be about 60gb-80gb of footage once it is rendered out it is 1gb. Delete the footage and boom 1gb video ready for youtube.
^this all the way
Plus if you want to record a large amount of shiz I would highly recommend buying an externel hard drive of about 1tb in size... Just send the recordings to this baby and Bob's your uncle! :3