As they have said, you can remove all of the emissions stuff.
EGR, AIR Pump, Catalytic Converters, etc.
It will only have a positive impact on performance, you'll have 1 less fan belt, the engine compartment will be much neater, and you'll have an easier time doing maintenance (parts will be easier to reach).
I've already done all of the things listed above, but I kept the PCV valve. That is not an emissions thing and it will help keep your oil clean.
Since this will be a dedicated road car, I would recommend changing the camshaft, rocker arms, carburetor (depending on what you have), headers, and intake. That combined with removing the emissions stuff will give you a serious boost in performance. Certain intake manifolds won't even have a place for EGR, so you won't need a block off plate (that's what I have).
Post any questions if you have them. Everyone is new when they start.
Good luck and Have fun!