ٌWhat is the Appropriate Modulation to transfer over 800Mbps data over Microwave Link?

It depend on the channel bandwidth and link fade margin, you can achieve 800 Mpbs in different channel bandwidth using the 256 bit Modulation.

Manufacturers compete between each others to produce fine tuned radios which allow for higher data rates at lower channel bandwidth while maintaining the bits error free rate @ highest.

2048QAM should do the job.