Is it Doctype mandatory to use in HTML5?

It depends on the target browsers. Webkit-browsers, Firefox and Opera will handle HTML5 elements quite normally even if your doctype is not HTML5.

IE9 on the other hand (I bet you saw this coming), may behave entirely differently on another type of doctype. If IE9 is not in IE9 Standards mode (it could be in quirks, IE8 compat, whatever), it will not support the HTML5 features it does in IE9 standards.

So essentially your main concern is that you need to make sure all browsers go into strict standards mode. The easiest way to achieve this is to use the HTML5 doctype, since it will trigger standards mode in all browsers - including older browsers that don't actually support HTML5.

Muhammad answer describes it completely, it is only IE that depends so heavily on the DOCTYPE, from a ASP.NET web forms perspective , postbacks and AJAX calls might fail as the server generated scripts become incompatible.

Yes, Because it defined that which version of HTML language is used.