When a company wants to offer more products, which option is the best: family branding or product line extension or launching a new brand?

It very much depends on the product.
If the new product line is very relevant to main product line, the first two are better options.
If it is a bit irrelevant and will stand well on its own, then launching a new brand makes more sense.

Family branding; as product line extension may dilute the brand where as launching a new brand may be expensive.