The formula of angle between two lines in space is:

\vec{a}=(a_1,a_2,a_3) ; \vec{b}=(b_1,b_2,b_3)

cos\alpha=\frac{|\vec{a} \vec{b}|}{|\vec{a}||\vec{b}|}

or out from there:

cos\alpha=\frac{|a_1b_1+a_2b_2+a_3b_3|}{\sqrt{a_1^ 2+a_2^2+a_3^2}\sqrt{b_1^2+b_2^2+b_3^2}}

Why it is |\vec{a}\vec{b}|[/itex]? Why not [tex]\vec{a} \vec{b}?

Scalar product of two vectors is \vec{a}\vec{b}=|\vec{a}||\vec{b}|cos(\vec{a},\vec{ b})

Because you're measuring the angle between -pi/2 and pi/2. a.b can be negative, but |a.b| is non-negative. Drawing a cos graph, you can see that if you fix cos(x) to be non-negative then you fix your angle to be between -pi/2 and pi/2.

Consider two lines, say the x axis and some line through the origin at an angle A to the x axis. As you increase A from 0 to pi/2, no problem. But when you get to A=pi/2 you can see that it's exactly the same as if you'd set A=-pi/2. Going to A = pi/2 + B is the same as having A = -pi/2 + B, because the line goes through the origin, not just from the origin out in one direction.

Hence, to have a single values well defined notion of angles in such a construction, you take |a.b|.

Here is one picture
http://pic.mkd.net/images/394221untitled12.JPG

Aren't they both correct?

That isn't what the | | on the a.b do.

Putting |a.b| rather than a.b doesn't make the angle always positive, it makes the cosine of the angle always positive.

Look at the cos graph :
http://upload.wikimedia.org/wikipedia/commons/thumb/3/38/Sine_cosine_plot.svg/300px-Sine_cosine_plot.svg.png

\cos(x) \geq 0 implies x \in [-\frac{\pi}{2}+ 2\pi n \; , \;\frac{\pi}{2}+ 2\pi n]. Therefore restricting a.b to |a.b| gives an angle in [-\frac{\pi}{2},\frac{\pi}{2}]. It can still be negative.

If we didn't do that restriction, you could have an angle in any interval of width 2\pi. But this isn't true for lines. Consider two lines, initially with an angle of pi/2 between them. They are at right angles. Now increase that angle more and more until they are at an angle pi from one another. They are now equal! The lines are on top of one another! Because they don't have directionality, unlike vectors.

Two vectors with an angle of pi between them are two arrows pointing in opposite directions, ie <---------->. But two lines don't have the 'arrow heads', so you end up with ----------. That's the same as ----------, ie two lines pointing in the same way.

Vectors have a.b because they have directions. Lines have |a.b| because they don't have directions.

And what will happen if I use angle from [-pi/2, pi]? Can you give me some example?

How do you know that is <---------->??? What tells you that is <---------->? ?