Is it big difference if I do not use "literally"

literally
used to emphasize that something, especially a large number, is actually true

• The Olympic Games were watched by literally billions of people.

Longman Dictionary of Contemporary English

Is it the same meaning if I do not use "literally" in the above sentence?

Solution 1:

From The Oxford Dictionaries Blog: 5 language arguments you can stop having

1. Literally
   Argument: Isn’t the use of literally when something isn’t actually real or happening incorrect?

For some people, there is nothing worse than the figurative literally. In standard use, literally means ‘in a literal manner or sense’ or ‘exactly’, but its extended use has become very common in the past several decades. In its figurative sense, literally is ‘used for emphasis or to express strong feeling while not being literally true’. For instance: ‘He was literally dying with laughter’. The subject of the sentence, of course, is not literally dying; the adverb is there in order to emphasize his extreme reaction.

For those plagued by this usage, it might help to point out that literary luminaries such as Mark Twain and F. Scott Fitzgerald have used the word in this sense. According to the Oxford English Dictionary (OED), the word has been used figuratively since the mid-18th century.

If something was "literally" watched by billions of people, the reporter should also provide data that supports this claim. If the writer is using a hyperbolic expression, then the adverb literally is being used correctly. Regardless, billions of viewers remains a huge number.

Summary

It's safe to use literally, even when you're literally exaggerating.

Correction: It is not literally safe to use the term literally. I would advise the OP to avoid using the term in that particular example he cited because it is ambiguous. If literally means "actually", as appears to be the case, some readers will probably object and feel confused as to whether the unspecified number (billions) is an exaggeration or in actual fact, true.

Solution 2:

Numbers like millions and billions are often hyperbole, meaning the speaker really has no idea how many, but there were a great many. Literally billions is supposed to mean that the speaker does in fact know how many, and it really was in the billions. So yes, there is an important difference.

Solution 3:

There is no real need to use 'literally' if what you are saying is obviously true in a literal sense. Considering your example, since the Olympic games are actually 'literally' watched by millions if not billions of people, you are right -- you don't need to use 'literally'! And as Dan Bron pointed out in comments, billions is probably too big a number, even for the Olympics, so I would advise you to change it to 'millions', or even 'hundreds of millions' as suggested by Mari-Lou A in comments.

The more orthodox usage of 'literally' is to give emphasis that something (that might be literally or figuratively expressed) is actually being used in a literal sense here. Example:

The concert literally brought the roof down (not figuratively but literally: the loud noise literally collapsed the roof!)

She literally kicked him out of the house.

Please note that the less orthodox but quite popular 'alternative use' of 'literally' is paradoxically meant to emphasise and intensify a figurative usage, as in

After the breakup, he was literally swimming in a sea of tears.

The headmaster literally burst with rage after seeing the boys' nasty trick.

IN SHORT, the 'literally' seems unnecessary in this context, though I expect the two sentences will not have the exact same meaning with and without 'literally.'