Well, probably. If you have a Google account, then it is. If you're using a smartphone and you use it to do just about anything on it, then it really is. Here's a look at my location map for the last month, which is the most that you can pull up in one go:
For good measure, I'm also showing you where I was on a specific day as well! You can choose different time periods, but it's still a maximum of a month at a time. You can see yours by going to the Google Maps Location History page (if you have one - you may need to log into your Google account at first) and then take a look.
Now, if you're freaked out by it, Google does allow you to delete your history for the time period that you're looking at, or for your entire history. If you're worried because you're using your smartphone, you can stop Google using this as well, by going to: Settings > Privacy > Location Services, and turn the slider switch off for Google Maps. If you're an Android user I can't help you directly because I don't use it, but Google has words of wisdom on it that you can use.
Is it something that you should be worried about? Depends entirely on your point of view really. I don't care in the slightest if Google knows where I am, and I actually think that it might be quite useful, certainly if I need to remember where I was on a particular day, and possibly also work out what I was doing. However, if you're particularly concerned about your privacy you can delete it all of course. And if you're that bothered, you'll really need to delete your Google account, and possibly give up using your smartphone as well. I've no knowledge or examples of how Google is using this information, if indeed they are to any great extent (obviously it's very helpful for me if Google knows where I am, so that I can use the mapping service properly), but in all honesty, if they want to, they're going to. And if you really want a one way ticket to paranoid city, even if you do get rid of the location history, how do you know that they don't have a mirror of it somewhere that they're not telling us about?
So - up to you, but I'm not that fussed personally.