"When I was in college..." Do you really mean college? Or university?
When someone in the US says "When I was in college..." he can mean "college" but he can also mean "university", so I've been told. If that's true, how can we know which one he is talking about? If I say my daughter is off to college, can it really mean either one?
While the OP specifies the US, some of the other answers are based on other countries. I'll respond mainly regarding the US, but I'll start with some comments regarding other countries. In many countries, college means secondary school, following on from primary school. It would never mean that to the average American.
Technically, a US college is a venue for obtaining post-high school (post-secondary) qualifications. It could be a 2-year degree, called an Associates degree, which may or may not be vocationally-based. You can get an AA or AS in welding, math, Spanish, etc. These colleges are usually known as community colleges.
The other type of college grants Bachelor's degrees (not to be confused with the bachillerato granted in many countries to what in the US would be called high school students, who are typically going on to what in the US would be called college). There are 4-year liberal arts colleges, and there are subject-oriented colleges (which are now rare).
In the US, a university would be composed of multiple colleges, including one or more bachelors colleges, as well as a grad school. You really have to offer a Masters degree to be able to call it a university.
Most Universities also offer Ph.D.s and possibly other advanced degrees (in the American academic world, we now call the MA and beyond terminal degrees), such as MD, JD, THD etc.
Some places, like Boston College, have chosen to keep College in their name, even though they are universities.
I would not say "I'm going to university"; I'd say "I'm in college" (though for my last degree I was actually taking classes in three colleges of the university).
I could say "I'm at the University of California", or "I go to Yale University", but not "I'm in university" (or "I'm in Uni") without it sounding a bit affected.
So if all I said was that my daughter was off to college, you wouldn't know if she was attending a small liberal arts school of 300 or a prestigious university, or even a local community college (but then why say she was "off to?") -- but you would know that she was studying at a level beyond high school.
People who are not in academia might use the term without knowing all these details.
In the US, there is little difference between a college and a university in common speech. Generally, institutes of higher learning with smaller student bodies call themselves colleges, while larger institutions call themselves universities, but the distinction is largely cosmetic. So you usually wouldn't know for sure, but that doesn't really matter.
In speech or writing, you would use college as the object of a preposition (e.g., "going to college") no matter whether the institution calls itself a college or a university; "going to university" would be understood, but sounds affected when spoken by a natural born American.
From a technical perspective, a university is a school of higher education that offers a graduate program, while a college does not. There are exceptions to this rule, but it mostly works.
In practice, though, these are seen as VERY SIMILAR academically in the US. In terms of my education, I can get a good education at a program that only offers a 4-year bachelors and I can also get a good education at a University that offers a post-graduate degree. I get the same education at either type of institution. That said, universities are seen as more prestigious because post-grad work creates the opportunities for research and in general, they will offer more programs and more variety of programs. That's why Americans don't see it as important to differentiate between which of the two schools they attended.
To answer your question, you cannot tell if someone went to a university or college based on a statement like "I'm in college" or "I went to college." However, if someone is talking about post-graduate work, they will say something like "In my post-grad..." or "For my masters..." In general, there's really no reason to care whether they went to a college or a university.
I feel compelled to add a non-US answer, as these words have quite distinct meanings in the UK.
A University is a seat of research and tertiary (normally age 18-21) education, which awards degrees and postgraduate qualifications.
A College can be either a place of secondary (normally age 16-18) education, with a bias toward awarding vocational qualifications. Or it can be an administrative subdivision of a University (most UK universities do not have colleges - Oxford and Cambridge famously do, and a few others have adopted that system). Which one is being spoken about when the word is used is usually obvious from the context.
The US-centric use of the word "College" as a direct alternative to "University" can cause considerable confusion to older UK English speakers, and indeed can feel annoyingly vague even to people who are aware of the different meaning across the Atlantic.
The common distinction made is that universities offer graduate programs and colleges do not, but that is not entirely true. Many colleges do have graduate programs, but they are typically applied masters programs with little to no research components. The true distinction of a university is that universities have research programs and PhD programs. In fact, universities tend to put a lot of emphasis on research.
Due to the emphasis on research, it is not uncommon for classes at universities to be taught by TAs and grad students, this is especially true of entry-level classes. The larger the university, the more likely the lower-level classes will be taught by aids and students. At smaller colleges, however, the classes are generally taught by the actual professors.
As with most things, there are exceptions. There are smaller universities with less research emphasis and there are larger colleges that do have research components.
It is also useful to note that a college can be a division of a university. For instance, you could study under the College of Arts and Sciences at Cornell University. And, to further confuse things, sometimes these divisions are called "schools," such as the NYU Tisch School of Arts.
All that said, in the US it isn't pejorative to say that someone went to "college."
http://www.becker.edu/global/college-vs-university/