How to retrieve tweets from command line?
From a-close-date on, Twitter won't let you into their API without having an OAuth key. But as a workaround you can use Search API. It is RESTful, so you can use curl to retrieve search results in JSON format. For example, if you want to retrieve @java's tweets, and save it to file ~/.tweets, this line of code can be used:
curl http://search.twitter.com/search.json?q=from:java&page=1&rpp=10&callback=? >> $HOME/.tweets
And you can parse the file, using any JSON parser.
The
rpp parameter is number af tweets to be retrieved. callback is javascript function to be executed on the resulting JSON. In case you're not using JavaScript with the api, you can leave it ?, but don't remove it. I will cause an error. More guidance on Search api can be found on https://dev.twitter.com/docs/api/1/get/search
There are tools to parse JSON from command line interface. Although I've never used one, I'll put some links to some resources, to help you find out the best suited tool:
- https://stackoverflow.com/questions/3858671/unix-command-line-json-parser
- http://www.conigliaro.org/2011/01/24/jazor-a-simple-command-line-json-parsing-tool/
- https://stackoverflow.com/questions/1955505/parsing-json-with-sed-and-awk
- http://kmkeen.com/jshon/
And as a little note, it is quicker to use some Python or Ruby (or others).
If you don't want to use the Twitter API, you could grab the RSS feed of the Twitter profile using a bash script and then proceed to format it from there.
- https://stackoverflow.com/questions/8632233/read-rss-feed-using-a-shell-script
Since Twitter API has deprecated the RSS feed, you can workaround this by generating RSS feed using the search results.
Here's the RSS feed of my tweets.
You would have to put together the necessary bash script though. From fetching the RSS feed to formatting the tweets as per your requirements.
This is my script made for screensaver usage
#!/bin/bash
user="$1"
user="${user:=pontifex_pl}"
last_status_url=$(lynx -dump https://twitter.com/$user \
| grep -i "$user/status/" \
| head -1 \
| awk '{print $2}')
lynx -dump "$last_status_url" \
| grep 'Twitter:' -m1 -A4 \
| tr -d '\n' \
| sed -e 's/[^"]*"//' -e 's/".*//' \
| tr -s ' '
echo
I made a tool that should do almost exactly what you described: twitter-screen-scrape . By default it will output in JSON, with metadata, but it's trivial to pipe the output through something like underscore-cli to strip out everything you don't want.
$ twitter-screen-scrape -u slang800 | underscore pluck text --outfmt text > outputfile
You could go the python + tweepy route by:
- Creating your own Twitter Application (To get API Keys)
- Create Access tokens for you twitter account
- Use a script such as this together with your credentials: https://gist.github.com/yanofsky/5436496
I just tested it and it works great. There are however users that protect their tweets, so it might not let you download everything. But that's a feature of twitter.
Limit still is 3200 and you get a CSV file.