PHP, iPad, and iPhone Developer - Ben Dodson » Code

TwitterFollowers PHP Class – A Better Way To Track Followers, Quitters, and Returning Followers on Twitter

Ben Dodson — Mon, 25 May 2009 18:10:40 +0000

Over the past few months, there have been a number of web apps that have popped up with the task of feeding your ego (or indeed deflating it) by telling you exactly who is following you on Twitter and giving you pretty graphs to show you how your followers are increasing – some of them even go so far as to estimate how many followers you are likely to have in a weeks time! However, the key thing for me that is missing from Twitter is the ability to see who has stopped following you and also those people that stopped but are now following you again as you don’t get email alerts from Twitter for these two things. This is a useful piece of information to have as it will let you know when people drop off and whether they are important (e.g. friends who don’t care what you are talking about thus suggesting you should stop talking crap) or not (e.g. spam bots).

I did a bit of research and the only real web app to fulfill this need that I could find was the beautifully designed Qwitter. However, the problem with Qwitter is that it only gives you the details for one person with the idea being that you say “tell me when this username stops following me” – it’s a little bit stalkerish in my opinion! Like any PHP developer, I decided that I could build my own little system to give me my Twitter ego boost and so have come up with the class below which you can all now take and use as you see fit.

Update: Turns out I was wrong about Qwitter as the username you put in to follow is supposed to be yours, not the person you want to watch when they leave you. They need to do better copywriting! In any case, the class below serves as a good demo of the public Twitter data and allows you to extend it the way you want.

Note: This won’t work straight out of the box – I’ve put in a few comments which say “SQL Required”. This is because you may well have your own schema (although I do provide one) and you may have your own framework or DB connection functions (I know I do). Once you’ve done those, just substitute the constants for your own details and it should all work

TwitterFollowers.class.php


 
/**
 * Crawls Twitter Followers and sends an email alert to show you who has started following, stopped following, and started re-following
 *
 * @author Ben Dodson
 **/
class TwitterFollowers
{
	// define constants
	const username	= 'bendodson';
	const email 	= 'ben@bendodson.com';
	const subject	= 'Twitter Updates';
	const from	= 'TwitterBot ';
 
	// define internal variables
	protected $actualFollowers = array();
	protected $internalFollowers = array();
	protected $followerChanges = array();
	protected $now = '';
 
	function __construct()
	{
		$this->now = date('Y-m-d H:i:s');
 
		$json = file_get_contents('http://twitter.com/followers/ids/'.self::username.'.json');
		$this->actualFollowers = json_decode($json);
		$this->internalFollowers = $this->getInternalFollowers();
 
		foreach ($this->actualFollowers as $actualFollower) {
			if (!in_array($actualFollower, $this->internalFollowers)) {
				if ($this->getTweeter($actualFollower)) {
					$this->followerChanges['returning follower'][] = $actualFollower;
					/*
					--SQL REQUIRED--
					UPDATE TwitterFollowers SET start = $this->now, end = NULL WHERE id = $actualFollower
					*/
				} else {
					$this->followerChanges['new follower'][] = $actualFollower;
					/*
					--SQL REQUIRED--
					INSERT INTO TwitterFollowers (id) VALUES ($actualFollower)
					*/
				}
			}
		}
 
		foreach ($this->internalFollowers as $internalFollower) {
			if (!in_array($internalFollower, $this->actualFollowers)) {
				$this->followerChanges['stopped following'][] = $internalFollower;
				/*
				--SQL REQUIRED--
				UPDATE TwitterFollowers SET end = $this->now WHERE id = $internalFollower
				*/
			}
		}
 
		$this->sendEmail();	
	}
 
	protected function getInternalFollowers()
	{
		$data = array();
		$raw = 
		/*
		--SQL REQUIRED--
		SELECT id FROM TwitterFollowers WHERE end IS NULL
		*/
		foreach ($raw as $r) {
			$data[] = $r['id'];
		}
		return $data;
	}
 
	protected function getTweeter($id)
	{
		return
		/*
		--SQL REQUIRED--
		SELET * FROM TwitterFollowers WHERE id = $id LIMIT 1
		*/		
	}
 
	protected function getTweeterDetails($id)
	{
		$json = file_get_contents('http://twitter.com/users/show/'.$id.'.json');
		$tweeter = json_decode($json);
		return $tweeter->name . ' ('.$tweeter->screen_name . ')';
	}
 
	protected function sendEmail()
	{
		$to      = self::email;
		$subject = self::subject;
		$headers = 'From: ' . self::from . "\r\n" . 'Reply-To: ' . self::from;
 
		$message = 'Hi,' . "\r\n\r\n";
		$message .= 'Here are your Twitter Updates:' . "\r\n";
 
		if (count($this->followerChanges) > 0) {
			foreach ($this->followerChanges as $changeType => $change) {
				$message .= "\r\n" . '--'.strtoupper($changeType).'--' . "\r\n\r\n";
				foreach ($change as $tweeter) {
					$message .= '*' . $this->getTweeterDetails($tweeter) . "\r\n";
				}
			}
		} else {
			$message .= "\r\n" . '--NO UPDATES FOUND--' . "\r\n";
		}
 
		mail($to, $subject, $message, $headers);
	}
 
}
 
?>

MySQL Database Schema

CREATE TABLE IF NOT EXISTS `TwitterFollowers` (
  `id` int(20) NOT NULL,
  `start` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
  `end` timestamp NULL DEFAULT NULL,
  PRIMARY KEY  (`id`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;

How does it work?

First of all, you will need to substitute the SQL sections for your own particular schema and database functions. Once you’ve done that, alter the class constants so that they are your own username and the email address you want to send your updates to. Finally, set up a CRON job so that it runs at a certain point every day. I currently have mine set to run at 9am every morning but I may well change it to run every time I post a tweet as then I’d be able to see which tweet had made people start or stop following me.

The script works by checking the publicly accessible JSON feed of your followers and getting all of their IDs. I say it’s publicly accessible, but I don’t think it is if you have protected your updates which will of course cause problems! Once it has all of the IDs, it checks this against the IDs stored in your database – if there aren’t any then everyone will show up as following you on the first run. If it finds an ID in your database that isn’t in your JSON string then you’ve been dumped! Conversely, if it finds an ID in the JSON string but not in the DB then, congratulations, you’ve got a new follower. The final instance is if it finds an ID in the JSON string that is in the DB but has an end datetime assigned to it. This means the person was following you, stopped, and has now decided to re-follow you.

The whole lot then gets packaged up and emailed to you with each section broken down so you can read it clearly. In order to do this, it looks up each ID that goes into the email against that persons publicly available Twitter information to give you both their “real name” and “username”.

Known Problems

* I don’t think it will work if you have your updates set to hidden.
* If one of your followers gets banned from Twitter, then their name won’t show up in the email (it will just be blank)
* This script won’t work if you have more than 5000 followers – this is because that is the maximum result set from the JSON string. You’d need to add paging information to get more than 5000 although this would be fairly easily. Alas I don’t have that many followers to be able to test that out!

Conclusion

So now you can easily (if you know PHP) get updates on all your Twitter followers and non-followers. Feel free to use all the above code and modify to your hearts content – if you found it to be useful, then please leave a comment below. Oh, and I couldn’t possibly write a post about Twitter without reminding you that you can follow me @bendodson

Getting Xbox Live Achievements Data: Part 2 – The AppleScript Solution

Ben Dodson — Tue, 19 May 2009 23:54:39 +0000

Following on from the first of this series of tutorials on how to extract Xbox Live achievement data using PHP and AppleScript, I thought I would use this second part to look at the AppleScript that powers one side of the system I’ve put together. In the next part, I’ll be explaining the PHP class I’ve built, and in the fourth part (the last of the series) I’ll be showing you how the two talk together and how you can use the collected data with other APIs such as Facebook Connect.

So, let’s have a look at some code!

XboxLive AppleScript

set urlFilePath to "http://externals.bendodson.com/XboxLive/urls.txt"
set dataFilePath to "server:XboxLive:data.txt"
set toCrawl to ""
 
set dataFile to open for access file dataFilePath with write permission
set eof of dataFile to 0
close access dataFile
 
tell application "Safari"
	open location urlFilePath
	delay 1
	do JavaScript "window.location.reload()" in the front
	delay 1
	try
		set toCrawl to (the text of the front document)
	end try
end tell
 
if length of toCrawl > 0 then
	tell application "Safari" to open location toCrawl
	delay 15
	tell application "Safari"
		set sourceCode to the source of front document as string
		set dataFile to open for access file dataFilePath with write permission
		write sourceCode to dataFile starting at eof
		close access dataFile
	end tell
end if
 
tell application "Safari" to close every window

This is the first time I’ve used AppleScript for anything other than just playing around and I have to say that as a language it’s incredibly good. Just by reading through the above, you’ll probably be able to work out what’s going on even if you’ve never seen any type of programming code in the past. Even so, I’ll go through each section and explain what it does along with the reasons why I decided to do it in this particular way rather than some of the other ways I could have chosen.

Note: All of this AppleScript is completely self-taught from searching around on the internet. I was going to buy the book AppleScript: The Missing Manual but I was able to read all the sections I needed on Google Books which was convenient – I’ll probably buy the book anyway to brush up on a few other areas. If you are an AppleScript guru and you know a way to optimize my code, please use the comments section below so others can learn and so I can update it.

Before we get on to looking at the code, it might be worth briefly recapping how everything will work. My server will check the XboxLive API in order to see if my gamerscore has increased. If it has, then it saves the URL of the achievements page for my most recently played game (which it can’t itself read due you needing to login to Xbox Live with javascript enabled – something cURL can’t do!) in a text file on the server. My mac mini at home then runs the above AppleScript in order to retrieve the saved URL, open it in Safari, and save the HTML in it’s own text file that is available via the internet. My server will then check this text file, parse the HTML, and save the achievements to a database.

How does it all work?

Now we’ve got that out of the way, let’s look at that AppleScript in more detail:

set urlFilePath to "http://externals.bendodson.com/XboxLive/urls.txt"
set dataFilePath to "server:XboxLive:data.txt"
set toCrawl to ""

These three lines of code are used to define variables which we will use later on in the code. The first one, urlFilePath, is the URL of the text file on my server that will tell our script what URL we need to retrieve the HTML from. You’ll see how we populate that text file with my XboxLive php class which will be discussed in part three of this four part series. The second variable, dataFilePath, is interesting as it contains the path to the file we are going to save the HTML to on the local machine. So why the strange syntax? This is referred to as Finder syntax and is a way for AppleScript to reference a particular section within Finder, in this case a text file. It is essentially the same as “/server/XboxLive/data.txt” which we could have used – the difference is that we would have had to have converted that into the Finder syntax in order to use some of the file editing commands we’ll use later so I thought it best just to save it correctly at the beginning.

set dataFile to open for access file dataFilePath with write permission
set eof of dataFile to 0
close access dataFile

These three lines are again fairly easy to follow. We set a variable of dataFile to be the handler of the file declared in the path of dataFilePath. Note that we have specifically mentioned we want to use write permissions as the default is just to use read permissions. The next line sets the eof or “end of file” within the handler to 0 whilst the following line tidies up by closing the file handler (which isn’t strictly necessary but good practice). The reason for setting eof to 0 is that we want to delete the contents of the file before we put anything else in later. This is practical for the simple reason that we don’t want our PHP script on the server to parse a load of data in the text file (or even download it) if it’s something it has already read as that would be a waste of processing power and bandwidth.

tell application "Safari"
	open location urlFilePath
	delay 1
	do JavaScript "window.location.reload()" in the front
	delay 1
	try
		set toCrawl to (the text of the front document)
	end try
end tell

Now we get to the first real section of the code that deals with our problem. In these lines, the application Safari is made to open the text file on the server that may contain the URLs, refresh that page using JavaScript, and then attempt to set the variable toCrawl to the text of the file. Before we even examine this in depth, you may be wondering something along the lines of “why don’t you download the file or read it with FTP rather than opening it in Safari” and this would be sensible. My initial thoughts on how to get the text on the server into a variable in my AppleScript was to access the file via FTP. OS X has very basic FTP support (read only unfortunately) that can be mounted through Finder and then accessed using the Finder syntax we used earlier on. I had originally written some AppleScript that would run in the startup process of the mac mini which would mount the drive. Then, this AppleScript read the file in using open for access file urlFilePath and set the variable that way. It all worked perfectly until the server changed the contents of the URL text file. No matter what I did, the text file came back the same as it had when first fetched and it was that that I realised that the FTP built into Finder is fundamentally flawed as everything is cached. If you don’t edit the file through Finder (e.g. by using a mac application that saves it again through Finder) then it will never know it’s updated and thus can’t be used in this scenario.

With that out of the way, let’s look at my workaround. The first and last lines are the opening and closing of a tell; a way to get an application to do what we want. In this instance we are going to tell Safari to open the URL saved in the variable urlFilePath and then delay for one second. This delay is needed as Safari may take this long to open the URL. Without the delay, we may be in danger of running code on a page that hasn’t loaded. In the next line, we tell JavaScript to reload the document before waiting another second for this to complete. This refresh is necessary to clear any caching of the document. The final section is used to set the variable toCrawl to the contents of the browser window. You may wonder why there is a try statement wrapped around it? This is because if the text file was empty and you tried to get the text of the front document, the script would error. To get around that, we initially set the variable (in the very first block of code if you remember?) and then use try to reset the variable only if no error would be caused in doing so. Very useful!

By the end of this block of code, we will have a variable which will either contain a URL if the text file on the server had one, or it will be empty, meaning that the server is not requesting any HTML. Let’s move on to the next section:

if length of toCrawl > 0 then
	tell application "Safari" to open location toCrawl
	delay 15
	tell application "Safari"
		set sourceCode to the source of front document as string
		set dataFile to open for access file dataFilePath with write permission
		write sourceCode to dataFile starting at eof
		close access dataFile
	end tell
end if

This is the core piece of functionality that I was trying to achieve all in this block of 11 lines. This script will open a URL in Safari, and then save the source code of the loaded page in a text file. You’ll notice that the first and last lines are an if statement relating to the length of the toCrawl variable. I don’t unlock achievements every 5 minutes of the day and so, more often than not, the toCrawl variable will be empty. If it is, then we want to completely ignore the next section of code as there is no reason whatsoever to run it!

The next line is a one line tell to make Safari open the URL we saved which is then followed by a 15 second delay. You’ll notice this is a lot longer than the 1 second delays earlier. The reason for this is that, in the first case, it was a simple text file with around 100 characters in it and so loaded incredibly quickly. This URL, conversely, will be a very large page (around the 100kb mark) that may go through a series of 5 redirects depending on how recently the page was accessed. This is because after 15 minutes of inactivity, the site forces you back to the login page but I have a cookie stored that will then automatically log me back in. It just takes a few seconds to go through the process of all the redirects to get to the actual page hence the long time delay.

The last section is a simple expansion of the code we used at the beginning. We tell Safari to set a variable of sourceCode to be the source of the page that’s open – we also tell it to be forced as a string in case there are any casting issues. Next, we open the dataFilePath and set a handler of dataFile so that we can then write the sourceCode variable into the file starting at the end of the file (which we all know is masquerading as the beginning of the file also as we set it earlier on… keep up!) before tidying up after ourselves and closing access to the handler. Easy!

tell application "Safari" to close every window

In the very final line of code, we tell Safari to close all of it’s windows in preparation for the next iteration. This may not seem terribly important, but trust me, if you neglect to put it in and then unlock 10 achievements, you’ll find your mac now has 20 open Safari windows!

Conclusion

So that’s all there is to this section – a large chunk of AppleScript and a rationale as to why I had to open Safari to get at a text document rather than doing it a slightly more simple way via FTP (due to massive caching issues). I hope this post has introduced some of you to AppleScript which I have found to be a rather powerful tool when it comes to mac development. It’s very easy to understand and is a great way of transitioning from a web-based language to a desktop-based one especially as you can save AppleScript as a standard mac application.

Join me again for part three of this four part series when I’ll be looking at this from the other angle; the PHP server that needs to parse the HTML we have gathered using this AppleScript. To make sure you don’t miss a section, you can subscribe to the RSS Feed or follow me on Twitter. Please feel free to leave any comments or suggestions on this page.

Getting Xbox Live Achievements Data: Part 1 – The PHP Problems

Ben Dodson — Tue, 12 May 2009 09:55:10 +0000

Those of you with an Xbox 360 (or indeed some “Games for Windows” titles) will know all to well about the achievements system prevalent in every game. For those that don’t know, every gamer has a profile which has a gamerscore. This score goes up by completing certain tasks within each game as laid down by the developer. This could be something you would do anyway such as “finish the game” or something random such as “destory 10 cars in 10 seconds”. Every full game can give out 1000 gamerpoints (1250 with expansion packs) and an Xbox Arcade title can give out 200. These points are somewhat of a geek badge of honour for most Xbox gamers who will try and do everything to get the full 1000 in each of their games – there are also those that want to increase the number as quickly as possible so you can find numerous guides online for the easiest way to get 1000 points (it seems Avatar is still the best way giving you the full 1000 with about 3 minutes of gameplay!)

When I was trying to compete with my ex-boss over the number of gamerpoints we each had (I lost by the way), I found that there was no public API from Microsoft to allow you to get at the Xbox Live data. There was however an internal API and one Microsoft associate had set up a restful API so that you could publicly call the internal one. This worked well enough for the basic site I put together to compare two gamerscores but I’ve been wanting to do more with the API for some time.

My overall idea is that I’ll be able to type in my userid and then have my server poll Xbox Live at a certain time and then update my Facebook Wall when I unlock new achievements. The message would be something along the lines of “Ben just completed the ‘Fuzz Off’ achievement in Banjo Kazooie: N&B and earned 20G” and would have the correct 32×32px image for the achievement. I initially thought that this would be fairly easy but I was unfortunately very wrong! In this series, I’m going to show you the problems I encountered as well as the final (rather complex) workaround I’m creating in order to get it all to work! If you’ve got any questions, please leave a comment or get in touch.

Attempt #1: Using the API

When I first sat down to work on this project, my initial thoughts were “I can just reuse the public API I used for my gamerscore comparison site – there’s bound to be an achievement section in the returned data”. After eagerly re-downloading all the code, I discovered that although there was some achievement data, it was nowhere near as detailed as the information that I would need. The problem was that the API only shows you your recently played games and how many achievements you have unlocked in each one as well as the overall number of points you have earned for that game. Theoretically, I could check the API every few minutes and compare the number of points with a local copy in order to work out when a new achievement had been unlocked but I’d only be able to say that an achievement had been unlocked in a certain game worth a number of points. To make things even trickier, if I unlocked more than one achievement within the timeframe of the API check, then the results would be wrong (e.g. it might say I’d unlocked one achievement worth 45G when in fact I’d done two; one for 20G and one for 25G). This would become even more complex if I unlocked an achievement, then switched games and unlocked one in that game before the API had been called. In short, the public API, useful though it can be, was not going to work for this.

Attempt #2: Screen Scraping

So now we move to option two; screen scraping. This is the process of getting the server to request pages from a website as if it were a browser and then just ripping the content out of the HTML. It’s messier than an API as it relies on the websites HTML not changing and it’s also a lot more processor intensive (as you’re parsing an entire XHTML page – possibly marked up invalidly – rather than a nice small XML or JSON file). I’ve done lots of screen scraping in the past, both for my Tube Updates API and for the Packrat Market Tracker (a tracking system for a Facebook game), so I didn’t think it would be too much hassle. But then I hadn’t banked on Microsoft…

The first hurdle is that although my Xbox Live data is set to be shown publicly, you still have to be logged in with a Windows Live account to view it. This is annoying because it means my script is going to have to log in to Windows Live in order to get the HTML of my achievements listings. The second hurdle is that there is no single page listing my latest unlocked achievements – the main profile page shows my last played game (and it’s last unlocked achievements) but that’s no good as they are not in order and it might be that I’ve switched games after unlocking something so the last achievement on the profile page may not be the last achievement I’ve unlocked. This isn’t such a big problem as there are pages for each game so I’ll just have to crawl each of my recently played games pages and get the achievements on each one but it’s slightly more hassle than having one page of latest achievements (as it means I have to make several requests thus increasing bandwidth and script run time).

Logging In to Windows Live

Generally, logging into a site is quite easy using cURL. You need to work out where the form is posting to, put all of the data to be posted in an array, and then make a cURL request that sends that array to that URL. You will also need to enable both a cookie file and a cookie jar (a basic text file that is used for all of the cookies during the requests) as you will probably only want to login once and then have each future request know that you are already logged in as this will save on overall requests per execution of the task.

The Windows Live login, on the other hand, is an entirely different kettle of fish! The URL you are posting to changes on each request as do the variables that you are posting. This means we need to make a request to the login page first of all and extract all of the data from the hidden input fields as well as the action attribute of the form. We can then go about posting that data (along with our email address and password) to the URL we just extracted. This POST goes through a HTTPS connection though, so we need to modify our cURL request further in order to ensure that SSL certificates are just accepted without question. Our overall cURL request, with all of these options, will look roughly like this:


// set up cURL request - the $url would be the action URL that you're POSTing to
$curl = curl_init($url);
 
// make sure the script follows all redirects and sets each one as the referer of the next request
curl_setopt($curl, CURLOPT_AUTOREFERER, true);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($curl, CURLOPT_HEADER, false);
 
// ssl options - don't verify each certificate, just accept it
curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false);
 
// fake the user-agent so the site thinks we are a browser, in this case Safari 3.2.1
curl_setopt($curl, CURLOPT_USERAGENT, 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_6; en-us) AppleWebKit/525.27.1 (KHTML, like Gecko) Version/3.2.1 Safari/525.27.1');
 
// tell cURL to use a text file for all cookies used in the request - $cookie should be a path to a txt file with 755 permissions
curl_setopt($curl, CURLOPT_COOKIEFILE, $cookie);
curl_setopt($curl, CURLOPT_COOKIEJAR, $cookie);
 
// post options - the data that is going to be sent to the server.  $post should be an array with key=>var pairs of each piece to be sent
foreach ($post as $key => $var)
{
	$postfields .= $key . '=' . urlencode($var) . '&';
}
curl_setopt($curl, CURLOPT_POST, true);
curl_setopt($curl, CURLOPT_POSTFIELDS, $postfields);
 
// make the request and save the result as $response - then close the request
$response = curl_exec($curl);
curl_close($curl);
?>

I had thought that this would be the end of it and that the returned data would be the first page after logging into Windows Live. Instead, I got nothing. Absolutely nothing. No matter what settings I tinkered with or parts of the code I changed, it was just returning blank. It was then that I noticed the rather unpleasant JavaScript files on the page and some suspicious