Following on from the first of this series of tutorials on how to extract Xbox Live achievement data using PHP and AppleScript, I thought I would use this second part to look at the AppleScript that powers one side of the system I've put together. In the next part, I'll be explaining the PHP class I've built, and in the fourth part (the last of the series) I'll be showing you how the two talk together and how you can use the collected data with other APIs such as Facebook Connect.

So, let's have a look at some code!

XboxLive AppleScript

set urlFilePath to "http://externals.bendodson.com/weblog/XboxLive/urls.txt"
set dataFilePath to "server:XboxLive:data.txt"
set toCrawl to ""

set dataFile to open for access file dataFilePath with write permission
set eof of dataFile to 0
close access dataFile

tell application "Safari"
	open location urlFilePath
	delay 1
	do JavaScript "window.location.reload()" in the front
	delay 1
	try
		set toCrawl to (the text of the front document)
	end try
end tell

if length of toCrawl > 0 then
	tell application "Safari" to open location toCrawl
	delay 15
	tell application "Safari"
		set sourceCode to the source of front document as string
		set dataFile to open for access file dataFilePath with write permission
		write sourceCode to dataFile starting at eof
		close access dataFile
	end tell
end if

tell application "Safari" to close every window

This is the first time I've used AppleScript for anything other than just playing around and I have to say that as a language it's incredibly good. Just by reading through the above, you'll probably be able to work out what's going on even if you've never seen any type of programming code in the past. Even so, I'll go through each section and explain what it does along with the reasons why I decided to do it in this particular way rather than some of the other ways I could have chosen.

Note: All of this AppleScript is completely self-taught from searching around on the internet. I was going to buy the book AppleScript: The Missing Manual but I was able to read all the sections I needed on Google Books which was convenient - I'll probably buy the book anyway to brush up on a few other areas. If you are an AppleScript guru and you know a way to optimize my code, please use the comments section below so others can learn and so I can update it.

Before we get on to looking at the code, it might be worth briefly recapping how everything will work. My server will check the XboxLive API in order to see if my gamerscore has increased. If it has, then it saves the URL of the achievements page for my most recently played game (which it can't itself read due you needing to login to Xbox Live with javascript enabled - something cURL can't do!) in a text file on the server. My mac mini at home then runs the above AppleScript in order to retrieve the saved URL, open it in Safari, and save the HTML in it's own text file that is available via the internet. My server will then check this text file, parse the HTML, and save the achievements to a database.

How does it all work?

Now we've got that out of the way, let's look at that AppleScript in more detail:

set urlFilePath to "http://externals.bendodson.com/weblog/XboxLive/urls.txt"
set dataFilePath to "server:XboxLive:data.txt"
set toCrawl to ""

These three lines of code are used to define variables which we will use later on in the code. The first one, urlFilePath, is the URL of the text file on my server that will tell our script what URL we need to retrieve the HTML from. You'll see how we populate that text file with my XboxLive php class which will be discussed in part three of this four part series. The second variable, dataFilePath, is interesting as it contains the path to the file we are going to save the HTML to on the local machine. So why the strange syntax? This is referred to as Finder syntax and is a way for AppleScript to reference a particular section within Finder, in this case a text file. It is essentially the same as "/server/XboxLive/data.txt" which we could have used - the difference is that we would have had to have converted that into the Finder syntax in order to use some of the file editing commands we'll use later so I thought it best just to save it correctly at the beginning.

set dataFile to open for access file dataFilePath with write permission
set eof of dataFile to 0
close access dataFile

These three lines are again fairly easy to follow. We set a variable of dataFile to be the handler of the file declared in the path of dataFilePath. Note that we have specifically mentioned we want to use write permissions as the default is just to use read permissions. The next line sets the eof or "end of file" within the handler to 0 whilst the following line tidies up by closing the file handler (which isn't strictly necessary but good practice). The reason for setting eof to 0 is that we want to delete the contents of the file before we put anything else in later. This is practical for the simple reason that we don't want our PHP script on the server to parse a load of data in the text file (or even download it) if it's something it has already read as that would be a waste of processing power and bandwidth.

tell application "Safari"
	open location urlFilePath
	delay 1
	do JavaScript "window.location.reload()" in the front
	delay 1
	try
		set toCrawl to (the text of the front document)
	end try
end tell

Now we get to the first real section of the code that deals with our problem. In these lines, the application Safari is made to open the text file on the server that may contain the URLs, refresh that page using JavaScript, and then attempt to set the variable toCrawl to the text of the file. Before we even examine this in depth, you may be wondering something along the lines of "why don't you download the file or read it with FTP rather than opening it in Safari" and this would be sensible. My initial thoughts on how to get the text on the server into a variable in my AppleScript was to access the file via FTP. OS X has very basic FTP support (read only unfortunately) that can be mounted through Finder and then accessed using the Finder syntax we used earlier on. I had originally written some AppleScript that would run in the startup process of the mac mini which would mount the drive. Then, this AppleScript read the file in using open for access file urlFilePath and set the variable that way. It all worked perfectly until the server changed the contents of the URL text file. No matter what I did, the text file came back the same as it had when first fetched and it was that that I realised that the FTP built into Finder is fundamentally flawed as everything is cached. If you don't edit the file through Finder (e.g. by using a mac application that saves it again through Finder) then it will never know it's updated and thus can't be used in this scenario.

With that out of the way, let's look at my workaround. The first and last lines are the opening and closing of a tell; a way to get an application to do what we want. In this instance we are going to tell Safari to open the URL saved in the variable urlFilePath and then delay for one second. This delay is needed as Safari may take this long to open the URL. Without the delay, we may be in danger of running code on a page that hasn't loaded. In the next line, we tell JavaScript to reload the document before waiting another second for this to complete. This refresh is necessary to clear any caching of the document. The final section is used to set the variable toCrawl to the contents of the browser window. You may wonder why there is a try statement wrapped around it? This is because if the text file was empty and you tried to get the text of the front document, the script would error. To get around that, we initially set the variable (in the very first block of code if you remember?) and then use try to reset the variable only if no error would be caused in doing so. Very useful!

By the end of this block of code, we will have a variable which will either contain a URL if the text file on the server had one, or it will be empty, meaning that the server is not requesting any HTML. Let's move on to the next section:

if length of toCrawl > 0 then
	tell application "Safari" to open location toCrawl
	delay 15
	tell application "Safari"
		set sourceCode to the source of front document as string
		set dataFile to open for access file dataFilePath with write permission
		write sourceCode to dataFile starting at eof
		close access dataFile
	end tell
end if

This is the core piece of functionality that I was trying to achieve all in this block of 11 lines. This script will open a URL in Safari, and then save the source code of the loaded page in a text file. You'll notice that the first and last lines are an if statement relating to the length of the toCrawl variable. I don't unlock achievements every 5 minutes of the day and so, more often than not, the toCrawl variable will be empty. If it is, then we want to completely ignore the next section of code as there is no reason whatsoever to run it!

The next line is a one line tell to make Safari open the URL we saved which is then followed by a 15 second delay. You'll notice this is a lot longer than the 1 second delays earlier. The reason for this is that, in the first case, it was a simple text file with around 100 characters in it and so loaded incredibly quickly. This URL, conversely, will be a very large page (around the 100kb mark) that may go through a series of 5 redirects depending on how recently the page was accessed. This is because after 15 minutes of inactivity, the site forces you back to the login page but I have a cookie stored that will then automatically log me back in. It just takes a few seconds to go through the process of all the redirects to get to the actual page hence the long time delay.

The last section is a simple expansion of the code we used at the beginning. We tell Safari to set a variable of sourceCode to be the source of the page that's open - we also tell it to be forced as a string in case there are any casting issues. Next, we open the dataFilePath and set a handler of dataFile so that we can then write the sourceCode variable into the file starting at the end of the file (which we all know is masquerading as the beginning of the file also as we set it earlier on... keep up!) before tidying up after ourselves and closing access to the handler. Easy!

tell application "Safari" to close every window

In the very final line of code, we tell Safari to close all of it's windows in preparation for the next iteration. This may not seem terribly important, but trust me, if you neglect to put it in and then unlock 10 achievements, you'll find your mac now has 20 open Safari windows!

Conclusion

So that's all there is to this section - a large chunk of AppleScript and a rationale as to why I had to open Safari to get at a text document rather than doing it a slightly more simple way via FTP (due to massive caching issues). I hope this post has introduced some of you to AppleScript which I have found to be a rather powerful tool when it comes to mac development. It's very easy to understand and is a great way of transitioning from a web-based language to a desktop-based one especially as you can save AppleScript as a standard mac application.

Join me again for part three of this four part series when I'll be looking at this from the other angle; the PHP server that needs to parse the HTML we have gathered using this AppleScript. To make sure you don't miss a section, you can subscribe to the RSS Feed or follow me on Twitter. Please feel free to leave any comments or suggestions on this page.