Downloading files from the Apple Developer website using wget (for poor connections or scheduling)

Recently we had some issues downloading the latest iphone SDK. Firstly due to a crappy 3G broadband connection; we never seemed to be able to download the entire 3.5 Gigabyte file without a dropout. Unfortunately for us, Apple seem to have overlooked this possibility with their developer website and they do not offer the download using a more robust facility. Note to Apple. Ftp would be nice.

Secondly with several team members located in the same city but not on the same LAN we wanted to distribute the update to all members via our shared linux server. Unfortunately, this said server is getting a little long in the tooth now and being a 32 bit linux distribution it does not support files of the size that is the Xcode and IOS dmg. We were going to have to split the file into more managable chunks. What a pain.

The command line tool wget often yields answers to these kinds of problems so consequently it was our initial foray into finding a solution. Firstly it can provide an extra layer of robustness for downloading files, secondly it's very easy to schedule downloads via cron. For those of you wondering why this is a consideration - welcome to the reality of living in the internet 3rd world - Australia. With the typical internet plans in Australia, heavy internet users such as ourselves find it important to spread our download usage between peak periods (any time you're likely to be awake) and offpeak periods (any time you're likely to be sleeping) to maximise our bandwidth allocation.

Frustratingly, downloading the IOS SDK via wget is complicated by the need for any web client connecting to the Apple Developer website to have been authenticated. The Apple website is known to use cookies to authenticate web clients, and several recipes for extracting authetication credentials from browser cookies into a file and using then via the wget command line interface are well known - at least for Firefox.

The basic procedure for accessing content using wget from a site requiring authentication involves logging into the said site using a standard web browser, once authenticated via a login page one can set abou extracting the authentication cookie from the browser. The extracted cookie is then fed to wget which can use the cookie for permission to download the desired content.

Being on a Macintosh system we are by default provided with Safari. Not bothering to install Firefox on every system one uses, we figured it was easier just to stick with Safari. Luckily the same technique can be performed with Safari as with Firefox. The technique for Safari is not as well known as the Firefox technique, so we'll cover it here.

Safari stores its cookies on a per user basis within a user's home directory. Specifically cookies are stored in a simple XML file. Have a look in Library/Cookies/Cookies.plist. You can see all of Safari's cookies in there.

To get the required cookie into the Cookies.plist before we proceed, use Safari to login to the Apple Developer Website using your Apple ID credentials. Safari should now have the requisite cookie. Opening the Cookies.plist with a text editor to view the cookie; we're looking for the one called ADCDownloadAuth.

With wget expecting it's cookie information in the nescape cookie.txt file format we'd like a quick and simple way to convert from one format to the other. Luckily this is relatively easy to do on a Macintosh system. As the language Ruby is preinstalled on Tiger, Leopard and Snow Leopard systems we may as well leverage the language to do this job.

Install the plist Ruby library and run the short ruby script listing below to convert the file to Firefox's cookies.txt format.

$ sudo gem install plist
$ irb
>> require 'plist'
>> result = Plist::parse_xml("Library/Cookies/Cookies.plist")
>> File.open("cookies.txt", "w") {f result.each {r f.write("#{r["Domain"]}\tTRUE\t#{r["Path"]}\tFALSE\t#{r["Expires"].strftime("%s")}\t#{r["Name"]}\t#{r["Value"]}\n")}}

Now that we have our cookies.txt file we can download the file we would like. Note that the URL for the sdk was found by looking at Apple's Download website to see where the download link led.

Below is the wget command line used to download the Xcode and iPhone SDK. Note that the command line variables for wget tell it to pipe the downloaded file to split which breaks the file up to make our venerable Linux file and webserver happy (2GB file limit). We're splitting the download into 512mb chunks here.

To make sure the authetication works I needed to use the header flag and insert the cookie value at the command line. Looking at the cookie.txt file to again find the ADCDownloadAuth key and it's datavalue we place this data in exchange of the "XXX" marked command line below for this recipe to work.

wget -qO- -U firefox -ct 0 --timeout=60 --waitretry=60 --load-cookies cookies.txt -c http://adcdownload.apple.com/ios/ios_sdk_4.2__final/xcode_3.2.5_and_ios_sdk_4.2_final.dmg --header="Cookie: ADCDownloadAuth=XXX" | split --bytes=512m - xcode_3.2.5_and_ios_sdk_4.2_final.dmg

You should now see your download commence and with everything to plan you'll have your dmg ready to install. You can ftp to you Macintosh. Once aboard you can
cat part1 part2 part3 ... > combined.dmg

In order to restore the split components. Happy Developing with the latest SDK!

Schedule Steam (and use your Offpeak bandwidth)

Antipodeans get a raw deal when it comes to the Internet. It's expensive and slow, not to mention it's become a political hot potatoe with several poorly thought out schemes relating to the internet being pushed through in Canberra.

Politics aside for now, that's not the reason for this post.

With the core market for Valve's Steam service no doubt being the American market little attention was paid to download manager style features such as bandwidth throttling and scheduling. Valve's bandwidth heavy Cornocopia of Software can chew through the average Australian/NZ household's monthly bandwidth allocation in hours.

Most Australian bandwidth plans have a Peak and Offpeak timesplit. Peak obviously falls into all those times you're likely to use the Internet. Unless you're a serious nightowl, at month's end you've probably still got a considerable amount of your Offpeak bandwidth allocation remaining.

Steam doesn't offer built in download scheduling so here is a short recipe on how you can coax Stream to schedule a download for the Offpeak internet hours.

I'm using Windows XP in this example, I can only assume it's similar for Vista and Windows 7.

Firstly on the Login screen of Steam make sure you have the "remember password" option checked. You'll need to do this to allow Steam to automatically login and resume or initiate a download.

Login in to steam initially, and go to the "My Games" tab and right click to "Install game..." on an undownloaded title you'd like to download and install. Steam will present you with details about the install, click next proceed. It will the process the file cache, and then ask you if you wish to create a shortcut to the desktop, which you should check (this is needed for later). The download will begin.

Now that you've compleated the manual steps to setup the download, you can exit Stream. Once it's shutdown we can schedule the download for Offpeak hours.

Go to your Desktop and find the shortcut you chose to create from Steam in the previous step. Right click on the shortcut and choose properties. You'll be presented with the shortcut properties. On this screen there will be a textbox labelled "Target". Highlight all the text in this textbox and copy it to your pasteboard. In this example the text I copied is (It's Half Life 2: Deathmatch):

"c:\Program Files\Steam\steam.exe\" -applaunch 240

Next choose Start->Run from the Main OS Menu, and type "cmd" in into the textbox presented in order to open a command window.

We'll be using the command line command "schtasks" to schedule the Steam download.

In the example below I've scheduled Steam to begin downloading at 2:00 AM in the morning. This is a typical Offpeak time, but you're might be different so you might want to check.

Now we can construct the command line:

schtasks /Create /TR "\"C:\Program Files\Steam\steam.exe\" -applaunch 240" /ST 02:00:00 /SC ONCE /TN Steam

I've used the text I copied from the desktop shortcut as you can see. Pay special notice to the "\" characters I've inserted before the quotes. These are required to be able to input the whole command properly.

The schtasks application should ask you for your user password to properly schedule Steam. If your user account doesn't have a password, you should probably set one. Typically schtasks won't run properly if you don't have a user password on your user account.

You can see the scheduled task(s) by just running "schtasks". It's generally worth trying it first by running it in a minute or two's time to test it's all working properly. Just logout of Steam and reschedule to the appropriate time after you're satisfied.