iTunes everywhere: Using Amazon S3 as your music library
Posted by Matt Thommes on 01/25/07 in Software, Tips
Lately I’ve been bothered by a couple of things, regarding my digital music library:
- I have to keep upgrading my external hard drive storage capacity, because new music is consistently being added.
- My music is only accessible from wherever the external hard drive is, which means I have to lug it around to work, on vacation, etc.
There are three things I wish to accomplish:
- Eliminate the limit on my music storage capacity.
- Access my music from anywhere in the world.
- Access my music directly from iTunes on my laptop - not with an iPod or external digital music device.
With iTunes and Amazon S3, I can accomplish these goals.
iTunes and Amazon S3
This article is geared towards Mac OS X users, but the same functionality can be achieved on Windows.
Protection and “access without excess”
My iTunes library is somewhere around 35 GB, and growing every day.
I’d like to eliminate the need for a personal external hard drive, for many reasons:
- Unless I spend a lot of time (which I don’t have) doing backups - in the event of a disaster (or drive failure), I’d lose all of my data.
- Carrying the hard drive around is cumbersome, and there’s a risk it could be damaged in transit.
“Access without excess”: the ability to access something without shouldering an excess burden. I don’t want to carry the hard drive around, but I still want access to my music.
So I need a solution that both protects my data, and allows me “access without excess.”
Amazon S3
Amazon S3 (Simple Storage Service) is a fast-growing service that frees users from the burden of self-storage. With an Amazon S3 account, you can store your personal or business data on safe and secure servers, all across the country.
You no longer have to worry about backups, nor do you have a limit on storage space. This immediately achieves the goals mentioned above.
Configuring S3 on Mac OS X
Setting up an Amazon S3 account is rather painless. If you already have an Amazon.com account, you can log-in and configure S3. Once you obtain your unique private access key, you can use this key to access your personal Amazon S3 storage space. You can connect to this storage space on Mac OS X, much like a network drive.
For this article, I tested JungleDisk to assist with connecting me in connecting to my S3 drive. JungleDisk is a free utility for Mac and Windows, which simply acts as a “middle man” between my local hard drive and Amazon S3 servers.
From the JungleDisk web site:
Download the JungleDisk application, and install it on your machine. Launch JungleDisk and set up your Amazon account ID information:
- Jungle Disk runs on your local machine as a WebDAV server. WebDAV is a protocol for remote file access supported by all modern operating systems.
- As you copy data to your Jungle Disk, it is transparently cached, encrypted, and uploaded to Amazon.com. Since uploads occur in the background, copying files to your Jungle Disk is as fast as a local hard drive.
- When you browse your Jungle Disk, if the data is already available in the local cache nothing is downloaded from Amazon, which makes Jungle Disk super efficient.
JungleDisk configuration
JungleDisk configuration complete
In the Finder, choose Go < Connect to Server, or hit Command + K:
Finder Connect to Server
Type in exactly: http://localhost:2667/:
Finder localhost address
This will now appear as a network drive on your computer:
Finder new network drive
You can simply drag and drop files to this folder, or point iTunes to use this drive as your music library - giving you unbridled access to your entire music library from anywhere in the world.
Configuring iTunes to point to S3
All that’s left to do is configure iTunes to use Amazon S3 as your music library.
In iTunes, go to Preferences, then the General tab, and change your iTunes Music folder location to point to your Amazon S3 network drive:
iTunes Advanced Preferences tab
You’ll have to manually copy your music library to the S3 drive though, which could take a while, depending on the size of your library.
And that’s it - you’re now free of storage restrictions, and you can access your music from anywhere that has an internet connection!
Benefits and drawbacks
If this seems too good to be true, it may be so. This idea is an experimental one, so there are some limitations to be aware of.
- Since you’re storing your entire music library on a remote web server, you’re susceptible to slow download/streaming, meaning your music may not sound as “immediate” as if it was on your local hard drive.
- The initial process of uploading all of your songs to Amazon S3 may take a considerable amount of time.
- I didn’t test burning CD’s from my S3 storage, but you can guess there’d be issues with burning remote files to a local CD.
- I didn’t test connecting an iPod and transferring the songs. Again, it will take more time, since the songs have to be downloaded first.
- Other applications like Front Row and Apple TV have yet to be tested as well.
The basic idea was to simply allow access to our music through the iTunes interface - and for that it works pretty well.
The “peace of mind” obtained from having your music securely stored on Amazon servers is worth the drawbacks outlined above.
WAN (Internet) speeds will only increase in the future, making the drawback of streaming a non-issue.
permalink | trackback url | 
Subscribe in a reader
Similar Posts
- Changing your network iTunes music library temporarily when importing
- Fetch iTunes artwork with Clutter
- One Backup Disk for Windows and OS X
- Amazon Web Services Minneapolis presentation
- Sell used technical books on Amazon
Michael | Jan 25, 2007 | Reply
…. but a bit cost prohibitive, I was looking at this as a solution out of my dilemma (mainly, running out of disk space), but to store the 500GB I currently have in digital music files alone I would pay $75/month, and that is before I have moved anything to the storage space yet or streamed it back.
For that amount of money I can buy myself a new HDD every three months and carry a huge chunk of my library with me on an external drive……
But still, VERY cool solution.
Tom | Jan 26, 2007 | Reply
Interesting - surely grabbing the metadata from every file on your initial import takes AGES?
kozi | Jan 26, 2007 | Reply
I think this is getting very expensive, or not?!
Anonymous | Jan 26, 2007 | Reply
I don’t see the advantages of this approach relative to your stated objectives: 2. Access my music from anywhere in the world
How are you going to access your music while on the beach or in transit (i.e. on the train, in the car or on the plane?) Mobile network acess is rather costly especially for the kind of connectivity you need to stream MP3’s. In the long run for the money you’d spend on bandwidth you could pay somebody to make backups for you.
I’d like to eliminate the need for a personal external hard drive, for many reasons:
You can just do incremental backups on DVD based on a smart playlist that only contains songs added in the last 30 days for example.
* Carrying the hard drive around is cumbersome, and there’s a risk it could be damaged in transit.
Carrying the laptop is rather more cumbersome than carrying around just the ipod on many occasions.
Kathy Davies | Jan 26, 2007 | Reply
This sounds great! But I keep all my music on an iPod. It’s not a free solution, but I have my music everywhere…in the car, at the beach…
But as a free solution…this is nice!
Mystech | Jan 26, 2007 | Reply
I’ve found myself in the same boat. I have an 800gb external disk drive for my “stationary collection”. While its not fully (yet), it is a hassle to occasionally have to lug around when my iPod isn’t quite enough. I opted for installing Hamachi (http://www.hamachi.cc/) instead of a third-party storage solution, because I like having physically access to my data. With a tiny footprint on each of machines (home computers, laptop, computer, etc) I have access to any shared drives on the network as well as the shared iTunes library (you can opt to have separate libraries on each machine, but I prefer keeping everything in sync).
James Newbery | Jan 26, 2007 | Reply
I’ve worked out roughly how much this would cost, based on the S3 usage fees, and some assumptions:
1. Storage Based on a 40Gb library at $0.15 per Gb-month = $6/month
2. Usage 6 hours of listening per day for 21 days a month (work days) = 126 hours / month Each hour of listening to an MP3 at roughly 84Mb / hour (based on 192Kbps mp3 encoding) = 10584 Mb / month = 10.34 Gb at $0.20 / Gb transferred = $2.07 / month
Giving a total of around $8 a month.
Now compare that with the costs of running an external drive (or breaking it by carrying it around everywhere).
Anonymous | Jan 26, 2007 | Reply
Does this allow multiple users to access the same files? Can you set it up as a remote mp3 jukebox?
Charlotte Web | Jan 26, 2007 | Reply
Internet-based storage is highly commoditized these days, and Amazon is charging a premium price for their service.
You could just open a GoDaddy hosting account. The Deluxe plan is 100 GB of storage plus 1 TB of transfer monthly for $6.29/mo (on a 1 year contract) or $75.48/yr. To match that storage & transfer on Amazon’s S3 service, you’d spend $15/mo for storage and $200/mo for transfer, or $2580/yr.
Hmmm… $75.48 (GoDaddy) or $2580 (Amazon). Not even in the same ballpark.
Jim Jones | Jan 26, 2007 | Reply
Just about any reputable hosting service will work for this. Dreamhost offers WebDav connectivity, 180gb of storage, and 1.8 terabytes of bandwidth for 8 bucks a month. Much cheaper than Amazon.
Jim RunFatBoy ( http://www.runfatboy.net ) - Exercise for the rest of us.
Thomas Edwards | Jan 26, 2007 | Reply
With the way things are looking now, I wouldn’t be surprised if our entire lives are kept on a sever somewhere in 10 years’ time.
Interesting experiment – expensive, yes – but will probably be the norm one day.
Anonymous | Jan 26, 2007 | Reply
The difference between S3 and hosting services is the redundancy built into S3. The data you upload and store is replicated to multiple servers, so you don’t have to worry about losing your data. It’s ideal for things you want to protect (e.g. pictures, music).
Anonymous | Jan 26, 2007 | Reply
I agree, why would I trust other services for this besides Amazon? Who cares about paying premium — it’s the redundant tech that I’m interested in. I would trust Amazon way more than any other service to actually be up 99% of the time also.
Anonymous | Jan 26, 2007 | Reply
“why would I trust other services for this besides Amazon?” What a ridiculous question! All legitimate hosting services provide data backups and RAID redundant storage. Amazon has a cool interface, but any hosting provider will work just as well. The only reason you trust amazon more is because its a big brand name. Do you think the companies all over the world who host their websites are trusting them with a host that doesn’t provide backup and redundancy? give your head a shake.
Anonymous | Jan 26, 2007 | Reply
Amazon’s S3 doesn’t just provide RAID redundancy, it provides redundancy by distributing the data to multiple machines at multiple locations. This results in even a smaller likelihood of losing data (e.g. if the machine blows up, the data is still somewhere else) and higher availability.
Anonymous | Jan 26, 2007 | Reply
why not just carry an iPod? i’ve had five of them, with an 80GB video my latest. the other, given to members of my family. none of them have every had any problems with fairly rigorous use. my music library is considerably bigger so I’d considered this solution, but if you have more than an ipod’s worth of music the price of ULing and streaming is too prohibitive.
Matt Deimel | Jan 26, 2007 | Reply
Between your calculations and those left by James Newbery above, it would appear that GoDaddy’s service allows for about 1,572 hours of music playback a month (at 192kbps), the only issues is that’s more than twice the number of hours in any month.
Not a realistic comparison.
Anonymous | Jan 26, 2007 | Reply
There’s no proof of Amazon’s claims on the S3 service. The service has had multiple partial and full outages, and that’s just in the past month since I started playing with it. There’s no signed SLA or anything remotely related.
It’s definitely not using “infrastructure that Amazon uses to run its own global network of web sites”, since their web site seemed to be doing just fine while it’s S3 users were struggling to put/get data from their S3 buckets.
Anonymous | Jan 26, 2007 | Reply
Another problem I see right now with S3 is upload reliability. Presently I’m evaluating S3 as storage space for online backups and I’m having a lot of trouble with uploading larger numbers of files at once (”error 500″). Neither Interarchy nor Jungle Disk (combined with rsync or SmartBackup) solve this. Also JD stores data on S3 in its own kind of format which a) doesn’t seem to handle all files (filenames?) I need/want to back up very well and b) which cannot be easily read (in a way that makes sense to a human being) by more “generic” S3 browser like Interarchy or, well, S3 Browser. Also JD loses you the ability to easily share files (which is quite simple via Interarchy).
Mind you, the upload troubles seem to be at least in part a problem on Amazon’s side but (e.g.) in the Interarchy mailing list archive (Yahoo! …) you’ll find the error 500 problem discussed back in November and now it’s almost February and it doesn’t look like things have gotten any better.
Personally I’m strongly considering Joyent’s Bingo! over S3 right now …
Dan | Jan 26, 2007 | Reply
While it’s really cool that services now make it possible for my iTunes library to be safe even if a major city or two gets nuked off the map, there’s a nagging feeling in the back of my mind that maybe, just maybe, some of my other data should get the benefit of this before a bunch of MP3s.
Matt Thommes | Jan 26, 2007 | Reply
Are you using JungleDisk on Windows, or Mac? My experience using it on OS X has been really great so far. I’ve even purposely interrupted uploads (by closing my MacBook and moving to another network at another location), and the upload resumes from where it left off! Pretty nifty I thought.
Andrew Stott | Jan 26, 2007 | Reply
Interesting idea… How is performance, though? Have you imported a CD or purchased something from the iTunes Store? How about sync’d your iPod after adding the new music?
Brad | Jan 26, 2007 | Reply
I’ve been using DOT.TUNES ( http://www.dottunes.net ) for over a year now for streaming my iTunes tracks to other computers, my PSP and Wii.
It even has a Flash Player so you can publish your iTunes tracks on your web space or MySpace etc.
You can also use DOT.TUNES to control iTunes - perfect for an iTunes/AirTunes setup.
http://www.dottunes.net
Adam K | Jan 26, 2007 | Reply
Yes, redundancy of data is important for ensuring no data loss, but many server accounts provide daily data backup. Amazon’s services aren’t really intended for singular end user use; they are priced for developers and businesses who want to leverage a highly available, redundant Internet datastore that provides total elasticity of scale. By elasticity of storage scale, I mean, you can store 1Mb or you can store 500Gb directly through the API. Pay for what you eat. That’s a feature you won’t find with server accounts that can dynamically change as fast as you can make service calls.
So when people complain that this is more than a Dreamhost account, it’s really like comparing apples and oranges.
Anonymous | Jan 27, 2007 | Reply
I was using the OS X client. But what bugs me even more about Jungle Disk is the more or less proprietary format that data is stored in on S3. Sure, AFAIR it’s open sourced but right now I don’t see that other people will adopt to it. On the other hand data I upload with Interarchy I can also access via S3 Browser (or Transmit, as soon as S3 support is ready there) and vice versa.
Also Jungle Disks (non-)handling of file and directory permissions wasn’t what I was looking for (everything ended up with (d)rwxrwxrwx …).
Island in the Net | Jan 28, 2007 | Reply
Explain the need to have 100 hours of portable music? Who the hell listens to 6 hours of music per day?
Anonymous | Jan 28, 2007 | Reply
Oh well, the file permission problem doesn’t really seem to be specific to JungleDisk/S3; the same happens with Bingo!—and probably other WebDAV-based solutions as well … I guess I was kind of spoiled by Strongspace.
Michael A. Vickers | Jan 31, 2007 | Reply
I’m not a Mac user (yet), but I’ve used JungleDisk (JD) on a PC and for those who are concerned with read/write performance to the JD, JD caches reads/writes locally. Writes out to the remote storage are delayed, and reads are checked against the cache before requesting remotely. Of course, the cache size is configurable.
I’m pulling this out of the nether areas, but I’d venture a guess that ripping to JD would not be any slower than a regular rip as it would be saving to the cache.
Another commenter asked about reading the file info on boot. Yes, I would wonder about that as well. The library is read in from the iTunes library file when booted, which shouldn’t take long on a cached copy. But iTunes does seem to check the availability of the files (at least on a PC) after the library is loaded.
As far as the overall idea of running your library off S3 — even though the cost isn’t a ton, I’m wary of little $10 dollar a month leeches on my credit card, and this would be another one which to me is entirely avoidable. Besides, is redundant, worldwide storage of their tunes?
I have a huge CD collection augmented with stuff I’ve purchased online. I rip CDs into the library, rate the songs, archive the ripped songs to DVD (using iTunes), and then delete files off the HD that don’t meet my rating threshold. After archiving to DVD I essentially have two copies of the music I own on CD - one on the original CD, and one on DVD. If my house burns down and both of those copies go, well, I have bigger things to worry about I guess.
Cheers,
Michael
Kevin | Feb 1, 2007 | Reply
Using JungleDisk and RSyncX to copy the data over. I was trying to move iTunes and iPhoto off my powerbook whose disk drive was full.
I was not able to get it to work because eventually JungleDisk would hang and stop copying. This was about a month ago. Also, while I understand why JungleDisk would use their own naming conventions I found it annoying because it was hard to use other tools to figure out what was going right/wrong. Also I keep seeing ’shadow’ versions of directories in the JungleDisk. There would be a ‘2007′ folder within the ‘2007′ folder but when I clicked it it would go away.
The combination of these factors led me to decide that the tech is not ready. I still think it is a good solution for someday.
Instead I bought an 300GB external FireWire RAID drive and moved everything on to that. It took about 45 minutes to copy everything (after weeks spent trying to get the S3 solution to work). It’s an expensive answer but cheaper than a new Powerbook.
So many solutions are designed for backup; but backup is not very helpful when you have run out of hard drive.
Ted Bongiovanni | Feb 3, 2007 | Reply
I copied my music library to S3–more interested in having an off-site backup but would love to be able to listen to my tunes while at work. Backup works like a charm–got my first bill from Amazon for the upload and end of month hosting–cost about $3–I have about 20 gigs worth of Tunes. Haven’t been able to get the library to work from the Jungle Disk, but can’t say that bothers me too much. I’m sleeping better knowing that the tunes are backed up.
Anonymous | Feb 3, 2007 | Reply
I’m a trucker & my ipod is running at least 6 hours a day, you could normally double that just working locally. Thats before taking into account nights out if I wire it up to my 12v tv to watch a video.
Barth | Feb 20, 2007 | Reply
What about multiple listeners?
The only reason I would upload my DB is for sharing ONE db with around 5 friends. Imagine the amount of usage.
And by the way, the best hosting is: servage.net
features:
-webdrive (e.g. S: on windows) for easy storage of your iTunes db -2.1TB data transfer -250GB storage
Metophile | Feb 24, 2007 | Reply
From how the Jungle Disk service presents it’s self it sounds like it is able to cash things that you use a lot. It claims that if the content you are searching for is held in the local cash it will not need to access the S3 server.I imagine that it also has settings about how large to make the cash. We can assume that iTunes uses things like the iTunes Library db file quite a lot so that would always be local and if you made the cash large, say 10 gigs. If it was that big I think it could quite easily hold all the music that you would listen to for some time, perhaps about one month. When you did need to listen to new music it would come in from the server.
I did a little test with my library of about 36 gigs to see how much different music I listen to in a month and in six months. I just used a smart play list with a Played last in the last n months clause. The results were a lot smaller than I had expected, in 1 month I had listened to about 1 gig of unique tracks. That trend was accurate for the last 6 months and year, being 6 and 12 gigs respectively. This is of course only accurate for me but assuming that most people use within an order of magnitude of the number of unique songs that I do per month the cost will still be pretty low. For me it would be roughly $5.45 per month or 65.4 per year, not including upload costs of course. It seems affordable and it seems like a worthwile service model.
I would love you hear if it works as hoped or not.
Stewart | Mar 4, 2007 | Reply
Yes. It’s like having a shared drive. In fact, it appears as a folder on My Network Places. You can install jungle disk on multiple computers and use the same S3 account info, so each computer can upload and download files to the account.
J Wynia | Apr 5, 2007 | Reply
6 hours a day really isn’t that odd. I probably hang out in that 5-7 hours a day myself. I listen for an hour or so in the morning while checking my email and having breakfast, in the car for a half hour each way to work, and for at least half of the working day while writing software. Throw in a bit at home in the evening and 6 hours a day is pretty reasonable.
Tech Blog | Jun 15, 2007 | Reply
I always wanted to listen itunes music directly on my Laptop. Amazon S3 rocks.
Jeff McNeill | Jun 15, 2007 | Reply
Not 100% of the time, and only about half the time while sleeping, but I have a variety of different kinds of music playing probably 50% of the time I am at home, and I do most of my work from home. So at least 6hrs a day and more like 12 hrs.
Jeff McNeill | Jun 15, 2007 | Reply
I use site5 and they have a 110gb/5tb for $5/mo. Pretty unbelievable, actually. I am very happy with them.
matt | Jul 1, 2007 | Reply
Comparing shared hosting services to Amazon S3 is the equivalent to comparing a floppy disk to replicated raid 0+1 storage.
You should also not consider the performance of Jungle disk or other solutions as the performance of S3. In the 6 months we have been using amazon S3 to store database snap shots 8 times a day, we have not had a single instance where we were unable to connect to the service. S3 is a highly reliable and scalable solution for use primarily for people or organizations who wish to develop their own tools using the API.
JungleDisk and others are tools built by 3rd parties to try and allow others to access the S3 service without the required custom development. These tools all have their own issues and it sounds like the problems people are dsecribing are related to the tools and not the service.
Amazon S3 is far and away the least expensive solution of its kind, comparing it to webdav on a shared web hosting provider is silly.
Keith | Jul 10, 2007 | Reply
I’m not sure why you’re connecting with localhost rather than with jungledisk - can’t you just start up jungle disk and go to your S3 from there? What would be the benefit of connecting to localhost?
PS Great piece! I followed all your instructions and now it’s like I have a brand new computer with no worries over losing my music and other important contact. Thank you!
Steve | Sep 21, 2007 | Reply
I actually use a networked hard drive connected to my wireless router here at my home, but of late I’ve been thinking of storing new or recently purchased music somewhere on the net for easy access when away from my home office. Very interesting solution… definitely best in small doses as I see a slow down even on a local router with iTunes.
jose Rivera | Jan 31, 2008 | Reply
Just use Hamachi and share your home library. You will be able to play your entirely library located in your home computer from anywhere in the world.
Alex | Feb 16, 2008 | Reply
is there a limit on how much music your itunes library can hold or can it hold as much as your hard drive can hold ?
Kevin | Sep 21, 2008 | Reply
I’m trying to take this one step further but I’m stuck. I have jungle disk mirroring my music drive so all 60GB is avalable from any computer I wish to use to access my music. This is very nice. But I would also like to grab that music by mapping a drive on my windows mobile 6 smartphone - a palm treo 750. I have an unlimited data plan and this would eliminate my 2GB memory card limit and allow me to queue up any song from my entire library any time I want. WM6 supports drive mapping but as far as I know no one has an API yet to access S3 from a WM6 device. Any skillfull people up for the challenge?