Pain in the Tech

.... but a bit cost prohibitive, I was looking at this as a solution out of my dilemma (mainly, running out of disk space), but to store the 500GB I currently have in digital music files alone I would pay $75/month, and that is before I have moved anything to the storage space yet or streamed it back. For that amount of money I can buy myself a new HDD every three months and carry a huge chunk of my library with me on an external drive...... But still, VERY cool solution.

Quick Link to this comment: http://TTIP.me/c4354

Interesting - surely grabbing the metadata from every file on your initial import takes AGES?

Quick Link to this comment: http://TTIP.me/c4355

I think this is getting very expensive, or not?!

Quick Link to this comment: http://TTIP.me/c4356

I don't see the advantages of this approach relative to your stated objectives: 2. Access my music from anywhere in the world How are you going to access your music while on the beach or in transit (i.e. on the train, in the car or on the plane?) Mobile network acess is rather costly especially for the kind of connectivity you need to stream MP3's. In the long run for the money you'd spend on bandwidth you could pay somebody to make backups for you. I’d like to eliminate the need for a personal external hard drive, for many reasons: * Unless I spend a lot of time (which I don’t have) doing backups - in the event of a disaster (or drive failure), I’d lose all of my data. You can just do incremental backups on DVD based on a smart playlist that only contains songs added in the last 30 days for example. * Carrying the hard drive around is cumbersome, and there’s a risk it could be damaged in transit. Carrying the laptop is rather more cumbersome than carrying around just the ipod on many occasions.

Quick Link to this comment: http://TTIP.me/c4357

This sounds great! But I keep all my music on an iPod. It's not a free solution, but I have my music everywhere...in the car, at the beach... But as a free solution...this is nice!

Quick Link to this comment: http://TTIP.me/c4359

I've found myself in the same boat. I have an 800gb external disk drive for my "stationary collection". While its not fully (yet), it is a hassle to occasionally have to lug around when my iPod isn't quite enough. I opted for installing Hamachi (http://www.hamachi.cc/) instead of a third-party storage solution, because I like having physically access to my data. With a tiny footprint on each of machines (home computers, laptop, computer, etc) I have access to any shared drives on the network as well as the shared iTunes library (you can opt to have separate libraries on each machine, but I prefer keeping everything in sync).

Quick Link to this comment: http://TTIP.me/c4360

I've worked out roughly how much this would cost, based on the S3 usage fees, and some assumptions: 1. Storage Based on a 40Gb library at $0.15 per Gb-month = $6/month 2. Usage 6 hours of listening per day for 21 days a month (work days) = 126 hours / month Each hour of listening to an MP3 at roughly 84Mb / hour (based on 192Kbps mp3 encoding) = 10584 Mb / month = 10.34 Gb at $0.20 / Gb transferred = $2.07 / month Giving a total of around $8 a month. Now compare that with the costs of running an external drive (or breaking it by carrying it around everywhere).

Quick Link to this comment: http://TTIP.me/c4361

Does this allow multiple users to access the same files? Can you set it up as a remote mp3 jukebox?

Quick Link to this comment: http://TTIP.me/c4362

Internet-based storage is highly commoditized these days, and Amazon is charging a premium price for their service. You could just open a GoDaddy hosting account. The Deluxe plan is 100 GB of storage plus 1 TB of transfer monthly for $6.29/mo (on a 1 year contract) or $75.48/yr. To match that storage & transfer on Amazon's S3 service, you'd spend $15/mo for storage and $200/mo for transfer, or $2580/yr. Hmmm... $75.48 (GoDaddy) or $2580 (Amazon). Not even in the same ballpark.

Quick Link to this comment: http://TTIP.me/c4364

Just about any reputable hosting service will work for this. Dreamhost offers WebDav connectivity, 180gb of storage, and 1.8 terabytes of bandwidth for 8 bucks a month. Much cheaper than Amazon. Jim RunFatBoy ( http://www.runfatboy.net ) - Exercise for the rest of us.

Quick Link to this comment: http://TTIP.me/c4358

With the way things are looking now, I wouldn't be surprised if our entire lives are kept on a sever somewhere in 10 years' time. Interesting experiment – expensive, yes – but will probably be the norm one day.

Quick Link to this comment: http://TTIP.me/c4365

The difference between S3 and hosting services is the redundancy built into S3. The data you upload and store is replicated to multiple servers, so you don't have to worry about losing your data. It's ideal for things you want to protect (e.g. pictures, music).

Quick Link to this comment: http://TTIP.me/c4366

I agree, why would I trust other services for this besides Amazon? Who cares about paying premium -- it's the redundant tech that I'm interested in. I would trust Amazon way more than any other service to actually be up 99% of the time also.

Quick Link to this comment: http://TTIP.me/c4367

"why would I trust other services for this besides Amazon?" What a ridiculous question! All legitimate hosting services provide data backups and RAID redundant storage. Amazon has a cool interface, but any hosting provider will work just as well. The only reason you trust amazon more is because its a big brand name. Do you think the companies all over the world who host their websites are trusting them with a host that doesn't provide backup and redundancy? give your head a shake.

Quick Link to this comment: http://TTIP.me/c4368

Amazon's S3 doesn't just provide RAID redundancy, it provides redundancy by distributing the data to multiple machines at multiple locations. This results in even a smaller likelihood of losing data (e.g. if the machine blows up, the data is still somewhere else) and higher availability.

Quick Link to this comment: http://TTIP.me/c4371

why not just carry an iPod? i've had five of them, with an 80GB video my latest. the other, given to members of my family. none of them have every had any problems with fairly rigorous use. my music library is considerably bigger so I'd considered this solution, but if you have more than an ipod's worth of music the price of ULing and streaming is too prohibitive.

Quick Link to this comment: http://TTIP.me/c4372

Between your calculations and those left by James Newbery above, it would appear that GoDaddy's service allows for about 1,572 hours of music playback a month (at 192kbps), the only issues is that's more than twice the number of hours in any month. Not a realistic comparison.

Quick Link to this comment: http://TTIP.me/c4373

There's no proof of Amazon's claims on the S3 service. The service has had multiple partial and full outages, and that's just in the past month since I started playing with it. There's no signed SLA or anything remotely related. It's definitely not using "infrastructure that Amazon uses to run its own global network of web sites", since their web site seemed to be doing just fine while it's S3 users were struggling to put/get data from their S3 buckets.

Quick Link to this comment: http://TTIP.me/c4374

Another problem I see right now with S3 is upload reliability. Presently I'm evaluating S3 as storage space for online backups and I'm having a lot of trouble with uploading larger numbers of files at once ("error 500"). Neither Interarchy nor Jungle Disk (combined with rsync or SmartBackup) solve this. Also JD stores data on S3 in its own kind of format which a) doesn't seem to handle all files (filenames?) I need/want to back up very well and b) which cannot be easily read (in a way that makes sense to a human being) by more "generic" S3 browser like Interarchy or, well, S3 Browser. Also JD loses you the ability to easily share files (which is quite simple via Interarchy). Mind you, the upload troubles seem to be at least in part a problem on Amazon's side but (e.g.) in the Interarchy mailing list archive (Yahoo! ...) you'll find the error 500 problem discussed back in November and now it's almost February and it doesn't look like things have gotten any better. Personally I'm strongly considering Joyent's Bingo! over S3 right now ...

Quick Link to this comment: http://TTIP.me/c4375

While it's really cool that services now make it possible for my iTunes library to be safe even if a major city or two gets nuked off the map, there's a nagging feeling in the back of my mind that maybe, just maybe, some of my other data should get the benefit of this before a bunch of MP3s. :)

Quick Link to this comment: http://TTIP.me/c4377

Are you using JungleDisk on Windows, or Mac? My experience using it on OS X has been really great so far. I've even purposely _interrupted_ uploads (by closing my MacBook and moving to another network at another location), and the upload resumes from where it left off! Pretty nifty I thought.

Quick Link to this comment: http://TTIP.me/c4378

Interesting idea... How is performance, though? Have you imported a CD or purchased something from the iTunes Store? How about sync'd your iPod after adding the new music?

Quick Link to this comment: http://TTIP.me/c4379

I've been using DOT.TUNES ( http://www.dottunes.net ) for over a year now for streaming my iTunes tracks to other computers, my PSP and Wii. It even has a Flash Player so you can publish your iTunes tracks on your web space or MySpace etc. You can also use DOT.TUNES to control iTunes - perfect for an iTunes/AirTunes setup. http://www.dottunes.net

Quick Link to this comment: http://TTIP.me/c4380

Yes, redundancy of data is important for ensuring no data loss, but many server accounts provide daily data backup. Amazon's services aren't really intended for singular end user use; they are priced for developers and businesses who want to leverage a highly available, redundant Internet datastore that provides total elasticity of scale. By elasticity of storage scale, I mean, you can store 1Mb or you can store 500Gb directly through the API. Pay for what you eat. That's a feature you won't find with server accounts that can dynamically change as fast as you can make service calls. So when people complain that this is more than a Dreamhost account, it's really like comparing apples and oranges.

Quick Link to this comment: http://TTIP.me/c4382

I was using the OS X client. But what bugs me even more about Jungle Disk is the more or less proprietary format that data is stored in on S3. Sure, AFAIR it's open sourced but right now I don't see that other people will adopt to it. On the other hand data I upload with Interarchy I can also access via S3 Browser (or Transmit, as soon as S3 support is ready there) and vice versa. Also Jungle Disks (non-)handling of file and directory permissions wasn't what I was looking for (everything ended up with (d)rwxrwxrwx ...).

Quick Link to this comment: http://TTIP.me/c4384

Explain the need to have 100 hours of portable music? Who the hell listens to 6 hours of music per day?

Quick Link to this comment: http://TTIP.me/c4387

Oh well, the file permission problem doesn't really seem to be specific to JungleDisk/S3; the same happens with Bingo!—and probably other WebDAV-based solutions as well … I guess I was kind of spoiled by Strongspace. ;-)

Quick Link to this comment: http://TTIP.me/c4388

I'm not a Mac user (yet), but I've used JungleDisk (JD) on a PC and for those who are concerned with read/write performance to the JD, JD caches reads/writes locally. Writes out to the remote storage are delayed, and reads are checked against the cache before requesting remotely. Of course, the cache size is configurable. I'm pulling this out of the nether areas, but I'd venture a guess that ripping to JD would not be any slower than a regular rip as it would be saving to the cache. Another commenter asked about reading the file info on boot. Yes, I would wonder about that as well. The library is read in from the iTunes library file when booted, which shouldn't take long on a cached copy. But iTunes does seem to check the availability of the files (at least on a PC) after the library is loaded. As far as the overall idea of running your library off S3 -- even though the cost isn't a ton, I'm wary of little $10 dollar a month leeches on my credit card, and this would be another one which to me is entirely avoidable. Besides, is redundant, worldwide storage of their tunes? I have a huge CD collection augmented with stuff I've purchased online. I rip CDs into the library, rate the songs, archive the ripped songs to DVD (using iTunes), and then delete files off the HD that don't meet my rating threshold. After archiving to DVD I essentially have two copies of the music I own on CD - one on the original CD, and one on DVD. If my house burns down and both of those copies go, well, I have bigger things to worry about I guess. Cheers, Michael

Quick Link to this comment: http://TTIP.me/c4390

Using JungleDisk and RSyncX to copy the data over. I was trying to move iTunes and iPhoto off my powerbook whose disk drive was full. I was not able to get it to work because eventually JungleDisk would hang and stop copying. This was about a month ago. Also, while I understand why JungleDisk would use their own naming conventions I found it annoying because it was hard to use other tools to figure out what was going right/wrong. Also I keep seeing 'shadow' versions of directories in the JungleDisk. There would be a '2007' folder within the '2007' folder but when I clicked it it would go away. The combination of these factors led me to decide that the tech is not ready. I still think it is a good solution for someday. Instead I bought an 300GB external FireWire RAID drive and moved everything on to that. It took about 45 minutes to copy everything (after weeks spent trying to get the S3 solution to work). It's an expensive answer but cheaper than a new Powerbook. So many solutions are designed for backup; but backup is not very helpful when you have run out of hard drive.

Quick Link to this comment: http://TTIP.me/c4394

I copied my music library to S3--more interested in having an off-site backup but would love to be able to listen to my tunes while at work. Backup works like a charm--got my first bill from Amazon for the upload and end of month hosting--cost about $3--I have about 20 gigs worth of Tunes. Haven't been able to get the library to work from the Jungle Disk, but can't say that bothers me too much. I'm sleeping better knowing that the tunes are backed up.

Quick Link to this comment: http://TTIP.me/c4398

I'm a trucker & my ipod is running at least 6 hours a day, you could normally double that just working locally. Thats before taking into account nights out if I wire it up to my 12v tv to watch a video.

Quick Link to this comment: http://TTIP.me/c4399

What about multiple listeners? ;-) The only reason I would upload my DB is for sharing ONE db with around 5 friends. Imagine the amount of usage. And by the way, the best hosting is: servage.net features: -webdrive (e.g. S: on windows) for easy storage of your iTunes db -2.1TB data transfer -250GB storage

Quick Link to this comment: http://TTIP.me/c4421

From how the Jungle Disk service presents it's self it sounds like it is able to cash things that you use a lot. It claims that if the content you are searching for is held in the local cash it will not need to access the S3 server.I imagine that it also has settings about how large to make the cash. We can assume that iTunes uses things like the iTunes Library db file quite a lot so that would always be local and if you made the cash large, say 10 gigs. If it was that big I think it could quite easily hold all the music that you would listen to for some time, perhaps about one month. When you did need to listen to new music it would come in from the server. I did a little test with my library of about 36 gigs to see how much different music I listen to in a month and in six months. I just used a smart play list with a Played last in the last n months clause. The results were a lot smaller than I had expected, in 1 month I had listened to about 1 gig of unique tracks. That trend was accurate for the last 6 months and year, being 6 and 12 gigs respectively. This is of course only accurate for me but assuming that most people use within an order of magnitude of the number of unique songs that I do per month the cost will still be pretty low. For me it would be roughly $5.45 per month or 65.4 per year, not including upload costs of course. It seems affordable and it seems like a worthwile service model. I would love you hear if it works as hoped or not.

Quick Link to this comment: http://TTIP.me/c4429

Yes. It's like having a shared drive. In fact, it appears as a folder on My Network Places. You can install jungle disk on multiple computers and use the same S3 account info, so each computer can upload and download files to the account.

Quick Link to this comment: http://TTIP.me/c4437

6 hours a day really isn't that odd. I probably hang out in that 5-7 hours a day myself. I listen for an hour or so in the morning while checking my email and having breakfast, in the car for a half hour each way to work, and for at least half of the working day while writing software. Throw in a bit at home in the evening and 6 hours a day is pretty reasonable.

Quick Link to this comment: http://TTIP.me/c4471

I always wanted to listen itunes music directly on my Laptop. Amazon S3 rocks.

Quick Link to this comment: http://TTIP.me/c4563

Not 100% of the time, and only about half the time while sleeping, but I have a variety of different kinds of music playing probably 50% of the time I am at home, and I do most of my work from home. So at least 6hrs a day and more like 12 hrs.

Quick Link to this comment: http://TTIP.me/c4565

I use site5 and they have a 110gb/5tb for $5/mo. Pretty unbelievable, actually. I am very happy with them.

Quick Link to this comment: http://TTIP.me/c4566

Comparing shared hosting services to Amazon S3 is the equivalent to comparing a floppy disk to replicated raid 0+1 storage. You should also not consider the performance of Jungle disk or other solutions as the performance of S3. In the 6 months we have been using amazon S3 to store database snap shots 8 times a day, we have not had a single instance where we were unable to connect to the service. S3 is a highly reliable and scalable solution for use primarily for people or organizations who wish to develop their own tools using the API. JungleDisk and others are tools built by 3rd parties to try and allow others to access the S3 service without the required custom development. These tools all have their own issues and it sounds like the problems people are dsecribing are related to the tools and not the service. Amazon S3 is far and away the least expensive solution of its kind, comparing it to webdav on a shared web hosting provider is silly.

Quick Link to this comment: http://TTIP.me/c4553

I'm not sure why you're connecting with localhost rather than with jungledisk - can't you just start up jungle disk and go to your S3 from there? What would be the benefit of connecting to localhost? PS Great piece! I followed all your instructions and now it's like I have a brand new computer with no worries over losing my music and other important contact. Thank you!

Quick Link to this comment: http://TTIP.me/c4579

I actually use a networked hard drive connected to my wireless router here at my home, but of late I've been thinking of storing new or recently purchased music somewhere on the net for easy access when away from my home office. Very interesting solution... definitely best in small doses as I see a slow down even on a local router with iTunes.

Quick Link to this comment: http://TTIP.me/c4614

Just use Hamachi and share your home library. You will be able to play your entirely library located in your home computer from anywhere in the world.

Quick Link to this comment: http://TTIP.me/c4672

is there a limit on how much music your itunes library can hold or can it hold as much as your hard drive can hold ?

Quick Link to this comment: http://TTIP.me/c4645

I'm trying to take this one step further but I'm stuck. I have jungle disk mirroring my music drive so all 60GB is avalable from any computer I wish to use to access my music. This is very nice. But I would also like to grab that music by mapping a drive on my windows mobile 6 smartphone - a palm treo 750. I have an unlimited data plan and this would eliminate my 2GB memory card limit and allow me to queue up any song from my entire library any time I want. WM6 supports drive mapping but as far as I know no one has an API yet to access S3 from a WM6 device. Any skillfull people up for the challenge?

Quick Link to this comment: http://TTIP.me/c4722

Hi,

You seems to be an advanced music listener. Let me tell you about Zicmama. As Jungledisk, It is a network drive. But it gets your mp3 copied from other music library ;)

Try it and open an account an http://www.zicmama.com Then connect your Webdav ZicDisk... Be gentle, this is still a beta french ;) Any interested translator?

Quick Link to this comment: http://TTIP.me/c4735

I am in a similar situation, but I doubt JungleDisk will work for the following reasons (some of which are already mentioned): a) cost - S3 charges for bandwidth usage. you'd probably risk getting metered by your ISP as well. b) performance - iTunes can barely handle my local network drive. c) reliability - I've lost several files trying to do a bulk copy (opposed to a backup). Network error during upload and I had a collection of 0 byte files. Was a pain to sort out.

Sugarsync (www.sugarsync.com) seems like a promising, but incomplete solution. They don't charge for bandwidth, and they support streaming non-DRM MP3's and AAC. But you can only backup from your local drive. You can stream from the cloud using their mobile app, but that's it. Your playlists, Apple TV, and iPod experience are shot.

Drives that have more capacity need to be more energy-efficient. Networks need to be faster. The real solution, IMO, is that compression envelope for the files needs to be pushed.

Quick Link to this comment: http://TTIP.me/c4905

I set this up a couple of days ago with three computers -- my work PowerMac computer (we have full OC3 connections to the Internet), my home iMac on a DSL line (3Mbps/256k) and my MacBook Air with Wifi used at locations like the public library and coffee shops.

It works, but not well enough to continue to using it. iTunes just doesn't handle remote access for the library well. When you have JungleDrive going, where you can actually see all the files that are constantly getting updated and rewritten, it becomes clear that anything other than local drive (or at least LAN) access is an issue.

I tried two different methods. First, I actually uploaded my entire catalog and the XML/indexes to S3, then simply made the local iTunes folder a symbolic link. This way both the library files and indexes are on the S3. The theory being that as I listen to files at work, and maybe add a new song file, then when I get home and fire up iTunes, it's identical. In a perfect world, this is the best solution. All computers treat the iTunes database as their own and it doesn't matter where changes are made.

Unfortunately, any change (including simply the fact iTunes increment the Times Played counter on every song as you listen) requires reuploading those index files. At work with huge pipes, not a huge deal. On the DSL or Wifi, major problem. After every song plays, it's uploading 50Meg worth of indexes.

With that option not working, I then undid the symbolic link, put the indexes back in a regular local iTunes folder, and simply set the actual library of songs to still use the Amazon drive.

In this mode, at least it's not updating after every song, but the benefit of having multiple computers with identical libraries immediately goes away. Any new song has to be added separately to each computer. The biggest problem is that some features, like Genius, suddenly decide to reindex which requires scanning your library (and sending info to Apple). With the songs in the cloud, iTunes basically causes Jungledisk to download each song to "scan", so suddenly my 50GB of songs are being re-downloaded one at a time so Genius can do a scan (no clue why it can't simply send the titles, I don't really know what it was doing that it wanted full file access.. but it does, so Jungledisk was busy downloading until I aborted that process Forcefully.)

Any updates to any song still require that download/upload, such as changing a tag. I changed the tags on three podcasts to be "Podcasts" instead of whatever they were before, which meant iTunes (JungleDisk) downloaded each file, rewrote the tag, and reuploaded those files. Even at work it took several minutes.

With my MacBook Air in particular, streaming music off S3 is very hit or miss. Many places I tried didn't have the best download. Even at the massive public library downtown, they throttle traffic during busy periods (like when there's literally 1000 people in there on their laptops and in-house computers), so my download speed was 128kpbs at peak times. Not a huge deal for web pages, but music no longer worked. (Streaming radio works because they're compressing heavily, but with iTunes downloading 3Meg files per song, performance was not good.)

So ... while everything mentioned here works in theory, real world performance just isn't up to the task. I'll keep my S3 drive because having unlimited backup space is great (especially since it just costs .10 cents per Gig/per Month to have it just sit there), but constant usage just isn't ready for prime time.

As a side note, this does prove that the DSL/Cable style of Broadband, i.e. fast download/slow upload because we're "consumers of data" really breaks down under the Cloud Computing or Storage model where we need full duplex access at high speeds. I have no doubt in my mind that cloud storage at least is the next major thing, but the infrastructure has to be there first.

Quick Link to this comment: http://TTIP.me/c4906