Anyone who has tried downloading a popular software or Linux
distribution on or just after release day, knows of the pain of the
phrase "connection timed out"�. Getting the software can be quite a
struggle, despite all the mirrors and BitTorrent Samaritans. Anthony
Bryan's Metalink is an open standard that makes downloading easier,
faster, and more reliable by helping users extract the last drop of
juice out of their connection.
I emailed a set of questions to Bryan to understand what separates
Metalink from other run-of-the-mill download accelerators and listen
in awe as he explains how Metalink combines traditional HTTP and FTP
methods of downloading files along with BitTorrent.
Mayank Sharma: Let's begin with the traditional what is question --
what is Metalink? Is it another way to download "stuff"?
Anthony Bryan: Yep, Metalink is a way to download "stuff." But it's
not a new transfer protocol like FTP or P2P method. It attempts to
make the traditional download process simpler by automating advanced
features and hiding complexity where possible. It's something that
download programs like Web browsers, FTP programs, download managers,
and P2P clients support. Information about the files to be downloaded
is stored in a simple XML file with a .metalink extension.
MS: So how exactly is using a metalink beneficial over other
traditional methods of downloading?
AB: In general, it should be easier and be error free. It can also be
much faster. The information a download program would need is in a
machine-processable format in the .metalink file, stuff like all the
different ways to download a file (from multiple mirrors to P2P) and
other things like the priority and geographical location of the
mirrors, checksums, publisher, license, OS, language, etc. More the
information, more the things that can be done!
So, instead of a single link to a file, you have many links. This
means higher availability. If many servers are down or very busy (like
on a release day), your download program can automatically check links
and see which ones are good. You don't have to manually gather and
check each link. This improves usability. Some programs can use the
multiple links to download different parts or segments of a file from
many places, which is called multi-source downloading. Usually, your
download will be much faster, but that all depends on your connection.
A single source download usually can't offer as fast download speeds.
Metalinks can also contain checksums or repair information to fix
corrupted downloads, either files downloaded with other programs or
repairing the download in progress. You can repair downloads with
rsync and some P2P, but with Metalink you can do it over regular
FTP/HTTP.
These benefits are mainly seen in large downloads like ISO files
(CD/DVD images), which can range from 500 MB to 4 GB or larger. Time
is money and sometimes you need things downloaded before you can get
work done. If there was an error in one of these files, it could be
painful to need to re-download it. This could be very helpful to movie
and music downloads, where (once this technology is included in
browsers) you'll be able to download huge files and recover from
errors, and download a whole album or multiple albums in one click,
within a browser and not some special application. For instance, when
you buy a bunch of albums on Bleep.com, you can download the files
individually or in an archive. The archive takes up resources on the
server and on your computer. With Metalink, you can list all the files
and have them added to a download queue in one click. This is also
useful for software like KDE that is available in multiple files.
MS: Whoa, those are some advantages. While it's good for the person
downloading, what about the person offering the download? Any reasons
why he should make a .metalink for his downloads?
AB: As we've already discussed, Metalink should offer your downloaders
an improved experience - more reliable and without errors.
This translates to cheaper bandwidth bills and support costs. Not very
exciting unfortunately, but something that needs to be done since it's
simple and can be automated. You can give different mirrors or P2P
sources a priority. If you list FTP/HTTP mirrors and P2P, the mirrors
can be used in case the person can't access P2P or if the file is no
longer on the P2P network. This can be handy in situations where a
business or university may block P2P because of copyright concerns or
that they don't want someone continuously uploading and using
bandwidth, or because of configuration issues with routers, etc. In
the end, you want people to be able to download whatever you are
distributing.
MS: Great! So how does one create a .metalink for their downloads?
AB: The easiest way to create a .metalink is with the Metalink Editor
which has a graphical interface. If you're going to be making a bunch,
you may want to use the automated tools which have a command line
interface or server tools which are meant to be run on mirrors. This
article on osresources details the non-graphical methods for .metalink
creation.
MS: Right. So do I need a special application to use a metalink
download? Won't it work with wget?
AB: Right now you do need a special application that supports reading
the .metalink file and using that info. Luckily, most download
managers support it right now, but no Web browsers natively. Support
in Opera and Firefox would make the benefits way more accessible to
regular people. wget doesn't support it yet, but aria2 is a similar
command-line program that does. The beauty of free and open source
software is that if people are interested enough, then support can be
added to a bunch of download programs quickly and easily.
MS: Do you track the number of projects using Metalink?
AB: Yes, mostly free and open source projects use it right now, but
some proprietary companies and game makes use it. OpenOffice.org,
openSUSE, cURL, Arch Linux, DesktopBSD, blag linux, StartCom Linux,
Berry Linux, PC-BSD, Linux Mint, Ubuntu Christian Edition, redWall
Firewall, GoboLinux, TrueBSD, PuppyLinux, UniProt Consortium, Eiffel
Software, and Ankama games have used Metalink.
MS: Any new feature Metalink users, both offering and downloading
stuff, should keep an eye out for?
AB: I view Metalink as basic infrastructure and hopefully it will get
to the point where people don't even know they are using it. Metalink
as it is now is Wave One (or maybe .5) of coming improvements. I think
we can build on it and offer some really cool things, which will again
work transparently for people. Not all clients support all the
features of Metalink, like using repair information - that's a really
important feature.
Phex is the first program with Metalink support that is primarily P2P
(other BitTorrent clients like aria2 and GetRight support it). You'll
be able to export your library, or parts of it. If you want to share
something with a friend, you can just email a .metalink or put it on a
server for download, and they'll get the same exact files since
they're identified with checksums. No weeding through search results
searching for similar files will be needed.
MS: Thanks for taking time out for the interview. Please share a
little information about yourself and your life outside of computers.
AB: I'm 29, just finished my Master's degree, and looking for work
related to Metalink with open source projects. :) I live in
southeastern Florida, in Pompano Beach. I'm pretty laid back and I
love the ocean. I'm really into music, mostly instrumental, from
electronic to Indian, dub, jazz, and even normal stuff. I like old
books and movies and I'm interested in technology and how it affects
our lives.
--------------------------------------------------------------------------------
Mayank Sharma is a freelance writer from New Delhi, India. He is blown
away by the power of Free and Open Source Software and its usefulness
to developing nations.
Check out his blog at http://www.geekybodhi.net/
--------------------------------------------------------------------------------
