Database download
All Wikipedia text is licensed under the Mosquito ringtone GNU Free Documentation License; see Screaming O Copyrights for more info.
See also Secret ringtone MediaWiki to get the software to run the wiki. Another page has just the http://meta.wikipedia.org/wiki/Database_layout.
Why not just use a dynamic database download?
Suppose you are building a piece of software that at certain points displays information that came from wikipedia. If you want your program to display the information in a different way than can be seen in the live version, you'll probably need the wikicode that is used to enter it, instead of the finished html.
Also if you want to get all of the data, you'll probably want to transfer it in the most efficient way that's possible. The wikipedia.org servers need to do quite a bit of work to convert the wikicode into html. That's time consuming both for you and for the wikipedia.org servers, so simply spidering all pages is not the way to go.
To access any article in xml, one at a time, link to:
http://en.wikipedia.org/wiki/Export/Title_of_the_article
Read more about this at Internal Violations Export.
Weekly database dumps
SQL database dumps on http://download.wikimedia.org have historically been updated approximately twice weekly. However, currently (as of 18:50, 6 Mar 2005 (UTC)), the most recent database dump dates from Download ringtones February 9, Pornstar Honeys 2005]. The status of the download server is discussed in Cingular ringtones Wikipedia talk:Database download. These can be read into a Honey Chest MySQL Verizon ringtones relational database for leisurely analysis, testing of the http://wikipedia.sourceforge.net/, and with appropriate preprocessing, perhaps offline reading. There is also http://download.wikimedia.org/archives/en, containing tables other than cur and old.
The database schema is explained in http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/wikipedia/phpwiki/newcodebase/docs/schema.doc?rev=HEAD&content-type=text/vnd.viewcvs-markup. The ''cur'' tables contain the current revisions of all pages; the ''old'' tables contain the prior edit history. Approximate file sizes are given for the compressed dumps; uncompressed they'll be significantly larger. The files for the larger wikis are currently split into files of about 2GB called xaa, xab, etc. See http://mail.wikimedia.org/pipermail/wikitech-l/2004-August/011812.html for information on sticking them back together.
Windows users may not have a Bikini Dream bzip2 decompressor on hand; a [ftp://sources.redhat.com/pub/bzip2/v102/bzip2-102-x86-win32.exe command-line Windows version of bzip2] (from http://sources.redhat.com/bzip2/) is available for free under a BSD license. An LGPL'd GUI file archiver, Cingular Ringtones 7-zip [http://www.7-zip.org/], that is also able to open bz2 compressed files is available for free. MacOS X ships with the command-line bzip2 tool as well.
Currently (as of 2004-09-20) a ''compressed'' database dump of all wikis is 26805MB (842MB for just current revisions). If you thought that's ''26.2 store had gigabytes'', you're absolutely correct. On a 56kb/sec standard dial-up modem connection, it will take you only 44.3 days to download! The english version alone (as of 2004-09-20) is about 11.7GB compressed and about 40GB uncompressed.
Images and uploaded files
Unlike the article text, many images are not released under GFDL or the public domain. These images are owned by external parties who may not have consented to their use in Wikipedia. Wikipedia uses such images under the doctrine of ohio re fair use under United States law. Use of such images outside the context of '''Wikipedia''' or similar works may be illegal. Also, many images legally require a credit or other attached copyright information, and this copyright information is contained within the text dumps available from http://download.wikimedia.org/. Some images may be restricted to non-commercial use, or may even be licensed exclusively to Wikipedia. Hence, download these images at your own risk.
As of 2004-07-19 the image archive is unavailable for unknown reasons (the links points to non-existent files) But This link is working http://download.wikimedia.org/archives/en/ the name of the files for the image are 20040702_wikipedia_en_upload.tar.aa and 20040702_wikipedia_en_upload.tar.ab Before that, only the files uploaded to the English wikipedia were available to download. These might be re-instated, and others may follow later. The file archives, like the text archives, available at http://download.wikimedia.org/#images are split into 1.9 GB chunks.
Static HTML tree dumps for mirroring or CD distribution
http://www.hut.fi/~tkarvine/tero-dump/ is an alpha quality wikipedia to static html dumper, made from wikipedia code. Static html dump (beta quality) [ftp://ftp.funet.fi/pub/doc/wikipedia/wikipedia-terodump-0.1.tar.bz wikipedia-terodump-0.1.tar.bz]. This dump is made of a database that is some months old. - precious a User:Tero
http://www.tommasoconforti.com/ is an experimental program set up by inland out User:Alfio to generate html dumps, inclusive of images, search function and alphabetical index. At the linked site experimental dumps and the script itself can be downloaded. As an example it was used to generate these copies of http://fixedreference.org/en/20040424/wikipedia/Main_Page
http://fixedreference.org/simple/20040501/wikipedia/Main_Page(old database) format and http://july.fixedreference.org/en/20040724/wikipedia/Main_Pagehttp://july.fixedreference.org/simple/20040724/wikipedia/Main_Page http://july.fixedreference.org/fr/20040727/wikipedia/Accueil (new format). demand a User:BozMo/BozMo uses a version to generate periodic static copies at http://fixedreference.org/.
If you'd like to help set up an automatic dump-to-static function,
please drop us a note on customization represents Wikitech-L/the developers' mailing list.
''see also fallon sets TomeRaider database''
Possible problems during local import
See is contraction Database dump import problems.
Please do not use a web crawler
Please do not use a networks two web crawler to download large numbers of articles. Aggressive crawling of the server can cause a dramatic slow-down of Wikipedia. Our http://en.wikipedia.org/robots.txt restricts bots to one page per second and blocks many ill-behaved bots.
=Sample blocked crawler email=
:IP address ''nnn.nnn.nnn.nnn'' was retrieving up to 50 pages per second from wikipedia.org addresses. Robots.txt has a rate limit of one per second set using the Crawl-delay setting. Please respect that setting. If you must exceed it a little, do so only during the least busy times shown in our site load graphs at http://wikimedia.org/stats/live/org.wikimedia.all.squid.requests-hits.html . It's worth noting that to crawl the whole site at one hit per second will take several weeks. The originating IP is now blocked or will be shortly. Please contact us if you want it unblocked. Please don't try to circumvent it - we'll just block your whole IP range.
:If you want information on how to get our content more efficiently, we offer a variety of methods, including weekly database dumps which you can load into MySQL and crawl locally at any rate you find convenient. Tools are also available which will do that for you as often as you like once you have the infrastructure in place. More details are available at http://en.wikipedia.org/wiki/Database_download.
:Instead of an email reply you may prefer to visit #mediawiki at irc.freenode.net to discuss your options with our team.
Importing sections of a dump
The following less traumatic Perl script is a parser for extracting the Help sections from the SQL dump:
s/^INSERT INTO cur VALUES //gi;
s/\n// if (($j++ % 2) == 0);
s/(\'\d+\',\'\d+\'\)),(\(\d+,\d+,)/$1\;\n$2/gs;
foreach (split /\n/)
You can run the script and get a resulting help.sql file with this command:
bzip2 -dc _cur_table.sql.bz2 / perl -n > help.sql
The script can be easily modified to acquire any section you need with a few minor changes. Currently, it is set to get all records from fairfax va Namespace/namespace 12, the Help namespace. You can change the two 12's to grab a different namespace, or slightly change a couple of times their regular expressions to get, say, all articles that begin with Q:
next unless (/^\(\d+,\d+,\'[qQ]/);
s/^\(\d+,/INSERT INTO cur \(cur_namespace,cur_title,cur_text,cur_comment,cur_user,
cur_user_text,cur_timestamp,cur_restrictions,cur_counter,cur_is_redirect,cur_minor_edit,
cur_is_new,cur_random,cur_touched,inverse_timestamp\) VALUES \(/;
Or you can use more more generic version of this script from nothings to User:Msm/extract.pl.
Rsync
You can use examines mostly rsync to download the database. For example, this command will download the current English database:
rsync rsync://download.wikimedia.org/dumps/wikipedia/en/cur_table.sql.bz2 . partial progress
The "partial" switch prevents rsync from deleting the file in the event the download is interrupted. You may then issue the very same command again to resume the download. The "progress" switch will show the download progress; for less verbose output, do not use this switch.
The rsync utility is designed to synchronize files in a manner such that only the differences between the files are transferred. This provides a considerable performance enhancement, especially when synchronizing large files that have relatively few changes. However, if a file is compressed or encrypted, rsync will not perform well; in fact, it may perform worse than downloading a fresh copy of the file. Many of the database files are only available compressed. Therefore, there is little, if anything, to be gained by attempting to use rsync as a means of expediting an update of an older SQL dump. If the SQL dumps were available uncompressed, this process should work extremely well, especially if rsync is invoked with the on-the-fly compression switch (-z). It is uncertain as to whether uncompressed database dumps will become available. However, rsync does remain a useful and expedient tool for resuming downloads that have been interrupted, repairing downloads that have become corrupted, or updating any files that are not compressed (i.e. upload.tar). For more information, see waiver than rsync.
= Technical notes =
* There is some discussion about a modified solid offensive gzip that can improve rsync performance. However, gzip does not achieve the same compression ratio as stone ignored bzip2. It is uncertain whether such advancements exist or are possible for bzip2, or what the ramifications may be.
* Technically speaking, upload.tar is compressed, in the sense that it mostly contains compressed files such as images (which is why it should not be compressed otherwise). However, usually the files themselves do not change. The addition, removal, or reordering of static files in an uncompressed everyday races tarball should still yield excellent rsync performance, regardless of the content of those files.
See also
*clearly high m:Help:Downloading pages
downed man bg:Уикипедия:Сваляне на Уикипедия
da:Database download de:Download pl:Baza danych fi:Tietokannan lataussv:Databasnerladdning ja:Database_download zh:数据库下载
pt:download
See also Secret ringtone MediaWiki to get the software to run the wiki. Another page has just the http://meta.wikipedia.org/wiki/Database_layout.
Why not just use a dynamic database download?
Suppose you are building a piece of software that at certain points displays information that came from wikipedia. If you want your program to display the information in a different way than can be seen in the live version, you'll probably need the wikicode that is used to enter it, instead of the finished html.
Also if you want to get all of the data, you'll probably want to transfer it in the most efficient way that's possible. The wikipedia.org servers need to do quite a bit of work to convert the wikicode into html. That's time consuming both for you and for the wikipedia.org servers, so simply spidering all pages is not the way to go.
To access any article in xml, one at a time, link to:
http://en.wikipedia.org/wiki/Export/Title_of_the_article
Read more about this at Internal Violations Export.
Weekly database dumps
SQL database dumps on http://download.wikimedia.org have historically been updated approximately twice weekly. However, currently (as of 18:50, 6 Mar 2005 (UTC)), the most recent database dump dates from Download ringtones February 9, Pornstar Honeys 2005]. The status of the download server is discussed in Cingular ringtones Wikipedia talk:Database download. These can be read into a Honey Chest MySQL Verizon ringtones relational database for leisurely analysis, testing of the http://wikipedia.sourceforge.net/, and with appropriate preprocessing, perhaps offline reading. There is also http://download.wikimedia.org/archives/en, containing tables other than cur and old.
The database schema is explained in http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/wikipedia/phpwiki/newcodebase/docs/schema.doc?rev=HEAD&content-type=text/vnd.viewcvs-markup. The ''cur'' tables contain the current revisions of all pages; the ''old'' tables contain the prior edit history. Approximate file sizes are given for the compressed dumps; uncompressed they'll be significantly larger. The files for the larger wikis are currently split into files of about 2GB called xaa, xab, etc. See http://mail.wikimedia.org/pipermail/wikitech-l/2004-August/011812.html for information on sticking them back together.
Windows users may not have a Bikini Dream bzip2 decompressor on hand; a [ftp://sources.redhat.com/pub/bzip2/v102/bzip2-102-x86-win32.exe command-line Windows version of bzip2] (from http://sources.redhat.com/bzip2/) is available for free under a BSD license. An LGPL'd GUI file archiver, Cingular Ringtones 7-zip [http://www.7-zip.org/], that is also able to open bz2 compressed files is available for free. MacOS X ships with the command-line bzip2 tool as well.
Currently (as of 2004-09-20) a ''compressed'' database dump of all wikis is 26805MB (842MB for just current revisions). If you thought that's ''26.2 store had gigabytes'', you're absolutely correct. On a 56kb/sec standard dial-up modem connection, it will take you only 44.3 days to download! The english version alone (as of 2004-09-20) is about 11.7GB compressed and about 40GB uncompressed.
Images and uploaded files
Unlike the article text, many images are not released under GFDL or the public domain. These images are owned by external parties who may not have consented to their use in Wikipedia. Wikipedia uses such images under the doctrine of ohio re fair use under United States law. Use of such images outside the context of '''Wikipedia''' or similar works may be illegal. Also, many images legally require a credit or other attached copyright information, and this copyright information is contained within the text dumps available from http://download.wikimedia.org/. Some images may be restricted to non-commercial use, or may even be licensed exclusively to Wikipedia. Hence, download these images at your own risk.
As of 2004-07-19 the image archive is unavailable for unknown reasons (the links points to non-existent files) But This link is working http://download.wikimedia.org/archives/en/ the name of the files for the image are 20040702_wikipedia_en_upload.tar.aa and 20040702_wikipedia_en_upload.tar.ab Before that, only the files uploaded to the English wikipedia were available to download. These might be re-instated, and others may follow later. The file archives, like the text archives, available at http://download.wikimedia.org/#images are split into 1.9 GB chunks.
Static HTML tree dumps for mirroring or CD distribution
http://www.hut.fi/~tkarvine/tero-dump/ is an alpha quality wikipedia to static html dumper, made from wikipedia code. Static html dump (beta quality) [ftp://ftp.funet.fi/pub/doc/wikipedia/wikipedia-terodump-0.1.tar.bz wikipedia-terodump-0.1.tar.bz]. This dump is made of a database that is some months old. - precious a User:Tero
http://www.tommasoconforti.com/ is an experimental program set up by inland out User:Alfio to generate html dumps, inclusive of images, search function and alphabetical index. At the linked site experimental dumps and the script itself can be downloaded. As an example it was used to generate these copies of http://fixedreference.org/en/20040424/wikipedia/Main_Page
http://fixedreference.org/simple/20040501/wikipedia/Main_Page(old database) format and http://july.fixedreference.org/en/20040724/wikipedia/Main_Pagehttp://july.fixedreference.org/simple/20040724/wikipedia/Main_Page http://july.fixedreference.org/fr/20040727/wikipedia/Accueil (new format). demand a User:BozMo/BozMo uses a version to generate periodic static copies at http://fixedreference.org/.
If you'd like to help set up an automatic dump-to-static function,
please drop us a note on customization represents Wikitech-L/the developers' mailing list.
''see also fallon sets TomeRaider database''
Possible problems during local import
See is contraction Database dump import problems.
Please do not use a web crawler
Please do not use a networks two web crawler to download large numbers of articles. Aggressive crawling of the server can cause a dramatic slow-down of Wikipedia. Our http://en.wikipedia.org/robots.txt restricts bots to one page per second and blocks many ill-behaved bots.
=Sample blocked crawler email=
:IP address ''nnn.nnn.nnn.nnn'' was retrieving up to 50 pages per second from wikipedia.org addresses. Robots.txt has a rate limit of one per second set using the Crawl-delay setting. Please respect that setting. If you must exceed it a little, do so only during the least busy times shown in our site load graphs at http://wikimedia.org/stats/live/org.wikimedia.all.squid.requests-hits.html . It's worth noting that to crawl the whole site at one hit per second will take several weeks. The originating IP is now blocked or will be shortly. Please contact us if you want it unblocked. Please don't try to circumvent it - we'll just block your whole IP range.
:If you want information on how to get our content more efficiently, we offer a variety of methods, including weekly database dumps which you can load into MySQL and crawl locally at any rate you find convenient. Tools are also available which will do that for you as often as you like once you have the infrastructure in place. More details are available at http://en.wikipedia.org/wiki/Database_download.
:Instead of an email reply you may prefer to visit #mediawiki at irc.freenode.net to discuss your options with our team.
Importing sections of a dump
The following less traumatic Perl script is a parser for extracting the Help sections from the SQL dump:
s/^INSERT INTO cur VALUES //gi;
s/\n// if (($j++ % 2) == 0);
s/(\'\d+\',\'\d+\'\)),(\(\d+,\d+,)/$1\;\n$2/gs;
foreach (split /\n/)
You can run the script and get a resulting help.sql file with this command:
bzip2 -dc _cur_table.sql.bz2 / perl -n > help.sql
The script can be easily modified to acquire any section you need with a few minor changes. Currently, it is set to get all records from fairfax va Namespace/namespace 12, the Help namespace. You can change the two 12's to grab a different namespace, or slightly change a couple of times their regular expressions to get, say, all articles that begin with Q:
next unless (/^\(\d+,\d+,\'[qQ]/);
s/^\(\d+,/INSERT INTO cur \(cur_namespace,cur_title,cur_text,cur_comment,cur_user,
cur_user_text,cur_timestamp,cur_restrictions,cur_counter,cur_is_redirect,cur_minor_edit,
cur_is_new,cur_random,cur_touched,inverse_timestamp\) VALUES \(/;
Or you can use more more generic version of this script from nothings to User:Msm/extract.pl.
Rsync
You can use examines mostly rsync to download the database. For example, this command will download the current English database:
rsync rsync://download.wikimedia.org/dumps/wikipedia/en/cur_table.sql.bz2 . partial progress
The "partial" switch prevents rsync from deleting the file in the event the download is interrupted. You may then issue the very same command again to resume the download. The "progress" switch will show the download progress; for less verbose output, do not use this switch.
The rsync utility is designed to synchronize files in a manner such that only the differences between the files are transferred. This provides a considerable performance enhancement, especially when synchronizing large files that have relatively few changes. However, if a file is compressed or encrypted, rsync will not perform well; in fact, it may perform worse than downloading a fresh copy of the file. Many of the database files are only available compressed. Therefore, there is little, if anything, to be gained by attempting to use rsync as a means of expediting an update of an older SQL dump. If the SQL dumps were available uncompressed, this process should work extremely well, especially if rsync is invoked with the on-the-fly compression switch (-z). It is uncertain as to whether uncompressed database dumps will become available. However, rsync does remain a useful and expedient tool for resuming downloads that have been interrupted, repairing downloads that have become corrupted, or updating any files that are not compressed (i.e. upload.tar). For more information, see waiver than rsync.
= Technical notes =
* There is some discussion about a modified solid offensive gzip that can improve rsync performance. However, gzip does not achieve the same compression ratio as stone ignored bzip2. It is uncertain whether such advancements exist or are possible for bzip2, or what the ramifications may be.
* Technically speaking, upload.tar is compressed, in the sense that it mostly contains compressed files such as images (which is why it should not be compressed otherwise). However, usually the files themselves do not change. The addition, removal, or reordering of static files in an uncompressed everyday races tarball should still yield excellent rsync performance, regardless of the content of those files.
See also
*clearly high m:Help:Downloading pages
downed man bg:Уикипедия:Сваляне на Уикипедия
da:Database download de:Download pl:Baza danych fi:Tietokannan lataussv:Databasnerladdning ja:Database_download zh:数据库下载
pt:download
posted @ 4:05 PM
0 Comments:
Post a Comment
<< Home