WP-MIRROR
Purpose
The Wikimedia Foundation offers wikipedias in nearly 300 languages.
WP-MIRROR is a free utility for mirroring any desired set of these wikipedias. That is, it builds a wikipedia farm that the user can browse locally. Many users need such off-line access, often for reasons of mobility, availability, and privacy. They currently use KIWIX which provides selected articles and thumbnail images.
WP-MIRROR builds a complete mirror with original size images. WP-MIRROR is robust, uses check-pointing to resume after interruption, and employs concurrency to accellerate mirroring of large wikipedias.
WP-MIRROR by default mirrors the simple wikipedia (Simple English means shorter sentences). The default should work `out-of-the-box' with no user configuration. It should build in 200ks (two days), occupy 60G of disk space, be served locally by a virtual host http://simple.wpmirror.site/, and update automatically every week. The default should be suitable for anyone who learned English as a second language (ESL).
The top ten wikipedias are the: en, de, fr, nl, it, es, pl, ru, ja, and pt wikipedias. Because WP-MIRROR uses original size image files, the top ten are too large to fit on a laptop with a single 500G disk, unless the user does not need the images (and this is configurable). The en wikipedia is the most demanding case. It should build in 5Ms (two months), occupy 3T of disk space, be served locally by a virtual host http://en.wpmirror.site/, and update automatically every month.
Most features are configurable, either through command-line options, or via a configuration file (/etc/wp-mirror/local.conf).
Use Cases
WP-MIRROR by default mirrors the simple wiki (Simple English means shorter sentences) which at 60G should fit on most laptops. Users may edit a configuration file (/etc/wp-mirror/local.conf) to specify any desired set of languages to be mirrored. For example:
Students learning English as a Second Language (ESL) might want Simple English side-by-side with their native language wikipedia:
- Arabic (ar simple)
- Hindi (hi simple)
- Vietnamese (vi simple)
- Chinese (zh-classical simple ) or (zh simple )
Someone interested in classical languages might configure:
Software developer might speed up the test cycle by choosing smaller languages:
Access
WP-MIRROR sets up a virtual host (e.g. http://simple.wpmirror.site/) so that the user may access the mirror locally using a web browser.
Process
WP-MIRROR is non-interactive and normally runs in background as a weekly cron job, updating the mirror whenever the Wikimedia Foundation posts new dump files.
WP-MIRROR maintains the state of the mirror in a transactional database (InnoDB which is the ACID compliant storage engine for MySQL). There are three advantages to this:
- Checkpointing. The state information is Durable (the `D' in ACID). When WP-MIRROR is interrupted (e.g. user closes laptop, power fails, cat walks across keyboard) the state information serves as a checkpoint. When WP-MIRROR is next started, it picks up where it left off.
- Concurrency. Multiple instances of WP-MIRROR can run concurrently. That is to say, each instance of WP-MIRROR is Isolated (the `I' in ACID) from every other instance. The concurrency feature is intended for desktop use when one is mirroring any of the top ten wikipedias (currently the en de fr nl it pl es ru ja and pt wikipedias).
- Monitoring. WP-MIRROR can also be run in monitor mode (concurrently with instances that are building mirrors). Instances running in monitor mode display the state of each wikipedia. If a suitable windowing system is present, progress bars are rendered using graphics in a separate window, and otherwise using ASCII characters in a console (see figures below).
WP-MIRROR is designed for robustness. WP-MIRROR asserts hardware and software prerequisites, skips over unparsable pages and bad file names, validates image files and sequesters those that are corrupt, waits for internet access when needed, and exits gracefully if disk space runs low.
Downloading WP-MIRROR
WP-MIRROR can be found on the main GNU server: http://download.savannah.gnu.org/releases/wp-mirror/ (via HTTP).
Documentation
Documentation for WP-MIRROR is available online. The WP-MIRROR Reference Manual is available in PDF format. If you install from a package, the documentation will be registered automatically with `doc-base' and readily found using `dhelp' or `dwww'.
You may also find more information about WP-MIRROR by running info wp-mirror or man wp-mirror, or by looking at /usr/share/doc/wp-mirror/, /usr/local/doc/wp-mirror/, or similar directories on your system. A brief summary is available by running wp-mirror --help.
Dependencies
WP-MIRROR has numerous dependences including: apache2, cURL, Graphics Magick, MediaWiki, and MySQL. For this reason, it is easiest for the user to install WP-MIRROR from a package.
WP-MIRROR 0.5 is available as a DEB package. It works `out-of-the-box' on Debian GNU/Linux 7.0 (wheezy) and Ubuntu 12.10 (quantal). Porting to other distributions may be considered for a future release.
WP-MIRROR 0.4 is available as a DEB package. It works `out-of-the-box' on Debian GNU/Linux 7.0 (wheezy).
WP-MIRROR 0.3 and earlier versions, were developed on a PC with the Debian GNU/Linux 6.0 (squeeze) distribution installed. User configuration of dependencies is required.
There are no plans to backport WP-MIRROR to earlier distributions.
Installation
Debian GNU/Linux 7.0 (wheezy), Ubuntu 12.10 (quantal)
1. Download the DEB package
Releases are found at http://download.savannah.gnu.org/releases/wp-mirror/. Select the most recent DEB package.
2. Install the DEB package
(root-shell)# dpkg --install wp-mirror-0.5-1_all.deb
3. Run
(root-shell)# wp-mirror --mirror
WP-MIRROR `just works'.
WP-MIRROR configuration is entirely automated. And that includes configuration of its dependencies such as `apache2', `cURL', `MediaWiki', `MySQL', and so on.
Debian GNU/Linux 6.0 (squeeze)
Mailing lists
WP-MIRROR has the following mailing lists:
- wp-mirror-announce is used to announce releases
- wp-mirror-devel is a closed list for developers and testers.
- wp-mirror-list is used to discuss most aspects of WP-MIRROR (e.g. feature requests and bug reports).
Getting involved
Development of WP-MIRROR, and GNU in general, is a volunteer effort, and you can contribute. For information, please read How to help GNU. If you'd like to get involved, it's a good idea to join the discussion mailing list (see above).
- Test releases
- Trying the latest test release (when available) is always appreciated. Test releases of WP-MIRROR can be found at http://download.savannah.gnu.org/releases/wp-mirror/ (via HTTP).
- Development
- For development sources, issue trackers, and other information, please see the WP-MIRROR project page at savannah.gnu.org.
- Translating WP-MIRROR
- To translate WP-MIRROR's messages into other languages, please see the Translation Project page for WP-MIRROR. If you have a new translation of the message strings, or updates to the existing strings, please have the changes made in this repository. Only translations from this site will be incorporated into WP-MIRROR. For more information, see the Translation Project.
- Maintainer
- WP-MIRROR is currently being maintained by Dr. Kent L. Miller. Please use the mailing lists for contact.
Licensing
WP-MIRROR is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 3 of the License, or (at your option) any later version.
