User:Magnus Manske/Migrating from toolserver
This page describes my personal experiences with porting my tools from the toolserver to labs. I hope it can help others.
Concept
Labs is based on projects, and (usually several) virtual machines (VMs) within each project. For migration from the toolserver, you want to become a member of project "tools". To achieve that:
- Create an account on this wiki; you will have both a "user name" and a "shell name". The "shell name" is your unix account.
- Create SSH keys and upload your public key here, as described at https://wikitech.wikimedia.org/wiki/Help:SSH
- Fill in the Tools Access Request form.
- Once your account has been accepted, you can login to tools-login: ssh -A [email protected].
Create a new tool
- Once you are part of project "tools", go to the Tool Labs webpage.
- Tools on the shared VMs are called "service groups". Each service group is a tool, and has:
- a UNIX user "local-TOOL"
- a UNIX group "local-TOOL"
- a directory /data/project/TOOL, with subdirectories public_html for HTML/JS/CSS, and cgi-bin for
Perl/PHP scriptsnot sure what would go here
- To create a new project, on the Tool Labs webpage (next to "Hosted tools"), click on "create new tool". Enter the TOOL name (without "local-"!). This will:
- create the tool directories. public_html will be visible under http://tools.wmflabs.org/TOOL, and cgi-bin under http://tools.wmflabs.org/TOOL/cgi-bin
- add your UNIX user to the UNIX group "local-TOOL"
- create the UNIX user "local-TOOL", also in that UNIX group. After logging out and in, you can act as that user via ssh with become TOOL, but you can also create and edit files under your "normal" UNIX name
Note: It seems that the system takes a few minutes until your "normal" UNIX name can write to the tool directories. Give it half an hour after tool creation to be on the safe side, or work as the "local-TOOL" user.
- There is help page about accessing Labs Tools with PuTTY and WinSCP. The tools project supports direct ssh login at tools-login.wmflabs.org, no need to connect through the labs bastion.
- For a full server/status list, see Nova Resource:Tools.
jargon
The jargon used for the TS and on Labs is slightly different. See below for some hints.
Toolserver | Labs |
---|---|
Cluster (e.g. 'the Toolserver Cluster') | Project (e.g. 'the Tools Project') |
Roots | Project Admins |
Multi-maintainer Project | Service group |
qsub et al
Tool Labs also uses SGE for job scheduling. There are a few things to take into account when moving a tool:
- qcronsub is not available - use qsub instead (same syntax, but doesn't stop multiple jobs) or jsub
- Tool Labs uses a different memory limit system - instead of virtual_free (which does not count shared libraries), it uses h_vmem. For instance, instead of
-l virtual_free=50M
, use-l h_vmem=256M
. - There are no Solaris servers!
PHP
PHP scripts (other languages probably as well) go into public_html, not cgi-bin. Also, the "local-TOOL" user has to be the owner of the file and the subdirectories within public_html, otherwise you'll get a 500 error!
Slow connections
If you're in Europe, and used to the Toolserver, you might notice the somewhat higher ping times (170-200 ms compared to 20 ms). To mitigate this, try using mosh (for windows, follow these somewhat more complicated instructions).
To connect, use
mosh -a tools-login.wmflabs.org
(-a to force predictive echo) instead of the normal
ssh tools-login.wmflabs.org
Connecting to different servers
If you want to ssh to specific bot instances other than tools-login.wmflabs.org, it is helpful to create a new SSH key:
$ ssh-keygen $ cat ~/.ssh/id_rsa.pub ssh-rsa .... user@host
Copy the ssh-rsa ... user@host
line to your authorized keys in the labs console.
Database access
Replicated databases
The sql command works for both you and the "tool user" on tools-login:
sql enwiki_p
For your bots and web tools, read the user name and password for the MySQL access from the
/data/project/TOOL/replica.my.cnf
file. The database name is the usual format
enwiki_p
and the server name is build by omitting the "_p" and appending ".labsdb" :
enwiki.labsdb
Create new databases
You can create databases on any of the replicas and on tools-db, as long as the database name starts with the username (that is the database username, not the tool user name!) from replica.my.cnf, followed by two underscores:
USERNAME__DATABASENAME
Each tool also has its own database on server
tools-db
The name of the database is the name of the tool with anything but letters and digits removed ("super-tool" → "supertool"). You cannot access this database with the replica credentials, but must use the credentials in ~/.my.cnf. New tools don't get the .my.cnf file, but the replica credentials will work.
See also documentation here.
GIT
WMF Labs does not offer an easy-to-use code repository. However, you can use a local git to version and keep backups, or BitBucket for unlimited git repositories (up to 5 users per git).
Note that in order to be able to push to BitBucket or GitHub from the Tool Labs server, the shell you use to ssh to tools-login needs to have both of your ssh keys available (if they're different). And, regardless of whether they're different, you need to have key forwarding enabled for your ssh session (this is done by passing -A
to ssh
, like ssh -A tools-login.wmflabs.org
BitBucket
- Create an account on https://bitbucket.org, e.g. MYBITBUCKETUSER
- Create new repository on bitbucket, e.g. MYTOOL
- ssh into tools-login, go to /data/project/MYTOOL
- Run
git init
- Run
git remote add origin https://[email protected]/MYBITBUCKETUSER/MYTOOL.git
- Add files to git, e.g.
git add public_html
- Run
git commit -m "Initial commit"
- Run
git push -u origin --all
GitHub
- Create an account on https://github.com, e.g. MYGITHUBUSER (e.g. this should be related to you, e.g. your Wikimedia user name, not the Tool)
- Create new repository on GitHub, e.g. MYTOOL
- ssh into tools-login, go to /data/project/MYTOOL
- Run
git init
- Add files to git, e.g.
git add public_html
- Run
git remote add origin [email protected]:MYGITHUBUSER/MYTOOL.git
- Run
git commit -m "Initial commit"
- Run
git push -u origin master
Local git
A local repository will keep versioned backups of your code, but if you accidentally delete the whole directory, this will go too!
- Run
git init
- Add files to git, e.g.
git add public_html
- Run
git commit -m "Initial commit"
Unsolved issues
- Wiki(m|p)edia page view stats would be nice-to-have :-)