Please note that these Trac pages are no longer being updated. Wiki contents/documentation have moved to GitHub.

Archive/SeattleGeniInstallation

Version 47 (modified by leonwlaw, 7 years ago)

--

Seattle Clearinghouse Installation HowTo

This document explains how to do a fresh install of the Seattle Clearinghouse portal on a machine that is running a Debian-like Linux operating system such as Ubuntu.

  • For info on deploying the production Seattle Clearinghouse code and for starting/restarting Apache, see SeattleGeniProductionHttp?


Install Dependencies


Seattle Clearinghouse requires at least the following software to be installed:

Most of these can be installed through a package manager. For example, on a Debian-based system:

$ apt-get install apache2 libapache2-mod-python
$ apt-get install mysql-server mysql-client python-mysqldb
$ apt-get install ntp

The version of Django required is 1.3, which is available at  https://www.djangoproject.com/download/.


Create MySQL databases


You need two mysql databases and seperate users with access to each.

  • First database name: seattlegeni
  • Second database name: keydb

For simplicity you can make the database usernames correspond to the database names.

Here's an example of creating a database and a user:

mysql -u root -p

mysql> create database seattlegeni;

mysql> GRANT ALL PRIVILEGES 
ON seattlegeni.* 
TO 'seattlegeni'@'localhost'
IDENTIFIED BY 'password';

where you would replace the 'password' with the password you want to use for the user 'seattlegeni'

Be sure to repeat the process for a separate database called keydb.


Deploying and running Seattle Clearinghouse


  • Initial preparation
    • Checkout the seattle trunk from svn.
      svn co http://seattle.cs.washington.edu/svn/seattle/trunk
      
    • Deploy all necessary files to a directory of your choice. The directory you deploy to should be the name of a directory that does not yet exist. For example:
      python ./trunk/seattlegeni/deploymentscripts/deploy_seattlegeni.py ./trunk /tmp/deploy
      
    • Change to the seattlegeni directory that is in the directory you deployed to. For example:
      cd /tmp/deploy/seattlegeni
      
      Note: all future steps/instructions will assume you are in this directory.
  • Future tip
    • Starting all of the processes described below is simplified by using the script deploymentscripts/start_seattlegeni_components.sh, whose use is described in the section on running start_seattlegeni_components.sh. For reference, we'll explain here how to start the individual components as well as include first-time configuration information. Note that when starting these manually, you need to start them in the order shown. Most importantly, first the lockserver, then the backend, then everything else. However, we're not going to describe the initial setup in this order.
  • Generating state keys Seven different state keys need to be created in order for all the backend scripts to work. The name of the seven key files should be:
    1. acceptdonation.publickey
    2. movingto_canonical.publickey
    3. canonical.publickey
    4. movingto_onepercentmanyevents.publickey
    5. onepercentmanyevents.publickey
    6. movingto_twopercent.publickey
    7. twopercent.publickey
    • Note that the privatekey of the key pairs is not necessary. To generate the keys follow these instructions:
      • Create a temporary folder.
      • Navigate to the trunk/ folder.
      • Run the command:
        python preparetest.py TEMP_FOLDER
        
        where the temp folder is the folder created in step 1.
      • Navigate to the TEMP_FOLDER.
      • Run the command:
        python generatekeys.py NAME_OF_YOUR_PUBKEY_PRIV_KEY
        
        An example would be:
        python generatekeys.py acceptdonation
        
      • Generate a key pair for all four state keys.
      • Create a folder named statekeys in your /tmp/deploy/seattlegeni/node_state_transitions/ folder and copy the seven publickeys over to that folder.
      • You can delete the privatekeys if you want to as we won't need them.
      • Delete the TEMP_FOLDER and all its content at this point if you want to.
  • Setup the website and start a development version of it
    • Be sure you've already created a mysql database for seattlegeni (e.g. called seattlegeni).
    • Edit website/settings.py
      • Specify the name of the seattlegeni database, as well as the database username, password, etc.
      • Set a long, random string the value of SECRET_KEY.
      • If this is a production launch, also set DEBUG to False, and uncomment and change the fields for sending ADMINS email.
    • Set your environment variables:
      export PYTHONPATH=$PYTHONPATH:/tmp/deploy:/tmp/deploy/seattle
      export DJANGO_SETTINGS_MODULE='seattlegeni.website.settings'
      
      Note: the /tmp/deploy path entry is to make available the two packages 'seattlegeni' and 'seattle' which the deployment script created in the /tmp/deploy directory. The /tmp/deploy/seattle path item is to ensure that repyhelper can find repy files in the python path, as the repy files were placed in this directory by the deployment script. You will have to do this in each of the shell terminals you work in for the steps below. Alternatively you can edit the .bashrc file in your home directory and add the two lines in:
      PYTHONPATH=$PYTHONPATH:/tmp/deploy:/tmp/deploy/seattle
      DJANGO_SETTINGS_MODULE='seattlegeni.website.settings'
      
    • Create the database structure:
      python website/manage.py syncdb
      
    • For development, start the django development webserver:
      python website/manage.py runserver
      
      You will now have a local development server running on port 8000. This is convenient for development and testing your seattlegeni instance but should not be used in production. Using runserver does not require configuring apache.
    • For production, run the site through apache: ConfigureApachesites?

  • Start the lockserver
    • Set your environment variables the same way as shown above.
    • In a new shell, start the lockserver:
      python lockserver/lockserver_daemon.py
      
  • Setup the key database and start the backend
    • Make sure have a mysql database to use for the key database (e.g. called keydb)
    • Make sure that the file keydb/config.py is not readable by the user the website is running as (this is only something to worry about for production launch, if you are just developing or testing, this is not required.)
    • Edit the file keydb/config.py and set the database information for the key database.
    • Create the key database structure by executing the contents of the file keydb/schema.sql on the new key database you created.
      mysql -u[username] -p --database=[keydbname] < keydb/schema.sql
      
    • Edit the file backend/config.py and set a value for the authcode.
    • For production launch, make sure that the file backend/config.py is not readable by the website.
    • Set your environment variables the same way as shown above.
    • In a new shell, start the backend_daemon from the backend directory (because the repy files need to be in the directory it is run from):
      cd backend
      python backend_daemon.py
      
  • Start the polling daemons
    • Set your environment variables the same way as shown above. (Not necessary if you have added it to the .bashrc file)
    • There's only one polling daemon at the moment. To start it:
      python polling/check_active_db_nodes.py
      
  • Start the node state transition scripts
    • Set your environment variables the same way as shown above. (Not necessary if you have added it to the .bashrc file)
    • Start each transition script you intend to run:
      python node_state_transitions/TRANSITION_SCRIPT_NAME.py
      
    • The four transitions scripts that you probably want running are:
      • transition_donation_to_canonical
      • transition_canonical_to_twopercent
      • transition_twopercent_to_twopercent
      • transition_onepercentmanyevents_to_canonical


Configure Apache sites


To provide encryption, Seattle Clearinghouse relies on SSL. Therefore you will need to setup two VirtualHost entries at the minimum: one for connections to port 80 (HTTP), and one for connections to port 443 (SSL).

Here is a somewhat minimal configuration to serve the geni website from  https://mysite/geni/. Note that in this snippet, the second VirtualHost entry assumes that you have a server certificate and key file setup, and the Location directive assumes that your geni installation lives in /home/geni/ and that your django settings module is seattlegeni.website.settings. The first VirtualHost is only provided to redirect all connections to http://mysite/geni/* to use https://mysite/geni/*. Also, the seattlegeni installation was deployed to the directory /home/geni/live rather than /tmp/deploy, which was the deployment directory given as an example in the steps above.

Depending on you configuration of apache, you may want to add the below code to: /etc/apache2/sites-available/default

NameVirtualHost *:80
NameVirtualHost *:443

# HTTP
<VirtualHost *:80>
    # Redirect requests for the server index page or that are geni-related
    # to the https site.
    RedirectMatch ^/$ https://mysite/geni/html/login
    RedirectMatch ^/geni https://mysite/geni/html/login
</VirtualHost>

# SSL
<VirtualHost *:443>
    ServerAdmin webmaster@localhost
    SSLEngine on
    SSLCertificateFile /etc/apache2/ssl/server.crt
    SSLCertificateKeyFile /etc/apache2/ssl/server.key
    SSLCertificateChainFile /etc/apache2/ssl/gd_intermediate.crt

    Alias /site_media "/home/geni/live/seattlegeni/website/html/media"
    <Location "/site_media">
        SetHandler None
    </Location>

    Alias /admin_media "/usr/local/lib/python2.5/site-packages/django/contrib/admin/media"
    <Location "/admin_media">
        SetHandler None
    </Location>

    <Location /geni/>
        SetHandler python-program
        PythonHandler django.core.handlers.modpython
        SetEnv DJANGO_SETTINGS_MODULE seattlegeni.website.settings
        PythonOption django.root /geni
        PythonDebug Off
        # We add /usr/local/lib/python2.5/site-packages to ensure that our
        # manual installation of django 1.1 to /usr/local is in the path
        # before any copy of django installed through the distro's repositories.
        PythonPath "['/home/geni/live/', '/home/geni/live/seattle', '/usr/local/lib/python2.5/site-packages'] + sys.path"
    </Location>

    # Make sure various locations people might request redirect somewhere that works.
    RedirectMatch ^/$ https://mysite/geni/html/login
    RedirectMatch ^/geni/?$ https://mysite/geni/html/login
    RedirectMatch ^/geni/html/?$ https://mysite/geni/html/login

    # Don't require a slash on the end of the admin url.
    RedirectMatch ^/geni/admin$ https://mysite/geni/admin/
</VirtualHost>

To use this configuration for your Seattle Clearinghouse installation, change "mysite" to your domain name, change "/home/geni/live/" in the site media location and Python path to the directory where you deployed Seattle Clearinghouse, and make sure "/admin_media" is aliased to a valid directory, as the exact location will vary depending on the version of Python installed and how you installed Django. A good place to put this configuration is the file /etc/apache2/sites-enabled/000-default.

To configure SSL you will probably need to install openssl to generate a private key/CSR (Certificate Signing Request), and then possibly purchase a certificate for your site. For more information see  this page. Put your SSL key and certificate in a directory named /etc/apache2/ssl (or change the configuration to correctly reference them). If you generate your own certificate instead of buying one, remove the line for "SSLCertificateChainFile".

Be sure to restart Apache after you are done changing the configuration files.

$ /etc/init.d/apache2 restart

If Apache gives the error:

Invalid command 'SSLEngine', perhaps misspelled or defined by a module not included in the server configuration

then you need to enable ssl by running:

$ sudo a2enmod ssl

For importing all the required modules, Apache requires read and execute permissions for the seattle and seattlegeni directories. To ensure Apache has the correct permissions, run the following command in your Seattle Clearinghouse directory:

$ chmod -R o+rx seattle/ seattlegeni/

If you try to access your Seattle Clearinghouse installation's website now, then it's possible you'll see an exception like this:

Cannot open file for translation '/home/geni/live/seattle/rsa.repy': [Errno 13] Permission denied: '/home/geni/live/seattle/rsa_repy.py'

This is because Apache doesn't have permission to overwrite the *_repy.py files when repyhelper get invoked, so as a workaround you can cd to the seattle directory and change the permissions:

$ chmod a+w *_repy.py


Running start_seattlegeni_components.sh


If you have all the components of Seattle Clearinghouse (including Apache) configured, the script deploymentscripts/start_seattlegeni_components.sh will start up all the individual components in the correct order.

Before running start_seattlegeni_components.sh, make sure to change PYTHONPATH, SEATTLEGENI_DIR, and LOG_DIR to the correct locations for your deployment, change geni in SUDO_CMD to the correct username, and create the directory for LOG_DIR if it doesn't already exist.

First, if one or more of the Seattle Clearinghouse scripts is already running, kill them before running start_seattlegeni_components.sh.

To run start_seattlegeni_components.sh, run the following commands with the correct directory substituted for /home/geni/live:

$ sudo -i
$ screen
$ cd /home/geni/live/seattlegeni/deploymentscripts
$ ./start_seattlegeni_components.sh

This will start the script in a new screen session running as root. Hit CTRL-A followed by D to detach the screen session.

To later reattach to the session in order to stop or restart start_seattlegeni_components.sh:

$ sudo -i
$ screen -r


Conclusion


Congratulations! You should now have a fully operational Seattle Clearinghouse installation that you can access at  https://mysite/geni/