welcome: please sign in
location: Moodle

Docs Migrated to: https://osmium.morningside.edu/confluence/x/GQA-AQ

Morningside College Moodle Technical Notes

http://moodle.morningside.edu is a LearningMangementSoftware (LMS) used on campus for undergraduate and graduate courses (both regular courses, hybrid, and online courses.)

Note

This page features technical documentation for the Moodle system. For user documentation, see the Moodle Knowledge Base.

Course Backups (How it works and How to restore them)

Moodle is configured to backup two copies of all changed classes to the automated course backup area. These files are not readily accessible from the CLI. They are, however, accessible from the course in question and this makes easy access by course instructors possible.

For courses older than 2 copies, the entire moodle system is backed up onto Hips. A mysqldump is run nightly by cron to ensure a consistent daily backup to Hips.

User Profiles and LDAP

Our Moodle implementation keeps track of two types of Profiles:

  1. Internal "Self Registered"
    1. (Past) Created by the student with a valid email address
    2. Created by an administrator for a user without "Mustang" credentials
      1. For a user "smith", please create a student account in the following fashion:
      2. Username: "smith-student", e-mail: "smith+student@morningside.edu". This consistency will allow faculty to receive the e-mail for their test user in their regular inbox.

  2. LDAP Profile
    1. Created automatically when a valid LDAP account is used to login
    2. Created automatically when the php /var/www/moodle.morningside.edu/moodle/auth/ldap/cli/sync_users.php< is run

With this added flexibility comes added complexity in troubleshooting. When troubleshooting be sure to find out if they are using their Mustang credentials or if this is an Internal "Self Registered" profile.

AutoEnrolments

We are able to syncronize Moodle enrollments to Aims via a helper script that Mike Husmann wrote which dumps STUD.SCHEDS. This list is parsed for current course registrations every 15 minutesand updates the information in a table in the moodle database called morn_autoenrol.

morn_autoenrol table

   1 CREATE TABLE `morn_autoenrol` (
   2   `userid` varchar(100) default NULL,
   3   `courseid` varchar(100) default NULL,
   4   `term` varchar(8) default NULL,
   5   `role` varchar(100) default NULL,
   6   PRIMARY KEY  (`userid`, `courseid`, `term`, `role`)
   7 ) ENGINE=MyISAM DEFAULT CHARSET=latin1;

Which will be changed in future revisions to

   1 CREATE TABLE `morn_autoenrol` (
   2  `idno` varchar(100) default NULL,
   3   `courseid` varchar(100) default NULL,
   4   `term` varchar(8) default NULL,
   5   `role` varchar(100) default NULL,
   6   PRIMARY KEY  (`userid`, `courseid`, `term`, `role`)
   7 ) ENGINE=MyISAM DEFAULT CHARSET=latin1;

Moodle reads this table and updates its internal enrolments:

  1. Every 24 hours at approximately 0630
  2. When a user logs in their own enrolments are checked.

To assist in debugging I've created a page at https://admin.morningside.edu/autoenrol.cgi which shows the enrollment information as Moodle knows it.

Users will not be on this list if:

Courses are identified by TERM-COURSEID where COURSEID is the catelog number of the course, such as MATH-101-001 and drawn from the fiscal yeal and the semester from the AimsTerms (eg: 2013/20 becomes 1320).

Backups

Important Moodle files are...

These files are backed up to Hips using the Ribs system.

Interesting SQL Queries for Moodle

Backup filename -> path conversion

Moodle uses a new (more irritating) scheme to lay out files on disk by content-hash instead of file-name. Thus, to map backup file names to backup file you need the database.

Files go in ${datadir}->filedir.

   1  SELECT * FROM mdl_files WHERE component LIKE "backup" LIMIT 10;

Files schema:

Field

Type

Null

Key

Default

Extra

id

bigint(10)

NO

PRI

NULL

auto_increment

contenthash

varchar(40)

NO

MUL

pathnamehash

varchar(40)

NO

UNI

contextid

bigint(10)

NO

MUL

NULL

component

varchar(100)

NO

MUL

filearea

varchar(50)

NO

itemid

bigint(10)

NO

NULL

filepath

varchar(255)

NO

filename

varchar(255)

NO

userid

bigint(10)

YES

MUL

NULL

filesize

bigint(10)

NO

NULL

mimetype

varchar(100)

YES

NULL

status

bigint(10)

NO

0

source

longtext

YES

NULL

author

varchar(255)

YES

NULL

license

varchar(255)

YES

NULL

timecreated

bigint(10)

NO

NULL

timemodified

bigint(10)

NO

NULL

sortorder

bigint(10)

NO

0

referencefileid

bigint(10)

YES

MUL

NULL

referencelastsync

bigint(10)

YES

NULL

referencelifetime

bigint(10)

YES

NULL

22 rows in set (0.00 sec)

Show duplicate course idnumbers

   1 SELECT idnumber 
   2   , id 
   3 FROM mdl_course 
   4 WHERE idnumber in 
   5   (
   6   SELECT idnumber 
   7   FROM 
   8     (
   9     SELECT idnumber, count(*) AS n 
  10     FROM mdl_course 
  11     WHERE idnumber != "" 
  12     GROUP BY idnumber 
  13     HAVING n > 1 
  14     ) as dups
  15   )
  16 ORDER BY idnumber,id;

Upgrading

Database & Files Backup

Note

Since these files are now greater than 400GB, it is best to do a VM snapshot to accomplish this.

Backup the MySQL database before doing an upgrade! Take a consistent database backup with mysqldump (This does lock the database while running, however.)

 # mysqldump -p moodle > /path/to/backup-file.sql

It can also be a good idea to copy the Moodle data from ./moodledata/ before an upgrade. This combined with the Database backup will allow you to painlessly roll back if the worst happens.

Maintenance Mode

While Moodle provides a built-in maintenance mode, it is not meant to keep users out during a version upgrade. My preference is to use Apache to completely redirect traffic away from the site while I work on the site.

Note

Be sure to edit maintenance-message.html as appropriate.

To deny new logins (while ignoring current users), create an empty file at /var/www/moodle.morningside.edu/moodle/nologin; this will use mod_rewrite to redirect access to login/index.php to maintenance-message.html as a 503 unavailable.

To deny all moodle activity (including currently logged-in users), create an empty file at /var/www/moodle.morningside.edu/moodle/maintenance.enable; this will also use mod_rewrite to redirect access to the site to maintenance-message.html as a 503 unavailable.

There is an extra condition in /var/www/moodle.morningside.edu/moodle/.htaccess to allow a given IP address to access the page during maintenance and no-login times.

GIT

The GIT versioning software is now used by moodle upstream. Git makes it easy to maintain software deviations from core between upgrades. Git can be a very tricky software and Morningside makes extensive use of it in organizing the Moodle code.

Git Organizational Hierarchy

Third-Party blocks and modules in Moodle are kept in their own repositories (therefore, they have a .git directory and can track upstream independently of the parent ./moodle directory. It can, therefore, be a little tedious to keep them organized all the time.

The git hierarchy on Moodle is as follows:

`moodle` (docroot, this code is checked out from upstream moodle git repository.)

`./mod`

Everything in /mod ships with moodle, except:

`./mod/attendance/`

Attendance replaces attforblock now that Modules and Blocks can have the same name. This is a supporting module and must be kept in tandem with blocks/attendance.

`./mod/respondusws/`

`./mod/journal`

During one of the upgrades, journal was moved out of Moodle Core and into its own module (because its functionality can be done with another built-in module). To ease the transition, the journal module was installed during that upgrade at Morningside.

`./mod/assignment/type/turnitin` (turn-it-in basic)

The turn-it-in basic plugin is deprecated. It is still installed, but uses Assignment2.2 which has been deprecated upstream. In fact, non-administrators cannot even create a new Assignment2.2 type, so this module is only used for backwards compatibility in courses that already used it.

TODO: Schedule a formal deprecation.

`./mod/turnitintool` (turn-it-in direct)

This is the current incarnation of turnitin tool, and integrates best with Moodle. It is also supported upstream. Because turnitin only provides .zip files, I keep a repository on git_hub to synchronize code between moodle and moodler as well as to ease upgrades.

`./mod/book/`

`./mod/wiziq`

WizIQ is currently under study for adoption as an e-portfolio solution by Graduate Studies in Education.

`./blocks/`

Everything in here ships with Moodle, except:

`./blocks/quickmail/`

QuickMail allows easy emailing to an entire class by the editingteacher.

`./blocks/attendance/`

This is the sister block to the attendance module, the two must be installed and upgraded in tandem.

`./blocks/exabis_eportfolio/`

Exabis eportfolio is being deprecated, I do not believe any new versions are available upstream.

TODO: Formal deprecation schedule.

General Commands

Make sure you have the latest sources (this is safe to run any time)

To see a list of changes between the present branch and the upstream...

To see the diffs of all changes between the present branch and the upstream:

To quickly stash work:

To get something out of the stash:

Minor Updates

Merge any new changes into the current branch:

When all files are ready, an upgrade is possible from the CLI:

Major Updates

Always make sure to have the latest sources (safe to run any time):

Make sure to be at the latest (previous) release before updating (see Minor Updates)

Create a new tracking branch (safe to run any time, does not change working branch):

Checkout the new remote-tracking branch (this is the dangerous part)

Here, re-apply contrib modules and code changes.

When all files are ready, an upgrade is possible from the CLI:

Caution

The CLI upgrade, historically, sometimes omits the same level of warning and error reporting that the browser-based does (such as for missing/incompatible plugins). This is dangerous.

Command line Moodle upgrade.
Please note you must execute this script with the same uid as apache!

Site defaults may be changed via local/defaults.php.

Options:
--non-interactive     No interactive questions or confirmations
--allow-unstable      Upgrade even if the version is not marked as stable yet,
                      required in non-interactive mode.
-h, --help            Print out this help

Example:
$sudo -u www-data /usr/bin/php admin/cli/upgrade.php

Installing

Notes on a from-scratch installation of Moodle.

RHEL packages:

Moodle 2 specific:

Install Morningside python modules:

File changes:

/etc/openldap/ldap.conf

   1  + TLS_REQCERT allow

/etc/postfix/main.cf

   1 --- main.cf-dist        2011-10-25 16:37:14.459512404 -0500
   2 +++ main.cf     2011-10-25 16:37:45.721498913 -0500
   3 @@ -74,6 +74,7 @@
   4  #
   5  #myhostname = host.domain.tld
   6  #myhostname = virtual.domain.tld
   7 +myhostname = webapps2.$mydomain
   8  
   9  # The mydomain parameter specifies the local internet domain name.
  10  # The default is to use $myhostname minus the first component.
  11 @@ -81,6 +82,7 @@
  12  # parameters.
  13  #
  14  #mydomain = domain.tld
  15 +mydomain = morningside.edu
  16  
  17  # SENDING MAIL
  18  # 
  19 @@ -316,6 +318,8 @@
  20  #relayhost = uucphost
  21  #relayhost = [an.ip.add.ress]
  22  
  23 +relayhost = rusty.morningside.edu
  24 +
  25  # REJECTING UNKNOWN RELAY USERS
  26  #
  27  # The relay_recipient_maps parameter specifies optional lookup tables

/etc/my.cnf

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0

# For Moodle utf8/unicode
default-character-set=utf8
default-collation=utf8_unicode_ci
character-set-server=utf8
collation-server=utf8_unicode_ci

# For Moodle db-sessions
max_allowed_packet=4M

# For InnoDB performance 
# (from http://www.mysqlperformanceblog.com/2007/11/01/
# innodb-performance-optimization-basics/ )
# innodb_buffer_pool_size=[70-80% of mem]
innodb_log_file_size=256M
innodb_log_buffer_size=4M
innodb_flush_log_at_trx_commit=2 # Can lose a transaction or two in a crash
# innodb_thread_concurrency=8
# innodb_flush_method=O_DIRECT # May be dangerous on non-stable storage
# innodb_file_per_table

# Some performance tweaks from mysqltuner.pl
#
max_connections=256

query_cache_type=1 # 1 = ON, enables caching except for SELECT SQL_NO_CACHE
query_cache_size=36M
query_cache_min_res_unit=2K #default = 4K

thread_cache_size=150 # In some cases this value can be very high (150+).
                      # Look at max_used_connections. If possible have this
                      # equal to the peak max_used connections.

# table_cache is related to max_connections. for 200 running connections, you 
# should have a table cache size of at least 200*N (max tables per join in any
# given query) 
table_cache=450 # based on 150 peak connections * 3

join_buffer_size=128K 

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

[client]
default-character-set=utf8

Crontab

# +---------------- minute (0 - 59)
# |  +------------- hour (0 - 23)
# |  |  +---------- day of month (1 - 31)
# |  |  |  +------- month (1 - 12)
# |  |  |  |  +---- day of week (0 - 7) (Sunday=0 or 7)
# |  |  |  |  |

# Mahara cron job. Minutely
* * * * * php /var/www/mahara.morningside.edu/mahara-1.8.0/htdocs/lib/cron.php > /var/log/mahara.log 2>&1

#
# Moodle maintenance jobs
#

# Perform Moodle-cron jobs every n-minutes. 
*/5 * * * * /var/www/moodle.morningside.edu/bin/moodle_cron_and_log.sh

# Every day, 0330, sync LDAP accounts to Moodle user database
30 3 * * * php /var/www/moodle.morningside.edu/moodle/auth/ldap/cli/sync_users.php > /dev/null

# Sync AIMS->morn_autoenrol table
*/15 * * * * /var/www/moodle.morningside.edu/bin/incremental_morn_autoenrol.py

# Update moodle course participants if morn_autoenroldb has been updated.
# (Checking for updates only once per hour.)
3 * * * * /var/www/moodle.morningside.edu/bin/conditional_enrol_sync.sh

# Moodle automated backup job
0 3 * * * /usr/bin/php /var/www/moodle.morningside.edu/moodle/admin/cli/automated_backups.php > /var/log/moodle/backup-`date '+\%a'`.log

#
# End of Moodle maintenance jobs
#

# For HIPs diagnostics
2 12 * * * /bin/date > /etc/hips
#2 12 * * * /usr/local/bin/perl_modules.pl > /etc/perl_modules_installed

# Look for Rodney's IP Address
#30 4 * * * grep 217.234.90.115 /var/log/httpd/*

# Batch copy script to clone Production Moodle into Moodler.
#0 3 9 8 * /var/www/moodler.morningside.edu/moodler/sync_files_and_db.sh

# Kickstarter script for 2.3 upgrade. 
# 1 0 10 8 * /root/kickstart_moodle2_upgrade.sh 

# SQL Dump in the middle of the night.
45 1 * * * mysqldump -p<dbpassword> -umoodle moodle > /var/www/moodle.morningside.edu/moodle-prod-backup.sql

Changelog

This is where I can keep a list of my changes to the Moodle system.

Note: Since I keep Moodle up-to-date with Git I can quickly print all local code changes with:

Hide non-visible courses from the "My Courses" block

   1 RCS file: /cvsroot/moodle/moodle/blocks/course_list/block_course_list.php,v
   2 retrieving revision 1.46.2.6
   3 diff -u -r1.46.2.6 block_course_list.php
   4 --- block_course_list.php       29 Aug 2008 04:23:38 -0000      1.46.2.6
   5 +++ block_course_list.php       25 Oct 2011 22:02:53 -0000
   6 @@ -40,6 +40,7 @@
   7              !isguest()) {    // Just print My Courses
   8              if ($courses = get_my_courses($USER->id, 'visible DESC, fullname ASC')) {
   9                  foreach ($courses as $course) {
  10 +                    if (!$course->visible) {continue;}
  11                      if ($course->id == SITEID) {
  12                          continue;
  13                      }

Course defaults to not enrollable

   1 --- course/edit_form.php.orig  2009-08-03 10:29:07.000000000 -0500
   2 +++ course/edit_form.php       2009-08-03 10:29:14.000000000 -0500
   3 @@ -217,7 +217,7 @@
   4          $radio[] = &MoodleQuickForm::createElement('radio', 'enrollable', null, get_string('enroldate'), 2);
   5          $mform->addGroup($radio, 'enrollable', get_string('enrollable'), ' ', false);
   6          $mform->setHelpButton('enrollable', array('courseenrollable2', get_string('enrollable')), true);
   7 -        $mform->setDefault('enrollable', 1);
   8 +        $mform->setDefault('enrollable', 0);
   9  
  10          $enroldatestartgrp = array();
  11          $enroldatestartgrp[] = &MoodleQuickForm::createElement('date_selector', 'enrolstartdate');

Course Settings - 'courseid' field too narrow

   1 --- course/edit_form.php.orig   2009-08-17 09:26:12.000000000 -0500
   2 +++ course/edit_form.php        2009-08-17 09:27:20.000000000 -0500
   3 @@ -90,7 +90,7 @@
   4              $mform->setConstant('shortname', $course->shortname);
   5          }
   6  
   7 -        $mform->addElement('text','idnumber', get_string('idnumbercourse'),'maxlength="100"  size="10"');
   8 +        $mform->addElement('text','idnumber', get_string('idnumbercourse'),'maxlength="100"  size="20"');
   9          $mform->setHelpButton('idnumber', array('courseidnumber', get_string('idnumbercourse')), true);
  10          $mform->setType('idnumber', PARAM_RAW);
  11          if ($course and !has_capability('moodle/course:changeidnumber', $coursecontext)) {

auth_ldap_sync_users.php failing silently (deprecated..? SRGM NOV 2011)

Cause:

Converting the moodle database to UNICODE apparently did not set the DEFAULT for tables in Moodle to the same collation so the temporary table created by auth_ldap_sync_users.php was not able to JOIN and compare the mdl_extuser with the mdl_user table.

Workaround:

   1 --- auth.php.orig       2009-07-06 13:51:30.000000000 -0500
   2 +++ auth.php    2009-08-21 13:10:01.000000000 -0500
   3 @@ -563,7 +563,9 @@
   4          switch (strtolower($CFG->dbfamily)) {
   5              case 'mysql':
   6                  $droptablesql[] = 'DROP TEMPORARY TABLE ' . $temptable; // sql command to drop the table (because session scope could be a problem)
   7 -                $createtemptablesql = 'CREATE TEMPORARY TABLE ' . $temptable . ' (username VARCHAR(64), PRIMARY KEY (username)) TYPE=MyISAM';
   8 +                // Changed from TYPE= to ENGINE= for this version of MySQL...
   9 +                $createtemptablesql = 'CREATE TEMPORARY TABLE ' . $temptable . ' (username VARCHAR(64), PRIMARY KEY (username)) ENGINE=MyISAM';
  10 +                $setutf8sql = 'alter table ' . $temptable . ' CONVERT TO CHARACTER SET utf8';
  11                  break;
  12              case 'postgres':
  13                  $droptablesql[] = 'DROP TABLE ' . $temptable; // sql command to drop the table (because session scope could be a problem)
  14 @@ -592,6 +594,9 @@
  15              exit;
  16          }
  17  
  18 +        // SHAUNS Workaround, set utf collation on our $temptable. 
  19 +        execute_sql($setutf8sql, false);
  20 +
  21          print "Connecting to ldap...\n";
  22          $ldapconnection = $this->ldap_connect();

Set Authenticated users to be able to post replies to news

Under Site Administration > Front Page > Front Page Roles > Override permissions for "Authenticated Users":

"Reply to News"

"Allow"

"Reply to posts"

"Allow"


CategoryServices CategoryMoodle

Moodle (last edited 2017-04-05 10:45:16 by meyersh)