User:Woozle/Redmine/1

2017-12-27

This is my second attempt to migrate-and-upgrade a Redmine site. The first, I was never able to get working. This time I'm trying a different approach: installing from the repository, which currently (Ubuntu 16.04) has the oldest version of Redmine that is still officially available, 3.2.1-2.

"apt install redmine" seems to assume you want sqlite for the back end, in that the packages it installs include redmine-sqlite, dbconfig-sqlite3, ruby-sqlite3, and sqlite3.

I want MySQL because I know how to work with that (and it's already installed), so I'm going to try "apt install redmine-mysql" first (which installs dbconfig-mysql, redmine-mysql, and ruby-mysql2 but also ruby-sqlite3 and sqlite3 anyway, who knows why).

The upgrade instructions on the official Redmine site assume you're installing from source, so I'm winging this...

During the install process, I get a text-dialog box which says:

The redmine/instances/default package must have a database installed and configured before it can be used. This can be optionally handled with dbconfig-common.

If you are an advanced database administrator and know that you want to perform this configuration manually, or if your database has already been installed and configured, you should refuse this option. Details on what needs to be done should most likely be provided in /usr/share/doc/redmine/instances/default.

Otherwise, you should probably choose this option.

Configure database for redmine/instances/default with dbconfig-common?

Given that I have a database I want it to use, I replied "no".

At the end of the installation, it gave me this message:

Errors were encountered while processing:
 redmine
E: Sub-process /usr/bin/dpkg returned an error code (1)

...which I assume is the result of me saying "no", and not an actual error.

Apparently this also installs redmine itself, as "apt install redmine" comes back with nothing new to install.

No sources seemed to be forthcoming with information about where file uploads are stored (so they can be migrated to a new server), so I figured it out the hard way by looking in the "attachments" table for a filename, and then searching the hard drive for that file. Apparently they are stored in /var/lib/redmine/default/files.

(Now I need to go migrate the data.)

2018-01-04

part 1

• Data migrated to vbz-rm on cloud4.
• Created MySQL user, gave it ALL privileges on vbz-rm.
• Running "dbconfig-common"... is not a command. Maybe an option?
  • Apparently it is a command that I don't have installed. Do I want to use it?
  • I think the answer is "no", because it looks like what it does is help with the process of creating the db and assigning a user to it -- which I already did (that's the part I do know how to do).
• What I don't know how to do is configure Redmine to use the db and user I've created.
  • Let me see if there's a config file in the old instance...
  • Yes there is, in /etc/redmine/default. I copied over configuration.yml (unchanged) and database.yml, then altered database.yml to use the new/local credentials.
• Now going back to "apt -f install" to see what the errors (still) are...
  • "`/var/www` is not writable." -- it's writeable by root; who else needs to write it? Here's more of the block of text:

Setting up redmine (3.2.1-2) ...
dbconfig-common: writing config to /etc/dbconfig-common/redmine/instances/default.conf
error: template infile /usr/share/redmine/templates/database-.yml.template does not exist
dbconfig-common: flushing administrative password
`/var/www` is not writable.
Bundler will use `/tmp/bundler/home/root' as your home directory temporarily.
rake aborted!
Gem::LoadError: Specified 'mysql' for database adapter, but the gem is not loaded. Add `gem 'mysql'` to your Gemfile (and ensure its version is at the minimum required by ActiveRecord).

The Gemfile is found in /usr/share/redmine/Gemfile; I manually added "gem 'mysql'" to the file with a text editor. Now it says this:

Setting up redmine (3.2.1-2) ...
Could not find gem 'mysql' in any of the gem sources listed in your Gemfile.
dpkg: error processing package redmine (--configure):
 subprocess installed post-installation script returned error exit status 7

Extrapolating from the Mastodon install instructions, I'm going to try "gem install mysql" (why didn't the apt install routine do this?) Tentatively this was the right thing to do, or at least part of it, but it's not the whole thing:

root@cloud4:~/scripts# gem install mysql
Fetching: mysql-2.9.1.gem (100%)
Building native extensions.  This could take a while...
ERROR:  Error installing mysql:
	ERROR: Failed to build gem native extension.

    current directory: /var/lib/gems/2.3.0/gems/mysql-2.9.1/ext/mysql_api
/usr/bin/ruby2.3 -r ./siteconf20180104-983-1fztnkg.rb extconf.rb
mkmf.rb can't find header files for ruby at /usr/lib/ruby/include/ruby.h

extconf failed, exit code 1

Gem files will remain installed in /var/lib/gems/2.3.0/gems/mysql-2.9.1 for inspection.
Results logged to /var/lib/gems/2.3.0/extensions/x86_64-linux/2.3.0/mysql-2.9.1/gem_make.out
root@cloud4:~/scripts# 

Clues here suggest that this is what is needed: "apt install ruby-dev"

"gem install mysql" then got a little further, but failed:

root@cloud4:~/scripts# gem install mysql
Building native extensions.  This could take a while...
ERROR:  Error installing mysql:
	ERROR: Failed to build gem native extension.

    current directory: /var/lib/gems/2.3.0/gems/mysql-2.9.1/ext/mysql_api
/usr/bin/ruby2.3 -r ./siteconf20180104-1311-4wf87b.rb extconf.rb
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lm... yes
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lz... yes
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lsocket... no
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lnsl... yes
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lmygcc... no
checking for mysql_query() in -lmysqlclient... no
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Provided configuration options:
	--with-opt-dir
	--without-opt-dir
	--with-opt-include
	--without-opt-include=${opt-dir}/include
	--with-opt-lib
	--without-opt-lib=${opt-dir}/lib
	--with-make-prog
	--without-make-prog
	--srcdir=.
	--curdir
	--ruby=/usr/bin/$(RUBY_BASE_NAME)2.3
	--with-mysql-config
	--without-mysql-config
	--with-mysql-dir
	--without-mysql-dir
	--with-mysql-include
	--without-mysql-include=${mysql-dir}/include
	--with-mysql-lib
	--without-mysql-lib=${mysql-dir}/lib
	--with-mysqlclientlib
	--without-mysqlclientlib
	--with-mlib
	--without-mlib
	--with-mysqlclientlib
	--without-mysqlclientlib
	--with-zlib
	--without-zlib
	--with-mysqlclientlib
	--without-mysqlclientlib
	--with-socketlib
	--without-socketlib
	--with-mysqlclientlib
	--without-mysqlclientlib
	--with-nsllib
	--without-nsllib
	--with-mysqlclientlib
	--without-mysqlclientlib
	--with-mygcclib
	--without-mygcclib
	--with-mysqlclientlib
	--without-mysqlclientlib

To see why this extension failed to compile, please check the mkmf.log which can be found here:

  /var/lib/gems/2.3.0/extensions/x86_64-linux/2.3.0/mysql-2.9.1/mkmf.log

extconf failed, exit code 1

Gem files will remain installed in /var/lib/gems/2.3.0/gems/mysql-2.9.1 for inspection.
Results logged to /var/lib/gems/2.3.0/extensions/x86_64-linux/2.3.0/mysql-2.9.1/gem_make.out
root@cloud4:~/scripts# 

Clues here suggested that "apt install libmysqlclient-dev" might be the thing. Did that, after which "gem install mysql" was successful.

So now back to "apt -f install" -- I still get the warning message about /var/www not being writeable, but that doesn't seem to be a show-stopper. The only other errors I get have to do with nginx, which I'm pretty sure came out of the attempted Mastodon install -- so I'm going to try "apt remove nginx-core nginx" (completed successfully) and then "apt -f install" again...

...which seems to be happy now; nothing needed doing. (I did "apt autoremove" just so it would stop mentioning those unneeded files.)

So now where are we? I guess I need to point test.rm.vbz.net at cloud4, and tell Apache that this is an alias for Redmine, so I can test this deployment without making the old one inaccessible... or is there an upgrade process which needs to be run from the CLI? Will have to check on that.

part 2

Finding definitive upgrade instructions for Redmine has not been fruitful; so far, this is the only thing I've been able to find, and it's for an old version and doesn't explain a lot of things.

Based on that, though, I tried this:

root@cloud4:/usr/share/redmine# rake db:migrate RAILS_ENV=production

...but nothing happened, which could mean either success or failure or "nothing needed doing".

So then the next step was to muck around with Apache's config to point it to Redmine and make it go. There were a couple of Rails directives that Apache didn't recognize and which were causing the reload to fail with no error. Apparently "libapache2-mod-passenger" needs to be installed.

part 3

Installed that, restored the missing Rails directives, reloaded Apache... and now I get a Rails error page, which is progress. Following it through, I end up doing this:

root@cloud4:~# passenger-config --detect-apache2
Looking for possible Apache installations...
 --> Looking for /usr/local/sbin/apxs2: not found
 --> Looking for /usr/local/bin/apxs2: not found
 --> Looking for /usr/sbin/apxs2: not found
 --> Looking for /usr/bin/apxs2: not found
 --> Looking for /sbin/apxs2: not found
 --> Looking for /bin/apxs2: not found
 --> Looking for /usr/games/apxs2: not found
 --> Looking for /usr/local/games/apxs2: not found
 --> Looking for /snap/bin/apxs2: not found
 --> Looking for /usr/local/sbin/apxs: not found
 --> Looking for /usr/local/bin/apxs: not found
 --> Looking for /usr/sbin/apxs: not found
 --> Looking for /usr/bin/apxs: not found
 --> Looking for /sbin/apxs: not found
 --> Looking for /bin/apxs: not found
 --> Looking for /usr/games/apxs: not found
 --> Looking for /usr/local/games/apxs: not found
 --> Looking for /snap/bin/apxs: not found
 --> Looking for : found

Analyzing ...
Detecting main Apache executable...
Detecting version...
 --> 2.4.18
Detecting control command...
Cannot find a usable Apache installation using .

Final autodetection results
Sorry, this program cannot find an Apache installation.

Please install Apache and its development headers.

 * To install Apache:
   Please install it with apt-get install apache2-mpm-worker

 * To install Apache development headers:
   Please install it with apt-get install apache2-threaded-dev

If you are sure that you have Apache installed, please read the documentation:
https://www.phusionpassenger.com/library/install/apache/customizing_compilation_process.html#forcing-location-of-command-line-tools-and-dependencies
root@cloud4:~# apt install apache2-mpm-worker
Reading package lists... Done
Building dependency tree       
Reading state information... Done
E: Unable to locate package apache2-mpm-worker
root@cloud4:~# 

The instructions are misleading. apache2-mpm-worker is actually the "worker" module already installed with Apache; it just needs to be activated.

Following instructions here:

root@cloud4:~# a2dismod mpm_prefork
Module mpm_prefork disabled.
To activate the new configuration, you need to run:
  service apache2 restart
root@cloud4:~# a2enmod mpm_worker
Considering conflict mpm_event for mpm_worker:
Considering conflict mpm_prefork for mpm_worker:
Enabling module mpm_worker.
To activate the new configuration, you need to run:
  service apache2 restart

The instructions then say to run the following command to see if it's working -- but they didn't say to restart Apache first, so I didn't:

root@cloud4:~# apachectl -V | grep -i mpm
[Fri Jan 05 00:00:19.913519 2018] [so:warn] [pid 21156] AH01574: module dav_module is already loaded, skipping
[Fri Jan 05 00:00:19.952379 2018] [:crit] [pid 21156:tid 140326418311040] Apache is running a threaded MPM, but your PHP Module is not compiled to be threadsafe.  You need to recompile PHP.
AH00013: Pre-configuration failed

The instructions say if you get that message, then do this:

root@cloud4:~# apt remove libapache2-mod-php5
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Package 'libapache2-mod-php5' is not installed, so not removed
The following package was automatically installed and is no longer required:
  libllvm4.0
Use 'apt autoremove' to remove it.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Well, okay, seems that package is already removed. Maybe I just needed to go ahead and restart Apache?

root@cloud4:~# service apache2 restart
Job for apache2.service failed because the control process exited with error code. See "systemctl status apache2.service" and "journalctl -xe" for details.

So, we're back there again. Okay, time to revert and consider other options.

root@cloud4:~# a2dismod mpm_worker
Module mpm_worker disabled.
To activate the new configuration, you need to run:
  service apache2 restart
root@cloud4:~# a2enmod mpm_prefork
Considering conflict mpm_event for mpm_prefork:
Considering conflict mpm_worker for mpm_prefork:
Enabling module mpm_prefork.
To activate the new configuration, you need to run:
  service apache2 restart
root@cloud4:~# service apapart 4che2 restart
root@cloud4:~# 

At least we're back to Apache being happy. (Yes, I keep testing other sites on the same server to make sure they are still working.)

2018-01-06

part 1

Somewhere along the line, I enabled an option to show the error message on the web page. The key part of the error message is this:

[!] There was an error parsing `Gemfile`: Permission denied @ rb_sysopen - config/database.yml. Bundler cannot continue.

Redmine is running (I think) as user vbz-net, which had no other group memberships, so I did usermod -a -G www-data vbz-net to add vbz-net to the www-data group.

This produced no apparent change in the web output.

The full error message seemed to indicate that Passenger is running as "nobody", which should never have special access to anything -- but database.yml inslcude the database password, and access to it should be carefully restricted.

I decided to research Passenger setup in order to figure out how this is supposed to work, and found this document which suggested that I use apt to install dirmngr and gnupg. I installed dirmngr, but gnupg was already installed/current.

At this point it occurred to me to check the hypothesis that file permissions are actually the problem, so I temporarily chmodded database.yml to 666 -- and got a different error message:

Permission denied @ rb_sysopen - /usr/share/redmine/instances/default/config/secret_key.txt (Errno::EACCES)

...so I think we're on the right track, here. (I then changed perms back to 640.)

Per the Passenger instructions, I then did:

apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 561F9B9CAC40B2F7 -- seems fine

apt install -y apt-transport-https ca-certificates -- both already the newest version

sudo sh -c 'echo deb https://oss-binaries.phusionpassenger.com/apt/passenger xenial main > /etc/apt/sources.list.d/passenger.list' -- I checked to make sure it wasn't already there, first (it wasn't)

apt update -- "2 packages can be upgraded. Run 'apt list --upgradable' to see them."

apt upgrade -- because I like to do things my way...

The following NEW packages will be installed:
  passenger-dev passenger-doc
The following packages will be upgraded:
  libapache2-mod-passenger passenger

Well, that takes care of the next instruction, which is to install "libapache2-mod-passenger".

I'm still getting this damn warning message, though:

`/var/www` is not writable.
Bundler will use `/tmp/bundler/home/root' as your home directory temporarily.

I'm running this as root, so obviously this message is coming from something that is not running as root and yet expects to have write access to /var/www -- so it would be really nice if it would tell me what user it is running as.

...but it doesn't seem to be a fatal error.

Next, the instructions want these:

sudo a2enmod passenger -- guess that might be necessary, huh (but what's the difference between this and mpm_worker, enabled earlier?)

Actually, not: Apache says it's already enabled.

sudo apache2ctl restart -- I usually do service apache2 restart, but I'll do it their way this time just to see...

Just in case something changed during all this, I went ahead and restarted Apache anyway.

So, okay, apache2ctl restart gives a message that service apache2 restart does not:

[Sat Jan 06 14:08:53.037559 2018] [so:warn] [pid 23132] AH01574: module dav_module is already loaded, skipping
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message

I put "ServerName cloud4.hypertwins.net" at the top of 000-default.conf to get rid of the 2nd message. The first one seems less urgent.

Error remains unchanged, but this is not surprising. Next we do this:

/usr/bin/passenger-config validate-install

I selected both. Apparently Passenger is fine, but I need to install apxs2 in order to validate Apache, and apparently this is in the apache2-dev package. So:

apt install apache2-dev

This results in:

The following additional packages will be installed:
  libapr1-dev libaprutil1-dev libldap2-dev libsctp-dev libsctp1 uuid-dev
Suggested packages:
  lksctp-tools
The following NEW packages will be installed:
  apache2-dev libapr1-dev libaprutil1-dev libldap2-dev libsctp-dev libsctp1 uuid-dev

...and now back to passenger-config validate-install, which now returns (among other things) "Everything looks good. :-)". (No change on the error page.)

/usr/sbin/passenger-memory-stats -- I see a handful of Apache processes, 0 nginx processes (I should certainly hope so, since I uninstalled nginx earlier), and one process each for "Passenger watchdog" and "Passenger core".

That seems to be all for actually installing Passenger, so maybe the problem is with the (Redmine) app configuration...

part 2 (success)

It turns out the key piece was to add the following to the virtual domain config in Apache:

PassengerUser vbz-net

The reference page for this is here, with some additional explanation here.

After making that change and doing service apache2 reload, Redmine gives me a log-in screen. I log in and everything seems to be working (and it's version 3.2.1.stable now).

part 3 (tidying up)

A couple of folders weren't write-accessible by Redmine, so I changed ownership to vbz-net:

chown -R vbz-net:vbz-net /var/lib/redmine/default

chown -R vbz-net:vbz-net /usr/share/redmine/public/plugin_assets/

Probably both of these folders -- /var/lib/redmine/default and /usr/share/redmine/public/ -- should be inside /home/vbz-net/ or even /home/vbz-net/domains/rm.vbz.net. I'll fix that when I have time to re-research how to modify Redmine's config so it knows to look for them in the appropriate spots.