User:Woozle/Mastodon/setup/take 3

Part 1

Variations from the instructions:

• Used Virtualmin to create virtual host (icmstdn.com) and system user (icmstdn) before starting.
• Cloned Mastodon from GitHub under /home/icmstn/apps/, with intent of creating a link to the new "mastodon" folder and naming it /home/icmstdn/public_html.

I know Mastodon is supposed to have a server all to itself, but this is just a test instance and shouldn't be dealing with much load... and even if I get it running and it turns out that merely federating is too much load, then at least I'll have learned two things: (1) yes, it really is too much and (2) I now know how to install Masto.

First problem arose when running this line:

apt -y install imagemagick ffmpeg libpq-dev libxml2-dev libxslt1-dev file git-core g++ libprotobuf-dev protobuf-compiler pkg-config nodejs gcc autoconf bison build-essential libssl-dev libyaml-dev libreadline6-dev zlib1g-dev libncurses5-dev libffi-dev libgdbm3 libgdbm-dev nginx redis-server redis-tools postgresql postgresql-contrib letsencrypt yarn libidn11-dev libicu-dev

Partial output:

Setting up nginx-common (1.10.3-0ubuntu0.16.04.2) ...
Setting up nginx-core (1.10.3-0ubuntu0.16.04.2) ...
Job for nginx.service failed because the control process exited with error code. See "systemctl status nginx.service" and "journalctl -xe" for details.
invoke-rc.d: initscript nginx, action "start" failed.
● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sun 2017-12-31 12:28:40 UTC; 12ms ago
  Process: 24202 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE)
  Process: 24198 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)

Dec 31 12:28:38 cloud4 nginx[24202]: nginx: [emerg] listen() to [::]:80, backlog 511 failed (98: Address already in use)
Dec 31 12:28:39 cloud4 nginx[24202]: nginx: [emerg] listen() to 0.0.0.0:80, backlog 511 failed (98: Address already in use)
Dec 31 12:28:39 cloud4 nginx[24202]: nginx: [emerg] listen() to [::]:80, backlog 511 failed (98: Address already in use)
Dec 31 12:28:39 cloud4 nginx[24202]: nginx: [emerg] listen() to 0.0.0.0:80, backlog 511 failed (98: Address already in use)
Dec 31 12:28:39 cloud4 nginx[24202]: nginx: [emerg] listen() to [::]:80, backlog 511 failed (98: Address already in use)
Dec 31 12:28:40 cloud4 nginx[24202]: nginx: [emerg] still could not bind()
Dec 31 12:28:40 cloud4 systemd[1]: nginx.service: Control process exited, code=exited status=1
Dec 31 12:28:40 cloud4 systemd[1]: Failed to start A high performance web server and a reverse proxy server.
Dec 31 12:28:40 cloud4 systemd[1]: nginx.service: Unit entered failed state.
Dec 31 12:28:40 cloud4 systemd[1]: nginx.service: Failed with result 'exit-code'.
dpkg: error processing package nginx-core (--configure):
 subprocess installed post-installation script returned error exit status 1
No apport report written because the error message indicates its a followup error from a previous failure.
                                                                                                          dpkg: dependency problems prevent configuration of nginx:
 nginx depends on nginx-core (>= 1.10.3-0ubuntu0.16.04.2) | nginx-full (>= 1.10.3-0ubuntu0.16.04.2) | nginx-light (>= 1.10.3-0ubuntu0.16.04.2) | nginx-extras (>= 1.10.3-0ubuntu0.16.04.2); however:
  Package nginx-core is not configured yet.
  Package nginx-full is not installed.
  Package nginx-light is not installed.
  Package nginx-extras is not installed.
 nginx depends on nginx-core (<< 1.10.3-0ubuntu0.16.04.2.1~) | nginx-full (<< 1.10.3-0ubuntu0.16.04.2.1~) | nginx-light (<< 1.10.3-0ubuntu0.16.04.2.1~) | nginx-extras (<< 1.10.3-0ubuntu0.16.04.2.1~); however:
  Package nginx-core is not configured yet.
  Package nginx-full is not installed.
  Package nginx-light is not installed.
  Package nginx-extras is not installed.

dpkg: error processing package nginx (--configure):
 dependency problems - leaving unconfigured
Setting up postgresql-contrib (9.5+173ubuntu0.1) ...
Setting up python-pyicu (1.9.2-2build1) ...
Setting up libjemalloc1 (3.6.0-9ubuntu1) ...
Setting up redis-tools (2:3.0.6-1) ...
Setting up redis-server (2:3.0.6-1) ...
Setting up i965-va-driver:amd64 (1.7.0-1) ...
Setting up va-driver-all:amd64 (1.7.0-1ubuntu0.1) ...
Setting up vdpau-driver-all:amd64 (1.1.1-3ubuntu1) ...
Setting up yarn (1.3.2-1) ...
Setting up libffi-dev:amd64 (3.2.1-4) ...
Setting up libprotobuf-dev:amd64 (2.6.1-1.3) ...
Setting up libyaml-dev:amd64 (0.1.6-3) ...
Setting up protobuf-compiler (2.6.1-1.3) ...
Processing triggers for libc-bin (2.23-0ubuntu9) ...
Processing triggers for libgdk-pixbuf2.0-0:amd64 (2.32.2-1ubuntu1.3) ...
Processing triggers for systemd (229-4ubuntu21) ...
Processing triggers for ureadahead (0.100.0-19) ...
Processing triggers for ufw (0.35-0ubuntu2) ...
Errors were encountered while processing:
 redmine
 nginx-core
 nginx
E: Sub-process /usr/bin/dpkg returned an error code (1)

I bet nginx doesn't like running on a system that's already running Apache. ("systemctl status nginx.service" just shows the above errors again.)

Forging on optimistically, let's see if everything else can be set up and I can coax Apache into running Ruby.

When it says to enter "type rbenv", I'm not sure what output it's supposed to produce -- but this looks reasonable:

icmstdn@cloud4:~/.rbenv$ type rbenv
rbenv is a function
rbenv () 
{ 
    local command;
    command="${1:-}";
    if [ "$#" -gt 0 ]; then
        shift;
    fi;
    case "$command" in 
        rehash | shell)
            eval "$(rbenv "sh-$command" "$@")"
        ;;
        *)
            command rbenv "$command" "$@"
        ;;
    esac
}
icmstdn@cloud4:~/.rbenv$ 

And now this seems to be taking rather a long time:

icmstdn@cloud4:~/.rbenv$ rbenv install 2.4.2
Downloading ruby-2.4.2.tar.bz2...
-> https://cache.ruby-lang.org/pub/ruby/2.4/ruby-2.4.2.tar.bz2
Installing ruby-2.4.2...

While waiting for that, I went ahead and opened up a new session to run the next command, "rbenv global 2.4.2". This returned immediately with no output. Dunno if that's good or bad.

Oh... and then the next line of instructions says "This will take some time. Go stretch for a bit and drink some water while the commands run."

And then I check on the first tab, and it's done:

Installed ruby-2.4.2 to /home/icmstdn/.rbenv/versions/2.4.2

icmstdn@cloud4:~/.rbenv$

Okay. (I re-ran "rbenv global 2.4.2" just to be sure; same result as before.)

Next it tells me to clone and install the Mastodon repository -- I'd already cloned it into ~/apps/mastodon rather than ~/live, so just following the rest of the instructions using that folder instead:

To execute the "git checkout" command, I had to give ownership of ~/apps/mastodon (and contents) to user icmstdn, since I had done the git clone operation as root.

Then:

• "bundle install --deployment --without development test" -- takes awhile on a couple of lines... and then we get this, all in dark red that's hard to read:

Installing cld3 3.2.1 with native extensions
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

    current directory: /home/icmstdn/apps/mastodon/vendor/bundle/ruby/2.4.0/gems/cld3-3.2.1/ext/cld3
/home/icmstdn/.rbenv/versions/2.4.2/bin/ruby -r ./siteconf20171231-5468-8sdlru.rb extconf.rb
creating Makefile

current directory: /home/icmstdn/apps/mastodon/vendor/bundle/ruby/2.4.0/gems/cld3-3.2.1/ext/cld3
make "DESTDIR=" clean

current directory: /home/icmstdn/apps/mastodon/vendor/bundle/ruby/2.4.0/gems/cld3-3.2.1/ext/cld3
make "DESTDIR="
compiling base.cc
cc1plus: warning: command line option ‘-Wimplicit-int’ is valid for C/ObjC but not for C++
cc1plus: warning: command line option ‘-Wdeclaration-after-statement’ is valid for C/ObjC but not for C++
cc1plus: warning: command line option ‘-Wimplicit-function-declaration’ is valid for C/ObjC but not for C++
cc1plus: warning: unrecognized command line option ‘-Wno-self-assign’
cc1plus: warning: unrecognized command line option ‘-Wno-constant-logical-operand’
cc1plus: warning: unrecognized command line option ‘-Wno-parentheses-equality’
cc1plus: warning: unrecognized command line option ‘-Wno-tautological-compare’
compiling embedding_feature_extractor.cc
cc1plus: warning: command line option ‘-Wimplicit-int’ is valid for C/ObjC but not for C++
cc1plus: warning: command line option ‘-Wdeclaration-after-statement’ is valid for C/ObjC but not for C++
cc1plus: warning: command line option ‘-Wimplicit-function-declaration’ is valid for C/ObjC but not for C++
virtual memory exhausted: Cannot allocate memory
Makefile:209: recipe for target 'embedding_feature_extractor.o' failed
make: *** [embedding_feature_extractor.o] Error 1

make failed, exit code 2

Gem files will remain installed in /home/icmstdn/apps/mastodon/vendor/bundle/ruby/2.4.0/gems/cld3-3.2.1 for inspection.
Results logged to /home/icmstdn/apps/mastodon/vendor/bundle/ruby/2.4.0/extensions/x86_64-linux/2.4.0-static/cld3-3.2.1/gem_make.out

An error occurred while installing cld3 (3.2.1), and Bundler cannot continue.
Make sure that `gem install cld3 -v '3.2.1'` succeeds before bundling.

In Gemfile:
  cld3
icmstdn@cloud4:~/apps/mastodon$

So... I'm thinking maybe I just need to crank up the RAM temporarily for the install. Gargron said this:

You need to have some swap because bundler, yarn (installation of dependencies) and the compilation of front-end assets takes a lot of RAM. However 512MB is enough of runtime memory. Please mind that the default settings are tuned higher than this (see tuning Mastodon documentation for details). 10-20 users is almost nothing in terms of load.

On a whim, I tried running it again, and this time it completed:

Bundle complete! 97 Gemfile dependencies, 163 gems now installed.
Gems in the groups development and test were not installed.
Bundled gems are installed into `./vendor/bundle`
Post-install message from paperclip:
##################################################
#  NOTE FOR UPGRADING FROM 4.3.0 OR EARLIER       #
##################################################

Paperclip is now compatible with aws-sdk >= 2.0.0.

If you are using S3 storage, aws-sdk >= 2.0.0 requires you to make a few small
changes:

* You must set the `s3_region`
* If you are explicitly setting permissions anywhere, such as in an initializer,
  note that the format of the permissions changed from using an underscore to
  using a hyphen. For example, `:public_read` needs to be changed to
  `public-read`.

For a walkthrough of upgrading from 4 to 5 and aws-sdk >= 2.0 you can watch
http://rubythursday.com/episodes/ruby-snack-27-upgrade-paperclip-and-aws-sdk-in-prep-for-rails-5
icmstdn@cloud4:~/apps/mastodon$

Next --

• "yarn install --pure-lockfile"... completed ok.
• postgres operations -- no problem

Now we come to the bit where we "configure nginx", and I get to see if I can translate that into Apache-speak...

Part 2

Thanks to KS for finding this set of official untested "alternative" config instructions. (For reference: mod_proxy)

• Configured Apache -- not exactly verbatim, because I'm modifying a file set up by Virtualmin.

Back to the canonical setup instructions:

• Created the 3 systemd config files
• "systemctl enable /etc/systemd/system/mastodon-*.service" - this apparently just creates symlinks?
• "systemctl start mastodon-*.service" - no output, seems ok
• "systemctl status mastodon-*.service" - no output, what does this mean?

Up next: Remote media attachment cache cleanup

...which I'll set up later, if I ever actually get this instance working.

The next task seems to be this: "Finally, to test this, spin up a Rails console (see the administration guide) [...]", but the administration guide doesn't actually explain how to "spin up a Rails console".

After some consultation, the following have come to light:

• you have to be in the mastodon folder when running this
• you have to be su as the mastodon user (icmstdn, in this case)

I got an error:

[!] There was an error parsing Gemfile: no implicit conversion of Symbol into Integer. Bundler cannot continue.

 #  from /home/icmstdn/apps/mastodon/Gemfile:4
 #  -------------------------------------------
 #  source 'https://rubygems.org/'
 >  ruby '>= 2.3.0', '< 2.5.0'
 #
 #  -------------------------------------------

An identical error was filed as a bug report back in June; running "gem install bundler" changed it to this:

/usr/lib/ruby/vendor_ruby/rails/app_rails_loader.rb:55:in `chdir': Permission denied @ dir_chdir - .. (Errno::EACCES)
	from /usr/lib/ruby/vendor_ruby/rails/app_rails_loader.rb:55:in `block in exec_app_rails'
	from /usr/lib/ruby/vendor_ruby/rails/app_rails_loader.rb:34:in `loop'
	from /usr/lib/ruby/vendor_ruby/rails/app_rails_loader.rb:34:in `exec_app_rails'
	from /usr/lib/ruby/vendor_ruby/rails/cli.rb:5:in `<top (required)>'
	from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
	from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
	from /usr/bin/rails:9:in `<main>'

Line 55 is the last line of this:

        # If we exhaust the search there is no executable, this could be a
        # call to generate a new application, so restore the original cwd.
        Dir.chdir(original_cwd) and return if Pathname.new(Dir.pwd).root?

        # Otherwise keep moving upwards in search of an executable.
        Dir.chdir('..')

After an intensive Amphetamine intervention, it turns out the correct command for invoking the Rails console is "bundle exec rails c", where "c" is short for "rails console". This seems to work.

The mailer test insists the SSL cert doesn't match the domain...

Part 2a

While trying to install Redmine, the vestiges of nginx were causing trouble, so I uninstalled them: "apt remove nginx-core nginx". No errors.

Part 3

At this point, I was getting a 500-something server configuration error. I was also getting this in /var/log/apache2/error.log:

[Sun Jan 07 13:02:05.199167 2018] [proxy:warn] [pid 1939] [client 50.111.109.24:11329] AH01144: No protocol handler was valid for the URL /. If you are using a DSO version of mod_proxy, make sure the proxy submodules are included in the configuration using LoadModule.
[Sun Jan 07 14:41:48.119496 2018] [authz_core:error] [pid 16984] [client 66.249.70.23:34410] AH01630: client denied by server configuration: /home/icmstdn/public_html/robots.txt
[Sun Jan 07 14:41:48.141139 2018] [authz_core:error] [pid 16984] [client 66.249.70.23:34410] AH01630: client denied by server configuration: /home/icmstdn/public_html/
[Sun Jan 07 15:31:59.666452 2018] [proxy:warn] [pid 23908] [client 98.101.39.22:39880] AH01144: No protocol handler was valid for the URL /. If you are using a DSO version of mod_proxy, make sure the proxy submodules are included in the configuration using LoadModule.

Clues on the internet suggested that this needed to be done:

a2enmod proxy_http (and then restart Apache)

Now I'm getting a "503 Service Unavailable" and the following in the log:

[Sun Jan 07 15:42:57.920911 2018] [proxy:error] [pid 24572] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:3000 (localhost) failed
[Sun Jan 07 15:42:57.920972 2018] [proxy:error] [pid 24572] AH00959: ap_proxy_connect_backend disabling worker for (localhost) for 60s
[Sun Jan 07 15:42:57.920986 2018] [proxy_http:error] [pid 24572] [client 98.101.39.22:40188] AH01114: HTTP: failed to make connection to backend: localhost

This sounds like maybe Passenger isn't running for this domain. Now I have to remember how to diagnose that...

Part 4

The back-end web server that Apache is trying to connect to is WEBrick; it's supposed to be serving WebSocket (ws://) on ports 3000 and 4000.

The configuration files for WEBrick are in /etc/systemd/system:

-rw-r--r--  1 root root  359 Dec 31 17:23 mastodon-sidekiq.service
-rw-r--r--  1 root root  292 Dec 31 17:23 mastodon-streaming.service
-rw-r--r--  1 root root  366 Dec 31 17:21 mastodon-web.service

This was done earlier, but included here for reference:

• systemctl enable /etc/systemd/system/mastodon-*.service

All three services can be started and stopped like this:

• systemctl start mastodon-*.service
• systemctl stop mastodon-*.service

You can check whether it's running or not with any of the following:

• systemctl status mastodon-*.service: combined status log for all three
• journalctl --unit=mastodon-*: another combined journal
• journalctl -x -u mastodon-web

...and it turns out it's not running - full paste here.

Clues here suggested that maybe sidekiq needed to be installed from apt -- this package apparently had not been installed:

apt install ruby-sidekiq -- resulted in additional package installations: libjs-jquery-metadata libjs-jquery-tablesorter libjs-twitter-bootstrap ruby-celluloid ruby-concurrent ruby-connection-pool ruby-hitimes ruby-redis ruby-redis-namespace ruby-timers

...but then another comment on that site suggests that maybe installing them that way is a bad idea. So maybe I should uninstall them.

Searching the web for "status=217/USER" pulled up this page, which suggested that the problem might be that my Mastodon username has a dash in it (which it does). This caused me to take a look at those three systemd config files to see if it was easy to change that -- and I found that they all assumed the username was "mastodon", and assume that everything is in /home/mastodon. Maybe this is a problem? Changing it to icmstdn (original contents: /paste-2).

There are multiple things which need changing in each file, actually:

User=icmstdn - all three

WorkingDirectory=/home/icmstdn/apps/mastodon - all three

ExecStart=/home/icmstdn/.rbenv/shims/bundle exec sidekiq -c 5 -q default -q mailers -q pull -q push - sidekiq config

ExecStart=/home/icmstdn/.rbenv/shims/bundle exec puma -C config/puma.rb - web config

No joy after doing that, though -- some dumps:

• journalctl --unit-mastodon-*: /paste-3
• journalctl -x -u mastodon-web: /paste-4 - why does it try to start the service more than once?

Running the startup command directly gave some more detailed errors that may be helpful:

• root@cloud4:/home/icmstdn/apps/mastodon# /home/icmstdn/.rbenv/shims/bundle exec puma -C config/puma.rb: /paste-5

That led me to install the Puma gem, and now we have /paste-6 (tried lots of things, pasted lots of output).

Part 5

The problem seems to be that Puma isn't starting. The problem with figuring out why is that running it "properly" seems to hide the error messages, and running the command specified by ExecStart directly seems to be running it in a different environment -- i.e. "development" -- with its own set of issues that would need to be resolved but which shouldn't need to be resolved for production purposes.

I finally hit how to get the standalone command to be in production mode: /paste-7

...and the upshot of that seemed to be that I was running out of RAM, so I temporarily upped the RAM allocation for cloud4 to 4GB and tried again: /paste-8

Part 6

...and then after looking at the PostgreSQL log, I determined that Masto was trying to use the wrong user (should be icmstdn, not mastodon) and the mastodon db had not yet been created...

No time to log everything, but important -- execute this as user icmstdn from /home/icmstdn/apps/mastodon:

export SAFETY_ASSURED=1

RAILS_ENV=production bundle exec rails db:setup

Next: restart mastodon from systemd yet again, I guess.

systemctl restart mastodon-*.service

No change.

Part 7: working!

The fact that I was getting 500 errors from Apache but no "500 error" web page led me to wonder: why isn't the "500.html" page being served?

It turns out that $mastodon/public/500.html is a link to $mastodon/public/assets/500.html -- but there is no $mastodon/public/assets folder.

Apparently this folder and its contents are generated by Mastodon on first run, and the way to force this to happen if it didn't happen earlier on is:

• make sure you're $mastodon_user
• cd to $mastodon
• RAILS_ENV=production bundle exec rails assets:precompile
  • results: /paste-9

After this, I now actually get a Mastodon-generated error page: "We're sorry, but something went wrong on our end."

I restart the mastodon processes with systemctl, and now I get a Mastodon login page (but without CSS)!

The problem there turned out to be that Apache was using $mastodon as its home folder, when it should have been using $mastodon/public. Upon changing that, I get a proper log-in screen.