User:Woozle/Mastodon/setup/take 3

Part 1

Variations from the instructions:

• Used Virtualmin to create virtual host (icmstdn.com) and system user (icmstdn) before starting.
• Cloned Mastodon from GitHub under /home/icmstn/apps/, with intent of creating a link to the new "mastodon" folder and naming it /home/icmstdn/public_html.

I know Mastodon is supposed to have a server all to itself, but this is just a test instance and shouldn't be dealing with much load... and even if I get it running and it turns out that merely federating is too much load, then at least I'll have learned two things: (1) yes, it really is too much and (2) I now know how to install Masto.

First problem arose when running this line:

apt -y install imagemagick ffmpeg libpq-dev libxml2-dev libxslt1-dev file git-core g++ libprotobuf-dev protobuf-compiler pkg-config nodejs gcc autoconf bison build-essential libssl-dev libyaml-dev libreadline6-dev zlib1g-dev libncurses5-dev libffi-dev libgdbm3 libgdbm-dev nginx redis-server redis-tools postgresql postgresql-contrib letsencrypt yarn libidn11-dev libicu-dev

Partial output:

Setting up nginx-common (1.10.3-0ubuntu0.16.04.2) ...
Setting up nginx-core (1.10.3-0ubuntu0.16.04.2) ...
Job for nginx.service failed because the control process exited with error code. See "systemctl status nginx.service" and "journalctl -xe" for details.
invoke-rc.d: initscript nginx, action "start" failed.
● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sun 2017-12-31 12:28:40 UTC; 12ms ago
  Process: 24202 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE)
  Process: 24198 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)

Dec 31 12:28:38 cloud4 nginx[24202]: nginx: [emerg] listen() to [::]:80, backlog 511 failed (98: Address already in use)
Dec 31 12:28:39 cloud4 nginx[24202]: nginx: [emerg] listen() to 0.0.0.0:80, backlog 511 failed (98: Address already in use)
Dec 31 12:28:39 cloud4 nginx[24202]: nginx: [emerg] listen() to [::]:80, backlog 511 failed (98: Address already in use)
Dec 31 12:28:39 cloud4 nginx[24202]: nginx: [emerg] listen() to 0.0.0.0:80, backlog 511 failed (98: Address already in use)
Dec 31 12:28:39 cloud4 nginx[24202]: nginx: [emerg] listen() to [::]:80, backlog 511 failed (98: Address already in use)
Dec 31 12:28:40 cloud4 nginx[24202]: nginx: [emerg] still could not bind()
Dec 31 12:28:40 cloud4 systemd[1]: nginx.service: Control process exited, code=exited status=1
Dec 31 12:28:40 cloud4 systemd[1]: Failed to start A high performance web server and a reverse proxy server.
Dec 31 12:28:40 cloud4 systemd[1]: nginx.service: Unit entered failed state.
Dec 31 12:28:40 cloud4 systemd[1]: nginx.service: Failed with result 'exit-code'.
dpkg: error processing package nginx-core (--configure):
 subprocess installed post-installation script returned error exit status 1
No apport report written because the error message indicates its a followup error from a previous failure.
                                                                                                          dpkg: dependency problems prevent configuration of nginx:
 nginx depends on nginx-core (>= 1.10.3-0ubuntu0.16.04.2) | nginx-full (>= 1.10.3-0ubuntu0.16.04.2) | nginx-light (>= 1.10.3-0ubuntu0.16.04.2) | nginx-extras (>= 1.10.3-0ubuntu0.16.04.2); however:
  Package nginx-core is not configured yet.
  Package nginx-full is not installed.
  Package nginx-light is not installed.
  Package nginx-extras is not installed.
 nginx depends on nginx-core (<< 1.10.3-0ubuntu0.16.04.2.1~) | nginx-full (<< 1.10.3-0ubuntu0.16.04.2.1~) | nginx-light (<< 1.10.3-0ubuntu0.16.04.2.1~) | nginx-extras (<< 1.10.3-0ubuntu0.16.04.2.1~); however:
  Package nginx-core is not configured yet.
  Package nginx-full is not installed.
  Package nginx-light is not installed.
  Package nginx-extras is not installed.

dpkg: error processing package nginx (--configure):
 dependency problems - leaving unconfigured
Setting up postgresql-contrib (9.5+173ubuntu0.1) ...
Setting up python-pyicu (1.9.2-2build1) ...
Setting up libjemalloc1 (3.6.0-9ubuntu1) ...
Setting up redis-tools (2:3.0.6-1) ...
Setting up redis-server (2:3.0.6-1) ...
Setting up i965-va-driver:amd64 (1.7.0-1) ...
Setting up va-driver-all:amd64 (1.7.0-1ubuntu0.1) ...
Setting up vdpau-driver-all:amd64 (1.1.1-3ubuntu1) ...
Setting up yarn (1.3.2-1) ...
Setting up libffi-dev:amd64 (3.2.1-4) ...
Setting up libprotobuf-dev:amd64 (2.6.1-1.3) ...
Setting up libyaml-dev:amd64 (0.1.6-3) ...
Setting up protobuf-compiler (2.6.1-1.3) ...
Processing triggers for libc-bin (2.23-0ubuntu9) ...
Processing triggers for libgdk-pixbuf2.0-0:amd64 (2.32.2-1ubuntu1.3) ...
Processing triggers for systemd (229-4ubuntu21) ...
Processing triggers for ureadahead (0.100.0-19) ...
Processing triggers for ufw (0.35-0ubuntu2) ...
Errors were encountered while processing:
 redmine
 nginx-core
 nginx
E: Sub-process /usr/bin/dpkg returned an error code (1)

I bet nginx doesn't like running on a system that's already running Apache. ("systemctl status nginx.service" just shows the above errors again.)

Forging on optimistically, let's see if everything else can be set up and I can coax Apache into running Ruby.

When it says to enter "type rbenv", I'm not sure what output it's supposed to produce -- but this looks reasonable:

icmstdn@cloud4:~/.rbenv$ type rbenv
rbenv is a function
rbenv () 
{ 
    local command;
    command="${1:-}";
    if [ "$#" -gt 0 ]; then
        shift;
    fi;
    case "$command" in 
        rehash | shell)
            eval "$(rbenv "sh-$command" "$@")"
        ;;
        *)
            command rbenv "$command" "$@"
        ;;
    esac
}
icmstdn@cloud4:~/.rbenv$ 

And now this seems to be taking rather a long time:

icmstdn@cloud4:~/.rbenv$ rbenv install 2.4.2
Downloading ruby-2.4.2.tar.bz2...
-> https://cache.ruby-lang.org/pub/ruby/2.4/ruby-2.4.2.tar.bz2
Installing ruby-2.4.2...

While waiting for that, I went ahead and opened up a new session to run the next command, "rbenv global 2.4.2". This returned immediately with no output. Dunno if that's good or bad.

Oh... and then the next line of instructions says "This will take some time. Go stretch for a bit and drink some water while the commands run."

And then I check on the first tab, and it's done:

Installed ruby-2.4.2 to /home/icmstdn/.rbenv/versions/2.4.2

icmstdn@cloud4:~/.rbenv$

Okay. (I re-ran "rbenv global 2.4.2" just to be sure; same result as before.)

Next it tells me to clone and install the Mastodon repository -- I'd already cloned it into ~/apps/mastodon rather than ~/live, so just following the rest of the instructions using that folder instead:

To execute the "git checkout" command, I had to give ownership of ~/apps/mastodon (and contents) to user icmstdn, since I had done the git clone operation as root.

Then:

• "bundle install --deployment --without development test" -- takes awhile on a couple of lines... and then we get this, all in dark red that's hard to read:

Installing cld3 3.2.1 with native extensions
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

    current directory: /home/icmstdn/apps/mastodon/vendor/bundle/ruby/2.4.0/gems/cld3-3.2.1/ext/cld3
/home/icmstdn/.rbenv/versions/2.4.2/bin/ruby -r ./siteconf20171231-5468-8sdlru.rb extconf.rb
creating Makefile

current directory: /home/icmstdn/apps/mastodon/vendor/bundle/ruby/2.4.0/gems/cld3-3.2.1/ext/cld3
make "DESTDIR=" clean

current directory: /home/icmstdn/apps/mastodon/vendor/bundle/ruby/2.4.0/gems/cld3-3.2.1/ext/cld3
make "DESTDIR="
compiling base.cc
cc1plus: warning: command line option ‘-Wimplicit-int’ is valid for C/ObjC but not for C++
cc1plus: warning: command line option ‘-Wdeclaration-after-statement’ is valid for C/ObjC but not for C++
cc1plus: warning: command line option ‘-Wimplicit-function-declaration’ is valid for C/ObjC but not for C++
cc1plus: warning: unrecognized command line option ‘-Wno-self-assign’
cc1plus: warning: unrecognized command line option ‘-Wno-constant-logical-operand’
cc1plus: warning: unrecognized command line option ‘-Wno-parentheses-equality’
cc1plus: warning: unrecognized command line option ‘-Wno-tautological-compare’
compiling embedding_feature_extractor.cc
cc1plus: warning: command line option ‘-Wimplicit-int’ is valid for C/ObjC but not for C++
cc1plus: warning: command line option ‘-Wdeclaration-after-statement’ is valid for C/ObjC but not for C++
cc1plus: warning: command line option ‘-Wimplicit-function-declaration’ is valid for C/ObjC but not for C++
virtual memory exhausted: Cannot allocate memory
Makefile:209: recipe for target 'embedding_feature_extractor.o' failed
make: *** [embedding_feature_extractor.o] Error 1

make failed, exit code 2

Gem files will remain installed in /home/icmstdn/apps/mastodon/vendor/bundle/ruby/2.4.0/gems/cld3-3.2.1 for inspection.
Results logged to /home/icmstdn/apps/mastodon/vendor/bundle/ruby/2.4.0/extensions/x86_64-linux/2.4.0-static/cld3-3.2.1/gem_make.out

An error occurred while installing cld3 (3.2.1), and Bundler cannot continue.
Make sure that `gem install cld3 -v '3.2.1'` succeeds before bundling.

In Gemfile:
  cld3
icmstdn@cloud4:~/apps/mastodon$

So... I'm thinking maybe I just need to crank up the RAM temporarily for the install. Gargron said this:

You need to have some swap because bundler, yarn (installation of dependencies) and the compilation of front-end assets takes a lot of RAM. However 512MB is enough of runtime memory. Please mind that the default settings are tuned higher than this (see tuning Mastodon documentation for details). 10-20 users is almost nothing in terms of load.

On a whim, I tried running it again, and this time it completed:

Bundle complete! 97 Gemfile dependencies, 163 gems now installed.
Gems in the groups development and test were not installed.
Bundled gems are installed into `./vendor/bundle`
Post-install message from paperclip:
##################################################
#  NOTE FOR UPGRADING FROM 4.3.0 OR EARLIER       #
##################################################

Paperclip is now compatible with aws-sdk >= 2.0.0.

If you are using S3 storage, aws-sdk >= 2.0.0 requires you to make a few small
changes:

* You must set the `s3_region`
* If you are explicitly setting permissions anywhere, such as in an initializer,
  note that the format of the permissions changed from using an underscore to
  using a hyphen. For example, `:public_read` needs to be changed to
  `public-read`.

For a walkthrough of upgrading from 4 to 5 and aws-sdk >= 2.0 you can watch
http://rubythursday.com/episodes/ruby-snack-27-upgrade-paperclip-and-aws-sdk-in-prep-for-rails-5
icmstdn@cloud4:~/apps/mastodon$

Next --

• "yarn install --pure-lockfile"... completed ok.
• postgres operations -- no problem

Now we come to the bit where we "configure nginx", and I get to see if I can translate that into Apache-speak...

Part 2

Thanks to KS for finding this set of official untested "alternative" config instructions. (For reference: mod_proxy)

• Configured Apache -- not exactly verbatim, because I'm modifying a file set up by Virtualmin.

Back to the canonical setup instructions:

• Created the 3 systemd config files
• "systemctl enable /etc/systemd/system/mastodon-*.service" - this apparently just creates symlinks?
• "systemctl start mastodon-*.service" - no output, seems ok
• "systemctl status mastodon-*.service" - no output, what does this mean?

Up next: Remote media attachment cache cleanup

...which I'll set up later, if I ever actually get this instance working.

The next task seems to be this: "Finally, to test this, spin up a Rails console (see the administration guide) [...]", but the administration guide doesn't actually explain how to "spin up a Rails console".

After some consultation, the following have come to light:

• you have to be in the mastodon folder when running this
• you have to be su as the mastodon user (icmstdn, in this case)

I got an error:

[!] There was an error parsing Gemfile: no implicit conversion of Symbol into Integer. Bundler cannot continue.

 #  from /home/icmstdn/apps/mastodon/Gemfile:4
 #  -------------------------------------------
 #  source 'https://rubygems.org/'
 >  ruby '>= 2.3.0', '< 2.5.0'
 #
 #  -------------------------------------------

An identical error was filed as a bug report back in June; running "gem install bundler" changed it to this:

/usr/lib/ruby/vendor_ruby/rails/app_rails_loader.rb:55:in `chdir': Permission denied @ dir_chdir - .. (Errno::EACCES)
	from /usr/lib/ruby/vendor_ruby/rails/app_rails_loader.rb:55:in `block in exec_app_rails'
	from /usr/lib/ruby/vendor_ruby/rails/app_rails_loader.rb:34:in `loop'
	from /usr/lib/ruby/vendor_ruby/rails/app_rails_loader.rb:34:in `exec_app_rails'
	from /usr/lib/ruby/vendor_ruby/rails/cli.rb:5:in `<top (required)>'
	from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
	from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
	from /usr/bin/rails:9:in `<main>'

Line 55 is the last line of this:

        # If we exhaust the search there is no executable, this could be a
        # call to generate a new application, so restore the original cwd.
        Dir.chdir(original_cwd) and return if Pathname.new(Dir.pwd).root?

        # Otherwise keep moving upwards in search of an executable.
        Dir.chdir('..')

After an intensive Amphetamine intervention, it turns out the correct command for invoking the Rails console is "bundle exec rails c", where "c" is short for "rails console". This seems to work.

The mailer test insists the SSL cert doesn't match the domain...

Part 2a

While trying to install Redmine, the vestiges of nginx were causing trouble, so I uninstalled them: "apt remove nginx-core nginx". No errors.

Part 3

At this point, I was getting a 500-something server configuration error. I was also getting this in /var/log/apache2/error.log:

[Sun Jan 07 13:02:05.199167 2018] [proxy:warn] [pid 1939] [client 50.111.109.24:11329] AH01144: No protocol handler was valid for the URL /. If you are using a DSO version of mod_proxy, make sure the proxy submodules are included in the configuration using LoadModule.
[Sun Jan 07 14:41:48.119496 2018] [authz_core:error] [pid 16984] [client 66.249.70.23:34410] AH01630: client denied by server configuration: /home/icmstdn/public_html/robots.txt
[Sun Jan 07 14:41:48.141139 2018] [authz_core:error] [pid 16984] [client 66.249.70.23:34410] AH01630: client denied by server configuration: /home/icmstdn/public_html/
[Sun Jan 07 15:31:59.666452 2018] [proxy:warn] [pid 23908] [client 98.101.39.22:39880] AH01144: No protocol handler was valid for the URL /. If you are using a DSO version of mod_proxy, make sure the proxy submodules are included in the configuration using LoadModule.

Clues on the internet suggested that this needed to be done:

a2enmod proxy_http (and then restart Apache)

Now I'm getting a "503 Service Unavailable" and the following in the log:

[Sun Jan 07 15:42:57.920911 2018] [proxy:error] [pid 24572] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:3000 (localhost) failed
[Sun Jan 07 15:42:57.920972 2018] [proxy:error] [pid 24572] AH00959: ap_proxy_connect_backend disabling worker for (localhost) for 60s
[Sun Jan 07 15:42:57.920986 2018] [proxy_http:error] [pid 24572] [client 98.101.39.22:40188] AH01114: HTTP: failed to make connection to backend: localhost

This sounds like maybe Passenger isn't running for this domain. Now I have to remember how to diagnose that...

Part 4

The back-end web server that Apache is trying to connect to is WEBrick; it's supposed to be serving WebSocket (ws://) on ports 3000 and 4000.

The configuration files for WEBrick are in /etc/systemd/system:

-rw-r--r--  1 root root  359 Dec 31 17:23 mastodon-sidekiq.service
-rw-r--r--  1 root root  292 Dec 31 17:23 mastodon-streaming.service
-rw-r--r--  1 root root  366 Dec 31 17:21 mastodon-web.service

This was done earlier, but included here for reference:

• systemctl enable /etc/systemd/system/mastodon-*.service

All three services can be started and stopped like this:

• systemctl start mastodon-*.service
• systemctl stop mastodon-*.service

You can check whether it's running or not with any of the following:

• systemctl status mastodon-*.service: combined status log for all three
• journalctl -x -u mastodon-web

...and it turns out it's not running:

● mastodon-sidekiq.service - mastodon-sidekiq
   Loaded: loaded (/etc/systemd/system/mastodon-sidekiq.service; enabled; vendor preset: enabled)
   Active: inactive (dead) (Result: exit-code) since Tue 2018-01-09 00:09:27 UTC; 46min ago
  Process: 4097 ExecStart=/home/mastodon/.rbenv/shims/bundle exec sidekiq -c 5 -q default -q mailers -q pull -q push (code=exited, status=217/USER)
 Main PID: 4097 (code=exited, status=217/USER)

Jan 09 00:09:27 cloud4 systemd[1]: mastodon-sidekiq.service: Service hold-off time over, scheduling restart.
Jan 09 00:09:27 cloud4 systemd[1]: Stopped mastodon-sidekiq.
Jan 09 00:09:27 cloud4 systemd[1]: mastodon-sidekiq.service: Start request repeated too quickly.
Jan 09 00:09:27 cloud4 systemd[1]: Failed to start mastodon-sidekiq.

● mastodon-streaming.service - mastodon-streaming
   Loaded: loaded (/etc/systemd/system/mastodon-streaming.service; enabled; vendor preset: enabled)
   Active: inactive (dead) (Result: exit-code) since Tue 2018-01-09 00:09:27 UTC; 46min ago
  Process: 4095 ExecStart=/usr/bin/npm run start (code=exited, status=217/USER)
 Main PID: 4095 (code=exited, status=217/USER)

Jan 09 00:09:27 cloud4 systemd[1]: mastodon-streaming.service: Service hold-off time over, scheduling restart.
Jan 09 00:09:27 cloud4 systemd[1]: Stopped mastodon-streaming.
Jan 09 00:09:27 cloud4 systemd[1]: mastodon-streaming.service: Start request repeated too quickly.
Jan 09 00:09:27 cloud4 systemd[1]: Failed to start mastodon-streaming.

● mastodon-web.service - mastodon-web
   Loaded: loaded (/etc/systemd/system/mastodon-web.service; enabled; vendor preset: enabled)
   Active: inactive (dead) (Result: exit-code) since Tue 2018-01-09 00:09:27 UTC; 46min ago
  Process: 4094 ExecStart=/home/mastodon/.rbenv/shims/bundle exec puma -C config/puma.rb (code=exited, status=217/USER)
 Main PID: 4094 (code=exited, status=217/USER)

Jan 09 00:09:27 cloud4 systemd[1]: mastodon-web.service: Service hold-off time over, scheduling restart.
Jan 09 00:09:27 cloud4 systemd[1]: Stopped mastodon-web.
Jan 09 00:09:27 cloud4 systemd[1]: mastodon-web.service: Start request repeated too quickly.
Jan 09 00:09:27 cloud4 systemd[1]: Failed to start mastodon-web.
lines 5-32/32 (END)