@c -*- coding: utf-8; mode: texinfo; -*-
@node Website work
@chapter Website work

@menu
* Introduction to website work::
* Uploading and security::
* Debugging website and docs locally::
* Translating the website::
@end menu


@node Introduction to website work
@section Introduction to website work

The website is @emph{not} written directly in HTML;
instead, the source is Texinfo, which is then generated into HTML,
PDF, and Info formats.  The sources are

@example
Documentation/web.texi
Documentation/web/*.texi
@end example

Unless otherwise specified, follow the instructions and policies
given in @ref{Documentation work}.  That chapter also contains a
quick introduction to Texinfo; consulting an external Texinfo
manual should be not necessary.

@subheading Exceptions to the documentation policies

@itemize

@item
Sectioning: the website only uses chapters and sections; no
subsections or subsubsections.

@item
@@ref@{@}s to other manuals (@@ruser, @@rlearning, etc): you can't
link to any pieces of automatically generated documentation, like
the IR or certain NR appendices.

@item
The bibliography in Community->Publications is generated automatically
from @file{.bib} files; formatting is done automatically by
@file{texi-web.bst}.

@item
@dots{}

@item
For anything not listed here, just follow the same style as the
existing website texinfo files.

@end itemize


@node Uploading and security
@section Uploading and security

@subheading Overall idea

To reduce the CPU burden on the shared host (as well as some
security concerns), we do not compile all of LilyPond.  The
website build process runs @code{texi2html}, but all media files
(be they graphical @code{lilypond} output, photos of people, or
pdfs) are copied from the @code{$LILYPOND_WEB_MEDIA_GIT}
repository.

All scripts and makefiles used for the website build are run from
a @qq{trusted} copy.  Any modification to those files in git needs
a human to review the changes (after they have been made in git)
before they are used on the server.


@subheading Building the website (quick local)

Initial setup: make sure that you have the environment variables
@code{$LILYPOND_GIT}, @code{$LILYPOND_BUILD_DIR} and
@code{$LILYPOND_WEB_MEDIA_GIT} set up correctly.  For more
information, see @ref{Environment variables}.

Once that is done,

@example
cd $LILYPOND_BUILD_DIR
make website
@end example

The website is in @file{out-website/website/index.html}.


@subheading Building the website (exactly as on the server)

@subsubheading Setting up (exactly as on the server)

Initial setup: you still need @code{$LILYPOND_GIT} and
@code{$LILYPOND_WEB_MEDIA_GIT}.

Once that's done, create:

@example
mkdir -p $HOME/lilypond/
mkdir -p $HOME/lilypond/bin/
mkdir -p $HOME/lilypond/cron/
mkdir -p $HOME/lilypond/trusted-scripts/
@end example

The add these files to @file{$HOME/lilypond/bin/}:

Update git repositories:

@smallexample
@verbatim
### update-git.sh
#!/bin/sh
cd $LILYPOND_GIT
git fetch origin
git merge origin/master
cd $LILYPOND_WEB_MEDIA_GIT
git fetch origin
git merge origin/master
@end verbatim
@end smallexample

Check for any updates to trusted scripts / files:

@smallexample
@verbatim
### check-git.sh
#!/bin/sh
DEST=$HOME/lilypond/trusted-scripts
diff -u $DEST/website.make \
  $LILYPOND_GIT/make/website.make
diff -u $DEST/lilypond-texi2html.init \
  $LILYPOND_GIT/Documentation/lilypond-texi2html.init
diff -u $DEST/extract_texi_filenames.py \
  $LILYPOND_GIT/scripts/build/extract_texi_filenames.py
diff -u $DEST/create-version-itexi.py \
  $LILYPOND_GIT/scripts/build/create-version-itexi.py
diff -u $DEST/create-weblinks-itexi.py \
  $LILYPOND_GIT/scripts/build/create-weblinks-itexi.py
diff -u $DEST/mass-link.py \
  $LILYPOND_GIT/scripts/build/mass-link.py
diff -u $DEST/website_post.py \
  $LILYPOND_GIT/scripts/build/website_post.py
diff -u $DEST/bib2texi.py \
  $LILYPOND_GIT/scripts/build/bib2texi.py
diff -u $DEST/langdefs.py \
  $LILYPOND_GIT/python/langdefs.py
diff -u $DEST/lilypond.org.htaccess \
  $LILYPOND_GIT/Documentation/web/server/lilypond.org.htaccess
diff -u $DEST/website-dir.htaccess \
  $LILYPOND_GIT/Documentation/web/server/website-dir.htaccess
@end verbatim
@end smallexample

If the changes look ok, make them trusted:

@smallexample
@verbatim
### copy-from-git.sh
#!/bin/sh
DEST=$HOME/lilypond/trusted-scripts
cp $LILYPOND_GIT/make/website.make \
  $DEST/website.make
cp $LILYPOND_GIT/Documentation/lilypond-texi2html.init \
  $DEST/lilypond-texi2html.init
cp $LILYPOND_GIT/scripts/build/extract_texi_filenames.py \
  $DEST/extract_texi_filenames.py
cp $LILYPOND_GIT/scripts/build/create-version-itexi.py \
  $DEST/create-version-itexi.py
cp $LILYPOND_GIT/scripts/build/create-weblinks-itexi.py \
  $DEST/create-weblinks-itexi.py
cp $LILYPOND_GIT/scripts/build/mass-link.py \
  $DEST/mass-link.py
cp $LILYPOND_GIT/scripts/build/website_post.py \
  $DEST/website_post.py
cp $LILYPOND_GIT/scripts/build/bib2texi.py \
  $DEST/bib2texi.py
cp $LILYPOND_GIT/python/langdefs.py \
  $DEST/langdefs.py
cp $LILYPOND_GIT/Documentation/web/server/lilypond.org.htaccess \
  $DEST/lilypond.org.htaccess
cp $LILYPOND_GIT/Documentation/web/server/website-dir.htaccess \
  $DEST/website-dir.htaccess
@end verbatim
@end smallexample

Build the website:

@smallexample
@verbatim
### make-website.sh
#!/bin/sh
DEST=$HOME/web/
BUILD=$HOME/lilypond/build-website
mkdir -p $BUILD
cd $BUILD
cp $HOME/lilypond/trusted-scripts/website.make .

make -f website.make WEBSITE_ONLY_BUILD=1 website
rsync -raO $BUILD/out-website/website/ $DEST/website/
cp $BUILD/out-website/pictures $DEST
cp $BUILD/out-website/.htaccess $DEST
@end verbatim
@end smallexample

Then in the @file{cronjob/} directory, put the cronjob to automate
the trusted portions:

@warning{@code{cron} will not inherit environment variables from
your main setup, so you must re-define any variables inside your
@code{crontab}.}

@smallexample
@verbatim
# website-rebuild.cron
LILYPOND_GIT=   ... fill this in
LILYPOND_WEB_MEDIA_GIT=   ... fill this in

11 * * * * $HOME/lilypond/trusted-scripts/update-git.sh >/dev/null 2>&1
22 * * * * $HOME/lilypond/trusted-scripts/make-website.sh >/dev/null 2>&1
@end verbatim
@end smallexample

As the final stage of the setup, run your @code{copy-from-git.sh}
script, assuming that you trust the current state of scripts in
lilypond git.

@subsubheading Normal maintenance

When there is a change to the build scripts and/or website
makefile, log in to the server (or your own home machine if you're
testing this there), and do

@example
update-git.sh
check-git.sh
@end example

After reviewing the changes carefully, you can update the trusted
scripts with @code{copy-from-git.sh}.


@subsubheading Building the website (exactly as on the server)

Run @code{make-website.sh}; the final version ends up in
@file{$HOME/web/}.

On the actual server, the website is generated hourly by user
@code{graham} the host @code{lilypond.org}.  You can set up the
cronjob by doing:

@example
crontab $HOME/lilypond/website-rebuild.cron
@end example


@subheading Initial setup for new users on actual serve

You should symlink your own @file{~/lilypond/} to
@file{~graham/lilypond/}

If this directory does not exist, make it.  Git master should go
in @file{~/lilypond/lilypond-git/} but make sure you enable:

@example
git config core.filemode false
@end example

If you have created any files in @file{~graham/lilypond/} then
please run:

@example
chgrp lilypond ~graham/lilypond/ -R
chmod 775 ~graham/lilypond/ -R
@end example



@subsubheading Additional information

Some information about the website is stored in
@file{~graham/lilypond/*.txt}; this information should not be
shared with people without trusted access to the server.


@node Debugging website and docs locally
@section Debugging website and docs locally

@itemize
@item
Install apache2, or any other http server.  These instructions
assume that you also enable @code{mod_userdir}, and use
@code{$HOME/public_html} as the location.

@item
Build the online docs and website:

@example
make WEB_TARGETS="offline online" doc
make website
@end example

This will make all the language variants of the website.  To save
a little time, just the English version can be made with the
command @code{make WEB_LANGS='' website} or the English and (for
example) the French with @code{make WEB_LANGS='fr' website}.

@item
Move the built stuff into those directories.  It's highly
recommended to have your build dir and www dir on the same
partition.  (make @code{$HOME/public_html/} a symlink if
necessary)

@example
mv out-website/website/ $HOME/public_html
mv $HOME/public_html/website/pictures $HOME/public_html/
mkdir -p $HOME/public_html/doc/v2.13/
mv out-www/online-root/* $HOME/public_html/doc/v2.13/
@end example

@end itemize


@node Translating the website
@section Translating the website

As it has much more audience, the website should be translated before
the documentation; see @ref{Translating the documentation}.

In addition to the normal documentation translation practices,
there are a few additional things to note:

@itemize

@item
Build the website with:

@example
make website
@end example

@noindent
however, please note that this command is not designed for being
run multiple times.  If you see unexpected output (mainly the page
footers getting all messed up), then delete your
@file{out-website} directory and run @code{make website} again.

@item
Some of the translation infrastructure is defined in python files;
you must look at the @code{### translation data} sections in:

@example
scripts/build/create-weblinks-itexi.py
scripts/build/website_post.py
@end example

@item
Translations are not included by default in @code{make website}.
To test your translation, edit the @code{WEB_LANGUAGES =} line in
@file{python/langdefs.py}.  You will need to copy this updated
script to @code{$LILYPOND_BUILD_DIR/python/out}.

Do not submit a patch to add your language to this file unless
@code{make website} completes with fewer than 5 warnings.

@item
Links to manuals are done with macros like
@code{@@manualDevelLearningSplit}.  To get translated links, you
must change that to @code{@@manualDevelLearningSplit-es} (for
es/Spanish translations, for example).

@end itemize



