http://www.edrdg.org/~smg/
The JMdictDB project is an informal project to put the contents
of Jim Breen's [*1] JMdict Japanese-English dictionary data [*2]
into a database, and provide a web-based maintenance system
for it.
Discussion takes place on the edict-jmdict@yahoo.com mailing
list (http://groups.yahoo.com/group/edict-jmdict/)
The software in this package is copyrighted by
Stuart McGraw, <jmdictdb@mtneva.com>
(except where otherwise noted)
and licensed under the GNU General Public License version 2.
See the file COPYING.txt for details.
JMdictDB comes with ABSOLUTELY NO WARRANTY.
The most recent version of this code may be downloaded at
http://www.edrdg.org/~smg/.
This package contains the following directories:
./ Package directory.
./doc/ Documentation.
./pg/ Database scripts.
./pg/data/ Database static data.
./python/ Command line apps.
./python/lib/ Library modules.
./python/lib/tmpl/ Web page templates.
./tools/ Scripts used by Makefiles.
./web/ Web related files.
./web/cgi/ CGI scripts.
======
STATUS
======
This code is under development and is alpha quality.
Everything here is subject to future change. Python code
is written for Python 3; Python 2 is no longer supported
(although an older Python 2 version is available from the
code repository, see INSTALLATION/Requirements below). The
web pages use Python/CGI.
The Python 2 to Python 3 conversion was only done recently
(2012-05 through 2012-11 approximately) thus there are likely
a number of conversion related errors remaining in less
frequently used parts of the code.
Development uses Mercurial (http://selenic.com/mercurial)
as a version control system. The development repository is
available for download, and the project's revision history
can be browsed at http://www.edrdg.org/~smg/.
The JMdictDB system is currently running on Jim Breen's
wwwjdic web sites
(http://www.edrdg.org/cgi-bin/wwwjdic/wwwjdic?1C and mirrors)
where it is used to accept additions and corrections to the
wwwjdic/JMdict data from wwwjdict users.
=============
DOCUMENTATION
=============
Overview and general information about JMdictDB:
README.txt -- This file.
Database schema:
doc/schema.odt(.pdf,.html) -- The schema is comprehensively
documented in schema.pdf (or schema.html). Both were
produced from the Open Office Writer document schema.odt.
doc/schema.dia(.png) -- A diagram of the database tables and
their relationships. schema.png was produced from
schema.dia by the open source Dia application.
========
PROGRAMS
========
The ./python/ directory contains a number of independent
programs:
The following tools find and display entries in the database.
shentr.py Command line tool for searching for and
displaying jmdict database entries. It
is well documented making it useful for
understanding the use of the API in a real
(if tiny) application. This program is kept
up-to-date.
srch.py, srch.tal, srch.xrc, srcht.tal, jmdbss.txt
GUI tool to search for and display dajmdict
database entries.
The following tools read an XML or text file and write a
file that can be loaded into a Postgresql database.
exparse.py Read examples.txt file and create loadable
Postgresql dump file.
jmparse.py Read JMdict or JMnedict XML file and create
loadable Postgresql dump file.
kdparse.py Read kanjidic2 XML file and create loadable
Postgresql dump file.
sndparse.py Read JMaudio XML file and create loadable
Postgresql dump file.
xresolv.py Resolve textual xrefs loaded into database
from JMdict files, to real xrefs.
The following tools will read information from the database and write
an XML file that can be loaded by the tools above.
entrs2xml.py Read entries from database and write to
XML file.
snds2xml.py Read Audio data from database and write
JMaudio XML file.
The following work with labeled audio produced by Audacity.
mklabels.py Generate a label file from a db sndfile entry
that can be imported into Audacity.
updsnds.py Update existing and add new snd records from
an Audacity label file.
============
INSTALLATION
============
Although this software was written and is maintained primarily
to support Jim Breen's JMdict and wwwjdic projects, you may
wish to install a local copy of this software:
- To contribute development work to the JMdict project.
- To use or adapt the code to a project of your own.
Requirements
------------
The code is currently developed and tested on Ubuntu using
Apache as a web server. The webserver should be configured
to run Python CGI scripts.
Regarding Microsoft Windows:
Up to mid 2014 the code also ran and was supported on Microsoft
Windows XP. However, current lack of access to a Windows machine
has required dropping Windows support but the Windows specific
code and documentation have been left in place in case support
is revived in the future. PLEASE BE AWARE THAT REFERENCES TO
MICROSOFT WINDOWS IN THIS AND OTHER DOCUMENTATION AND CODE ARE
UNSUPPORTED AND MAY BE WRONG.
JMdictDB requires Python 3; Python 2 is no longer supported
although the last working Python 2 version is available in
the code repository in the branch, "py2-maint".
Some additional Python modules are also needed. Version
numbers are the versions currently in use in the author's
development environment -- the software may work fine with
earlier or later versions, but this has not been verified.
Postgresql [9.6]
Python [3.6] (known not to work before 3.3).
Additional Python packages:
psycopg2-2.7.3 Python-Postgresql connector.
http://initd.org/projects/psycopg2/
http://stickpeople.com/projects/python/win-psycopg/ (Windows)
ply-3.9 -- YACC'ish parser generator.
http://www.dabeaz.com/ply/
lxml-4.0.0 -- XML/XSLT library. Used by xslfmt.py for doing
xml->edict2 conversion.
jinja2-2.9.6 -- Template engine for generating web pages.
Apache [2.4] (on Unix/Linux/Windows systems) or
IIS [5.0] (on MS Windows systems)
make -- Gnu make is required if you want to use the provided
Makefile's to automate parts of the installation.
wget -- Used by Makefile to download the JMdict_e.gz,
JMnedict.gz, and examples.utf8.gz file from the Monash
site. If not available, you can download the needed
files manually.
iconv -- Not required but very useful when dealing with
character encoding conversions that are frequenly required
when working with Japanese language text files.
The principle author had Cygwin (http://cygwin.com) installed on
his Windows development machine and used the make, wget, etc.,
programs provided by that package. A smaller (though untested)
alternative might be to use the programs provided by the Gnuwin32
project: http://gnuwin32.sourceforge.net.
Database Authentication
-----------------------
Any program that accesses the database needs a username
and possibly a pasword to do so. In a standard Postgresql
install, local connections made with user "postgres" do
not need a password, but your installation may require
you to use a different username and password.
Most command line programs supplied by Posgresql, such
as psql, allow one to specify a user name but not a
password; the password will either be interactively
prompted for, or read from the user's ~/.pgpass [*3]
file. Command line tools that are part of the JMdictDB
system generally allow a "-p" option for supplying a
password. Using it on a multi-user machine is usually
a bad idea since another user, using "ps" or other
such commands, can view it. The safest way of supplying
passwords is to use a .pgpass file. See [*4] for more
info.
The database is accessed by the JMdictDB system in three
contexts:
- When running the Makefile to install the JMdictDB
system.
- When cgi scripts are executed by the web server.
- When a local (or remote if permitted) user runs
the command line or GUI tools.
When the Makefile target "init" is run, it will create
two database users (by default, "jmdictdb" and "jmdictdbv").
The other targets create and load databases as user
"jmdictdb". The "jmdictdbv" user is given read-only
access to the databases and is for use by the cgi scripts
and not further used by the Makefile.
When CGI scripts access the database, they do so using
a username obtained from the file config.ini (in python/lib
or the cgi lib/ directory.) You need to create this file
from the config.ini.sample file supplied. The usernames
in config.ini should match the usernames used by the
Makefile "init" target.
Passwords for these usernames may also be supplied in the
config.ini file, but since the file must be readable by
the operating system user that the web server runs as,
you will want to limit read access to the file to only
the web server user. Alternatively, you can install a
.pgpass file in the home directory of the web server user
to provide the passwords.
Editor Authentication
---------------------
The CGI scripts allow unauthenticated users to submit
unapproved edited or new entries, but to approve or
reject entries, a user must be logged in as an editor.
The CGI scripts use a separate database named "jmsess" for
storing editor user info and active sessions. This database
need only be setup once.
Procedure
---------
Note: relative file paths below (except in command
lines) are relative to the package top level directory.
A Makefile is provided that automates the loading and
updating of JMdictDB database. It is presumed that
there is a functioning Postgresql instance, and that
you have access to the database "postgres" account or
some account with enough privledges to create and drop
databases.
The Makefile is usable on both *nix and Windows systems
but the latter requires a working Gnu 'make' program.
The Cygwin package (http://www.cygwin.com) provides a
full unix environment under Windows, including 'make'.
Alternatively, stand-alone native versions of Gnu 'make'
are available (see http://unxutils.sourceforge.net/ or
http://www.mingw.org/ for example.)
By default, the currently active database is named
"jmdict". The makefile targets that load data do so
into a database named "jmnew" so as to not destroy
any working database in the event of a problem. A
make target, "activate" is provided to move the newly
loaded database to "jmdict".
No provision is made for concurrent access while loading
data; we assume that only the access to the database being
loaded is by the procedures used for the loading. Use of
databases other than the one being loaded can continue as
usual during loading.
1. Choose passswords to use for Postgreql users "jmdictdb"
and "jmdictdbv".
2. Copy the file python/lib/config.ini.sample to config.ini
in the same directory. Review it and make any changes
neccessary. Uncomment and change the "pw" and "sel_pw"
passwords in the "db_*" sections to the values chosen in
step (1) above if you wish to supply passwords via this
file (note warnings above.) Otherwise create a .pgpass
file in the web server user's home directory. The .pgpass
file should have two lines in it:
localhost:*:*:jmdictdb:xxxxxx
localhost:*:*:jmdictdbv:xxxxxx
Change the "xxxxxx"s to match the passwords chosen in
step 1. Permissions on the file must be 600 (rw-------)
or Postgresql will ignore it.
3. When you run the Makefile in step 6 below, if there
are passwords on the 'jmdictdb" and "postgres" accounts
(or their equivalents if you've changed them in Makefile)
and Postgresql does not know the passwords, you will be
prompted to enter them (many times). To prevent the
prompting, tell postgresql the passwords by creating a
(or editing a preexisting) .pgpass file in your home
directory and add a line like:
localhost:*:*:jmdictdb:xxxxxx
localhost:*:*:postgres:xxxxxx
Change the "xxxxxx"s to match the "jmdictdb" password
chosen in step 1, and the "postgres" user password. If
PG_SUPER in the Makefile is changed (in next step) from
"postgres" to some other user, adjust the second line
above appropriately.
Permissions on the file must be 600 (rw-------) or
Postgresql will ignore it.
4. Check the settings in Makefile. There are some
configuration settings in the Makefile that you may
want to change. Read the comments therein. In
particular, the cgi directory is assumed to be
~/public_html/cgi-bin/. You may wish to change that
if you will be using the cgi files. There are also
some options for the Postgresql database server
connections, including authentication settings.
If you are running on Microsoft Windows you will
need to change the value of DBLOCALE from "ja_JP.utf8"
to "japanese" or specify DBLOCALE when you run "make"
in step 8 below.
5. Set (or modify) the enviroment variable PYTHONPATH
so that it contains an absolute path to the python/lib
directory. For example, if you installed the jmdictdb
software in /home/joe/jmdictdb/, then PYTHONPATH must
contain (possibly in addition to other directories)
/home/joe/jmdictdb/python/lib.
6. If you have not done so before, in the top-level directory,
run
make init
to create the users/sessions database. It will also
create a single user with administration privilege,
"admin" with password "admin". As soon as the install
is done, you need at a minimum to change the password.
Changing the userid and password is better still.
See the see section "Operation / User Management" below
for more details.
7. (Optional) In the python/lib, run
make
This will make sure that the JEL parser files are
up-to-date. You can generally skip this step if you
are running unmodified copy of the source (since an
attempt is made to keep the distributed support
files updated) but must do this if you've changed
any of the support files' dependencies.
8. In the top level directory, run "make" which won't
do anything other than list the available targets
that will do something.
If you are running on Microsoft Windows you should
first set the client encoding for Postgresql by
setting an environment variable:
set PGCLIENTENCODING=utf-8
To load JMdict, JMnedict, and Examples on a Unix-like
machine, run:
make loadall
Similarly but on a Windows machine:
make DBLOCALE=japanese loadall
"make loadall" will create a database named "jmnew", download
the needed XML files, then parse and load the JMdict, JMnedict,
and Examples files into it and recreate the necessary foreign
key constraints and indexes which were disabled during loading
for performance reasons. If any of the prerequistite files
are already present (such as the .pgi files produced by the
parsers), it will use them. To force a complete reloading
from scratch (except for the fetching which will be done only
if the needed XML file are not present), use
make reloadall
To load a different set of corpora or in a different order
you'll need to do the steps explicitly. For example, to
load JMdict and Kanjidic2, only, run make four times with
the targets:
make jmnew # Create empty jmdictdb database.
make loadjm # Load JMdict
make loadkd # Load Kanjidic2
make postload # Resolve xrefs and update sequences.
In particular "make postload" should always be run last to
finalize a sequence of "make loadxx" operations.
After the above "make" commands have completed sucessfully
you will have a database named "jmnew" which can be examined
to confirm the data is as expected.
The "make" commands generate a lot of output and it is
normal to see as fair number of warning and a few error
messages while "make" is running -- files and database
objects are often deleted or recreated to be sure that
the environment is in a consistent state, and messages
are produced if the objects are already gone or present.
Unfortunately it is hard to tell what is a problem and
what is normal short of experience running the install
a number of times.
Some of the more significant Makefile targets are:
jmnew:
Create a new database named "jmnew" with all
jmdictdb tables and other database objects needed
and ready to load data into.
newdb:
Create an cnmpletely empty database named "jmnew".
(This can be useful if one wants to restore a
jmdictdb database previously saved with pg_dump.)
data/jmdict.xml:
Download the current JMdict_e.gz file from the
Moash FTP site, and unpack it.
data/jmdict.pgi:
Make target jmdict.xml if neccessary, then parse
the jmdict.xml file, generating a rebasable
jmdict.pgi file and jmdict.log.
loadjm:
Make target jmdict.pgi if neccessary, then load
the .pgi file into preexisting database "jmnew"
and do all the post-load tasks like creating
indexes, resolving xref's etc. After this, the
database should be fully loaded and functional,
but is still named "jmnew" to avoid clobbering
any existing and in-use "jmdict" database.
loadall:
Create database "jmnew" and load JMdict, JMnedict,
and Examples into it.
activate:
Renames the "jmnew" database produced above to
"jmdict", making it accessible to all the tools
and cgi scripts.
There are similar sets of data/* and load* targets for
loading JMnedict, the Examples file and Kanjidic2 (though
kanjidic2 support, while usable, is still incomplete).
Note that these targets expect to load their data into
the "jmnew" database and thus should be executed before
doing a "make activate". Or alternatively, you can have
them load directly into the active database (and losing
the opportunity to validate the data before bringing it
to the production database) by doing, for example,
"make DB=jmdict loadex"
Makefile will download JMdict_e.gz (or JMdict.gz if so
configured), JMnedict.gz, and examples.utf8.gz as needed
depending on the make targets used, using the 'wget'
program. If wget is not available you can download
the needed files manually, and put them in the ./data/
directory.
9. The makefile will parse the data files, create a database
named "jmnew", load the jmdictdb schema, and finally load
all the parsed data into it. If everything was loaded
sucessfully, run
make activate
which will rename any existing "jmdict" database to "jmold"
(any existing "jmold" database is deleted), and rename the
"jmnew" database to "jmdict", thus making it the active
database and the one accessed by default by the cgi web
pages. There must be no active users in any of these
databases or the "make activate" command will fail.
10. If you plan on using the cgi files with a web server,
double check the settings in the Makefile (see step #1)
and then run:
make web
to install the web CGI files.
Note that it is also possible to configure your web server
to serve the cgi files directly from the development directory
making this step unnecessary.
11. Create a config.ini file in the cgi directory where the
lib files were copied based the python/lib/config.ini.sample
file and adjusted for your installation. It should be readable
by the web server process and not readable by world (it will
contain database passwords.)
Create a log file as described in the OPERATION section below.
It should be writable by the the web server process.
You should now be able to go to the url corresponding to
srchform.py and do searches for jmdict entries. The url
corresponding to edform.py will let you add new entries.
=========
OPERATION
=========
Web access to the JMdictDB system can be suspended temporarily
by creating a control file in the installed CGI directory named
"status_maint" or "status_load". If either file exists, any
web access to a CGI script will result in a redirect to
"status_maint.html" or "status_load.html" which present the
user with a message that the system is unavailable due to
maintenance or excessive load, respectively.
The directory in which the CGI scripts look for the control
files can be set in the config.ini file. The location of the
html files is not customizable although you can of course
modify their contents.
It is up to you to create and and remove the control files as
appropriate.
Log Files:
----------
The CGI scripts log events to a log file whose name and location
are given in the config.ini file (see python/lib/config.ini/sample
for details.) If no logfile is given in the config.ini file the
default is "jmdictdb.log" in the current directory when the script
is executed by the web server. For Apache-2.4 this will often be
in the CGI directory itself. If no logfile level is given, the
defaulr is "debug".
The logfile must by manually created, the CGI scripts won't create
it if it doesn't exist. It must also have permissions that allow
writing by the web server process owner.
If the log file is not writable when a CGI script starts, the
script will write a message to that effect to stderr and disable
further logging during that scrupt's execution. The initial
message on most web servers will be written to the web server's
log file and may help identify where the JMdictDB log file is.
The format of JMdictDB log file messages start with a timestamp
using the format: "YYMMDD-hhmmss". The processes number is also
provided in square brackets: "[pid]". When a non-fatal error
occurs it is logged in the log file and an error page is presented
to the user that will have an error id number of the form:
"YYMMDD-hhmmss-pid". This allows its correlation to the log file
message which may have moe information such as a Python traceback.
The log file is not truncated or rotated periodically; you must
arrange for that.
Updates:
--------
Updates occur periodically to the code and to the database.
Program code updates including website scripts are generally done by:
$ cd [...]/jmdictdb
$ hg pull & hg update
$ make web
Database updates are generally done by:
$ cd [...]/jmdictdb
$ psql -d jmdict -U jmdictdb -f patches/nnn-xxxxxx.sql
IMPORTANT: read the patch file contents before applying the above
commands. There are sometimes exceptions to the sequence shown that
will be documented in the update file itself.
See the file ./patches/README.txt for more details.
User management:
----------------
IMPORTANT: The first time the jmsess database is created by
running 'make init', a single user named "admin" with password
"admin" is created. You must at a minimum change this user's
password before making JMdictDB accessible in other than a local
trusted envirionment.
There is not yet any tools or webpages for adding, removing
ot updating users and these activities need to be performed
by direct manipulation of the user data in the "jmsess" database
using Postgresql's 'psql' command.
To add a new user:
INSERT INTO users VALUES (
'jones', 'Bob Jones', 'bjones@ntt.co.jp',
crypt('plaintext-password', gen_salt('bf')),
FALSE, 'E', NULL);
To disable a user (prevent him/her from logging in):
UPDATE users SET disabled=True WHERE userid='jones';
To change a password:
UPDATE users SET pw=crypt('plaintext-password', gen_salt('bf'))
WHERE userid='jones';
======================================================================
Notes:
[*1]
http://www.csse.monash.edu.au/~jwb/japanese.html
[*2]
http://www.csse.monash.edu.au/~jwb/edict_doc.html
[*3]
On Windows the Postgresql password file is typically in
"C:\Documents and Settings\<your_windows_user_name>\ -
Application Data\Postgresql\pgpass.conf". For brevity
we will refer simply to "~/.pgpass" in this document.
[*4]
For more information on usernames, passwords, and the .pgpass
file, see the Postgresql docs:
31.15 Client Interfaces / libpq / The Password File
31.1 Client Interfaces / libpq / Database Connection -
Control Functions
19 Server Administration / Client Authentication
sec VI Reference / Postgresql Client Applications / -
psql / Usage / Connecting to a Database
Note that chapter numbers are Postgresql version dependent.
Numbers given are for Postgres version 9.2.
===
EOF
===
