| Revision History | ||
|---|---|---|
| Revision 0.6 | 2007/05/13 | |
| Revision 0.5 | 2006/11/20 | |
| Revision 0.4 | 2006/04/03 | |
| Revision 0.3 | 2005/03/06 | |
| Revision 0.2 | 2004/10/07 | |
| Revision 0.1 | 2004/06/05 | |
Sinhala is the main language of Sri Lanka. This guide describes the level of Sinhala support available in GNU/Linux. It also describes how to enable improved Sinhala support and the tasks that still require attention.
This guide is GNOME and Debian/Ubuntu centric. Most of the explanations and suggestions should also be applicable to other distributions.
The level of Sinhala support in GNU/Linux distributions have improved significantly. On most modern distributions, simply installing a Unicode Sinhala font should be sufficient to read and type in Sinhala.
Debian 4.0 (Etch) and Ubuntu 7.04 (Feisty) ship with Unicode Sinhala (SLS1134) support on the desktop. This includes Sinhala support in the web browser, Firefox [Section 3.3], and the office suite, Open Office [Section 3.4]. Both distributions ship with a phonetic Sinhala keyboard layout and a basic, incomplete, Unicode Sinhala font [Section 3.2].
To improve the level of Sinhala support in Debian and Ubuntu, simply follow the instructions in this document [Section 3].
Silva, A.W.L. (2003). SINHALESE for Beginners. Sri Lanka: Pubudu Printers Kandana
The first chapter has excellent coverage of the Sinhala letters.
ISO 639-2: Codes for the Representation of Names of Languages
SLS1134 - Sinhala Character Code for Information Interchange (SCCII) (not available online)
SLS1134 - Sinhala Character Code for Information Interchange (SCCII) - Part 1: Collation Sequence (not available online)
SLS1134 - Sinhala Character Code for Information Interchange (SCCII) - Part 2 : Requirements and Methods of Test (not available online)
The Sinhala GNU/Linux users and developers use these mailing lists for announcements, discussions, debugging and reviews.
Therefore, the archives of the mailing lists contain very useful information for both new users and new developers.
Please search the archives for the answers to your questions before posting to the mailing lists.
Many individuals have contributed to the project that began in 2003 on an LKLUG mailing list. Some of the notable contributors are:
Dushara Jayasinghe
Harshula Jayasuriya
Chamath Keppitiyagama
Anuradha Ratnaweera
Harsha Senanayake
Naoto Takahashi
LKLUG Unicode Sinhala font
SCIM transliterated input method
Automatically sets the required environment variables in /etc/environment
Add to /etc/apt/sources.list
On Debian 4.0 (Etch)
deb http://sinhala.sourceforge.net/debian/i386/etch/ ./ |
On Debian testing (Lenny)
deb http://sinhala.sourceforge.net/debian/i386/lenny/ ./ |
On Ubuntu 7.10 (Gutsy) i386
deb http://sinhala.sourceforge.net/ubuntu/i386/gutsy/ ./ |
On Ubuntu 7.10 (Gutsy) amd64
deb http://sinhala.sourceforge.net/ubuntu/amd64/gutsy/ ./ |
Update repository metadata:
apt-get update |
Install Sinhala packages:
apt-get install sinhala-gnu-linux |
Upgrade relevant packages:
apt-get upgrade |
Logout and login again. Environment variables need to be set/updated (NO NEED TO REBOOT)
Visit http://si.wikipedia.org/ and see if the Sinhala letters render correctly.
Copy and paste some of the content from Sinhala wikipedia to Open Office Writer. Then highlight the Sinhala text and choose the LKLUG font to display them.
To test SCIM, press Control-space whilst you are running a GNOME application. Then select one of the Sinhala input methods.
Download a Unicode Sinhala font:
If you are using a modern GNU/Linux version and it has fontconfig installed, all you have to do is make a .fonts directory in your home directory:
mkdir ~/.fonts |
and copy the True/Open Type font into that directory.
If you want to make the font available to all users of the system, become root and copy the font to:
/usr/share/fonts |
In both the above cases, run:
fc-cache -fv |
To check which font file provides the Sinhala support, run:
fc-list :lang=si file |
Immediately you'll be able to read Unicode Sinhala in these programs (You may have to restart the program.):
Anything gtk2 based
evolution
gedit
gucharmap
Firefox/Mozilla (built with gtk2, FreeType2 and Pango support)
If you have Pango 1.8.2 and greater, you will have full SLS1134 Sinhala support.
Pango is enabled by default in Debian 4.0.
Ubuntu 6.06 users can enable Pango in Firefox by setting an environment variable:
MOZ_DISABLE_PANGO=0 |
Ubuntu 6.10 users can enable Pango in Firefox by setting an environment variable:
MOZ_DISABLE_PANGO=0 |
Or by simply installing the Ubuntu package:
language-pack-si-base |
Firefox and Mozilla can be enabled with pango rendering support, which enables many text layout features, including the rendering of CTL (Complex Text Layout) such as Indic languages. To enable this, set the following environment variable when running Firefox or Mozilla:
MOZ_ENABLE_PANGO=1 [1]
Pango is enabled by default so you don't have to do anything extra:
7.2. Pango Text Renderer for Firefox
Fedora is building Firefox with the Pango system as the text renderer. This provides better support for certain language scripts, such as Indic and some CJK scripts. Pango is included with with permission of the Mozilla Corporation. This change is known to break rendering of MathML, and may negatively impact performance on some pages. To disable the use of Pango, set your environment before launching Firefox:
MOZ_DISABLE_PANGO=1 /usr/bin/firefox
...
23.4. Pango Support in Firefox
Firefox in Fedora Core is built with Pango, which provides better support for certain scripts, such as Indic and some CJK scripts. Fedora has the permission of the Mozilla Corporation to use the Pango system for text renderering.
To disable the use of Pango, set MOZ_DISABLE_PANGO=1 in your environment before launching Firefox. [2]
Debian 4.0 ships with ICU 3.6 which contains Sinhala support. Furthermore, Open Office has been patched to not filter out ZWJ.
Ubuntu 7.04 ships with Open Office 2.2 and ICU 3.6 which contain Sinhala support.
Fedora Core 6 ships with ICU 3.6 which contains Sinhala support. Furthermore, Open Office has been patched to not filter out ZWJ. However, there is a bug in ICU that can trigger Open Office to crash when typing in Sinhala.
To test multi-lingual input methods in gtk2 based programs, run:
gedit |
To check which input methods are available for gtk2 based programs, run:
/usr/bin/gtk-query-immodules-2.0 |
which comes with gtk2.
Install scim-m17n and m17n-db (In Ubuntu, these packages are in the Universe repository. For help read: http://www.ubuntuguide.org/)
Download the m17n Wijesekera input method from:
Copy the file to:
/usr/share/m17n/ |
Make sure the file /etc/environment contains:
XMODIFIERS=@im=SCIM GTK_IM_MODULE=scim QT_IM_MODULE=scim |
Logout and Login again, no need to restart
Select the M17N-si-wijesekera-preedit input method in SCIM
Type away
You should have at least XFree86 4.3 or Xorg 6.7. To familiarise yourself with this keyboard layout, read:
The X Keyboard Extension only allows one-to-one mappings between keys and codepoints, therefore rakaaranshaya, yansaya and repaya, which consist of multiple codepoints, have to be manually constructed. See the comments in the Sinhala X Keyboard Extension layout file.
The aforementioned layout is already included in Xorg 6.9 and above and distributions that ship with xkeyboard-config 0.6 and above.
Debian Etch:
/usr/share/X11/xkb/symbols/lk |
Ubuntu 5.10:
/etc/X11/xkb/symbols/lk |
Fedora Core 5:
/usr/share/X11/xkb/symbols/lk |
The latest version of the keyboard layout can be downloaded from CVS:
Read the comments in the lk file to see how to create rakaaranshaya, yansaya and repaya.
The window manager should come with a program which allows the user to choose multiple keyboard layouts.
In the example below I have chosen the SHIFT keys to switch between the Sinhala phonetic layout and the US QWERTY layout. Hold one of the SHIFT keys down and then press the other SHIFT key, this should toggle between the layouts.
Using the GUI in GNOME:
Run:
gnome-keyboard-properties |
Choose the “Layout” tab and click on the “Add” button. This will open a new window which contains a list of layouts ordered by country.
Scroll down the list till you find “Sri Lanka” and then highlight it by clicking on it. The Sinhala layout is the default in the Sri Lanka layouts file, so you do NOT need to click the expand triangle icon. Then press “OK”.
Choose the “Layout Options” tab and click on the text “Group Shift/Lock behavior”. A list will expand below this text
Scroll down the list till you find the text “Both Shift keys together change group”. Click on the corresponding checkbox.
If you wish to use an LED to indicate the toggling of keyboard layouts, click on the text "Use keyboard LED to show alternative group". A list will expand below this text
Scroll down the list till you find the text “ScrollLock LED shows alternative group”.
Using the command line in X:
In an xterm do:
setxkbmap -layout "us,lk" -option "grp:shifts_toggle,grp_led:scroll" |
Alternately, you can directly modify /etc/X11/xorg.conf:
To add the new lk keyboard layout, look for this line:
Section "InputDevice" |
There will probably be two such lines, one for the keyboard and another for the mouse. Go to the keyboard related line.
Then add 'lk' to a line that looks like:
Option "XkbLayout" "us,lk" |
Also add a mechanism to switch between 'us' and 'lk' and indicate which LED should be used:
Option "XkbOptions" "grp:shifts_toggle,grp_led:scroll" |
If asked by the window manager, reset keyboard defaults to the X defaults.
In the example below I have chosen the ALT keys to switch between the Sinhala phonetic layout and the US QWERTY layout. Hold one of the ALT keys down and then press the other ALT key, this should toggle between the layouts.
Download the keyboard layout from:
Copy the keyboard layout to:
/etc/X11/xkb/symbols/pc/ |
There are two options:
In an xterm do:
setxkbmap -layout "sin,us" -option "grp:alts_toggle,grp_led:scroll" |
Or alternately, edit the /etc/X11/XF86Config or /etc/X11/xorg.conf file.
To add the new 'sin' keyboard layout, look for this line:
Section "InputDevice" |
There will probably be two such lines, one for the keyboard and another for the mouse. Go to the keyboard related line.
Then add 'sin' to a line that looks like:
Option "XkbLayout" "sin,us" |
Also add a mechanism to switch between 'us' and 'sin' and indicate which LED should be used:
Option "XkbOptions" "grp:alts_toggle,grp_led:scroll" |
If asked by the window manager, reset keyboard defaults to the X defaults.
You can use a Unicode Character Map program to copy and paste Sinhala characters into your program/document. Available programs are:
gucharmap
The ability to alphabetically sort words in a database is essential. Till recently databases containing Sinhala words could not be sorted according to the Sinhala sorting order as established by SLS1134 - Part 1: Collation Sequence. Now, Sinhala words can be sorted in MySQL (from version 5.2).
Instead of running this query:
SELECT * FROM table1 ORDER BY column1; |
run a slightly modified query that looks like:
SELECT * FROM table1 ORDER BY column1 COLLATE utf8_sinhala_ci; |
Savannah Sinhala Project: http://savannah.nongnu.org/projects/sinhala/
Sourceforge Sinhala Project: http://sourceforge.net/projects/sinhala/
You can use this PHP script to generate Unicode Sinhala letters:
If you wish to use TeX font glyphs to create a new font, mftrace will be useful. In most circumstances fontforge alone will be sufficient for most of your needs.
The top of tree Pango (since 1.8.2) & ICU (since 3.6) now support SLS1134.
Pango's Indic renderer is based on ICU's Indic renderer.
The original patch to add Sinhala support was created by Harsha Senanayake for ICU [3]and later ported to Pango. The Pango patch was ported to the latest version of Pango by Chamath Keppitiyagama. It was submitted to bugzilla by Anuradha Ratnaweera[4]. Harshula Jayasuriya modified the Pango state table & ZWJ handling [5] & [6].
The Pango code for Sinhala and Indic rendering is common and can be found in the Pango source at:
modules/indic/ |
One of the most important files to understand is:
modules/indic/indic-ot-class-tables.c |
Particularly how the function:
indic_ot_find_syllable() |
works.
Next have a look at the file:
modules/indic/indic-ot.c |
and the function:
indic_ot_reorder() |
Owen Taylor (Pango) submitted the Pango Sinhala patch to the ICU project [7]. Eric Mader (ICU) ported the Pango patch to ICU and checked-in the changes to ICU 3.6. Then Eric added the state table & ZWJ modifications from Pango to ICU 3.6 [8] & [9].
There was an issue with U+0DDD (dependent vowel diga o) that can cause Open Office to crash. Opening this text file will crash Open Office and ICU 3.6:
The worstCaseExpansion for Sinhala was set to 3 when it should have been set to 4. The dependent vowel 'oo' (U+0DDD) consists of (kombuva)(dotted-circle)(aela-pilla)(al-lakuna) which are 4 glyphs. As a result of the worstCaseExpansion being 3, memory was probably being allocated for 3 glyphs when memory was required for 4 glyphs. The actual crash occurred when unallocated memory was being freed.
Caolan McNamara also found this bug and fixed it first. [10]
source/layoutex/ParagraphLayout.cpp
ParagraphLayout::ParagraphLayout(const LEUnicode chars[], le_int32 count, const FontRuns *fontRuns, const ValueRuns *levelRuns, const ValueRuns *scriptRuns, const LocaleRuns *localeRuns, UBiDiLevel paragraphLevel, le_bool vertical, LEErrorCode &status)
source/layout/LayoutEngine.cpp:
LayoutEngine *LayoutEngine::layoutEngineFactory(const LEFontInstance *fontInstance, le_int32 scriptCode, le_int32 languageCode, LEErrorCode &success)
LayoutEngine *LayoutEngine::layoutEngineFactory(const LEFontInstance *fontInstance, le_int32 scriptCode, le_int32 languageCode, le_int32 typoFlags, LEErrorCode &success)
IndicOpenTypeLayoutEngine::IndicOpenTypeLayoutEngine(const LEFontInstance *fontInstance, le_int32 scriptCode, le_int32 languageCode, le_int32 typoFlags, const GlyphSubstitutionTableHeader *gsubTable)
IndicOpenTypeLayoutEngine::IndicOpenTypeLayoutEngine(const LEFontInstance *fontInstance, le_int32 scriptCode, le_int32 languageCode, le_int32 typoFlags)
source/layout/LayoutEngine.cpp
le_int32 LayoutEngine::layoutChars(const LEUnicode chars[], le_int32 offset, le_int32 count, le_int32 max, le_bool rightToLeft, float x, float y, LEErrorCode &success)
le_int32 LayoutEngine::computeGlyphs(const LEUnicode chars[], le_int32 offset, le_int32 count, le_int32 max, le_bool rightToLeft, LEGlyphStorage &glyphStorage, LEErrorCode &success)
source/layout/IndicLayoutEngine.cpp
le_int32 IndicOpenTypeLayoutEngine::characterProcessing(const LEUnicode chars[], le_int32 offset, le_int32 count, le_int32 max, le_bool rightToLeft, LEUnicode *&outChars, LEGlyphStorage &glyphStorage, LEErrorCode &success)
source/layout/IndicReordering.cpp
le_int32 IndicReordering::reorder(const LEUnicode *chars, le_int32 charCount, le_int32 scriptCode, LEUnicode *outChars, LEGlyphStorage &glyphStorage, MPreFixups **outMPreFixups)
engine->getGlyphs(fStyleRunInfo[run].glyphs, layoutStatus);
engine->getGlyphPositions(fStyleRunInfo[run].positions, layoutStatus);
engine->getCharIndices(&fGlyphToCharMap[glyphBase], runStart, layoutStatus);
Interestingly, Debian, Fedora Core and Ubuntu decided to address enabling Pango in Firefox in completely different ways.
Ubuntu 5.10 enabled Pango by default. Have a look at:
/usr/bin/mozilla-firefox |
which contains the code:
## ## Set MOZ_ENABLE_PANGO ## MOZ_ENABLE_PANGO=1 export MOZ_ENABLE_PANGO |
On the other hand Ubuntu 6.06, decided to disable Pango in Firefox by default except for a pre-determined list of locales. The extensive discussion can be found here:
Have a look at:
/usr/bin/mozilla-firefox |
which contains the code:
if [ "x${MOZ_DISABLE_PANGO}" = x ]; then
if egrep '^(bn|gu|hi|kn|ml|mr|ne|pa|ta|te)_' \
/var/lib/locales/supported.d/*[^~] >/dev/null 2>&1; then
MOZ_DISABLE_PANGO=0
else
MOZ_DISABLE_PANGO=1
fi
export MOZ_DISABLE_PANGO
fi
if [ "x${MOZ_DISABLE_PANGO}" = x0 ]; then
unset MOZ_DISABLE_PANGO
fi |
This means that Ubuntu 6.06 users that need Pango enabled in Firefox need to set an environment variable
MOZ_DISABLE_PANGO=0 |
You can see the difference by running Firefox at the command line like so:
# MOZ_DISABLE_PANGO=0 mozilla-firefox |
Since Fedora Core 4, Pango is enabled in Firefox by default. In order to disable Pango in Firefox an environment variable has to be set:
MOZ_DISABLE_PANGO=1 |
You can see the difference by running Firefox at the command line like so:
# MOZ_DISABLE_PANGO=1 firefox |
Have a look at:
/usr/bin/firefox |
for an explanation.
## ## Set MOZ_ENABLE_PANGO is no longer used because Pango is enabled by default ## you may use MOZ_DISABLE_PANGO=1 to force disabling of pango ## #MOZ_DISABLE_PANGO=1 #export MOZ_DISABLE_PANGO |
## ## In order to better support certain scripts (such as Indic and some CJK ## scripts), Fedora builds its Firefox, with permission from the Mozilla ## Corporation, with the Pango system as its text renderer. This change ## is known to break rendering of MathML, and may negatively impact ## performance on some pages. To disable the use of Pango, set ## MOZ_DISABLE_PANGO=1 in your environment before launching Firefox. ## # # MOZ_DISABLE_PANGO=1 # export MOZ_DISABLE_PANGO # |
Changelog:
2006-01-27 Christian Persch <chpe at cvs dot gnome dot org>
* src/ephy-main.c: (main):
Disable pango rendering by default, unless MOZ_ENABLE_PANGO env
var is set. Bug #328844.
|
src/ephy-main.c:
/* Work around bug #328844, and avoid the gecko+pango performance problem */
env = g_getenv ("MOZ_ENABLE_PANGO");
enable_pango = env != NULL &&
env[0] != '\0' &&
g_ascii_strtoull (env, NULL, 10) != 0;
if (eel_gconf_get_boolean (CONF_GECKO_ENABLE_PANGO))
{
g_print ("NOTE: Enabling gecko pango renderer; this may cause performance degradation.\n"
"You can set " CONF_GECKO_ENABLE_PANGO " to \"false\" to disable it.\n");
}
else if (!enable_pango)
{
g_setenv ("MOZ_DISABLE_PANGO", "1", TRUE);
} |
Epiphany also has a file, data/epiphany-pango.schemas containing a list of locales which require Pango to be enabled by default.
Whilst working on the patches for adding Sinhala support to ICU, the renderer of Open Office, I observed that the ZWJ characters do not appear to reach ICU. [13]
Then, Caolan McNamara found the Open Office file that filters ZWJ and ZWNJ. [14]
The source file:
vcl/source/gdi/sallayout.cxx |
contains a function:
inline bool IsControlChar( sal_Unicode cChar ) |
This function tells a caller that characters U+200B to U+200F are control characters.
In the source file:
linguistic/source/misc.cxx |
two functions,
static INT16 GetOrigWordPos( const OUString &rOrigWord, INT16 nPos ) |
and
INT32 GetPosInWordToCheck( const OUString &rTxt, INT32 nPos ) |
call
inline bool IsControlChar( sal_Unicode cChar ) |
when doing lingustic analysis for what appears to be spelling purposes. Even found some comments written in, I assume, German.
In the source file:
vcl/source/gdi/sallayout.cxx |
there is a function:
void ImplLayoutArgs::AddRun( int nCharPos0, int nCharPos1, bool bRTL ) |
which calls the function:
inline bool IsControlChar( sal_Unicode cChar ) |
it's purpose is to:
// add a run after splitting it up to get rid of control chars |
It should be noted that this function handles RTL text in a different way to LTR text. My initial reaction is that should not be the case. However, I have not looked into it any further.
Compiling Open Office 2.0.4 on Debian Etch on a Pentium M 2.13 GHz with 1 GiB RAM took approximately 10 hours and required 10 GBs of additional hard drive space for the source and the compiled files.
The recommended infrastructure for keyboard layouts are XKB, XIM, IIIMF [15], m17n [16] and SCIM [17]. The recommended infrastructures are XKB, for simple one-to-one keyboard layouts, and SCIM/m17n for complex keyboard layouts. XKB is a component of Xorg.
Sinhala letters which define the start of a new 'syllable':
All independent vowels (U+0d85 - U+0d96)
Kombuva (U+0dd9) - except if preceded by a kombuva.
All consonants (U+0d9a - U+0dc6) - except if preceded by kombuva or kombuva deka (U+0ddb)
Kunddaliya (U+0df4)
All non-Sinhala characters/codepoints - except ZWJ (U+200D)
Wijesekera Compatible
ASCII Compatible Wijesekera
Phonetic (Static)
Phonetic (Dynamic)
A Unicode Sinhala Font has to be installed in order to read the Keyboard Layouts.
You can use showkey in linux to display the scancode.
57 - space
56 - left alt
100 - right alt
29 - left ctrl
97 - right ctrl
42 - left shift
54 - right shift
Look in the linux source:
drivers/char/keyboard.c |
Look for the function:
getkeycode() |
All you need to do is just copy the keyboard layout file into the correct directory:
/etc/X11/xkb/symbols/ |
or
/etc/X11/xkb/symbols/pc/ |
or
/usr/share/X11/xkb/symbols |
However, for completeness some files in these directories:
/etc/X11/ |
or
/usr/X11R6/lib/X11/ |
or
/usr/share/X11/ |
need to be modified, namely these files:
xkb/rules/{xorg,xfree86}
xkb/rules/{xorg,xfree86}.lst
xkb/rules/{xorg,xfree86}.xml
xkb/symbols.dir |
To test a loaded keyboard layout:
setxkbmap -print | xkbcomp -w 10 -xkb - <outfile> |
SCIM can be used as the frontend, which is exposed to the user, and the backend that maps keycodes to codepoints. Or SCIM can be used as a frontend for other backends. e.g. m17n can be a backend.
The m17n backend keyboard layout definition file is a text file. The documentation can be found:
The xmodmap keyboard layout is not fully functional, hence it is recommended you use the X Keyboard Extension keyboard layout. To familiarise yourself with this keyboard layout, read:
Download the keyboard layout from:
Then run xmodmap:
xmodmap sin.xmodmap |
To familiarise yourself with this keyboard layout, read:
Download the keyboard layout and redirector from:
Copy the keyboard layout and redirector to ~/.vim/keymap/
Start gvim
Need to disable the menu so that you can use the 'alt' key.
set guioptions-=m |
Select the new keyboard layout, using the redirector, by typing:
set keymap=sinhala |
or select the new keyboard layout directly by typing:
set keymap=sinhala-phonetic_utf-8 |
To toggle between the Sinhala keyboard layout and the standard ASCII keyboard layout, press <Ctrl> <6> whilst in insert mode.
The Sinhala letters in the Unicode chart can be categorised as:
independent vowels: U+0D85 - U+0D96
consonants: U+0D9A - U+0DC6
dependent vowels: U+0DCA - U+0DF3
consonant modifiers: U+0D82 - U+0D83
The collation order of the groups can be broadly described as:
independent vowels
consonant modifiers
consonants
dependent vowels.
The order of the groups and the order of the letters within the groups do not correspond to the collation order. Hence, tailoring is required.
The first tailoring rule is the minimal description of Sinhala tailoring. The second tailoring rule is the complete description of Sinhala tailoring as required by MySQL[19].
/*
SCCII Part 1 : Collation Sequence (SLS1134)
2006/11/24
Harshula Jayasuriya <harshula at gmail dot com>
Language Technology Research Lab, University of Colombo / ICTA
*/
#if 1
static const char sinhala[]=
"& \\u0D96 < \\u0D82 < \\u0D83"
"& \\u0DA5 < \\u0DA4"
"& \\u0DD8 < \\u0DF2 < \\u0DDF < \\u0DF3"
"& \\u0DDE < \\u0DCA";
#else
static const char sinhala[]=
"& \\u0D96 < \\u0D82 < \\u0D83 < \\u0D9A < \\u0D9B < \\u0D9C < \\u0D9D"
"< \\u0D9E < \\u0D9F < \\u0DA0 < \\u0DA1 < \\u0DA2 < \\u0DA3"
"< \\u0DA5 < \\u0DA4 < \\u0DA6"
"< \\u0DA7 < \\u0DA8 < \\u0DA9 < \\u0DAA < \\u0DAB < \\u0DAC"
"< \\u0DAD < \\u0DAE < \\u0DAF < \\u0DB0 < \\u0DB1"
"< \\u0DB3 < \\u0DB4 < \\u0DB5 < \\u0DB6 < \\u0DB7 < \\u0DB8"
"< \\u0DB9 < \\u0DBA < \\u0DBB < \\u0DBD < \\u0DC0 < \\u0DC1"
"< \\u0DC2 < \\u0DC3 < \\u0DC4 < \\u0DC5 < \\u0DC6"
"< \\u0DCF"
"< \\u0DD0 < \\u0DD1 < \\u0DD2 < \\u0DD3 < \\u0DD4 < \\u0DD6"
"< \\u0DD8 < \\u0DF2 < \\u0DDF < \\u0DF3 < \\u0DD9 < \\u0DDA"
"< \\u0DDB < \\u0DDC < \\u0DDD < \\u0DDE < \\u0DCA";
#endif |
SHOW CHARACTER SET;
SHOW COLLATION;
SHOW COLLATION like 'ucs%';
SHOW COLLATION like 'utf8%';
SET NAMES 'utf8'; // after connecting to server if the server has NOT set 'skip-character-set-client-handshake'
SHOW CREATE TABLE <table-name>;
SHOW VARIABLES;
\s
Edit the file /etc/mysql/my.cnf and add to the [mysqld] section:
default-character_set=utf8
skip-character-set-client-handshake
This is done to ensure that UTF-8 is the default encoding for the server and client.
Test Swedish (MySQL is a Swedish company) collation algorithm:
http://en.wikipedia.org/wiki/Swedish_alphabet
CREATE TABLE t1 (
id SERIAL PRIMARY KEY,
letter VARCHAR(10) NOT NULL
) CHARACTER SET utf8; |
Running this collation handler:
SELECT * FROM t1 ORDER BY letter COLLATE utf8_swedish_ci; |
results in non-English letters appearing at the end of the sorted alphabet as expected.
Test new Sinhala collation algorithm:
http://en.wikipedia.org/wiki/Sinhala_alphabet
CREATE TABLE t2 (
id SERIAL PRIMARY KEY,
letter VARCHAR(10) NOT NULL
) CHARACTER SET utf8; |
Load the data from this file:
Running this collation handler:
SELECT * FROM t2 ORDER BY letter COLLATE utf8_sinhala_ci; |
results match the SCCII Part 1 : Collation Sequence (SLS1134).
Download the output file from:
mysql/strings/ctype-uca.c
/*
Collation language is implemented according to
subset of ICU Collation Customization (tailorings):
http://icu.sourceforge.net/userguide/Collate_Customization.html
Collation language elements:
Delimiters:
space - skipped
<char> := A-Z | a-z | \uXXXX
Shift command:
<shift> := & - reset at this letter.
Diff command:
<d1> := < - Identifies a primary difference.
<d2> := << - Identifies a secondary difference.
<d3> := <<< - Idenfifies a tertiary difference.
Collation rules:
<ruleset> := <rule> { <ruleset> }
<rule> := <d1> <string>
| <d2> <string>
| <d3> <string>
| <shift> <char>
<string> := <char> [ <string> ]
An example, Polish collation:
&A < \u0105 <<< \u0104
&C < \u0107 <<< \u0106
&E < \u0119 <<< \u0118
&L < \u0142 <<< \u0141
&N < \u0144 <<< \u0143
&O < \u00F3 <<< \u00D3
&S < \u015B <<< \u015A
&Z < \u017A <<< \u017B
*/ |
mysql/include/m_ctype.h
typedef struct charset_info_st
{
uint number;
uint primary_number;
uint binary_number;
uint state;
const char *csname;
const char *name;
const char *comment;
const char *tailoring;
uchar *ctype;
uchar *to_lower;
uchar *to_upper;
uchar *sort_order;
uint16 *contractions;
uint16 **sort_order_big;
uint16 *tab_to_uni;
MY_UNI_IDX *tab_from_uni;
MY_UNICASE_INFO **caseinfo;
uchar *state_map;
uchar *ident_map;
uint strxfrm_multiply;
uchar caseup_multiply;
uchar casedn_multiply;
uint mbminlen;
uint mbmaxlen;
uint16 min_sort_char;
uint16 max_sort_char; /* For LIKE optimization */
uchar pad_char;
my_bool escape_with_backslash_is_dangerous;
MY_CHARSET_HANDLER *cset;
MY_COLLATION_HANDLER *coll;
} CHARSET_INFO; |
mysys/charset.c: CHARSET_INFO *all_charsets[256]
mysys/charset.c
init_available_charsets()
mysys/charset-def.c
init_compiled_charsets()
mysys/charset.c
add_compiled_collation()
init_state_maps()
strings/ctype-uca.c
my_coll_init_uca()
create_tailoring()
my_coll_rule_parse()
my_coll_lexem_init()
[Copy the default weights data to the new weights data structure]
mysys/charset.c
get_charset_by_name()
get_collation_number()
get_collation_number_internal()
get_internal_charset()
mysql/config/ac-macros/character_sets.m4
mysql/mysys/charset-def.c
mysql/strings/ctype-uca.c
mysql/configure (generated)
mysql/mysql-test/t/ctype_utf8.test
mysql/mysql-test/r/ctype_utf8.result
The supported locales are listed in the file:
/usr/share/i18n/SUPPORTED |
The locale definition is available in:
/usr/share/i18n/locales/ |
These files are provided by the locales package.
The locale definitions need to be compiled before before they can be used. To compile a particular locale, first edit:
/etc/locale.gen |
Then run the program:
/usr/sbin/locale-gen |
Following that step you can then see what a particular program looks like when it is localised:
LANG=si_LK.UTF-8 gedit LANG=fr_FR.UTF-8 gedit |
The actual translated message strings are stored in:
/usr/share/locale/<ISO639 2 character lang code>/LC_MESSAGES/ |
Each program adds a file containing translated message strings.
The default locale is set in the file:
/etc/default/locale |
To understand issues involving config.guess and config.sub, refer to /usr/share/doc/autotools-dev/README.Debian.gz .
In debian/rules, for the target binary-arch, you'll probably have to uncomment dh_install. Or you can just change the DESTDIR of:
$(MAKE) DESTDIR=$(CURDIR)/debian/tmp install |
To extract the content of a .deb file:
ar -x <name>.deb |
To create a Debianized source tree from a Debian source package:
dpkg-source -x <name>.dsc |
To generate a .deb from within a Debianized source tree:
dpkg-buildpackage -b -uc -us |
To generate the .dsc and the .diff.gz run:
dpkg-source -b <dir> |
on the untarred Debianized source tree. Remember to leave the renamed original source tarball of the package in the parent directory.
Don't forget to sign the source package:
debsign -m<maintainer> <name>.dsc |
irc.oftc.net, channel #debian-mentors
Renderer (Layout Engine)
Pango
Create and submit Pango patch - don't implicitly create conjuncts
Inform bengalinux-core team the implications of the fix to Pango Bug 145233 – Zero-width non-joiner is displayed (http://bugzilla.gnome.org/show_bug.cgi?id=145233).
Create and submit Pango patch - Enable touching letters in Sinhala rendering
Create and submit Pango patch - Worst case expansion for Sinhala
Create and submit Pango patch - Cursor positioning for Sinhala is broken
ICU
Provide test files and images for ICU Sinhala support
Convince Pango and ICU maintainers to emit ZWJ to the font lookup stage.
ICU: ZWJ Processing in Sinhala / Implement PR 37 ZWJ/ZWNJ Behavior
Find ICU bug - Indic Reordering State Table Allows ZWJ Virama ZWJ
Port Pango patches, which add Sinhala support, to ICU for immediate use in ICU 3.4
Create and submit ICU patch - SF_MPRE_FIXUP causing conjuncts to be split in Sinhala
Open Office
Discovered Open Office was filtering ZWJ
Convince Open Office developers to stop filtering ZWJ and ZWNJ - ZWJ: The zero width joiner shouldn't be filtered out
Inform of Open Office filtering ZWJ and ZWNJ - Incorrect Bengali rendering of ra+japhala
Inform Debian of Open Office bug - The zero width joiner shouldn't be filtered out
Create and submit Epiphany patch - Add si (Sinhala) to the list of locales requiring Pango
Create and submit Ubuntu Firefox patch - Add si (Sinhala) to the list of locales requiring Pango
Input Methods
Submit XKB keyboard layout to X Keyboard Configuration Database
Submit vim keyboard layout to Bram
Submit XKB keyboard layout to xorg
Submit XKB keyboard layout to xfree86
Submit phonetic static keyboard layout to m17n
Test and provide feedback on the m17n Wijesekera input method.
Update XKB keyboard layout
Sync m17n-contrib from Debian to Ubuntu
Developed Sinhala Phonetic keyboard layouts and a Transliteration scheme.
Implemented phonetic and transliteration input methods for m17n.
Fonts
Discovered that the printing problem was due to the font being an OTF font. Once the LKLUG font was changed to a TTF, the printing problem disappeared.
Added a glyph for “Kunddaliya” to LKLUG font.
Reorganised glyphs containing “Repaya” and added corresponding lookups to LKLUG font.
Standards
Amended ISO639 to include 'Sinhala' in the languages list alongside 'Sinhalese'.
Reported errors in the SCCII - Part 1: Collation Sequence (SLS1134)
Collation
Add Sinhala script (Sri Lanka) collation to MySQL
Renderer (Layout Engine)
Support touching letters in QT Renderer.
Pango tries to display a 25CC, which is a dotted circle, when a dependent vowel is displayed without a consonant. However the glyph for 25CC does not appear.
see chapter 9 of TUS Unicode - apparently discusses ZWJ/ZWNJ.
What does Thunderbird use for rendering?
Is Firefox 3 filtering ZWJ?
Input Methods
See if XKB can be extended to allow multiple codepoints per keycode.
Fonts
Learn about OT features/order.
Discover why the appearance of the Unicode Sinhala fonts deteriorate at smaller sizes. Is it a smoothing or hinting issue in the font. Or an issue with the renderers.
Improve the range and correctness of the Unicode Sinhala section in the freefont.
Develop a standard lookup table for font developers.
Sorting
Implement established standards
String matching - a consonant followed by dependent vowel 'o' should not match the same consonant followed by dependent vowel 'oo'.
Can the DUCET be updated with the correct Sinhala collation sequence?
MySQL Sinhala locale needed.
Other GNU/Linux Infrastructure
Add Sinhala to /usr/include/X11/keysymdef.h
OTF printing problem.
Printing
Firefox & Thunderbird can't print Sinhala.
Misc
Submit corrections to Unicode. e.g. aae Vs aee.
UTF-8 should be declared the standard file encoding.
Develop Sinhala IPA transliteration for documents
Develop Sinhala literary transliteration for documents
English Locale for Sri Lanka
We have made significant progress in providing Sinhala support in GNU/Linux. There are still areas that require attention before GNU/Linux can be deployed more widely in Sri Lanka.