Eksport unidecode国际字符ascii等价物数据库

Question

如何从unidecodepython模块导出数据以在另一种语言中使用？你知道吗

此模块将unicode字符转换为拉丁（ascii）字符，大致保留如下语音含义：

kožušček => kozuscek
北亰 -> Bei Jing
Москва -> Moskva

例如，这对于为国际网页创建URL-s非常有用。还有其他语言的端口，比如UnidecodeSharp，但是质量不是很好。你知道吗

Answer 1

下面是一个Python程序unidecode_sqlite.py，用于将unidecode数据导出到SQLite数据库，它可以用在每种主要语言中：

#!/usr/bin/env python

'''Export unidecode data to SQLite'''

from __future__ import print_function, unicode_literals

import inspect
import os, sys, re
import sqlite3
import unidecode, unicodedata

def unidecode_sqlite(filename):
    '''Export unidecode data to filename'''

    if os.path.exists(filename):
        raise RuntimeError('File exists: %s' % filename)

    conn = sqlite3.connect(filename)
    conn.execute(
        '''create table if not exists unidecode (
            c text primary key,
            category text not null,
            ascii text not null
        )'''
    )

    unidecode_path = os.path.dirname(inspect.getfile(unidecode))

    # Python 2 compatibility
    if 'unichr' in dir(__builtins__):
        unichr_ = unichr
    else:
        unichr_ = chr

    for filename in sorted(os.listdir(unidecode_path)):
        if not os.path.isfile(os.path.join(unidecode_path, filename)):
            continue
        filename_match = re.match(
            r'^x([0-9a-f]{3})\.py$',
            filename,
            re.IGNORECASE
        )
        if not filename_match:
            continue
        section = filename_match.group(1)
        section_start = int("0x"+section, 0)*0x100
        for char_position in range(0x100):
            character = unichr_(section_start+char_position)
            unidecoded_character = unidecode.unidecode(character)
            if unidecoded_character is None or unidecoded_character == '[?]':
                continue
            conn.execute(
                '''insert into unidecode (c, category, ascii)
                    values (?,?,?)''',
                (
                    character,
                    unicodedata.category(character),
                    unidecoded_character
                )
            )
    conn.commit()
    conn.execute('vacuum')

if __name__ == "__main__":
    if len(sys.argv) != 2:
        print('USAGE: %s FILE' % sys.argv[0])
        sys.exit(0)

    try:
        unidecode_sqlite(sys.argv[1])
    except (OSError, RuntimeError) as error:
        print('ERROR: %s' % error, file=sys.stderr)
        sys.exit(1)

它可以像这样在任何使用python的计算机上使用（2或3，我不确定Windows），并创建1,3MB文件：

virtualenv venv
venv/bin/pip install unidecode
venv/bin/python unidecode_sqlite.py unidecode.sqlite