加载大型翻译内存交换（TMX）文件

2024-09-28 03:15:26 发布

您现在位置：Python中文网/ 问答频道 /正文
3143 3

如何在不破坏内存的情况下加载TMX文件

在ParaCrawl datasets中加载时，我希望使用TMX格式。为了加载TMX文件，我一直在使用这个[SO post] （TMX(Translation Memory eXchange) files in python）

然而，对于像瑞典语到英语这样的语言，简单地加载数据集需要66GB的内存，根据数据集的大小，我预测从西班牙语到英语加载几乎需要500GB的内存。最大的AWS实例是128 GB内存，看起来我做错了什么


Tags： 文件数据内存insoexchange格式情况
1条回答
网友
1楼 · 发布于 2024-09-28 03:15:26

使用XML parser with streaming似乎效果很好。TMX文件中有3个级别<tmx><body><tu>

import xmltodict
from gzip import GzipFile


language_pairs = []

def get_lang_pair(_, tree):
    lang_pair = {}
    for elem in tree['tuv']:
        language = elem['@xml:lang']
        text = elem['seg']
        lang_pair[language] = text

    language_pairs.append(lang_pair)
    return True

# <tmx>
# <body>
# <tu>
xmltodict.parse(
    GzipFile('en-da.tmx.gz'),
    item_depth=3, item_callback=get_lang_pair,
)

输出：

[{'en': 'Galaxy S9 price: $799.99 full price or $33.33/month for 24 months with 0 percent APR Galaxy S9 Plus price: $929.99 full price or $38.74/month for 24 months with 0 percent APR Pre-order launch date: 1 March, 9:01 p.m Pacific Launch date: 16 March Trade-in promo: Up to $350 credit (iPhone X, 8, 8 Plus; Samsung GS8, GS8+, Note8) Other promos: $150 prepaid MasterCard for porting in a number, $10 off accessories for Verizon Up customers, 50 percent off Samsung tablet with 2-year agreement, $100 off Gear S3, and Samsung fast charge power bundle for $99.99',
  'da': 'Galaxy S9 pris: $799.99 fuld pris eller $33.33/måned i 24 måneder med 0% p.a Galaxy S9 Plus pris: $929.99 fuld pris eller $38.74/måned i 24 måneder med 0% p.a Forud for lanceringen dato: 1 Marts, 9:01 p.m Pacific Lanceringen dato: 16 Marts Handel-promo: Op til $350 kredit (iPhone X, 8, 8, Plus, Samsung GS8, GS8+, Note8) Andre promotions: $150 prepaid MasterCard for portering af et nummer, $10 off tilbehør til Verizon kunder Op, 50% off Samsung tablet med 2-årig aftale, $100 off Gear S3, og Samsung hurtig opladning power bundle for $99.99'},
 {'en': 'Room facilities: Air conditioning, Bath or Shower, Cable Channels, Desk, Dining table, Entire unit located on ground floor, Fan, Flat-screen TV, Free toiletries, Heating, Linen, Microwave, Private bathroom, Private entrance, Refrigerator, Seating Area, Shower, TV, Tile/Marble floor, Toilet, Toilet paper, Towels Max persons: 3 Size of the room: 13.94 m² Beds: 1x Large bed(s) (King size) bed Rate: US$ 55.71 ...',
  'da': 'Værelses faciliteter: Aircondition, Badekar eller bruser, Bruser, Egen indgang, Eget badeværelse, Fjernsyn, Fladskærms-tv, Flise-/marmorgulv, Gratis toiletartikler, Hele boligen er beliggende i stueetagen, Håndklæder, Kabelkanaler, Køleskab, Loftvifte, Mikrobølgeovn, Sengelinned, Siddekrog, Skrivebord, Spisebord, Toilet, Toiletpapir , Varme Max antal personer: 3 Størrelsen af \u200b\u200brummet: 13.94 m² Senge: 1x Large bed(s) (King size) bed Pris: US$ 55.71 ...'},

相关问题 更多 >

    编程相关推荐

    热门问题