RFC2279 日本語訳
2279 UTF-8, a transformation format of ISO 10646. F. Yergeau. January 1998. (Format: TXT=21634 bytes) (Obsoletes RFC2044) (Obsoleted by RFC3629) (Status: DRAFT STANDARD)
プログラムでの自動翻訳です。
RFC一覧
英語原文
Network Working Group F. Yergeau Request for Comments: 2279 Alis Technologies Obsoletes: 2044 January 1998 Category: Standards Track
UTF-8, a transformation format of ISO 10646
Status of this Memo
This document specifies an Internet standards track protocol for the Internet community, and requests discussion and suggestions for improvements. Please refer to the current edition of the "Internet Official Protocol Standards" (STD 1) for the standardization state and status of this protocol. Distribution of this memo is unlimited.
Copyright Notice
Copyright (C) The Internet Society (1998). All Rights Reserved.
Abstract
ISO/IEC 10646-1 defines a multi-octet character set called the Universal Character Set (UCS) which encompasses most of the world's writing systems. Multi-octet characters, however, are not compatible with many current applications and protocols, and this has led to the development of a few so-called UCS transformation formats (UTF), each with different characteristics. UTF-8, the object of this memo, has the characteristic of preserving the full US-ASCII range, providing compatibility with file systems, parsers and other software that rely on US-ASCII values but are transparent to other values. This memo updates and replaces RFC 2044, in particular addressing the question of versions of the relevant standards.
1. Introduction
ISO/IEC 10646-1 [ISO-10646] defines a multi-octet character set called the Universal Character Set (UCS), which encompasses most of the world's writing systems. Two multi-octet encodings are defined, a four-octet per character encoding called UCS-4 and a two-octet per character encoding called UCS-2, able to address only the first 64K characters of the UCS (the Basic Multilingual Plane, BMP), outside of which there are currently no assignments.
It is noteworthy that the same set of characters is defined by the Unicode standard [UNICODE], which further defines additional character properties and other application details of great interest to implementors, but does not have the UCS-4 encoding. Up to the
Yergeau Standards Track [Page 1] RFC 2279 UTF-8 January 1998
present time, changes in Unicode and amendments to ISO/IEC 10646 have tracked each other, so that the character repertoires and code point assignments have remained in sync. The relevant standardization committees have committed to maintain this very useful synchronism.
The UCS-2 and UCS-4 encodings, however, are hard to use in many current applications and protocols that assume 8 or even 7 bit characters. Even newer systems able to deal with 16 bit characters cannot process UCS-4 data. This situation has led to the development of so-called UCS transformation formats (UTF), each with different characteristics.
UTF-1 has only historical interest, having been removed from ISO/IEC 10646. UTF-7 has the quality of encoding the full BMP repertoire using only octets with the high-order bit clear (7 bit US-ASCII values, [US-ASCII]), and is thus deemed a mail-safe encoding ([RFC2152]). UTF-8, the object of this memo, uses all bits of an octet, but has the quality of preserving the full US-ASCII range: US-ASCII characters are encoded in one octet having the normal US- ASCII value, and any octet with such a value can only stand for an US-ASCII character, and nothing else.
UTF-16 is a scheme for transforming a subset of the UCS-4 repertoire into pairs of UCS-2 values from a reserved range. UTF-16 impacts UTF-8 in that UCS-2 values from the reserved range must be treated specially in the UTF-8 transformation.
UTF-8 encodes UCS-2 or UCS-4 characters as a varying number of octets, where the number of octets, and the value of each, depend on the integer value assigned to the character in ISO/IEC 10646. This transformation format has the following characteristics (all values are in hexadecimal):
- Character values from 0000 0000 to 0000 007F (US-ASCII repertoire) correspond to octets 00 to 7F (7 bit US-ASCII values). A direct consequence is that a plain ASCII string is also a valid UTF-8 string.
- US-ASCII values do not appear otherwise in a UTF-8 encoded character stream. This provides compatibility with file systems or other software (e.g. the printf() function in C libraries) that parse based on US-ASCII values but are transparent to other values.
- Round-trip conversion is easy between UTF-8 and either of UCS-4, UCS-2.
Yergeau Standards Track [Page 2] RFC 2279 UTF-8 January 1998
- The first octet of a multi-octet sequence indicates the number of octets in the sequence.
- The octet values FE and FF never appear.
- Character boundaries are easily found from anywhere in an octet stream.
- The lexicographic sorting order of UCS-4 strings is preserved. Of course this is of limited interest since the sort order is not culturally valid in either case.
- The Boyer-Moore fast search algorithm can be used with UTF-8 data.
- UTF-8 strings can be fairly reliably recognized as such by a simple algorithm, i.e. the probability that a string of characters in any other encoding appears as valid UTF-8 is low, diminishing with increasing string length.
UTF-8 was originally a project of the X/Open Joint Internationalization Group XOJIG with the objective to specify a File System Safe UCS Transformation Format [FSS-UTF] that is compatible with UNIX systems, supporting multilingual text in a single encoding. The original authors were Gary Miller, Greger Leijonhufvud and John Entenmann. Later, Ken Thompson and Rob Pike did significant work for the formal UTF-8.
A description can also be found in Unicode Technical Report #4 and in the Unicode Standard, version 2.0 [UNICODE]. The definitive reference, including provisions for UTF-16 data within UTF-8, is Annex R of ISO/IEC 10646-1 [ISO-10646].
2. UTF-8 definition
In UTF-8, characters are encoded using sequences of 1 to 6 octets. The only octet of a "sequence" of one has the higher-order bit set to 0, the remaining 7 bits being used to encode the character value. In a sequence of n octets, n>1, the initial octet has the n higher-order bits set to 1, followed by a bit set to 0. The remaining bit(s) of that octet contain bits from the value of the character to be encoded. The following octet(s) all have the higher-order bit set to 1 and the following bit set to 0, leaving 6 bits in each to contain bits from the character to be encoded.
The table below summarizes the format of these different octet types. The letter x indicates bits available for encoding bits of the UCS-4 character value.
Yergeau Standards Track [Page 3] RFC 2279 UTF-8 January 1998
UCS-4 range (hex.) UTF-8 octet sequence (binary) 0000 0000-0000 007F 0xxxxxxx 0000 0080-0000 07FF 110xxxxx 10xxxxxx 0000 0800-0000 FFFF 1110xxxx 10xxxxxx 10xxxxxx
0001 0000-001F FFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx 0020 0000-03FF FFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 0400 0000-7FFF FFFF 1111110x 10xxxxxx ... 10xxxxxx
Encoding from UCS-4 to UTF-8 proceeds as follows:
1) Determine the number of octets required from the character value and the first column of the table above. It is important to note that the rows of the table are mutually exclusive, i.e. there is only one valid way to encode a given UCS-4 character.
2) Prepare the high-order bits of the octets as per the second column of the table.
3) Fill in the bits marked x from the bits of the character value, starting from the lower-order bits of the character value and putting them first in the last octet of the sequence, then the next to last, etc. until all x bits are filled in.
The algorithm for encoding UCS-2 (or Unicode) to UTF-8 can be obtained from the above, in principle, by simply extending each UCS-2 character with two zero-valued octets. However, pairs of UCS-2 values between D800 and DFFF (surrogate pairs in Unicode parlance), being actually UCS-4 characters transformed through UTF-16, need special treatment: the UTF-16 transformation must be undone, yielding a UCS-4 character that is then transformed as above.
Decoding from UTF-8 to UCS-4 proceeds as follows:
1) Initialize the 4 octets of the UCS-4 character with all bits set to 0.
2) Determine which bits encode the character value from the number of octets in the sequence and the second column of the table above (the bits marked x).
3) Distribute the bits from the sequence to the UCS-4 character, first the lower-order bits from the last octet of the sequence and proceeding to the left until no x bits are left.
If the UTF-8 sequence is no more than three octets long, decoding can proceed directly to UCS-2.
Yergeau Standards Track [Page 4] RFC 2279 UTF-8 January 1998
NOTE -- actual implementations of the decoding algorithm above should protect against decoding invalid sequences. For instance, a naive implementation may (wrongly) decode the invalid UTF-8 sequence C0 80 into the character U+0000, which may have security consequences and/or cause other problems. See the Security Considerations section below.
A more detailed algorithm and formulae can be found in [FSS_UTF], [UNICODE] or Annex R to [ISO-10646].
3. Versions of the standards
ISO/IEC 10646 is updated from time to time by published amendments; similarly, different versions of the Unicode standard exist: 1.0, 1.1 and 2.0 as of this writing. Each new version obsoletes and replaces the previous one, but implementations, and more significantly data, are not updated instantly.
In general, the changes amount to adding new characters, which does not pose particular problems with old data. Amendment 5 to ISO/IEC 10646, however, has moved and expanded the Korean Hangul block, thereby making any previous data containing Hangul characters invalid under the new version. Unicode 2.0 has the same difference from Unicode 1.1. The official justification for allowing such an incompatible change was that no implementations and no data containing Hangul existed, a statement that is likely to be true but remains unprovable. The incident has been dubbed the "Korean mess", and the relevant committees have pledged to never, ever again make such an incompatible change.
New versions, and in particular any incompatible changes, have q conseuences regarding MIME character encoding labels, to be discussed in section 5.
4. Examples
The UCS-2 sequence "A<NOT IDENTICAL TO><ALPHA>." (0041, 2262, 0391, 002E) may be encoded in UTF-8 as follows:
41 E2 89 A2 CE 91 2E
The UCS-2 sequence representing the Hangul characters for the Korean word "hangugo" (D55C, AD6D, C5B4) may be encoded as follows:
ED 95 9C EA B5 AD EC 96 B4
Yergeau Standards Track [Page 5] RFC 2279 UTF-8 January 1998
The UCS-2 sequence representing the Han characters for the Japanese word "nihongo" (65E5, 672C, 8A9E) may be encoded as follows:
E6 97 A5 E6 9C AC E8 AA 9E
5. MIME registration
This memo is meant to serve as the basis for registration of a MIME character set parameter (charset) [CHARSET-REG]. The proposed charset parameter value is "UTF-8". This string labels media types containing text consisting of characters from the repertoire of ISO/IEC 10646 including all amendments at least up to amendment 5 (Korean block), encoded to a sequence of octets using the encoding scheme outlined above. UTF-8 is suitable for use in MIME content types under the "text" top-level type.
It is noteworthy that the label "UTF-8" does not contain a version identification, referring generically to ISO/IEC 10646. This is intentional, the rationale being as follows:
A MIME charset label is designed to give just the information needed to interpret a sequence of bytes received on the wire into a sequence of characters, nothing more (see RFC 2045, section 2.2, in [MIME]). As long as a character set standard does not change incompatibly, version numbers serve no purpose, because one gains nothing by learning from the tag that newly assigned characters may be received that one doesn't know about. The tag itself doesn't teach anything about the new characters, which are going to be received anyway.
Hence, as long as the standards evolve compatibly, the apparent advantage of having labels that identify the versions is only that, apparent. But there is a disadvantage to such version-dependent labels: when an older application receives data accompanied by a newer, unknown label, it may fail to recognize the label and be completely unable to deal with the data, whereas a generic, known label would have triggered mostly correct processing of the data, which may well not contain any new characters.
Now the "Korean mess" (ISO/IEC 10646 amendment 5) is an incompatible change, in principle contradicting the appropriateness of a version independent MIME charset label as described above. But the compatibility problem can only appear with data containing Korean Hangul characters encoded according to Unicode 1.1 (or equivalently ISO/IEC 10646 before amendment 5), and there is arguably no such data to worry about, this being the very reason the incompatible change was deemed acceptable.
Yergeau Standards Track [Page 6] RFC 2279 UTF-8 January 1998
In practice, then, a version-independent label is warranted, provided the label is understood to refer to all versions after Amendment 5, and provided no incompatible change actually occurs. Should incompatible changes occur in a later version of ISO/IEC 10646, the MIME charset label defined here will stay aligned with the previous version until and unless the IETF specifically decides otherwise.
実際には、次に、バージョンインディペンデント・レーベルは保証されます、ラベルがAmendment5の後のすべてのバージョンを示すのが理解されて、どんな非互換な変化も実際に起こらないなら。 そして、非互換な変化がISO/IEC10646の後のバージョンに起こるはずであると、ここで定義されたMIME charsetラベルが旧バージョンで並べられた状態で残る、IETFが別の方法で明確に決めない場合。
It is also proposed to register the charset parameter value "UNICODE-1-1-UTF-8", for the exclusive purpose of labelling text data containing Hangul syllables encoded to UTF-8 without taking into account Amendment 5 of ISO/IEC 10646 (i.e. using the pre-amendment 5 code point assignments). Any other UTF-8 data SHOULD NOT use this label, in particular data not containing any Hangul syllables, and it is felt important to strongly recommend against creating any new Hangul-containing data without taking Amendment 5 of ISO/IEC 10646 into account.
また、それがcharsetパラメタ価値を示すために提案される、「ユニコード1-1UTF、8インチ、テキストをISO/IEC10646(すなわち、プレ修正の5コード・ポイントの課題を使用します)の修正5を考慮に入れないでUTF-8にコード化されたハングル音節を含むデータに分類する排他的な目的、」 いかなる他のUTF-8データSHOULD NOTもどんなハングル音節も含まない特定のデータでこのラベルを使用します、そして、それは、ISO/IEC10646のAmendment5をアカウントに連れて行かないでどんな新しいハングルを含むデータも作成しないように強く推薦するために重要であると感じられます。
6. Security Considerations
6. セキュリティ問題
Implementors of UTF-8 need to consider the security aspects of how they handle illegal UTF-8 sequences. It is conceivable that in some circumstances an attacker would be able to exploit an incautious UTF-8 parser by sending it an octet sequence that is not permitted by the UTF-8 syntax.
UTF-8の作成者は、セキュリティがそれらがどう不法なUTF-8系列を扱うかに関する局面であると考える必要があります。 いくつかの事情では、攻撃者がUTF-8構文で受入れられない八重奏系列をそれに送ることによって軽率なUTF-8パーサを利用できるだろうというのが想像できます。
A particularly subtle form of this attack could be carried out against a parser which performs security-critical validity checks against the UTF-8 encoded form of its input, but interprets certain illegal octet sequences as characters. For example, a parser might prohibit the NUL character when encoded as the single-octet sequence 00, but allow the illegal two-octet sequence C0 80 and interpret it as a NUL character. Another example might be a parser which prohibits the octet sequence 2F 2E 2E 2F ("/../"), yet permits the illegal octet sequence 2F C0 AE 2E 2F.
この攻撃の特に微妙なフォームはパーサに対して行われて、どれがUTF-8に対してセキュリティ批判的なバリディティチェックを実行するかが、入力のフォームをコード化しますが、キャラクタとしてある不法な八重奏系列を解釈するということであるかもしれません。 例えば、ただ一つの八重奏系列00としてコード化されると、パーサはNULキャラクタを禁止するかもしれません、不法な2八重奏に系列C0 80を許容して、NULキャラクタとしてそれを解釈するのを除いて。 別の例は2八重奏系列2 2E E2F F(「//」)禁止しますが、まだ不法な八重奏系列2F C0 AEを2 2E F可能にしているパーサであるかもしれません。
Acknowledgments
承認
The following have participated in the drafting and discussion of this memo:
以下はこのメモの草稿と議論に参加しました:
James E. Agenbroad Andries Brouwer Martin J. D|rst Ned Freed David Goldsmith Edwin F. Hart Kent Karlsson Markus Kuhn Michael Kung Alain LaBonte John Gardiner Myers Murray Sargent Keld Simonsen Arnold Winkler
ジェームスE.Agenbroad AndriesブルーエルマーチンJ.D|rstネッド・フリード・デヴィッド・ゴールドスミス・エドウィン・F.ハート・ケントカールソン・マーカス・キューン・マイケル・キュング・アラン・LaBonteジョン・ガーディナー・マイアーズマレーサージェント・Keldシモンセン・アーノルド・ウィンクラー
Yergeau Standards Track [Page 7] RFC 2279 UTF-8 January 1998
Yergeau標準化過程[7ページ]RFC2279UTF-1998年1月8日
Bibliography
図書目録
[CHARSET-REG] Freed, N., and J. Postel, "IANA Charset Registration Procedures", BCP 19, RFC 2278, January 1998.
[CHARSET-REG] フリード、N.、およびJ.ポステル、「IANA Charset登録手順」、BCP19、RFC2278、1月1998。
[FSS_UTF] X/Open CAE Specification C501 ISBN 1-85912-082-2 28cm. 22p. pbk. 172g. 4/95, X/Open Company Ltd., "File System Safe UCS Transformation Format (FSS_UTF)", X/Open Preleminary Specification, Document Number P316. Also published in Unicode Technical Report #4.
[FSS_UTF] X/Open CAE仕様C501 ISBN1-85912-082-2 28cm。 22p. pbk。 172g。 4/95(X/Open会社株式会社、「ファイルのシステムの安全なUCS変化形式(FSS_UTF)」、X/Open Preleminary仕様)は数のP316を記録します。 また、ユニコードTechnical Report#4で、発行されています。
[ISO-10646] ISO/IEC 10646-1:1993. International Standard -- Information technology -- Universal Multiple-Octet Coded Character Set (UCS) -- Part 1: Architecture and Basic Multilingual Plane. Five amendments and a technical corrigendum have been published up to now. UTF-8 is described in Annex R, published as Amendment 2. UTF-16 is described in Annex Q, published as Amendment 1. 17 other amendments are currently at various stages of standardization.
[ISO-10646]ISO/IEC10646-1:1993。 国際規格--情報技術--普遍的なMultiple-八重奏Coded文字コード(UCS)--第1部: 構造と基本多言語水準。 5つの修正と技術的な間違いはこれまで発行されました。 UTF-8はAmendment2として発行されたAnnex Rで説明されます。 UTF-16はAmendment1として発行されたAnnex Qで説明されます。 17 現在、様々なステージの標準化には他の修正があります。
[MIME] Freed, N., and N. Borenstein, "Multipurpose Internet Mail Extensions (MIME) Part One: Format of Internet Message Bodies", RFC 2045. N. Freed, N. Borenstein, "Multipurpose Internet Mail Extensions (MIME) Part Two: Media Types", RFC 2046. K. Moore, "MIME (Multipurpose Internet Mail Extensions) Part Three: Message Header Extensions for Non-ASCII Text", RFC 2047. N. Freed, J. Klensin, J. Postel, "Multipurpose Internet Mail Extensions (MIME) Part Four: Registration Procedures", RFC 2048. N. Freed, N. Borenstein, " Multipurpose Internet Mail Extensions (MIME) Part Five: Conformance Criteria and Examples", RFC 2049. All November 1996.
解放された[MIME]、N.、およびN.Borenstein、「マルチパーパスインターネットメールエクステンション(MIME)は1つを分けます」。 「インターネットメッセージ本体の形式」、RFC2045。 N。 解放されたN.Borenstein、「マルチパーパスインターネットメールエクステンション(MIME)は2を分けます」。 「メディアタイプ」、RFC2046。 K。 ムーア、「パートThreeをまねてください(マルチパーパスインターネットメールエクステンション)」 「非ASCIIテキストのためのメッセージヘッダー拡張子」、RFC2047。 N。 解放されたJ.Klensin、J.ポステル、「マルチパーパスインターネットメールエクステンション(MIME)は4を分けます」。 「登録手順」、RFC2048。 N。 解放されたN.Borenstein、「マルチパーパスインターネットメールエクステンション(MIME)は5を分けます」。 「順応評価基準と例」、RFC2049 すべての1996年11月。
[RFC2152] Goldsmith, D., and M. Davis, "UTF-7: A Mail-safe Transformation Format of Unicode", RFC 1642, Taligent inc., May 1997. (Obsoletes RFC1642)
[RFC2152] ゴールドスミス、D.、およびM.デイヴィス、「UTF-7:」 「ユニコードのメール安全なTransformation Format」、RFC1642、Taligent inc、5月1997日 (RFC1642を時代遅れにします)
[UNICODE] The Unicode Consortium, "The Unicode Standard -- Version 2.0", Addison-Wesley, 1996.
[ユニコード] ユニコード共同体、「バージョン2インチ、アディソン・ウエスリー、ユニコード規格--1996。」
[US-ASCII] Coded Character Set--7-bit American Standard Code for Information Interchange, ANSI X3.4-1986.
[米国-ASCII]は文字コードをコード化しました--7ビットの情報交換用米国標準コード、ANSI X3.4-1986。
Yergeau Standards Track [Page 8] RFC 2279 UTF-8 January 1998
Yergeau標準化過程[8ページ]RFC2279UTF-1998年1月8日
Author's Address
作者のアドレス
Francois Yergeau Alis Technologies 100, boul. Alexis-Nihon Suite 600 Montreal QC H4M 2P2 Canada
フランソアYergeau Alis Technologies100、boul。 アレックサス-日本スイート600モントリオールQC H4M 2P2カナダ
Phone: +1 (514) 747-2547 Fax: +1 (514) 747-2561 EMail: fyergeau@alis.com
以下に電話をしてください。 +1 (514) 747-2547Fax: +1 (514) 747-2561 メールしてください: fyergeau@alis.com
Yergeau Standards Track [Page 9] RFC 2279 UTF-8 January 1998
Yergeau標準化過程[9ページ]RFC2279UTF-1998年1月8日
Full Copyright Statement
完全な著作権宣言文
Copyright (C) The Internet Society (1998). All Rights Reserved.
Copyright(C)インターネット協会(1998)。 All rights reserved。
This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English.
それに関するこのドキュメントと翻訳は、コピーして、それが批評するか、またはそうでなければわかる他のもの、および派生している作品に提供するか、または準備されているかもしれなくて、コピーされて、発行されて、全体か一部広げられた実現を助けるかもしれません、どんな種類の制限なしでも、上の版権情報とこのパラグラフがそのようなすべてのコピーと派生している作品の上に含まれていれば。 しかしながら、このドキュメント自体は何らかの方法で変更されないかもしれません、インターネット協会か他のインターネット組織の版権情報か参照を取り除くのなどように、それを英語以外の言語に翻訳するのが著作権のための手順がインターネットStandardsの過程で定義したどのケースに従わなければならないか、必要に応じてさもなければ、インターネット標準を開発する目的に必要であるのを除いて。
The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns.
上に承諾された限られた許容は、永久であり、インターネット協会、後継者または案配によって取り消されないでしょう。
This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
このドキュメントとそして、「そのままで」という基礎とインターネットの振興発展を目的とする組織に、インターネット・エンジニアリング・タスク・フォースが速達の、または、暗示しているすべての保証を放棄するかどうかというここにことであり、他を含んでいて、含まれて、情報の使用がここに侵害しないどんな保証も少しもまっすぐになるという情報か市場性か特定目的への適合性のどんな黙示的な保証。
Yergeau Standards Track [Page 10]
Yergeau標準化過程[10ページ]
一覧
スポンサーリンク