Jump to content

Extended ASCII: Difference between revisions

From Wikipedia, the free encyclopedia
Content deleted Content added
CP 1252: Add ref. Perhaps the text needs to be updated to better reflect what the source says.
No edit summary
 
(33 intermediate revisions by 10 users not shown)
Line 1: Line 1:
{{Short description|Nick-name for 8-bit ASCII-derived character sets}}
{{Short description|Nickname for 8-bit ASCII-derived character sets}}
[[File:Table ascii extended.png|thumb|Output of the program {{mono|ascii}} in [[Cygwin]]]]
[[File:Table ascii extended.png|thumb|Output of the program {{mono|ascii}} in [[Cygwin]]]]
'''Extended ASCII''' is a repertoire of [[character encoding]]s that include (most of) the original 96 [[ASCII]] character set, plus up to 128 additional characters. There is no formal definition of "extended ASCII", and even use of the term is sometimes criticized,<ref>{{cite mailing list |url=http://cygwin.com/ml/cygwin/2001-02/msg01526.html |title=Re: Cygwin Termcap information involving extended ascii charicters |author=Benjamin Riefenstahl |mailing-list=cygwin |date=26 Feb 2001 |access-date=2 December 2012 |archive-url=https://web.archive.org/web/20130711002757/http://cygwin.com/ml/cygwin/2001-02/msg01526.html |archive-date=11 July 2013 |url-status=live }}</ref><ref>{{cite web |url=https://community.oracle.com/tech/apps-infra/discussion/comment/10231529/#Comment_10231529 |title=Print Extended ASCII Codes in sql*plus |author=S. Wolicki |date=Mar 23, 2012 |access-date=May 17, 2022}}</ref><ref>{{cite newsgroup |url=https://groups.google.com/g/comp.editors/c/PgFq_AuW4EI/m/Pa54Lv4nX4cJ |newsgroup=comp.editors |title=vim: how to type extended-ascii? |author=Mark J. Reed |date=March 28, 2004 |access-date=May 17, 2022}}</ref> because it can be mistakenly interpreted to mean that the [[American National Standards Institute]] (ANSI) had updated its {{nobr|[[ANSI X3.4-1986]]}} standard to include more characters, or that the term identifies a single unambiguous encoding, neither of which is the case.
'''Extended ASCII''' is a repertoire of [[character encoding]]s that include (most of) the original 96 [[ASCII]] character set, plus up to 128 additional characters. There is no formal definition of "extended ASCII", and even use of the term is sometimes criticized,<ref>{{cite mailing list |url=http://cygwin.com/ml/cygwin/2001-02/msg01526.html |title=Re: Cygwin Termcap information involving extended ascii charicters |author=Benjamin Riefenstahl |mailing-list=cygwin |date=26 Feb 2001 |access-date=2 December 2012 |archive-url=https://web.archive.org/web/20130711002757/http://cygwin.com/ml/cygwin/2001-02/msg01526.html |archive-date=11 July 2013 |url-status=live }}</ref><ref>{{cite web |url=https://community.oracle.com/tech/apps-infra/discussion/comment/10231529/#Comment_10231529 |title=Print Extended ASCII Codes in sql*plus |author=S. Wolicki |date=Mar 23, 2012 |access-date=May 17, 2022}}</ref><ref>{{cite newsgroup |url=https://groups.google.com/g/comp.editors/c/PgFq_AuW4EI/m/Pa54Lv4nX4cJ |newsgroup=comp.editors |title=vim: how to type extended-ascii? |author=Mark J. Reed |date=March 28, 2004 |access-date=May 17, 2022}}</ref> because it can be mistakenly interpreted to mean that the [[American National Standards Institute]] (ANSI) had updated its {{nobr|[[ANSI X3.4-1986]]}} standard to include more characters, or that the term identifies a single unambiguous encoding, neither of which is the case.
Line 5: Line 5:
The ISO standard [[ISO 8859]] was the first international standard to formalise a (limited) expansion of the ASCII character set: of the many language variants it encoded, [[ISO 8859-1]] ("ISO Latin 1"){{snd}}which supports most Western European languages{{snd}} is best known in the West. There are many other extended ASCII encodings (more than 220 DOS and Windows [[codepage]]s). [[EBCDIC]] ("the other" major character code) likewise developed many extended variants (more than 186 EBCDIC codepages) over the decades.
The ISO standard [[ISO 8859]] was the first international standard to formalise a (limited) expansion of the ASCII character set: of the many language variants it encoded, [[ISO 8859-1]] ("ISO Latin 1"){{snd}}which supports most Western European languages{{snd}} is best known in the West. There are many other extended ASCII encodings (more than 220 DOS and Windows [[codepage]]s). [[EBCDIC]] ("the other" major character code) likewise developed many extended variants (more than 186 EBCDIC codepages) over the decades.


All modern [[operating system]]s use [[Unicode]] which supports thousands of characters. However, extended ASCII remains important in the [[history of computing]].
All modern [[operating system]]s use [[Unicode]] which supports thousands of characters. However, extended ASCII remains important in the [[history of computing]], and supporting multiple extended ASCII character sets required software to be written in ways that made it much easier to support the [[UTF-8]] encoding method later on.


==History==
==History==
Line 15: Line 15:
The ASCII character set is barely large enough for US English use and lacks many glyphs common in [[typesetting]], and far too small for universal use. Many more letters and symbols are desirable, useful, or required to directly represent letters of alphabets other than English, more kinds of punctuation and spacing, more mathematical operators and symbols (× ÷ ⋅ ≠ ≥ ≈ π etc.), some unique symbols used by some programming languages, [[ideogram]]s, [[logogram]]s, box-drawing characters, etc.
The ASCII character set is barely large enough for US English use and lacks many glyphs common in [[typesetting]], and far too small for universal use. Many more letters and symbols are desirable, useful, or required to directly represent letters of alphabets other than English, more kinds of punctuation and spacing, more mathematical operators and symbols (× ÷ ⋅ ≠ ≥ ≈ π etc.), some unique symbols used by some programming languages, [[ideogram]]s, [[logogram]]s, box-drawing characters, etc.


The biggest problem for computer users around the world was other alphabets. ASCII's English alphabet almost accommodates European languages, if accented letters are replaced by non-accented letters or two-character approximations such as {{char|ss}} for {{char|ß}}. Modified variants of 7-bit ASCII appeared promptly, trading some lesser-used symbols for highly desired symbols or letters, such as replacing "#" with "£" on UK Teletypes, "\" with "¥" in Japan or "₩" in Korea, etc. At least 29 variant sets resulted. 12 code points were modified by at least one modified set, leaving only [[ISO/IEC 646|82 "invariant" codes]]. Programming languages however had assigned meaning to many of the replaced characters, work-arounds were devised such as C [[digraphs and trigraphs|three-character sequences]] "??<" and "??>" to represent "{" and "}".<ref>{{cite book|section-url=https://www.lysator.liu.se/c/rat/b.html#2-2-1-1|section=2.2.1.1 Trigraph sequences|url=https://www.lysator.liu.se/c/rat/title.html|title=Rationale for American National Standard for Information Systems - Programming Language - C|access-date=2019-02-08|archive-url=https://web.archive.org/web/20180929010444/http://www.lysator.liu.se/c/rat/title.html|archive-date=2018-09-29|url-status=live}}</ref> Languages with dissimilar basic alphabets could use transliteration, such as replacing all the Latin letters with the [[Volapük encoding|closest match Cyrillic letters]] (resulting in odd but somewhat readable text when English was printed in Cyrillic or vice versa). Schemes were also devised so that two letters could be overprinted (often with the [[backspace]] control between them) to produce accented letters. Users were not comfortable with any of these compromises and they were often poorly supported.{{Citation needed|date=August 2019}}
The biggest problem for computer users around the world was other alphabets. ASCII's English alphabet almost accommodates European languages, if accented letters are replaced by non-accented letters or two-character approximations such as {{char|ss}} for {{char|ß}}. Modified variants of 7-bit ASCII appeared promptly, trading some lesser-used symbols for highly desired symbols or letters, such as replacing "#" with "£" on UK Teletypes, "\" with "¥" in Japan or "₩" in Korea, etc. At least 29 variant sets resulted. 12 code points were modified by at least one modified set, leaving only [[ISO/IEC 646|82 "invariant" codes]]. Programming languages however had assigned meaning to many of the replaced characters, work-arounds were devised such as C [[digraphs and trigraphs (programming)#C|three-character sequences]] "??<" and "??>" to represent "{" and "}".<ref>{{cite book|section-url=https://www.lysator.liu.se/c/rat/b.html#2-2-1-1|section=2.2.1.1 Trigraph sequences|url=https://www.lysator.liu.se/c/rat/title.html|title=Rationale for American National Standard for Information Systems - Programming Language - C|access-date=2019-02-08|archive-url=https://web.archive.org/web/20180929010444/http://www.lysator.liu.se/c/rat/title.html|archive-date=2018-09-29|url-status=live}}</ref> Languages with dissimilar basic alphabets could use transliteration, such as replacing all the Latin letters with the closest match Cyrillic letters (resulting in odd but somewhat readable text when English was printed in Cyrillic or vice versa). Schemes were also devised so that two letters could be overprinted (often with the [[backspace]] control between them) to produce accented letters. Users were not comfortable with any of these compromises and they were often poorly supported.{{Citation needed|date=August 2019}}


When computers and peripherals standardized on eight-bit [[byte]]s in the 1970s, it became obvious that computers and software could handle text that uses 256-character sets at almost no additional cost in programming, and no additional cost for storage. (Assuming that the unused 8th bit of each byte was not reused in some way, such as error checking, Boolean fields, or packing 8 characters into 7 bytes.) This would allow ASCII to be used unchanged and provide 128 more characters. Many manufacturers devised 8-bit character sets consisting of ASCII plus up to 128 of the unused codes: encodings which covered all the more used Western European (and Latin American) languages, such as Danish, Dutch, French, German, Portuguese, Spanish, Swedish and more could be made.
When computers and peripherals standardized on eight-bit [[byte]]s in the 1970s, it became obvious that computers and software could handle text that uses 256-character sets at almost no additional cost in programming, and no additional cost for storage. (Assuming that the unused 8th bit of each byte was not reused in some way, such as error checking, Boolean fields, or packing 8 characters into 7 bytes.) This would allow ASCII to be used unchanged and provide 128 more characters. Many manufacturers devised 8-bit character sets consisting of ASCII plus up to 128 of the unused codes: encodings which covered all the more used Western European (and Latin American) languages, such as Danish, Dutch, French, German, Portuguese, Spanish, Swedish and more could be made.
Line 31: Line 31:
The [[TRS-80 character set]] for the [[TRS-80]] home computer added 64 [[semigraphics]] characters (0x80 through 0xBF) that implemented low-resolution block graphics. (Each block-graphic character displayed as a 2x3 grid of pixels, with each block pixel effectively controlled by one of the lower 6 bits.)<ref>{{cite web |author-first=Ira |author-last=Goldklang |date=2015 |title=Graphic Tips & Tricks |url=http://www.trs-80.com/wordpress/zaps-patches-pokes-tips/graphics/ |access-date=2017-07-29 |url-status=live |archive-url=https://web.archive.org/web/20170729202445/http://www.trs-80.com/trs80-zaps-hardware-model4.htm |archive-date=2017-07-29}}</ref>
The [[TRS-80 character set]] for the [[TRS-80]] home computer added 64 [[semigraphics]] characters (0x80 through 0xBF) that implemented low-resolution block graphics. (Each block-graphic character displayed as a 2x3 grid of pixels, with each block pixel effectively controlled by one of the lower 6 bits.)<ref>{{cite web |author-first=Ira |author-last=Goldklang |date=2015 |title=Graphic Tips & Tricks |url=http://www.trs-80.com/wordpress/zaps-patches-pokes-tips/graphics/ |access-date=2017-07-29 |url-status=live |archive-url=https://web.archive.org/web/20170729202445/http://www.trs-80.com/trs80-zaps-hardware-model4.htm |archive-date=2017-07-29}}</ref>


[[IBM]] introduced eight-bit extended ASCII codes on the original [[IBM PC]] and later produced variations for different languages and cultures. IBM called such character sets ''[[code pages]]'' and assigned numbers to both those they themselves invented as well as many invented and used by other manufacturers. Accordingly, character sets are very often indicated by their IBM code page number. In ASCII-compatible code pages, the lower 128 characters maintained their standard US-ASCII values, and different pages (or sets of characters) could be made available in the upper 128 characters. [[DOS]] computers built for the North American market, for example, used [[code page 437]], which included accented characters needed for French, German, and a few other European languages, as well as some graphical line-drawing characters. The larger character set made it possible to create documents in a combination of languages such as [[English language|English]] and [[French language|French]] (though French computers usually use [[code page 850]]), but not, for example, in English and [[Greek language|Greek]] (which required [[code page 737]]).
[[IBM]] introduced eight-bit extended ASCII codes on the original [[IBM PC]] and later produced variations for different languages and cultures. IBM called such character sets ''[[code pages]]'' and assigned numbers to both those they themselves invented as well as many invented and used by other manufacturers. Accordingly, character sets are very often indicated by their IBM code page number. In ASCII-compatible code pages, the lower 128 characters maintained their standard ASCII values, and different pages (or sets of characters) could be made available in the upper 128 characters. [[DOS]] computers built for the North American market, for example, used [[code page 437]], which included accented characters needed for French, German, and a few other European languages, as well as some graphical line-drawing characters. The larger character set made it possible to create documents in a combination of languages such as [[English language|English]] and [[French language|French]] (though French computers usually use [[code page 850]]), but not, for example, in English and [[Greek language|Greek]] (which required [[code page 737]]).


[[Apple Computer]] introduced their own eight-bit extended ASCII codes in [[Mac OS]], such as [[Mac OS Roman]]. The [[Apple LaserWriter]] also introduced the [[PostScript Standard Encoding|Postscript character set]].
[[Apple Computer]] introduced their own eight-bit extended ASCII codes in [[Mac OS]], such as [[Mac OS Roman]]. The [[Apple LaserWriter]] also introduced the [[PostScript Standard Encoding|Postscript character set]].
Line 38: Line 38:


==ISO 8859==
==ISO 8859==
{{Main|ISO/IEC 8859}}
Eventually, the members of the [[International Organization for Standardization]] (ISO) agreed a set of standards for eight-bit ASCII extensions, [[ISO 8859]]. The most popular of these was [[ISO 8859-1]] (also called "ISO Latin&nbsp;1") which contains characters sufficient for the most common Western European languages. Other standards in the 8859 group included [[ISO 8859-2]] for Eastern European languages using the [[Latin script]] and [[ISO 8859-5]] for languages using the [[Cyrillic script]], and others.
In 1987, the [[International Organization for Standardization]] (ISO) published a set of standards for eight-bit ASCII extensions, ISO 8859. The most popular of these was [[ISO 8859-1]] (also called "ISO Latin&nbsp;1") which contains characters sufficient for the most common Western European languages. Other standards in the 8859 group included [[ISO 8859-2]] for Eastern European languages using the [[Latin script]] and [[ISO 8859-5]] for languages using the [[Cyrillic script]], and others.


One notable way in which the ISO standards differ from some vendor-specific extended ASCII is that the 32 character positions 80<sub>16</sub> to 9F<sub>16</sub>, which correspond to the ASCII [[control character]]s with the high-order bit 'set', are reserved by ISO for control use and unused for printable characters (they are also reserved in Unicode<ref>{{cite book |title=The Unicode Standard, Version 15.1 |chapter=C1 Controls and Latin-1 Supplement {{!}} Range: 0080–00FF |publisher=[[Unicode Consortium]] | chapter-url=https://www.unicode.org/charts/PDF/U0080.pdf}}</ref>). This convention was almost universally ignored by other extended ASCII sets.
One notable way in which the ISO standards differ from some vendor-specific extended ASCII is that the 32 character positions 80<sub>16</sub> to 9F<sub>16</sub>, which correspond to the ASCII [[control character]]s with the high-order bit 'set', are reserved by ISO for control use and unused for printable characters (they are also reserved in Unicode<ref>{{cite book |title=The Unicode Standard, Version 15.1 |chapter=C1 Controls and Latin-1 Supplement {{!}} Range: 0080–00FF |publisher=[[Unicode Consortium]] | chapter-url=https://www.unicode.org/charts/PDF/U0080.pdf}}</ref>). This convention was almost universally ignored by other extended ASCII sets.


==Windows-1252==
ISO 8859-1 is the common 8-bit character encoding used by the [[X Window System]], and most [[Internet]] standards used it before [[Unicode]].
{{Main|Windows-1252}}


Microsoft intended to use ISO 8859 standards in Windows,{{cn|date=September 2023}} but soon replaced the unused C1 control characters with additional characters, making the proprietary Windows-1252 character set, which is sometimes mislabeled as ''ANSI''. The added characters included "curly" [[quotation mark]]s and other typographical elements like [[em dash]], the [[euro sign]], and letters missing from French and Finnish. This became the most-used extended ASCII in the world, and often is used on the web even when 8859-1 is specified.<ref>{{cite web |title=HTML Character Sets |website= W3 Schools |url=https://www.w3schools.com/charsets/ |quote=When a browser detects ISO-8859-1 it normally defaults to Windows-1252, because Windows-1252 has 32 more international characters.}}</ref><ref name="WHATWG">{{cite web |url=https://encoding.spec.whatwg.org/#names-and-labels |title=Encoding |at=sec. 5.2 Names and labels |publisher=[[WHATWG]] |date=27 January 2015 |access-date=4 February 2015 |archive-url=https://web.archive.org/web/20150204174315/https://encoding.spec.whatwg.org/#names-and-labels |archive-date=4 February 2015 |url-status=live}}</ref>
==CP 1252==
Microsoft intended to use ISO 8859 standards in Windows, but soon replaced the unused C1 control characters with characters sought by customers but not in the ISO set, thus making the proprietary [[code page 1252]]. Added characters included "curly" [[quotation mark]]s and other typographical elements like [[em dash]], the [[euro sign]], and letters missing from French and Finnish. Because of Microsoft's market dominance, this set quickly became the character set most used on the internet, even in text claiming to use ISO 8859-1.<ref>https://www.w3schools.com/charsets/</ref>


==Character set confusion==
==Character set confusion==
Line 55: Line 56:


Many [[communications protocol]]s, most importantly [[SMTP]] and [[HTTP]], require the character encoding of content to be tagged with [[Internet Assigned Numbers Authority|IANA]]-assigned character set identifiers.
Many [[communications protocol]]s, most importantly [[SMTP]] and [[HTTP]], require the character encoding of content to be tagged with [[Internet Assigned Numbers Authority|IANA]]-assigned character set identifiers.

==Usage in computer-readable languages==
{{original research|section|date=September 2023}}
For programming languages and document languages such as [[C (programming language)|C]] and [[HTML]], the principle of extended ASCII is important, since it enables many different encodings and therefore many human languages to be supported with little extra programming effort in the software that interprets the computer-readable language files. The languages can rely on all ASCII bytes (0x00 to 0x7F) have the same meaning in ''all'' variants of extended ASCII, and must not assign any meaning to the bytes with the high bit set, allowing them only in free-form text such as string constants and comments.

Before extended ASCII was supported, lots of software would mangle non-ASCII text, most often by removing the high bit. Supporting extended ASCII forced the compilers to be fixed to preserve the bytes in the source unchanged. This has been a benefit for Unicode, as it is relatively easy to support [[UTF-8]] in the same software.


==See also==
==See also==
*[[ASCII]]
* [[ASCII art]]
* [[Digraphs and trigraphs (programming)]]
*[[ASCII art]]
*[[Digraphs and trigraphs]]
* [[Input method]]
* [[List of Unicode characters]]
*[[Input method]]
* [[KOI-8]]
*[[List of Unicode characters]]
*[[UTF-8]]
*[[KOI-8]]


==References==
==References==
Line 75: Line 68:


==External links==
==External links==
*[http://czyborra.com/ Roman Czyborra's Unicode and extended ASCII information pages]
*[https://czyborra.com/ Roman Czyborra's Unicode and extended ASCII information pages]
*[http://www.cplusplus.com/doc/ascii/ A short page on ASCII, with the OEM 8-bit chart and the ANSI 8-bit chart]
*[https://www.cplusplus.com/doc/ascii/ A short page on ASCII, with the OEM 8-bit chart and the ANSI 8-bit chart]


{{character encoding}}
{{character encoding}}

Latest revision as of 23:39, 21 June 2024

Output of the program ascii in Cygwin

Extended ASCII is a repertoire of character encodings that include (most of) the original 96 ASCII character set, plus up to 128 additional characters. There is no formal definition of "extended ASCII", and even use of the term is sometimes criticized,[1][2][3] because it can be mistakenly interpreted to mean that the American National Standards Institute (ANSI) had updated its ANSI X3.4-1986 standard to include more characters, or that the term identifies a single unambiguous encoding, neither of which is the case.

The ISO standard ISO 8859 was the first international standard to formalise a (limited) expansion of the ASCII character set: of the many language variants it encoded, ISO 8859-1 ("ISO Latin 1") – which supports most Western European languages – is best known in the West. There are many other extended ASCII encodings (more than 220 DOS and Windows codepages). EBCDIC ("the other" major character code) likewise developed many extended variants (more than 186 EBCDIC codepages) over the decades.

All modern operating systems use Unicode which supports thousands of characters. However, extended ASCII remains important in the history of computing, and supporting multiple extended ASCII character sets required software to be written in ways that made it much easier to support the UTF-8 encoding method later on.

History

[edit]

ASCII was designed in the 1960s for teleprinters and telegraphy, and some computing. Early teleprinters were electromechanical, having no microprocessor and just enough electromechanical memory to function. They fully processed one character at a time, returning to an idle state immediately afterward; this meant that any control sequences had to be only one character long, and thus a large number of codes needed to be reserved for such controls. They were typewriter-derived impact printers, and could only print a fixed set of glyphs, which were cast into a metal type element or elements; this also encouraged a minimum set of glyphs.

Seven-bit ASCII improved over prior five- and six-bit codes. Of the 27=128 codes, 33 were used for controls, and 95 carefully selected printable characters (94 glyphs and one space), which include the English alphabet (uppercase and lowercase), digits, and 31 punctuation marks and symbols: all of the symbols on a standard US typewriter plus a few selected for programming tasks. Some popular peripherals only implemented a 64-printing-character subset: Teletype Model 33 could not transmit "a" through "z" or five less-common symbols ("`", "{", "|", "}", and "~"). and when they received such characters they instead printed "A" through "Z" (forced all caps) and five other mostly-similar symbols ("@", "[", "\", "]", and "^").

The ASCII character set is barely large enough for US English use and lacks many glyphs common in typesetting, and far too small for universal use. Many more letters and symbols are desirable, useful, or required to directly represent letters of alphabets other than English, more kinds of punctuation and spacing, more mathematical operators and symbols (× ÷ ⋅ ≠ ≥ ≈ π etc.), some unique symbols used by some programming languages, ideograms, logograms, box-drawing characters, etc.

The biggest problem for computer users around the world was other alphabets. ASCII's English alphabet almost accommodates European languages, if accented letters are replaced by non-accented letters or two-character approximations such as ss for ß. Modified variants of 7-bit ASCII appeared promptly, trading some lesser-used symbols for highly desired symbols or letters, such as replacing "#" with "£" on UK Teletypes, "\" with "¥" in Japan or "₩" in Korea, etc. At least 29 variant sets resulted. 12 code points were modified by at least one modified set, leaving only 82 "invariant" codes. Programming languages however had assigned meaning to many of the replaced characters, work-arounds were devised such as C three-character sequences "??<" and "??>" to represent "{" and "}".[4] Languages with dissimilar basic alphabets could use transliteration, such as replacing all the Latin letters with the closest match Cyrillic letters (resulting in odd but somewhat readable text when English was printed in Cyrillic or vice versa). Schemes were also devised so that two letters could be overprinted (often with the backspace control between them) to produce accented letters. Users were not comfortable with any of these compromises and they were often poorly supported.[citation needed]

When computers and peripherals standardized on eight-bit bytes in the 1970s, it became obvious that computers and software could handle text that uses 256-character sets at almost no additional cost in programming, and no additional cost for storage. (Assuming that the unused 8th bit of each byte was not reused in some way, such as error checking, Boolean fields, or packing 8 characters into 7 bytes.) This would allow ASCII to be used unchanged and provide 128 more characters. Many manufacturers devised 8-bit character sets consisting of ASCII plus up to 128 of the unused codes: encodings which covered all the more used Western European (and Latin American) languages, such as Danish, Dutch, French, German, Portuguese, Spanish, Swedish and more could be made.

128 additional characters is still not enough to cover all purposes, all languages, or even all European languages, so the emergence of many proprietary and national ASCII-derived 8-bit character sets was inevitable. Translating between these sets (transcoding) is complex (especially if a character is not in both sets); and was often not done, producing mojibake (semi-readable resulting text, often users learned how to manually decode it). There were eventually attempts at cooperation or coordination by national and international standards bodies in the late 1990s, but manufacturer-proprietary sets remained the most popular by far, primarily because the international standards excluded characters popular in or peculiar to specific cultures.

Proprietary extensions

[edit]

Various proprietary modifications and extensions of ASCII appeared on non-EBCDIC mainframe computers and minicomputers, especially in universities.

Hewlett-Packard started to add European characters to their extended 7-bit / 8-bit ASCII character set HP Roman Extension around 1978/1979 for use with their workstations, terminals and printers. This later evolved into the widely used regular 8-bit character sets HP Roman-8 and HP Roman-9 (as well as a number of variants).

Atari and Commodore home computers added many graphic symbols to their non-standard ASCII (Respectively, ATASCII and PETSCII, based on the original ASCII standard of 1963).

The TRS-80 character set for the TRS-80 home computer added 64 semigraphics characters (0x80 through 0xBF) that implemented low-resolution block graphics. (Each block-graphic character displayed as a 2x3 grid of pixels, with each block pixel effectively controlled by one of the lower 6 bits.)[5]

IBM introduced eight-bit extended ASCII codes on the original IBM PC and later produced variations for different languages and cultures. IBM called such character sets code pages and assigned numbers to both those they themselves invented as well as many invented and used by other manufacturers. Accordingly, character sets are very often indicated by their IBM code page number. In ASCII-compatible code pages, the lower 128 characters maintained their standard ASCII values, and different pages (or sets of characters) could be made available in the upper 128 characters. DOS computers built for the North American market, for example, used code page 437, which included accented characters needed for French, German, and a few other European languages, as well as some graphical line-drawing characters. The larger character set made it possible to create documents in a combination of languages such as English and French (though French computers usually use code page 850), but not, for example, in English and Greek (which required code page 737).

Apple Computer introduced their own eight-bit extended ASCII codes in Mac OS, such as Mac OS Roman. The Apple LaserWriter also introduced the Postscript character set.

Digital Equipment Corporation (DEC) developed the Multinational Character Set, which had fewer characters but more letter and diacritic combinations. It was supported by the VT220 and later DEC computer terminals. This later became the basis for other character sets such as the Lotus International Character Set (LICS), ECMA-94 and ISO 8859-1.

ISO 8859

[edit]

In 1987, the International Organization for Standardization (ISO) published a set of standards for eight-bit ASCII extensions, ISO 8859. The most popular of these was ISO 8859-1 (also called "ISO Latin 1") which contains characters sufficient for the most common Western European languages. Other standards in the 8859 group included ISO 8859-2 for Eastern European languages using the Latin script and ISO 8859-5 for languages using the Cyrillic script, and others.

One notable way in which the ISO standards differ from some vendor-specific extended ASCII is that the 32 character positions 8016 to 9F16, which correspond to the ASCII control characters with the high-order bit 'set', are reserved by ISO for control use and unused for printable characters (they are also reserved in Unicode[6]). This convention was almost universally ignored by other extended ASCII sets.

Windows-1252

[edit]

Microsoft intended to use ISO 8859 standards in Windows,[citation needed] but soon replaced the unused C1 control characters with additional characters, making the proprietary Windows-1252 character set, which is sometimes mislabeled as ANSI. The added characters included "curly" quotation marks and other typographical elements like em dash, the euro sign, and letters missing from French and Finnish. This became the most-used extended ASCII in the world, and often is used on the web even when 8859-1 is specified.[7][8]

Character set confusion

[edit]

The meaning of each extended code point can be different in every encoding. In order to correctly interpret and display text data (sequences of characters) that includes extended codes, hardware and software that reads or receives the text must use the specific extended ASCII encoding that applies to it. Applying the wrong encoding causes irrational substitution of many or all extended characters in the text.

Software can use a fixed encoding selection, or it can select from a palette of encodings by defaulting, checking the computer's nation and language settings, reading a declaration in the text, analyzing the text, asking the user, letting the user select or override, and/or defaulting to last selection. When text is transferred between computers that use different operating systems, software, and encodings, applying the wrong encoding can be commonplace.

Because the full English alphabet and the most-used characters in English are included in the seven-bit code points of ASCII, which are common to all encodings (even most proprietary encodings), English-language text is less damaged by interpreting it with the wrong encoding, but text in other languages can display as mojibake (complete nonsense). Because many Internet standards use ISO 8859-1, and because Microsoft Windows (using the code page 1252 superset of ISO 8859-1) is the dominant operating system for personal computers today,[citation needed][when?] unannounced use of ISO 8859-1 is quite commonplace, and may generally be assumed unless there are indications otherwise.

Many communications protocols, most importantly SMTP and HTTP, require the character encoding of content to be tagged with IANA-assigned character set identifiers.

See also

[edit]

References

[edit]
  1. ^ Benjamin Riefenstahl (26 Feb 2001). "Re: Cygwin Termcap information involving extended ascii charicters". cygwin (Mailing list). Archived from the original on 11 July 2013. Retrieved 2 December 2012.
  2. ^ S. Wolicki (Mar 23, 2012). "Print Extended ASCII Codes in sql*plus". Retrieved May 17, 2022.
  3. ^ Mark J. Reed (March 28, 2004). "vim: how to type extended-ascii?". Newsgroupcomp.editors. Retrieved May 17, 2022.
  4. ^ "2.2.1.1 Trigraph sequences". Rationale for American National Standard for Information Systems - Programming Language - C. Archived from the original on 2018-09-29. Retrieved 2019-02-08.
  5. ^ Goldklang, Ira (2015). "Graphic Tips & Tricks". Archived from the original on 2017-07-29. Retrieved 2017-07-29.
  6. ^ "C1 Controls and Latin-1 Supplement | Range: 0080–00FF" (PDF). The Unicode Standard, Version 15.1. Unicode Consortium.
  7. ^ "HTML Character Sets". W3 Schools. When a browser detects ISO-8859-1 it normally defaults to Windows-1252, because Windows-1252 has 32 more international characters.
  8. ^ "Encoding". WHATWG. 27 January 2015. sec. 5.2 Names and labels. Archived from the original on 4 February 2015. Retrieved 4 February 2015.
[edit]