Binary versus ASCII (Plain Text) Files, and converting them

 

The Difference Between Binary and ASCII Files; Converting them

At heart all files are binary files -- that is, a collection of 1s and 0s. But there's a subset of binary files we call ASCII, or plain text files. ASCII is short for American Standard Code for Information Interchange, which allocates a number to each letter, digit and symbol. A plain text file contains no formatting codes whatsoever, no fonts, bold, italics or underlines, headers, footers or graphics. The only 'formatting' that can be applied is to use spaces to pad lines out so that they are centered or right justified, or to add extra blank lines.

Let's look at some example - different file types containing the word 'hello' followed by a new line. To see the differences, we will use a hex display, to show us exactly what is in each file, and highlight the word hello in the output so we can see where it appears. A hex display shows us the offset, ASCII code and the actual characters contained in a file (where those characters are printable - it shows a period where they are not). These hex displays were generated by TextPipe Pro (Filters Menu\Convert\Hex dump).

Plain Text File - hello.txt (7 bytes long)

This is the simplest file - the ASCII codes for the letters 'hello' followed by the ASCII codes for a carriage return and line feed.

00000000 68 65 6C 6C 6F 0D 0A                            hello..

Rich Text Format (RTF) File - hello.rtf (168 bytes long)

You can see that an RTF file includes lots of extra guff. Generally, all the letters of the word will be together. However, if you have two or more words, other codes can appear between the words, making them difficult to locate.

00000000 7B 5C 72 74 66 31 5C 61 6E 73 69 5C 61 6E 73 69 {\rtf1\ansi\ansi 
00000010 63 70 67 31 32 35 32 5C 64 65 66 66 30 5C 64 65 cpg1252\deff0\de 
00000020 66 6C 61 6E 67 33 30 38 31 7B 5C 66 6F 6E 74 74 flang3081{\fontt 
00000030 62 6C 7B 5C 66 30 5C 66 73 77 69 73 73 5C 66 63 bl{\f0\fswiss\fc 
00000040 68 61 72 73 65 74 30 20 41 72 69 61 6C 3B 7D 7D harset0 Arial;}} 
00000050 0D 0A 7B 5C 2A 5C 67 65 6E 65 72 61 74 6F 72 20 ..{\*\generator 
00000060 4D 73 66 74 65 64 69 74 20 35 2E 34 31 2E 31 35 Msftedit 5.41.15 
00000070 2E 31 35 30 33 3B 7D 5C 76 69 65 77 6B 69 6E 64 .1503;}\viewkind 
00000080 34 5C 75 63 31 5C 70 61 72 64 5C 66 30 5C 66 73 4\uc1\pard\f0\fs 
00000090 32 30 20 68 65 6C 6C 6F 5C 70 61 72 0D 0A 5C 70 20 hello\par..\p 
000000A0 61 72 0D 0A 7D 0D 0A 00                         ar..}... 

Microsoft Word Document - hello.doc (19,968 bytes long)

The file below, even without any formatting, is huge, so we've removed large sections of it for clarity. A major point we have to make here is that Word relies on the exact position of various aspects of the file being fixed, such as font tables, symbol tables and other internal references. If these positions are changed (e.g. by searching for 'hello' and replacing it with a shorter string such as 'bye' or a longer string such as 'hello there') then the document will be corrupted and MS Word will not be able to load the document again. Recovery may not be possible. This is why you CANNOT use a text editor or text tool on Word documents. You must use a specific tool that knows how to maintain the correct offsets, such as WordPipe for MS Word, ExcelPipe for MS Excel or PowerPointPipe for MS PowerPoint.

An additional point to note is that the word 'Symbol' is stored in the Word document in Unicode format (see below), so a text editor or text tool will not find it. Since this file contains mixed sections of ASCII and Unicode, it is crucial that the file positions are left unchanged.

00000000 D0 CF 11 E0 A1 B1 1A E1 00 00 00 00 00 00 00 00 ÐÏ.ࡱ.á........ 
00000010 00 00 00 00 00 00 00 00 3E 00 03 00 FE FF 09 00 ........>...þÿ.. 
00000020 06 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 ................ 
...
000009F0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 
00000A00 68 65 6C 6C 6F 0D 0D 00 00 00 00 00 00 00 00 00 hello........... 
00000A10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 
...
00001A40 00 53 00 79 00 6D 00 62 00 6F 00 6C 00 00 00 33 .S.y.m.b.o.l...3 
...
00004DF0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 

Unicode Plain Text File - hello.txt (16 bytes long)

ASCII is being replaced in many applications by Unicode, which uses 16 bits (2 bytes) per character to represent non-Roman alphabets like Japanese, Chinese, and Cyrillic. A text editor or text tool won't find 'hello' in this file. TextPipe Pro provides Unicode search and replace facilities, in addition to ASCII search and replace, so it can find both forms of 'hello'.

00000000 FF FE 68 00 65 00 6C 00 6C 00 6F 00 0D 00 0A 00 ÿþh.e.l.l.o..... 

Convert binary files to text files

Now, to convert a binary file to a useful text form, you need to strip out all the binary characters - the formatting, control and other gobbledygook stuff.  TextPipe Pro provides a simple filter for this under Filters\Remove\Binary characters.

You can also generate your own custom filter that only removes the binary characters you specify by using Filters\Maps\New map.

 

You may freely link to this page, but you may not copy its content.