# Hidden character cleaner: exactly what it removes, and what it misses

> The precise character lists behind each option, computed results for real cases, how emoji, Persian and Indic joiners are kept by default, and the few invisible characters this tool still does not touch.

- Tool: [Hidden Character Cleaner](https://toolsg.com/en/hidden-character-cleaner)
- Language: en (available: en, zh, ms, ta)
- Last verified against official sources: 2026-10-20
- Source site: ToolSG — a planning guide, not official or financial advice. Cite the agencies listed under Sources.

## Key figures at a glance

| Fact | Value | Source |
| --- | --- | --- |
| Zero-width option | U+200B, U+2060, and U+200C / U+200D outside emoji and non-Latin words — By default ZWJ and ZWNJ are kept inside emoji sequences and between non-Latin letters (Persian, Tamil); untick Keep ZWJ/ZWNJ to remove them all. |  |
| BOM option | U+FEFF anywhere in the text — The byte order mark is only meaningful at the start of a file. |  |
| Control characters | C0 and C1 controls except tab, line feed and carriage return — Includes U+0000 to U+0008, U+000B, U+000C, U+000E to U+001F and U+007F to U+009F; U+2028 and U+2029 become line breaks. |  |
| Still not removed | Other special spaces, invisible operators, variation selectors and others — See the list below before assuming the text is fully clean. |  |

## What the buttons do

**Highlight Hidden Characters** shows your text with every match of the ticked options replaced by a red `[•]` marker, and reports how many were found. **Clean Text** returns the text without them. Nothing is uploaded. Both work only on the options that are ticked; "Normalize whitespace" is off by default and affects only cleaning.

> **Tip:** Copy and Download always use the cleaned text, never the `[•]` markers. If you press Copy while the box shows the highlighted view, the tool copies your text with the ticked options applied and says so in the status line. Highlight first to see what would go, then Copy or Clean Text.

## What each option removes

| Option (default) | Characters |
| --- | --- |
| Zero-width (on) | U+200B zero width space, U+2060 word joiner, and U+200C zero width non-joiner / U+200D zero width joiner unless kept (next row) |
| Keep ZWJ/ZWNJ (on) | Sub-option of zero-width. Keeps U+200D between emoji (an emoji, variation selector or skin-tone modifier before it, an emoji after it) and keeps U+200C / U+200D when the letters on both sides are not Latin (Persian, Urdu, Tamil, Hindi and so on) |
| BOM (on) | U+FEFF anywhere in the text |
| Control characters (on) | U+0000 to U+0008, U+000B, U+000C, U+000E to U+001F, U+007F to U+009F (everything except tab, LF, CR). U+2028 line separator and U+2029 paragraph separator are replaced by a line feed instead of being deleted |
| Invisible Unicode (on) | U+00AD soft hyphen, U+034F combining grapheme joiner, U+061C Arabic letter mark, U+115F and U+1160 Hangul fillers, U+3164 Hangul filler, U+17B4 and U+17B5 Khmer inherent vowels, U+180E Mongolian vowel separator. U+00A0 non-breaking space is replaced by a normal space instead of being deleted |
| Direction markers (on) | U+200E, U+200F, U+202A to U+202E (embedding, override and pop controls) and the bidirectional isolates U+2066 to U+2069 |
| Normalize whitespace (off) | Every run of spaces and tabs becomes one space; three or more consecutive line feeds become two |

## Computed results

Results produced by the page's own code with the default options:

| Input | Found | Result |
| --- | --- | --- |
| `pass` U+200B `word` | 1 | `password` |
| U+FEFF `id,name` | 1 | `id,name` |
| `co` U+00AD `operate` | 1 | `cooperate` |
| `invoice` U+202E `txt.exe` | 1 | `invoicetxt.exe` |
| `a` U+0007 `b` U+0000 `c` | 2 | `abc` |
| `a` tab `b` LF `c` CR `d` | 0 | unchanged (tab, LF and CR are kept) |
| `a` U+0085 `b` | 1 | `ab` (U+0085 NEXT LINE is a C1 control) |
| `one` U+00A0 `two` | 1 | `one two` (a normal space) |
| `a` U+2028 `b` U+2029 `c` | 2 | `a`, line feed, `b`, line feed, `c` |
| `a` U+2060 `b` | 1 | `ab` |
| `a` U+2066 `b` U+2069 `c` | 2 | `abc` |
| `a` U+3164 `b` | 1 | `ab` |
| `👨‍👩‍👧` (family emoji) | 0 | unchanged |
| `می‌خواهم` (Persian "I want") | 0 | unchanged |
| `ab` U+200D `cd` | 1 | `abcd` |

The right-to-left override case is the kind used in file-name and source-code spoofing (the "Trojan Source" attack, CVE-2021-42574, uses bidirectional controls). Cleaning removes the control but you should still treat the source as untrusted.

## Emoji, Persian and Indic scripts

U+200D (zero width joiner) is the glue in emoji sequences such as the family emoji, and U+200C (zero width non-joiner) is part of the spelling of Persian and some Indic words. Removing them blindly damages such text, so the **Keep ZWJ/ZWNJ** option, on by default, leaves them alone where they do something:

- `👨‍👩‍👧` (five code points) and `🏳️‍🌈` (rainbow flag) are left intact.
- `می‌خواهم` keeps its half-space and `க்‍ஷ` keeps its joiner.
- The same joiners are removed when they sit between Latin letters (`ab` U+200D `cd` becomes `abcd`), at the start or end of the text, or next to spaces and digits.
- Untick **Keep ZWJ/ZWNJ** to remove every joiner: the family emoji then falls apart into three separate emoji `👨👩👧` and `می‌خواهم` becomes `میخواهم` with the letters joined.

The rule is a heuristic based on the characters on each side, not a language detector. If a Latin-script text uses ZWNJ on purpose, untick the whole zero-width option.

## Other side effects

- **Non-breaking space becomes a normal space.** French and other typography put U+00A0 before `!`, `?`, `:` and `;` on purpose. Untick **Invisible Unicode** if you want to keep it, at the cost of keeping the other invisible characters in that list.
- **Soft hyphen** U+00AD only marks where a word may break, so removing it is normally what you want.
- **Whitespace normalisation is blunt.** It changes tabs to spaces, so tab-indented code loses its tabs, and it does not trim line ends.
- Counts: **Total Characters** is the JavaScript string length (UTF-16 units; `🙂` counts 2). **Removed** is the difference in length, so a non-breaking space turned into a normal space is found but not "removed", and with normalisation on it includes collapsed spaces.

## Hidden characters this tool does not remove

Selecting every option will still not touch:

- Other special spaces: U+2000 to U+200A, U+202F, U+205F and U+3000. (The [copy-paste sanitizer](https://toolsg.com/en/copy-paste-sanitizer)'s non-ASCII option turns these into normal spaces.)
- The invisible operators U+2061 to U+2064
- Emoji variation selectors such as U+FE0F
- Blank-looking characters such as U+2800 Braille blank and U+FFA0 halfwidth Hangul filler
- Tag characters (U+E0000 range)

Use your editor for those, and a hex view such as [text to hex](https://toolsg.com/en/text-to-hex) to inspect what remains.

## Where hidden characters come from

Copying from web pages, PDFs, chat apps and word processors, exporting spreadsheets as UTF-8 with a BOM, and right-to-left text editing are common sources. A BOM (U+FEFF) is legitimate as the first character of a file, where it signals UTF-8 (per the Unicode FAQ); in the middle of text it acts as a zero-width no-break space.

## Common mistakes

- Unticking Keep ZWJ/ZWNJ and then cleaning emoji or Persian text.
- Expecting the non-breaking space to be deleted; it becomes a normal space.
- Believing "no hidden characters found" means the text is free of every invisible character.
- Expecting this to remove tabs, line breaks or double spaces; it does not unless normalisation is on, and tabs and line feeds are never removed.

## Related tools

[Unicode explorer](https://toolsg.com/en/unicode-explorer) identifies one character at a time. [Text diff](https://toolsg.com/en/text-diff) shows invisible differences between two versions. [Word counter](https://toolsg.com/en/word-counter) reveals hidden characters through unexpected counts. [HTML entity encoder](https://toolsg.com/en/html-entity-encoder) can turn invisible characters into visible references.

## Frequently asked questions

**What are hidden characters?**

Hidden characters are Unicode characters that have no visible representation but exist in text. Common examples include zero-width spaces (ZWSP), byte order marks (BOM), and control characters. These can cause problems with text processing, code execution, data validation, and more.

**Why should I remove hidden characters?**

Hidden characters should be removed because they can break code syntax, cause data import failures, prevent text from matching in searches, trigger validation errors, and cause unexpected behavior in applications. They're especially problematic when copying text from websites or PDFs.

**Is my text data sent to any server?**

No, absolutely not. All text processing happens entirely in your web browser using JavaScript. Your data never leaves your device, making this tool completely safe for sensitive documents, source code, API keys, and confidential content.

**How do I know if my text has hidden characters?**

Use the "Highlight Hidden Characters" button to visually mark where hidden characters appear in your text (shown in red). The statistics panel also shows the count of hidden characters detected. If text behaves unexpectedly or has character count mismatches, it likely contains hidden characters.

**What's the difference between ZWSP, ZWNJ, and ZWJ?**

Zero-Width Space (ZWSP) allows line breaks without visible space. Zero-Width Non-Joiner (ZWNJ) prevents character joining in complex scripts. Zero-Width Joiner (ZWJ) forces character joining, especially in emoji. All are invisible but serve different formatting purposes.

**Will cleaning remove all my spaces?**

No, this tool only removes invisible and hidden characters. Normal spaces, line breaks, and tabs are preserved unless you enable the "Normalize whitespace" option, which only removes excessive whitespace while keeping text readable. A non-breaking space (U+00A0) is turned into a normal space rather than deleted, so words never run together.

**Can I use this for cleaning code?**

Yes, this tool is excellent for cleaning code copied from Stack Overflow, documentation sites, or other sources that might include hidden characters. It removes problematic characters while preserving code formatting and regular spaces.

**What is a Byte Order Mark (BOM)?**

BOM (U+FEFF) is a special character placed at the beginning of text files to indicate byte order encoding. While useful for some applications, BOM often causes parsing errors in programming languages, APIs, and web applications, so it's frequently removed.

**Why do websites add hidden characters?**

Websites use hidden characters for proper text rendering, line breaking, and internationalization support. However, when you copy text from websites, these formatting characters come along and can cause issues when pasted into code editors, forms, or databases.

**Can this tool damage my text?**

No, this tool only removes invisible characters while preserving all visible content. By default the joiners ZWJ and ZWNJ are kept inside emoji sequences and between letters of non-Latin scripts such as Persian or Tamil, and removed elsewhere; untick "Keep ZWJ/ZWNJ" to remove every one. Other zero-width characters are always removed when that option is on, so keep a backup if unsure.

**How do hidden characters get into my text?**

Hidden characters typically enter text through copy-pasting from websites, PDFs, Microsoft Word documents, or rich text editors. They can also appear in files saved with certain encodings, or when text is processed by translation tools or content management systems.

**Do I need to clean text every time I copy from a website?**

Not always, but it's recommended for critical use cases like code, email addresses, API requests, data imports, or form submissions. For casual text, hidden characters usually don't cause issues. When in doubt, run a quick check with this tool.


## Sources

- [Unicode FAQ — UTF-8, UTF-16, UTF-32 and BOM](https://www.unicode.org/faq/utf_bom.html) — Explains U+FEFF as a signature and as ZERO WIDTH NO-BREAK SPACE.
- [Unicode UTS #51 — Unicode Emoji](https://www.unicode.org/reports/tr51/) — ZWJ sequences form single emoji; without ZWJ they fall back to separate emoji.
- [Unicode UAX #9 — Bidirectional Algorithm](https://www.unicode.org/reports/tr9/)
- [Trojan Source (CVE-2021-42574)](https://trojansource.codes/) — Attack that hides code logic using bidirectional control characters.
