Tag: buffers

  • Can TypeBox handle binary data or buffer-based values?

    Can TypeBox handle binary data or buffer-based values?

    Yes, with a clear understanding of where the boundaries are. TypeBox binary data support exists because real applications deal with more than text and numbers: file contents, images, cryptographic material and encoded payloads all travel as bytes. TypeBox includes dedicated types for byte-based values, so a schema can describe a field as binary, generate a matching TypeScript type, and check the value properly when it is validated in memory.

    The boundary is JSON itself. Binary values do not survive a trip through plain JSON, so when data has to cross a JSON boundary it is normally converted to text first, most often base64, and the schema describes that text instead. Knowing which side of that boundary your data sits on is the whole trick, and this article explains how TypeBox represents binary values, what the types look like, and how validation treats them.

    Why Binary Data Does Not Fit the Usual JSON Shapes

    JSON was designed around a small set of shapes: text, numbers, true and false, lists, and objects made of named fields. A sequence of raw bytes is none of those.

    Bytes can hold any value from zero to 255, including many that have no printable character and no safe representation as ordinary text. Forcing them into a JSON string corrupts them or inflates them beyond recognition.

    Why an Array of Numbers Does Not Work

    The naive workaround is an array of numbers, one per byte. It is technically valid and practically miserable.

    A modest image becomes hundreds of thousands of array entries, the payload balloons in size, and every consumer has to rebuild the original bytes before doing anything useful. Nobody who has watched a network tab fill with numbers does this twice.

    The Two Honest Options

    So the ecosystem settled on two honest options. Inside a running program, bytes are bytes, held in a buffer-like container and handled as a single value.

    Across a JSON boundary, bytes are encoded as text, usually base64, which trades a size increase of roughly a third for safe transport through any JSON channel. A schema library has to serve both worlds, and that is exactly the split you will see in how TypeBox approaches the problem: real binary types for values in memory, and ordinary string schemas for encoded values on the wire.

    How TypeBox Represents Buffer-Based Values in a Schema

    TypeBox treats binary as a first-class category alongside its other built-in types. Rather than pretending a buffer is an array of numbers, it provides dedicated schema builders for byte array values and for the underlying buffer containers that hold them. A field declared with one of these builders is described as a binary value in its own right, with its own identity in the schema.

    This matters because the description stays truthful. A reader of the schema can see immediately that the field carries file contents or raw bytes, not a list of small integers.

    Size Expectations and Exported Schemas

    Tooling that consumes the schema sees the same thing. The schema can also express expectations about the value, such as size limits, in the same spirit as length limits on text or item limits on arrays, which is useful when a field is supposed to hold a small thumbnail rather than an unbounded upload.

    There is an important nuance when schemas are exported as pure JSON Schema documents. JSON Schema has no native binary shape, so exported descriptions lean on conventions: encoded content is usually described as text with a declared format, while the in-memory binary types rely on the validator understanding what a byte array instance is. Teams that share schemas with non-TypeScript tools should decide early which convention they are using, so the contract means the same thing on both sides.

    Binary Fields Inside Ordinary Records

    None of this changes the shape of the surrounding schema. A record can mix ordinary fields, a name here and a size there, with one or two binary fields where the actual content lives. Composition works as usual, and binary fields can sit inside optional properties, arrays of records and the rest of the normal toolkit.

    What the Generated TypeScript Type Looks Like for Binary Fields

    The static type generated for a binary field is the natural TypeScript counterpart: a byte array type for buffer-based values, and the buffer container type where the schema describes a whole buffer. In practice this means code that consumes validated data can treat the field as bytes immediately, passing it to file writers, hashing functions, image processors and network calls without any conversion ceremony.

    This is a noticeable quality-of-life difference from the encoded-text approach. When binary travels as base64 text, the static type is just a string, and nothing in the type tells the reader that the string represents bytes, which encoding it uses, or that it should be decoded before use.

    How the Two Styles Compose

    Every consumer has to know the convention by heart. With a true binary type, the type itself carries that information, and misuse, like accidentally displaying raw bytes as text, becomes easier to spot in review.

    The two styles also compose differently. A schema describing an upload record with a binary content field produces a record type whose content field is bytes and whose metadata fields are ordinary text and numbers. TypeScript developers reading that type understand the whole arrangement at a glance, which is the same readability benefit TypeBox aims for everywhere else.

    The Caution at the JSON Boundary

    One caution is worth repeating: the elegant in-memory type only exists while the data stays in memory. The moment the same record is serialised to JSON for storage or transport, the binary field needs an encoding step, and the type on the receiving side is whatever that side’s schema says it is. Keeping the two representations clearly named in a codebase prevents a whole category of confusion.

    How Validation Treats Binary Values at Runtime

    Runtime validation of a binary field is an identity check rather than a content inspection. The validator confirms that the value actually is a byte array or buffer of the expected kind, rather than, say, a string that merely looks like one.

    If size expectations were declared in the schema, those are checked as well. The contents of the bytes are not interpreted; a schema cannot tell a valid image from random noise, only that the value is genuinely binary and the right sort of size.

    What the Schema Cannot Judge

    That distinction shapes how you should design checks around uploads. The schema layer answers “is this really bytes, and is it within the allowed size”, while format-specific verification, such as confirming an image header or a file signature, belongs in ordinary code after validation. Expecting the schema to judge content is a category error that leads to disappointment.

    Performance and the Encoded-Text Case

    Performance is generally a non-issue for binary fields, because checking a value’s kind and length is cheap compared with walking a large object field by field. Teams wondering about the overall cost of checking large payloads on every request are really asking whether validation slows a busy API down, which depends far more on object size and schema complexity than on the presence of a binary field or two.

    Encoded values behave differently, of course. A base64 field validates as a string, so the schema can check its length and pattern but cannot confirm the decoded bytes are meaningful until code decodes them. If the distinction matters to your application, validate the text at the boundary, decode it, and then treat the result as binary data inside.

    Practical Places Binary Fields Appear, Such as File Uploads

    File uploads are the everyday case. A user submits a photo, a document or an attachment, and the receiving code holds the contents as bytes long before anything is stored. A schema with a binary field describes that payload honestly: the content is bytes, the filename is text, the size is a number, and each part validates according to its own nature.

    Generated and derived files form a second case. Systems that produce PDFs, export spreadsheets or render images end up holding the results as buffers before sending or saving them. Describing those results with binary fields keeps internal message shapes truthful as they move between services in the same process or across a direct binary channel.

    Security Material and the Hybrid Pattern

    Cryptographic and identity material is a third case. Hashes, signatures, keys and tokens at the byte level are all binary values, even when they are short. Schemas for security-sensitive messages benefit from saying so explicitly, because a hash described as bytes cannot be quietly confused with the human-readable text it might be printed as.

    Finally, there is the hybrid pattern most APIs actually use. Bytes cross the network encoded as text, then are decoded and handled as binary inside. A well-kept codebase marks the boundary clearly: the external schema describes the encoded text, the internal schema describes the binary value, and one deliberate conversion step connects them.

    Frequently Asked Questions

    Can a TypeBox schema describe a file upload directly?

    Yes, when the upload is handled in memory. The content field can be declared as binary alongside text fields for the filename and other metadata, and each part validates according to its kind.

    Does binary data survive being sent as JSON?

    No. JSON has no binary shape, so bytes are encoded as text, usually base64, before transport and described as a string in the schema on that side of the boundary.

    Can validation check that binary data is a valid image?

    Not at the schema level. Validation confirms the value is genuinely bytes and within any declared size expectations. Checking image headers or file signatures belongs in code after validation.

    What type does a binary field produce in TypeScript?

    A byte array or buffer type. It matches how the value is actually used in code, so consumers can pass it straight to file, hashing and network operations.

    Conclusion

    TypeBox handles binary data and buffer-based values as first-class citizens inside a running program: dedicated schema types describe them truthfully, the generated TypeScript types treat them as bytes, and validation checks their kind and size quickly. Across JSON boundaries the same data travels as encoded text under a string schema. Keep those two worlds clearly separated in your schemas, and binary fields stop being a special case and become just another well-described part of your data.