Go command support for embedded static assets (files)

Go command support for embedded static assets (files) — Draft Design

Russ Cox
Brad Fitzpatrick
July 2020

This is a Draft Design, not a formal Go proposal, because it describes a potential large change that addresses the same need as many third-party packages and could affect their implementations (hopefully by simplifying them!). The goal of circulating this draft design is to collect feedback to shape an intended eventual proposal.

This design builds upon the file system interfaces draft design.

We are using this change to experiment with new ways to scale discussions about large changes. For this change, we will use a Go Reddit thread to manage Q&A, since Reddit’s threading support can easily match questions with answers and keep separate lines of discussion separate.

There is a video presentation of this draft design.

The prototype code is available for trying out.

Abstract

There are many tools to embed static assets (files) into Go binaries. All depend on a manual generation step followed by checking in the generated files to the source code repository. This draft design eliminates both of these steps by adding support for embedded static assets to the go command itself.

Background

There are many tools to embed static assets (files) into Go binaries. One of the earliest and most popular was github.com/jteeuwen/go-bindata and its forks, but there are many more, including (but not limited to!):

Clearly there is a widespread need for this functionality.

The go command is the way Go developers build Go programs. Adding direct support to the go command for the basic functionality of embedding will eliminate the need for some of these tools and at least simplify the implementation of others.

Goals

It is an explicit goal to eliminate the need to generate new Go source files for the assets and commit those source files to version control.

Another explicit goal is to avoid a language change. To us, embedding static assets seems like a tooling issue, not a language issue. Avoiding a language change also means we avoid the need to update the many tools that process Go code, among them goimports, gopls, and staticcheck.

It is important to note that as a matter of both design and policy, the go command never runs user-specified code during a build. This improves the reproducibility, scalability, and security of builds. This is also the reason that go generate is a separate manual step rather than an automatic one. Any new go command support for embedded static assets is constrained by that design and policy choice.

Another goal is that the solution apply equally well to the main package and to its dependencies, recursively. For example, it would not work to require the developer to list all embeddings on the go build command line, because that would require knowing the embeddings needed by all of the dependencies of the program being built.

Another goal is to avoid designing novel APIs for accessing files. The API for accessing embedded files should be as close as possible to *os.File, the existing standard library API for accessing native operating-system files.

Design

This design adds direct support for embedded static assets into the go command itself, building on the file system draft design.

That support consists of:

• A new //go:embed comment directive naming the files to embed.
• A new embed package, which defines the type embed.Files, the public API for a set of embedded files. The embed.Files implements fs.FS from the file system interfaces draft design, making it directly usable with packages like net/http and html/template.
• Go command changes to process the directives.
• Changes to go/build and golang.org/x/tools/go/packages to expose information about embedded files.

//go:embed directives

A new package embed, described in detail below, provides the type embed.Files. One or more //go:embed directives above a variable declaration of that type specify which files to embed, in the form of a glob pattern. For example:

package server

// content holds our static web server content.
//go:embed image/* template/*
//go:embed html/index.html
var content embed.Files

The go command will recognize the directives and arrange for the declared embed.Files variable (in this case, content) to be populated with the matching files from the file system.

The //go:embed directive accepts multiple space-separated glob patterns for brevity, but it can also be repeated, to avoid very long lines when there are many patterns. The glob patterns are in the syntax of path.Match; they must be unrooted, and they are interpreted relative to the package directory containing the source file. The path separator is a forward slash, even on Windows systems. To allow for naming files with spaces in their names, patterns can be written as Go double-quoted or back-quoted string literals.

If a pattern names a directory, all files in the subtree rooted at that directory are embedded (recursively), so the above example is equivalent to:

package server

// content is our static web server content.
//go:embed image template html/index.html
var content embed.Files

An embed.Files variable can be exported or unexported, depending on whether the package wants to make the file set available to other packages. Similarly, an embed.Files variable can be a global or a local variable, depending on what is more convenient in context.

• When evaluating patterns, matches for empty directories are ignored (because empty directories are never packaged into a module).
• It is an error for a pattern not to match any file or non-empty directory.
• It is not an error to repeat a pattern or for multiple patterns to match a particular file; such a file will only be embedded once.
• It is an error for a pattern to contain a .. path element.
• It is an error for a pattern to contain a . path element (to match everything in the current directory, use *).
• It is an error for a pattern to match files outside the current module or that cannot be packaged into a module, like .git/* or symbolic links (or, as noted above, empty directories).
• It is an error for a //go:embed directive to appear except before a declaration of an embed.Files. (More specifically, each //go:embed directive must be followed by a var declaration of a variable of type embed.Files, with only blank lines and other //-comment-only lines between the //go:embed and the declaration.)
• It is an error to use //go:embed in a source file that does not import "embed" (the only way to violate this rule involves type alias trickery).
• It is an error to use //go:embed in a module declaring a Go version before Go 1.N, where N is the Go version that adds this support.
• It is not an error to use //go:embed with local variables declared in functions.
• It is not an error to use //go:embed in tests.
• It is not an error to declare an embed.Files without a //go:embed directive. That variable simply contains no embedded files.

The embed package

The new package embed defines the Files type:

// A Files provides access to a set of files embedded in a package at build time.
type Files struct { … }

The Files type provides an Open method that opens an embedded file, as an fs.File:

func (f Files) Open(name string) (fs.File, error)

By providing this method, the Files type implements fs.FS and can be used with utility functions such as fs.ReadFile, fs.ReadDir, fs.Glob, and fs.Walk.

As a convenience for the most common operation on embedded files, the Files type also provides a ReadFile method:

func (f Files) ReadFile(name string) ([]byte, error)

Because Files implements fs.FS, a set of embedded files can also be passed to template.ParseFS, to parse embedded templates, and to http.HandlerFS, to serve a set of embedded files over HTTP.

Go command changes

The go command will change to process //go:embed directives and pass appropriate information to the compiler and linker to carry out the embedding.

The go command will also add six new fields to the Package struct exposed by go list:

EmbedPatterns      []string
EmbedFiles         []string
TestEmbedPatterns  []string
TestEmbedFiles     []string
XTestEmbedPatterns []string
XTestEmbedFiles    []string

The EmbedPatterns field lists all the patterns found on //go:embed lines in the package’s non-test source files; TestEmbedPatterns and XTestEmbedPatterns list the patterns in the package’s test source files (internal and external tests, respectively).

The EmbedFiles field lists all the files, relative to the package directory, matched by the EmbedPatterns; it does not specify which files match which pattern, although that could be reconstructed using path.Match. Similarly, TestEmbedFiles and XTestEmbedFiles list the files matched by TestEmbedPatterns and XTestEmbedPatterns. These file lists contain only files; if a pattern matches a directory, the file list includes all the files found in that directory subtree.

go/build and golang.org/x/tools/go/packages

In the go/build package, the Package struct adds only EmbedPatterns, TestEmbedPatterns, and XTestEmbedPatterns, not EmbedFiles, TestEmbedFiles, or XTestEmbedFiles, because the go/build package does not take on the job of matching patterns against a file system.

In the golang.org/x/tools/go/packages package, the Package struct adds one new field: EmbedFiles lists the embedded files. (If embedded files were added to OtherFiles, it would not be possible to tell whether a file with a valid source extension in that list—for example, x.c—was being built or embedded or both.)

Rationale

As noted above, the Go ecosystem has many tools for embedding static assets, too many for a direct comparison to each one. Instead, this section lays out the affirmative rationale in favor of each of the parts of the design. Each subsection also addresses the points raised in the helpful preliminary discussion on golang.org/issue/35950. (The Appendix at the end of this document makes direct comparisons with a few existing tools and examines how they might be simplified.)

It is worth repeating the goals and constraints mentioned in the background section:

• No generated Go source files.
• No language change, so no changes to tools processing Go code.
• The go command does not run user code during go build.
• The solution must apply as well to dependency packages as it does to the main package.
• The APIs for accessing embedded files should be close to those for operating-system files.

Approach

The core of the design is the new embed.Files type annotated at its use with the new //go:embed directive:

//go:embed *.jpg
var jpgs embed.Files

This is different from the two approaches mentioned at the start of the preliminary discussion on golang.org/issue/35950. In some ways it is a combination of the best parts of each.

The first approach mentioned was a directive along the lines of

//go:genembed Logo logo.jpg

that would be replaced by a generated func Logo() []byte function, or some similar accessor.

A significant drawback of this approach is that it changes the way programs are type-checked: you can’t type-check a call to Logo unless you know what that directive turns into. There is also no obvious place to write the documentation for the new Logo function. In effect, this new directive ends up being a full language change: all tools processing Go code have to be updated to understand it.

The second approach mentioned was to have a new importable embed package with standard Go function definitions, but the functions are in effect executed at compile time, as in:

var Static = embed.Dir("static")
var Logo = embed.File("images/logo.jpg")
var Words = embed.CompressedReader("dict/words")

This approach fixes the type-checking problem—it is not a full language change—but it still has significant implementation complexity. The go command would need to parse the entire Go source file to understand which files need to be made available for embedding. Today it only parses up to the import block, never full Go expressions. It would also be unclear to users what constraints are placed on the arguments to these special calls: they look like ordinary Go calls but they can only take string literals, not strings computed by Go code, and probably not even named constants (or else the go command would need a full Go expression evaluator).

Much of the preliminary discussion focused on deciding between these two approaches. This design combines the two and avoids the drawbacks of each.

The //go:embed comment directive follows the established convention for Go build system and compiler directives. The directive is easy for the go command to find, and it is clear immediately that the directive can’t refer to a string computed by a function call, nor to a named constant.

The embed.Files type is plain Go code, defined in a plain Go package embed. All tools that type-check Go code or run other analysis on it can understand the code without any special handling of the //go:embed directive.

The explicit variable declaration provides a clear place to write documentation:

// jpgs holds the static images used on the home page.
//go:embed *.jpg
var jpgs embed.Files

(As of Go 1.15, the //go:embed line is not considered part of the doc comment.)

The explicit variable declaration also provides a clear way to control whether the embed.Files is exported. A data-only package might do nothing but export embedded files, like:

package web

// Styles holds the CSS files shared among all our websites.
//go:embed style/*.css
var Styles embed.Files

Modules versus packages

In the preliminary discussion, a few people suggested specifying embedded files using a new directive in go.mod.

The design of Go modules, however, is that go.mod serves only to describe information about the module’s version requirements, not other details of a particular package. It is not a collection of general-purpose metadata. For example, compiler flags or build tags would be inappropriate in go.mod. For the same reason, information about one package’s embedded files is also inappropriate in go.mod: each package’s individual meaning should be defined by its Go sources. The go.mod is only for deciding which versions of other packages are used to resolve imports.

Placing the embedding information in the package has benefits that using go.mod would not, including the explicit declaration of the file set, control over exportedness, and so on.

Glob patterns

It is clear that there needs to be some way to give a pattern of files to include, such as *.jpg. This design adopts glob patterns as the single way to name files for inclusion. Glob patterns are common to developers from command shells, and they are already well-defined in Go, in the APIs for path.Match, filepath.Match, and filepath.Glob. Nearly all file names are valid glob patterns matching only themselves; using globs avoids the need for separate //go:embedfile and //go:embedglob directives. (This would not be the case if we used, say, Go regular expressions as provided by the regexp package.)

Directories versus ** glob patterns

In some systems, the glob pattern ** is like * but can match multiple path elements. For example images/**.jpg matches all .jpg files in the directory tree rooted at images/. This syntax is not available in Go’s path.Match or in filepath.Glob, and it seems better to use the available syntax than to define a new one. The rule that matching a directory includes all files in that directory tree should address most of the need for ** patterns. For example, //go:embed images instead of //go:embed images/**.jpg. It’s not exactly the same, but hopefully good enough.

If at some point in the future it becomes clear that ** glob patterns are needed, the right way to support them would be to add them to path.Match and filepath.Glob; then the //go:embed directives would get them for free.

Dot-dot, module boundaries, and file name restrictions

In order to build files embedded in a dependency, the raw files themselves must be included in module zip files. This implies that any embedded file must be in the module’s own file tree. It cannot be in a parent directory above the module root (like ../../../etc/passwd), it cannot be in a subdirectory that contains a different module, and it cannot be in a directory that would be left out of the module (like .git). Another implication is that it is not possible to embed two different files that differ only in the case of their file names, because those files would not be possible to extract on a case-insensitive system like Windows or macOS. So you can’t embed two files with different casings, like this:

//go:embed README readme

But //go:embed dir/README other/readme is fine.

Because embed.Files implements fs.FS, it cannot provide access to files with names beginning with .., so files in parent directories are also disallowed entirely, even when the parent directory named by .. does happen to be in the same module.

Codecs and other processing

The preliminary discussion raised a large number of possible transformations that might be applied to files before embedding, including: data compression, JavaScript minification, TypeScript compilation, image resizing, generation of sprite maps, UTF-8 normalization, and CR/LF normalization.

It is not feasible for the go command to anticipate or include all the possible transformations that might be desirable. The go command is also not a general build system; in particular, remember the design constraint that it never runs user programs during a build. These kinds of transformations are best left to an external build system, such as Make or Bazel, which can write out the exact bytes that the go command should embed.

A more limited version of this suggestion was to gzip-compress the embedded data and then make that compressed form available for direct use in HTTP servers as gzipped response content. Doing this would force the use of (or at least support for) gzip and compressed content, making it harder to adjust the implementation in the future as we learn more about how well it works. Overall this seems like overfitting to a specific use case.

The simplest approach is for Go’s embedding feature to store plain files, let build systems or third-party packages take care of preprocessing before the build or postprocessing at runtime. That is, the design focuses on providing the core functionality of embedding raw bytes into the binary for use at run-time, leaving other tools and packages to build on a solid foundation.

Compression to reduce binary size

A popular question in the preliminary discussion was whether the embedded data should be stored in compressed or uncompressed form in the binary. This design carefully avoids assuming an answer to that question. Instead, whether to compress can be left as an implementation detail.

Compression carries the obvious benefit of smaller binaries. However, it also carries some less obvious costs. Most compression formats (in particular gzip and zip) do not support random access to the uncompressed data, but an http.File needs random access (ReadAt, Seek) to implement range requests. Other uses may need random access as well. For this reason, many of the popular embedding tools start by decompressing the embedded data at runtime. This imposes a startup CPU cost and a memory cost. In contrast, storing the embedded data uncompressed in the binary supports random access with no startup CPU cost. It also reduces memory cost: the file contents are never stored in the garbage-collected heap, and the operating system efficiently pages in necessary data from the executable as that data is accessed, instead of needing to load it all at once.

Most systems have more disk than RAM. On those systems, it makes very little sense to make binaries smaller at the cost of using more memory (and more CPU) at run time.

On the other hand, projects like TinyGo and U-root target systems with more RAM than disk or flash. For those projects, compressing assets and using incremental decompression at runtime could provide significant savings.

Again, this design allows compression to be left as an implementation detail. The detail is not decided by each package author but instead could be decided when building the final binary. Future work might be to add -embed=compress as a go build option for use in limited environments.

Go command changes

Other than support for //go:embed itself, the only user-visible go command change is new fields exposed in go list output.

It is important for tools that process Go packages to be able to understand what files are needed for a build. The go list command is the underlying mechanism used now, even by golang.org/x/tools/go/packages. Exposing the embedded files as a new field in Package struct used by go list makes them available both for direct use and for use by higher level APIs.

Command-line configuration

In the preliminary discussion, a few people suggested that the list of embedded files could be specified on the go build command line. This could potentially work for files embedded in the main package, perhaps with an appropriate Makefile. But it would fail badly for dependencies: if a dependency wanted to add a new embedded file, all programs built with that dependency would need to adjust their build command lines.

Potential confusion with go:generate

In the preliminary discussion, a few people pointed out that developers might be confused by the inconsistency that //go:embed directives are processed during builds but //go:generate directives are not.

There are other special comment directives as well: //go:noinline, //go:noescape, // +build, //line. All of these are processed during builds. The exception is //go:generate, because of the design constraint that the go command not run user code during builds. The //go:embed is not the special case, nor does it make //go:generate any more of a special case.

For more about go generate, see the original proposal and discussion.

The embed package

Import path

The new embed package provides access to embedded files. Previous additions to the standard library have been made in golang.org/x first, to make them available to earlier versions of Go. However, it would not make sense to use golang.org/x/embed instead of embed: the older versions of Go could import golang.org/x/embed but still not be able to embed files without the newer go command support. It is clearer for a program using embed to fail to compile than it would be to compile but not embed any files.

File API

Implementing fs.FS enables hooking into net/http, text/template, and html/template, without needing to make those packages aware of embed.

Code that wants to change between using operating system files and embedded files can be written in terms of fs.FS and fs.File and then use os.DirFS as an fs.FS or use a *os.File directly as an fs.File.

Direct access to embedded data

An obvious extension would be to add to embed.Files a ReadFileString method that returns the file content as a string. If the embedded data were stored in the binary uncompressed, ReadFileString would be very efficient: it could return a string pointing into the in-binary copy of the data. Callers expecting zero allocation in ReadFileString might well preclude a future -embed=compress mode that trades binary size for access time, which could not provide the same kind of efficient direct access to raw uncompressed data. An explicit ReadFileString method would also make it more difficult to convert code using embed.Files to use other fs.FS implementations, including operating system files. For now, it seems best to omit a ReadFileString method, to avoid exposing the underlying representation and also to avoid diverging from fs.FS.

Another extension would be to add to the returned fs.File a WriteTo method. All the arguments against ReadFileString apply equally well to WriteTo. An additional reason to avoid WriteTo is that it would expose the uncompressed data in a mutable form, []byte instead of string.

The price of this flexibility—both the flexibility to move easily between embed.Files and other file systems and also the flexibility to add -embed=compress later (perhaps that would useful for TinyGo)—is that access to data requires making a copy. This is at least no less efficient than reading from other file sources.

Writing embedded files to disk

In the preliminary discussion, one person asked about making it easy to write embedded files back to disk at runtime, to make them available for use with the HTTP server, template parsing, and so on. While this is certainly possible to do, we probably should avoid that as the suggested way to use embedded files: many programs run with limited or no access to writable disk. Instead, this design builds on the file system draft design to make the embedded files available to those APIs.

Compatibility

This is all new API. There are no conflicts with the compatibility guidelines.

It is worth noting that, as with all new API, this functionality cannot be adopted by a Go project until all developers building the project have updated to the version of Go that supports the API. This may be a particularly important concern for authors of libraries. If this functionality ships in Go 1.15, library authors may wish to wait to adopt it until they are confident that all their users have updated to Go 1.15.

Implementation

The implementation details are not user-visible and do not matter nearly as much as the rest of the design.

A prototype implementation is available.

Appendix: Comparison with other tools

A goal of this design is to eliminate much of the effort involved in embedding static assets in Go binaries. It should be able to replace the common uses of most of the available embedding tools. Replacing all possible uses is a non-goal. Replacing all possible embedding tools is also a non-goal.

This section examines a few popular embedding tools and compares and contrasts them with this design.

go-bindata

One of the earliest and simplest generators for static assets is github.com/jteeuwen/go-bindata. It is no longer maintained, so now there are many forks and derivatives, but this section examines the original.

Given an input file hello.txt containing the single line hello, world, go-bindata hello.txt produces 235 lines of Go code. The generated code exposes this exported API (in the package where it is run):

func Asset(name string) ([]byte, error)
    Asset loads and returns the asset for the given name. It returns an error if
    the asset could not be found or could not be loaded.

func AssetDir(name string) ([]string, error)
    AssetDir returns the file names below a certain directory embedded in the
    file by go-bindata. For example if you run go-bindata on data/... and data
    contains the following hierarchy:

        data/
          foo.txt
          img/
            a.png
            b.png

    then AssetDir("data") would return []string{"foo.txt", "img"}
    AssetDir("data/img") would return []string{"a.png", "b.png"}
    AssetDir("foo.txt") and AssetDir("notexist") would return an error
    AssetDir("") will return []string{"data"}.

func AssetInfo(name string) (os.FileInfo, error)
    AssetInfo loads and returns the asset info for the given name. It returns an
    error if the asset could not be found or could not be loaded.

func AssetNames() []string
    AssetNames returns the names of the assets.

func MustAsset(name string) []byte
    MustAsset is like Asset but panics when Asset would return an error. It
    simplifies safe initialization of global variables.

func RestoreAsset(dir, name string) error
    RestoreAsset restores an asset under the given directory

func RestoreAssets(dir, name string) error
    RestoreAssets restores an asset under the given directory recursively

This code and exported API is duplicated in every package using go-bindata-generated output. One benefit of this design is that the access code can be in a single package shared by all clients.

The registered data is gzipped. It must be decompressed when accessed.

The embed API provides all this functionality except for “restoring” assets back to the local file system. See the “Writing embedded assets to disk” section above for more discussion about why it makes sense to leave that out.

statik

Another venerable asset generator is github.com/rakyll/statik. Given an input file public/hello.txt containing the single line hello, world, running statik generates a subdirectory statik containing an import-only package with a func init containing a single call, to register the data for asset named "hello.txt" with the access package github.com/rakyll/statik/fs.

The use of a single shared registration introduces the possibility of naming conflicts: what if multiple packages want to embed different static hello.txt assets? Users can specify a namespace when running statik, but the default is that all assets end up in the same namespace.

This design avoids collisions and explicit namespaces by keeping each embed.Files separate: there is no global state or registration.

The registered data in any given invocation is a string containing the bytes of a single zip file holding all the static assets.

Other than registration calls, the statik/fs package includes this API:

func New() (http.FileSystem, error)
    New creates a new file system with the default registered zip contents data.
    It unzips all files and stores them in an in-memory map.

func NewWithNamespace(assetNamespace string) (http.FileSystem, error)
    NewWithNamespace creates a new file system with the registered zip contents
    data. It unzips all files and stores them in an in-memory map.

func ReadFile(hfs http.FileSystem, name string) ([]byte, error)
    ReadFile reads the contents of the file of hfs specified by name. Just as
    ioutil.ReadFile does.

func Walk(hfs http.FileSystem, root string, walkFn filepath.WalkFunc) error
    Walk walks the file tree rooted at root, calling walkFn for each file or
    directory in the tree, including root. All errors that arise visiting files
    and directories are filtered by walkFn.

    As with filepath.Walk, if the walkFn returns filepath.SkipDir, then the
    directory is skipped.

The embed API provides all this functionality (converting to http.FileSystem, reading a file, and walking the files).

Note that accessing any single file requires first decompressing all the embedded files. The decision in this design to avoid compression is discussed more above, in the “Compression to reduce binary size” section.

go.rice

Another venerable asset generator is github.com/GeertJohan/go.rice. It presents a concept called a rice.Box which is like an embed.Files filled from a specific file system directory. Suppose box/hello.txt contains hello world and hello.go is:

package main

import rice "github.com/GeertJohan/go.rice"

func main() {
        rice.FindBox("box")
}

The command rice embed-go generates a 44-line file rice-box.go that calls embedded.RegisterEmbeddedBox to registers a box named box containing the single file hello.txt. The data is uncompressed. The registration means that go.rice has the same possible collisions as statik.

The rice embed-go command parses the Go source file hello.go to find calls to rice.FindBox and then uses the argument as both the name of the box and the local directory containing its contents. This approach is similar to the “second approach” identified in the preliminary discussion, and it demonstrates all the drawbacks suggested above. In particular, only the first of these variants works with the rice command:

rice.FindBox("box")

rice.FindBox("b" + "o" + "x")

const box = "box"
rice.FindBox(box)

func box() string { return "box" }
rice.FindBox(box())

As the Go language is defined, these should all do the same thing. The limitation to the first form is fine in an opt-in tool, but it would be problematic to impose in the standard toolchain, because it would break the orthogonality of language concepts.

The API provided by the rice package is:

type Box struct {
        // Has unexported fields.
}
    Box abstracts a directory for resources/files. It can either load files from
    disk, or from embedded code (when `rice --embed` was ran).

func FindBox(name string) (*Box, error)
    FindBox returns a Box instance for given name. When the given name is a
    relative path, it’s base path will be the calling pkg/cmd’s source root.
    When the given name is absolute, it’s absolute. derp. Make sure the path
    doesn’t contain any sensitive information as it might be placed into
    generated go source (embedded).

func MustFindBox(name string) *Box
    MustFindBox returns a Box instance for given name, like FindBox does. It
    does not return an error, instead it panics when an error occurs.

func (b *Box) Bytes(name string) ([]byte, error)
    Bytes returns the content of the file with given name as []byte.

func (b *Box) HTTPBox() *HTTPBox
    HTTPBox creates a new HTTPBox from an existing Box

func (b *Box) IsAppended() bool
    IsAppended indicates whether this box was appended to the application

func (b *Box) IsEmbedded() bool
    IsEmbedded indicates whether this box was embedded into the application

func (b *Box) MustBytes(name string) []byte
    MustBytes returns the content of the file with given name as []byte. panic’s
    on error.

func (b *Box) MustString(name string) string
    MustString returns the content of the file with given name as string.
    panic’s on error.

func (b *Box) Name() string
    Name returns the name of the box

func (b *Box) Open(name string) (*File, error)
    Open opens a File from the box If there is an error, it will be of type
    *os.PathError.

func (b *Box) String(name string) (string, error)
    String returns the content of the file with given name as string.

func (b *Box) Time() time.Time
    Time returns how actual the box is. When the box is embedded, it’s value is
    saved in the embedding code. When the box is live, this methods returns
    time.Now()

func (b *Box) Walk(path string, walkFn filepath.WalkFunc) error
    Walk is like filepath.Walk() Visit http://golang.org/pkg/path/filepath/#Walk
    for more information

type File struct {
        // Has unexported fields.
}
    File implements the io.Reader, io.Seeker, io.Closer and http.File interfaces

func (f *File) Close() error
    Close is like (*os.File).Close() Visit http://golang.org/pkg/os/#File.Close
    for more information

func (f *File) Read(bts []byte) (int, error)
    Read is like (*os.File).Read() Visit http://golang.org/pkg/os/#File.Read for
    more information

func (f *File) Readdir(count int) ([]os.FileInfo, error)
    Readdir is like (*os.File).Readdir() Visit
    http://golang.org/pkg/os/#File.Readdir for more information

func (f *File) Readdirnames(count int) ([]string, error)
    Readdirnames is like (*os.File).Readdirnames() Visit
    http://golang.org/pkg/os/#File.Readdirnames for more information

func (f *File) Seek(offset int64, whence int) (int64, error)
    Seek is like (*os.File).Seek() Visit http://golang.org/pkg/os/#File.Seek for
    more information

func (f *File) Stat() (os.FileInfo, error)
    Stat is like (*os.File).Stat() Visit http://golang.org/pkg/os/#File.Stat for
    more information

type HTTPBox struct {
        *Box
}
    HTTPBox implements http.FileSystem which allows the use of Box with a
    http.FileServer.

        e.g.: http.Handle("/", http.FileServer(rice.MustFindBox("http-files").HTTPBox()))

func (hb *HTTPBox) Open(name string) (http.File, error)
    Open returns a File using the http.File interface

As far as public API, go.rice is very similar to this design. The Box itself is like embed.Files, and the File is similar to fs.File. This design avoids HTTPBox by building on HTTP support for fs.FS.

Bazel

The Bazel build tool includes support for building Go, and its go_embed_data rule supports embedding a file as data in a Go program. It is used like:

go_embed_data(
        name = "rule_name",
        package = "main",
        var = "hello",
        src = "hello.txt",
)

or

go_embed_data(
        name = "rule_name",
        package = "main",
        var = "files",
        srcs = [
                "hello.txt",
                "gopher.txt",
        ],
)

The first form generates a file like:

package main

var hello = []byte("hello, world\n")

The second form generates a file like:

package main

var files = map[string][]byte{
        "hello.txt": []byte("hello, world\n"),
        "gopher.txt": []byte("ʕ◔ϖ◔ʔ\n"),
}

That’s all. There are configuration knobs to generate string instead of []byte, and to expand zip and tar files into their contents, but there’s no richer API: just declared data.

Code using this form would likely keep using it: the embed API is more complex.

However, it will still be important to support this //go:embed design in Bazel. The way to do that would be to provide a go tool embed that generates the right code and then either adjust the Bazel go_library rule to invoke it or have Gazelle (the tool that reads Go files and generates Bazel rules) generate appropriate genrules. The details would depend on the eventual Go implementation, but any Go implementation of //go:embed needs to be able to be implemented in Bazel/Gazelle in some way.

from Hacker News https://ift.tt/30Omk5l